06 — Org-Level Tuning Scaffolding & Overfit-Mitigation Evals

Directive: all matching/tuning config and eval scaffolding live at the organization (Ordermatic customer) level. Overfitting to CK is acceptable because config is org-scoped and cannot touch other customers — but we add evals to keep that honest.

Today: config is fragmented across three substrates

1. organization_settings.fuzzy_matching_config (org-scoped JSON) — only the nameOverlap sub-key is actually read by the matcher; the rest of the schema (prefixRules/thresholds) is dead at the Typesense layer.
2. erp_connections.extra_config (connection-scoped JSONB) — only discountConfig.enabled is read at runtime (auto-validate.ts:569). And its PUT handler doesn’t persist extra_config (api/erp/connections/[id].ts) — a ship-blocker for any connection flag.
3. Flagsmith connection-identity flags (price_tier_review_enabled, auto-validate.ts:612).

Plus hard-coded constants in the matcher (nearTie 0.97, UPC name-overlap 0.3, num_typos, candidate breadth) and the new packAwareRerank / gs1UpcRecovery options.

Design — one org-scoped matchingConfig

Reuse the existing organization_settings JSON-column pattern (no migration): extend packages/db/src/schemas/fuzzy-matching-config.ts (or a sibling matching-config.ts) with every matcher knob, defaulted OFF / identity, so a config-less org behaves exactly as today:

matchingConfig = {
  customerResolution: { senderDomainMap, disambiguation: { poPrefixRules, useShipToAddress } },
  candidate: { vectorRecall: false, vectorMaxDistance, broadenUnion: false, candidateBreadth },
  rerank: { llmSelectOrZero: false, packAwareRerank: false },
  upc: { gs1UpcRecovery: false, nameOverlap, upcNameOverlap: 0.3 },
  thresholds: { nearTie: 0.97, numTypos },
  connectionOverrides: { <connectionId>: { ...partial } }   // CK's single connection
}

• Read once at match time via OrganizationSettingsService and threaded into MatchGateOptions. connectionOverrides lets CK’s one WhereFour connection be tuned without depending on the broken erp_connections.extra_config PUT path.
• Defaults preserve current behavior org-wide → CK tuning is provably isolated.
• Also land the minimal PUT-handler fix so extra_config persists (for the discount/ override gates that legitimately live there).

Eval scaffolding — per-org + de-oracled

Extend run-match-correctness-eval.ts:

• --matching-config <path|inline> — a run uses the exact org config (not just {metric,threshold}), so the eval measures the configured matcher.
• --resolve-customer — resolve the customer from _emailMetadata.from instead of the oracle (doc 05); report customer-resolution accuracy + end-to-end recall/wrong under resolved customers.
• Per-org aggregation so results are reported per Ordermatic customer.

Overfit-mitigation evals

• Held-out split keyed on threadId — split train/test by _emailMetadata.threadId (NOT by item), so near-duplicate lines from the same PO/thread don’t leak across the split. Tune on train, report on held-out test. New packages/eval/src/... split util.

• Per-org metrics + regression detection — reuse packages/eval/src/scorers/ regression-detector.ts; a CK tuning change must show no regression on a basket of other orgs’ fixtures before it can ship (guard).

• Coverage of the no-pin / broker cases measured separately so a CK win isn’t an artifact of the easy single-domain accounts.

• Determinism control (measured need). The harness shows run-to-run variance of ~±1–2pp / ±5 aligned lines (Typesense ranking ties + async order). Single-run deltas under ~2pp are not significant. Add seed/order control and report each metric as a mean over N repeats with a spread — without this, small tuning wins can’t be trusted (e.g. the gs1 +1.2pp needs re-confirmation; the de-oracle −0.6pp is within noise).

  Update (expanded-207 corpus): variance is now ZERO. run-ck-eval-repeats.sh (n=3) on the 207-order corpus produced byte-identical aggregates across all three runs — recall 0.5637898686679175, wrong 0.3873598369011213, unmatched 0.07973733583489681 every time (correct=601, wrong=380, aligned=1066), and oracle vs resolved were also byte-identical to 5+ decimals. The earlier ~±1–2pp was small-corpus sensitivity (66 orders / 328 lines — a handful of Typesense ties flipping moved the rate visibly). On the larger corpus the tie-flips wash out. Consequence: on the 207 corpus, sub-percentage-point deltas ARE now trustworthy (no longer noise-floored at ~2pp), so previously-borderline tuning wins (e.g. gs1 +1.2pp) can be re-confirmed here as real. Keep reporting n=3 with spread as the guard, but the spread is now ~0.

Why this satisfies the directive

Every lever in 04-fidelity-roadmap.md becomes an org-scoped, default-off flag in one place; CK can be tuned aggressively (even overfit) with a held-out split to watch it, and a config-less org is byte-for-byte unchanged.