Decision Log — CK Matching

Each entry: the decision, and why (for traceability).

#	Decision	Why
D1	Measure with history ground truth (email→prod WhereFour), not operator labels	CK has ~no operator-validated erp_item_id (auto-submit fail-closed). The history pipeline already exists and yields 5,541 labeled lines.
D2	Reject auto-validation-runner.ts as the match-quality measure	It feeds the answer-key SKUs into the matcher (tautology, ~90% flatter). Built a new extract→match→align→score harness that feeds the document lines.
D3	Two-pass scorer (align-then-score), separating extraction-miss from match error	Extract-then-match conflates the two; per-line attribution keeps a bad extraction from being scored as a matcher miss.
D4	Candidate recall is the lever, not ranking	Measured: 58% of wrong binds never had the right product retrieved; pack-aware re-rank was neutral on the real matcher.
D5	Pack-aware re-rank kept gated + default-off, conservative tie-break only	Aggressive version regressed; conservative was neutral. Not worth as primary, but safe and useful once recall is fixed.
D6	UPC bucket is the strongest single lever where a UPC exists (~56% of lines)	Harness: 77.1% coverage / 63.1% recall / 23.6% wrong vs name-only 38%/50%.
D7	Built a shared GS1 normalizer (gs1Upc12Candidates), TS + Python	Recovers ~74% of corrupted UPCs deterministically; one helper for both the matcher and the catalog sync.
D8	Catalog UPC: master-file override stays canonical, GS1 only conservative gap-fill	Verified on prod: clean each not in the API (74% recoverable, 86% single-pick precision, case codes ~30% derivable). The API can’t be sole source of truth.
D9	Replace the static CSV with a DB-backed override + internal upload UI + audit	The frozen 92-row CSV goes stale silently; a validated, self-service, auditable, self-refreshing source is the robust fix.
D10	Wrong retailer/customer must be nailed first via the email from: lever	It’s the dominant production failure (multi-DC retailers) and cascades into wrong matches; hidden by the eval’s oracle customer.
D11	All tuning config + eval scaffolding at the org (Ordermatic customer) level	Overfitting to CK is acceptable only because config is org-scoped and can’t touch other customers. Add held-out + per-org overfit-mitigation evals.
D12	Vector recall is a cold-start no-UPC complement, pure-vector + precision gate, never the decider	Live-verified it works (whole-foods→WFM 365), but naive hybrid demoted a correct match and a flat distance gate makes wrong-retailer worse.
D13	Deterministic customer-part# xref from history = the durable no-UPC moat, but supply-limited	Once confirmed, repeats bind ~100%; but CK history is thin (~171 lines/17 customers), so it compounds over time, with vector covering cold start.
D14	Document the body of work in checked-in md (docs/designs/ck-matching/)	Findings + motivations must be traceable when we commit and build.
D15	pricePackSelect (unit-price pick-side lever) ships paired with broadenUnion, gated+off	Pack token absent on 94% of lines (pack-rerank inert); unit price is the surviving signal. Measured: bu+pps wrong 38.7%→26.3%, recall→71.1%; neither alone helps (doc 11). 133 wrong→correct, 0 correct→wrong.

Open items / prereqs

• api/erp/connections/[id].ts PUT handler does not persist extra_config → blocks toggling any connection-scoped flag in prod. Ship-blocking for rollout.
• Extend the eval corpus beyond the 66 pre-extracted POs (extract the other ~633) for population-final numbers.
• Reconcile the 10 API↔master UPC disagreements with CK.
• Verify/port name_embedding schema onto our branch before building the vector bucket.