Cleveland Kitchen Item-Matching — Investigation & Roadmap
This directory documents the investigation into Cleveland Kitchen (CK) item-matching quality: how we measure it, the measured baseline and root causes, the UPC data pipeline, and the ranked roadmap of fidelity improvements. It exists so the findings and the motivation behind each approach are traceable when we commit and build.
Status: investigation + measurement complete; first safe improvements built and A/B-verified in an eval harness; production rollout not yet wired. All matcher changes are gated off by default and connection/org-scoped.
Headline
Section titled “Headline”- Measured CK matching (real matcher, history ground truth, 66 orders / 328 lines): recall 60.4%, wrong-match 26.1%. Extraction is fine (~88% of answer lines surfaced); matching is the weak link — confirming the original concern.
- Root cause is candidate RECALL, not ranking: 58% of wrong binds never had the correct product in the candidate set. Pack-aware re-rank alone was neutral on the real matcher. 74% of wrong binds are wrong pack-size/variant of the right family.
- The catalog splits in two: ~56% of order lines carry a UPC (where matching is strong); ~43% (all Walmart DCs ~18% of volume, Harris Teeter, Stop & Shop) carry no UPC and match poorly (~38% recall / ~50% wrong) — this is the half to fix.
- Wrong retailer/customer is the dominant, cross-cutting failure (many retailers = multiple DCs as distinct WhereFour customers) and is the first thing to nail.
Documents
Section titled “Documents”| File | Scope |
|---|---|
| 01-measuring-match-quality.md | How we measure: history ground truth (email→prod WhereFour), the extract→match→align→score harness, metrics, caveats |
| 02-baseline-and-root-cause.md | The 60.4%/26.1% baseline, the candidate-recall root cause, the pack-size + magnet failure modes |
| 03-upc-pipeline.md | UPC data provenance (prod-verified), GS1 recovery (built + measured), and the maintainable DB-backed + upload-UI system |
| 04-fidelity-roadmap.md | Ranked levers: broaden recall → LLM rerank → deterministic customer-part# xref → vector recall (live-verified) |
| 05-wrong-retailer-and-email-context.md | The email from: → customer-narrowing lever (priority) + other source-email levers |
| 06-org-level-tuning-and-evals.md | Org-scoped tuning config + overfit-mitigation evals |
| 07-dagster-pipeline-work.md | Pipeline-level work: DB-backed UPC override, embedding/xref materialization, audits |
| 08-extraction-enhancements.md | Broker extraction: supplier-vs-retailer prompt + structured ship-to schema (the resolver’s input) |
| 09-ship-to-resolution.md | Ship-to resolver measured end-to-end: 93.5% top-1 under extracted ship-to (0.6pp tax), AWG Whse→DC table, prod impl plan |
| 10-pack-variant-matcher.md | Recall-vs-ranking split (54.7% rank / 45.3% recall), strategy bake-off, and the broadenUnion candidate-recall change |
| 11-price-pack-selector.md | pricePackSelect: unit-price pick-side lever; bu+pps cuts wrong-match 38.7%→26.3% (recall 56.5%→71.1%), measured A/B |
| decisions.md | Decision log — what we chose and why |
Artifacts (code)
Section titled “Artifacts (code)”- Eval harness:
apps/webapp/scripts/run-ck-match-eval.ts(real matcher, history truth) - Candidate experiments:
apps/webapp/scripts/run-ck-candidate-experiments.ts(--strategy) - Two-pass scorer:
packages/eval/src/scorers/ck-match-scorer.ts - GS1 normalizer (shared):
apps/webapp/src/utils/matching-helpers.ts(gs1Upc12Candidates) - Catalog UPC derivation:
apps/dagster/erp_pipeline/assets/typesense.py - Results JSON:
packages/eval/reports/ck-*.json