Skip to content
Ordermatic Docs

01 — Measuring CK Match Quality

Why a new measurement was needed

Section titled “Why a new measurement was needed”

“Validation is trash” was the hypothesis; we needed a number. The existing packages/eval/src/runners/auto-validation-runner.ts looked like the tool but is a tautology: it feeds the WhereFour answer-key SKUs straight into the matcher, so it measures “given the right SKU, can Typesense re-find it” (~90%+, flattering), not “given the document, do we find the right product.” Its scorer header even says it: “the dataset has no canonical expected erpProductId per line.”

Ground truth: history, not operator labels

Section titled “Ground truth: history, not operator labels”

CK has effectively no operator-validated match labels (auto-submit is fail-closed; only ~17 validated lines on conn 4f234677), so the webapp eval:match path (which keys on 90-day operator-confirmed extracted_order_items.erp_item_id) is empty for CK.

Instead we reconstruct truth from history: CK Gmail emails matched to the actual prod WhereFour sales orders. This pipeline already existed (pull-ck-emails → enrich-ck-emails → compile-ck-golden) and produced cleveland-kitchen-golden-dataset.json: 1,055 emails → 823 WhereFour orders → 5,541 line items with item_ids ([sku, upc]) + unit_price per line and _wherefourMatch (customer, PO, method) per entry.

The harness: extract → match → align → score

Section titled “The harness: extract → match → align → score”

apps/webapp/scripts/run-ck-match-eval.ts:

  1. Corpus — golden entries that already had extracted document lines in the DB (extracted_order_items), joined to the WhereFour answer by PO number.
  2. Match — run the real production matcher (findProductMatchesBatch + fuzzy) on the extracted document lines (never the answer-key SKUs), scoped to the WhereFour customer.
  3. Two-pass align + score (packages/eval/src/scorers/ck-match-scorer.ts):
    • Pass 1 (extraction layer): align each extracted line to an answer line on document-side fields. A line with no confident answer partner = extraction_miss, not a matcher miss.
    • Pass 2 (matcher layer): on aligned pairs, compare the matcher’s predicted product to the answer by SKU → UPC → normalized-name → correct / wrong / unmatched.
  4. Aggregate — match recall / precision / wrong-match-among-matched + extraction-miss
    • a reproducibility snapshot (connection, Typesense counts, dataset sha, git rev).

Metrics

Section titled “Metrics”
  • Candidate coverage = % of aligned answer lines whose correct product is retrieved into the candidate set. This is the metric of record for the recall problem.
  • Match recall = correct / aligned; precision = correct / matched; wrong-match-among-matched = wrong / matched (highest severity — confident wrong binds). NOT “false auto-submit” (CK is fail-closed; nothing auto-ships).

Honest caveats

Section titled “Honest caveats”
  • Oracle customer. The harness resolves the WhereFour customer by name, so it measures item matching, not customer resolution. Customer/retailer disambiguation (the dominant production failure) is hidden here — see 05-wrong-retailer-and-email-context.md.
  • 66 of 699 golden POs had pre-extracted lines in the DB. Numbers are a real but partial slice; population-final numbers need extracting the rest.
  • WhereFour-as-truth = the fulfilled order, which can differ from the email (substitutions, standing-template qty-0 lines; fix-ck-ground-truth.ts strips some).