Skip to content
Ordermatic Docs

02 — Baseline & Root Cause

Baseline (real matcher, history truth)

Section titled “Baseline (real matcher, history truth)”

66 orders / 328 aligned lines on conn 4f234677:

MetricValue
Match recall (correct / aligned)60.4% (198/328)
Match precision (correct / matched)73.9%
Wrong-match among matched26.1% (70) — ~1 in 4 binds wrong
Unmatched (no bind)18.3%
Extraction-miss12.5% (extraction surfaced ~88% of lines)

Lands in the range of the June-1 manual ENG-467 read (64.9% / 51.7%). Extraction is fine; matching is the weak link.

Root cause #1 — candidate RECALL, not ranking

Section titled “Root cause #1 — candidate RECALL, not ranking”

Diagnostic across the candidate experiments: 58% of wrong binds never had the correct product in the candidate set. No re-ranking can fix a line whose right answer isn’t retrieved. This is the binding constraint.

Corroboration: a pack-size-aware re-rank, built and A/B’d on the real matcher, was neutral (recall 60.4%→60.4% conservative; the aggressive version slightly regressed). Re-ranking only helps once the right candidate is present.

Root cause #2 — wrong pack-size / variant (74% of wrong binds)

Section titled “Root cause #2 — wrong pack-size / variant (74% of wrong binds)”

Of 70 wrong matches, 52 (74%) are the right product family, wrong size; 18 are wrong product entirely. Concrete:

  • 2 gallon pail → bound to 9/36 oz Pack
  • 6/16 oz Case Pickled Red Onion → bound to Pura Vida 2 Gallon
  • Classic Kimchi 6/16 oz → bound to Whiskey Dill Kraut 6/16 oz

False-match “magnets” (one catalog entry absorbing many distinct ordered SKUs): 5 Gallon Kimchi Pickled Chips (5), Classic Caraway Kraut 9/36 oz (3), Sprouts Raw Sauerkraut (private label, 3), Costco-branded, Pura Vida.

Root cause #3 — the no-UPC half

Section titled “Root cause #3 — the no-UPC half”
  • ~56% of CK order lines carry a UPC; matching is strong there (UPC/identifier bucket reaches 77.1% coverage / 63.1% recall / 23.6% wrong).
  • ~43% carry NO UPC — all Walmart DCs (~18% of total volume), Harris Teeter Greensboro, Stop & Shop — and match at ~38% recall / ~50% wrong (name-only). UPC presence is bimodal by customer: distributors/grocers ~100%, big retailers ~0%.
  • The supposed no-UPC fallback (cross_references) isn’t real: of 2,546 CK xrefs, 96% are UPC restatements; only 110 are true customer-part-number aliases.

Root cause #4 (production-only, hidden by the eval) — wrong retailer/customer

Section titled “Root cause #4 (production-only, hidden by the eval) — wrong retailer/customer”

Many retailers are multiple WhereFour customers (Giant Eagle Bedford DC vs Pittsburgh DC, etc.). The matcher is customer-scoped (xref + pricing), so a wrong customer cascades into wrong matches. The eval uses an oracle customer, so this is invisible here but is the dominant production failure — see 05-wrong-retailer-and-email-context.md.

Expanded-corpus result (207 orders, broker-inclusive)

Section titled “Expanded-corpus result (207 orders, broker-inclusive)”

The 66-order set is biased toward easy single-domain accounts. We expanded to 207 orders / 1,134 answer lines by extracting the broker docs (awginc / johare / kehe / unfi / empirefoods) the old corpus couldn’t see. Deterministic (n=3, zero variance — see doc 06):

Metriceasy-66expanded-207
Match recall (correct / aligned)60.4%56.4% (601/1066)
Match precision (correct / matched)73.9%61.3%
Wrong-match among matched26.1%38.7% (380/981) — ~2 in 5 binds wrong
Unmatched (no bind)18.3%8.0% (85)
Extraction-miss12.5%6.0% (68)

Failure decomposition (expanded-207): of 1,134 answer lines, 601 correct, 380 wrong, 85 unmatched, 68 extraction-miss. Of the 601 correct, 586 matched via the SKU/UPC bucket, only 15 name-only — name matching barely contributes. All 380 wrong have no matchedVia (the matcher returned a real product whose SKU+UPC+name all disagree with truth — a genuine mis-bind, not a missing-id miss).

Why it is harsher than easy-66 (not a metric artifact):

  • It’s the same weakness, more of it. Splitting by sender domain, the pre-broker baseline orders already show 33.1% wrong-match (91/275); the added broker orders show 40.9% (289/706). Brokers move the headline only +5.6pp — the pack/variant confusion predates the expansion and is corpus-wide, not manufactured by it.
  • More valid-UPC, pack-ambiguous lines — the opposite of “no-UPC mush.” Broker truth lines are ~100% UPC-covered; 86–93% of wrong binds are right-family / wrong-pack (the documented two-series problem): 1616CC 'Classic Caraway Kraut 6/16 oz' → 774537 '9/36 oz Pack'; 'Pickled Red Onion 6/16 oz Case' → 'Pura Vida 2 Gallon'; 'Kimchi Pickle Chips 12/16 fl oz' → 'Costco 12/3.5 fl oz'.
  • Wrong-match is concentrated, confirming it’s a discrimination problem in specific duplicate-series accounts, not a uniform floor: johare alone = 226/380 (59.5%) of all wrong binds; fsproduce 80.8%; sirna 100%.
  • Alignment is not forcing it: wrong-match lines have mean alignment score 0.976 (correct 1.040); only 1.8% sit below 0.6 and 50.5% are exact id-hits — well-aligned real lines, not fabricated by mis-alignment.

The conservative denominator (wrong / all answer lines = 380/1134) is still 33.5%, so even the most charitable framing leaves a large real matcher weakness. Caveat: this is a clean product-matcher-quality figure only — the eval scores product-id/name and never prices, and customerId is inert in the data path (0 xref-bucket wins), so it says nothing about wrong-retailer cost (doc 05).

What this implies

Section titled “What this implies”

Fix in this order: (1) get the right candidate retrieved (recall); (2) get the right customer (wrong-retailer); (3) disambiguate pack/variant among candidates (precision). Levers in 04-fidelity-roadmap.md.