05 — Wrong Retailer & Source-Email Context (the #1 lever)

May contain stale references

This document was audited on 2026-06-22 and contains references that need updating: workflow now includes resolve_customer_by_shipto. Cross-check critical details against the current codebase before relying on them.

Thesis: fixing which WhereFour customer an order is for is the highest-leverage, cross-cutting change — and the source email’s from: address is the strongest untapped signal for it. Do this before any item-matcher tuning.

Why it leads — the customer cascade

The matcher takes customerId as an input and is customer-scoped on both axes:

• cross-reference item lookup is hard-filtered customer_id:=[customerId,__UNIVERSAL__] (typesense-search-service.ts:517-519, 1739-1741), and
• pricing is keyed by the same id (auto-price-resolve.ts:108-109: gold + customer-tier prices).

So one wrong customer poisons every line’s matches and prices simultaneously — the dominant production failure. ENG-467 already documented it: PO 406206 “Giant Eagle” maps to both Giant Eagle – Bedford DC (415230) and Pittsburgh DC (415231).

Today: name-only, and oracle’d in the eval

• Production resolves the customer from Gemini’s extracted customer_name only (pdf_extraction.py:490-497 → match_customer(customer_name, connection_id)); the envelope sender is never used. (Note: the temporal-worker match_customer is a stub; the real resolution is by-name via TypesenseSearchService / the LLM validator’s search_customers tool / the operator pick in review-v2.)
• The email from-address is available at resolution time but unused: email_events.from/from_email/from_fullname, linked via pdf_documents.email_event_id and extracted_orders.email_event_id.
• The eval oracles the customer (run-match-correctness-eval.ts:108-110 and dataset-entry-adapter.ts:102-114 both read the ground-truth customerId), so the 60.4%/26.1% numbers are measured given the right customer — the wrong-retailer failure is invisible. De-oracling the eval is the first build.

Quantified narrowing power (1,055 golden entries, real from: + truth customer)

• Sender domain alone pins exactly one WhereFour customer for 56.1% of orders; narrows to ≤3 candidates for 72.6% (mean candidate set 2.80, 107 distinct customers).
• 41 of 61 domains uniquely identify one customer.
• 20 domains are shared — mostly brokers/3PLs/portals representing many buyers: awginc.com→9, kehe.com→8, johare.com→7 (101 orders), empirefoods.com→6 (all Kroger DCs), unfi.com→6, misfitsmarket.com→4.
• Multi-DC discriminator is data-dependent: PO-number prefix perfectly encodes the DC for Misfits (BWI1/CHI1/LAX1/SAT1) but covers only ~13% of ambiguous orders; domain
  • PO-prefix lifts the pin rate to just 61.8%. PO-prefix is a high-value rule for a few accounts, not a general solution.
• Ship-to address — the general disambiguator (ENG-467’s fix) — is MISSING from the golden set (0/1055 entries carry any address; only {customer_name}). Extraction captures it (extracted_orders.customerShippingAddress; a shipping_addresses Typesense collection exists), but the compile pipeline dropped it. The golden set must be re-compiled with ship-to before broker cases can be measured/solved (doc 07).

Design — SenderCustomerResolver (narrow, don’t pick)

1. New pre-matching step plugged in before match_customer (pdf_extraction.py:490-497): pass the sender from-address; domain-derived candidates are the primary signal, name match the fallback. Mirror it in the review-v2 customer picker (pre-rank) and the LLM validator (new search_customers_by_domain tool + a SYSTEM_PROMPT step-1 change).
2. Disambiguation tiers when a domain → >1 customer: (a) ship-to match against the customer’s shipping_addresses Typesense collection — the general solution; (b) org-configured PO-prefix rules; (c) down-weight brokers and fall back to name + ship-to.
3. Org-scoped config (doc 06): a customerResolution block in the org’s matching config: { senderDomainMap, disambiguation: { poPrefixRules, useShipToAddress } }. Build CK’s initial map by inverting the golden from:→customerId data (or, in prod, GROUP BY sender-domain over historical extracted_orders.erp_customer_id).
4. De-oracle the eval: add a --resolve-customer mode to the eval that resolves the customer from _emailMetadata.from (not the truth), scoring customer-resolution accuracy and end-to-end recall/wrong-match under resolved customers.

Other source-email levers (lower priority, same plumbing)

• Recurring-order priors: boost candidates that appear in this customer’s approved order history (per (connection, erp_customer_id) SKU set).
• Customer-specific part-number schemes: learn that e.g. Four Seasons sends both a vendor code and the CK SKU; Giant Eagle uses 3-digit codes — populate the unused customer_part_number from learned schemes.
• Broker vs direct: the classifier already separates broker_order and extracts the retailer distinct from the broker sender (enrich-ck-emails.ts:404-405,414) — resolve the retailer, not the broker, for those.
• PO-format priors per sender: PO_PATTERNS are already vendor-specific (enrich-ck-emails.ts:202-208); use them as a per-sender prior.

De-oracle eval results (built + measured)

run-ck-match-eval.ts --resolve-customer builds the sender-domain→customer affinity map from all golden threads except the eval orders’ own threads (no leakage), resolves each order’s customer from _emailMetadata.from, scores resolution accuracy, and feeds the resolved customer to the matcher.

On the 66-order corpus (back-to-back A/B):

	recall	wrong-match	unmatched
oracle (true customer name)	61.6%	25.5%	17.4%
sender-domain resolved (held-out)	61.0%	26.3%	17.2%

• Customer resolution from sender domain: 86.4% top-1 correct, 93.9% within top-3, 60.6% domain-pinned to one candidate (2 orders had no candidate).
• De-oracling costs ~nothing here (−0.6pp recall, within noise) — on the measurable corpus the customer is not the bottleneck; domain resolution is already accurate.

Why, and the redirect: the 66-order set is biased toward easy single-domain accounts (Giant Eagle, Rainforest — which also carry UPCs). The hard broker/multi-DC/no-UPC accounts (Walmart via kehe/unfi/awginc) are under-represented because their orders lack pre-extracted lines and UPCs. So the wrong-retailer cost is real but concentrated where this corpus can’t see it. The highest-value next step is therefore expand the corpus (extract the Walmart/broker docs) + re-compile the golden set with ship-to (currently 0/1055) so the broker/multi-DC cases become measurable — then build the resolver + ship-to disambiguation against a corpus that actually exercises them.

Measurement caveat (recorded): the eval has run-to-run nondeterminism (~±1–2pp / ±5 aligned lines). Single-run deltas under ~2pp are not significant — needs seed control

• repeated runs (see doc 06).

Ship-to backfill (built) — the disambiguator, quantified

packages/eval/scripts/backfill-ck-golden-shipto.ts re-fetches each golden order’s shipping_address from prod WhereFour (read-only) and writes it into the golden (_wherefourMatch.shipTo + approvedOrders[0].customer.ship_to). Populated 1,055/1,055 entries. The WhereFour order’s shipping_address uniquely identifies the DC (e.g. order 5136846 → “Associated Wholesale Grocers – Nashville, 500 South Cartwright St”).

Disambiguation lift (n=1,055):

signal	pins exactly one WhereFour customer
sender domain alone	64.8%
domain + ship-to	100.0%

Ship-to is the complete disambiguator — it resolves every shared-domain / multi-DC / broker case. Caveat: this 100% uses the ground-truth ship-to from the WhereFour order. Production must (a) extract the ship-to from the document and (b) match it to the customer’s shipping_addresses (Typesense collection exists). So 100% is the ceiling; the real number depends on ship-to extraction + address-match quality — which is the next thing to measure (needs the corpus expansion below).

Corpus expansion — status & blocker

The hard accounts (broker/no-UPC) are identified: 180 broker-domain v1 docs (awginc/kehe/unfi/johare/empirefoods) with r2Keys ready in the golden set — e.g. PO 55700 (AWG Nashville). The extraction path is the same one the current 66 orders came from: POST /api/eval/upload-from-r2 (X-Eval-Token) → Temporal extraction → extracted_order_items in the staging DB (ep-muddy-wind), which the eval reads.

Blocker: the EVAL_SERVICE_TOKEN in the local .env (len 20) returns 401 against staging.ordermatic.co, and extracting against prod (app.ordermatic.co) would write to the prod DB, not the staging DB the eval reads. Cross-using the prod eval token against staging was (correctly) blocked as cross-environment credential use. To unblock, either: (a) supply the staging EVAL_SERVICE_TOKEN; (b) run the 180-doc extraction batch in a proper staging context; or (c) point a local webapp+Temporal at the staging DB and extract via localhost. Once lines land for the broker docs, run-ck-match-eval.ts picks them up by PO automatically, and --resolve-customer will then measure the wrong-retailer cost where it actually bites.

Expanded-corpus measurement + the customerId-inertness gap

The corpus was expanded (broker docs extracted) to 207 orders / 1,134 lines. The wrong-retailer hypothesis can now be attempted — and the headline result is that the eval, as built, cannot measure it, for a precise mechanical reason.

Resolution collapses to 39% — but it’s a hold-out artifact, not a signal problem

De-oracled customer resolution (from sender domain, threadId hold-out) on the 207 corpus:

signal	top-1	within top-3	domain-pinned	no-candidate
sender domain alone	39.1% (81/207)	57.5% (119)	22.7% (47)	71/207

The drop from 86.4% (easy-66) to 39.1% is entirely a leakage artifact, not lost signal. All 71/207 no-candidate orders are “held-out gap” (zero genuinely-unseen): they come from 4 broker domains whose entire golden footprint sits inside the eval set (awginc 37, kehe 25, unfi 7, chefswarehouse 2), so after the threadId hold-out the domain map has nothing left to learn from. Brokers fan out to many buyers (awginc→9 truth customers, johare→7, unfi→6), which domain-alone cannot split.

Ship-to is the fix, and it’s a clean 1:1 map. Every one of the 133 distinct (domain, ship-to) keys in the corpus maps to exactly one truth customer (0 ambiguous corpus-wide; awginc 10/10, johare 8/8, unfi 6/6, kehe 8/8 pure). Under the production-realistic leave-one-out-by-order regime (any prior order to that ship-to exists), domain + ship-to lifts top-1 to 93.7% with only 13/207 no-candidate — a +54.6pp gain over domain-alone. The 13 residual misses are all genuine first-occurrence ship-tos (true cold-start DCs). So the 39.1% headline is a methodology artifact of threadId-only hold-out; key resolution on ship-to history, not the domain map.

The inertness gap — why oracle → resolved is exactly 0pp

The de-oracled run fed a wrong-or-empty customer for 126 of 207 orders (61%), yet produced byte-identical predictions to the oracle run: 1,134/1,134 lines same predictedErpProductId, 1,134/1,134 same verdict, aggregates identical to 5 decimals (recall 0.5638 / wrong 0.3874 in both). customerId is inert for everything this eval measures:

• The eval correctly resolves the truth customer name to a numeric erp_customer_id (run-ck-match-eval.ts:339 via resolveCustomerId :77-99, query_by:'name', include_fields:'erp_customer_id') and passes that id into findProductMatchesBatch (:236). The name is only the scorer label (:343 customerId: ans.customerName).
• But customerId’s only effect on product selection is the cross-reference (xref) bucket filter: typesense-search-service.ts:1485-1487 (batch) and :1104 (fuzzy fallback), customer_id:=[${customerId},__UNIVERSAL__]; findFuzzyMatch just forwards it (fuzzy-matching.ts:214,232). The product/UPC/name buckets (priorities P2–P4, :1697-2026) score on connection-scoped text_match + Jaccard only — customerId never enters.
• That xref bucket fired 0/1,134 times across both runs (matchedVia: sku=586, name=15, none=533; zero typesense_xref / cross_reference_match). CK lines carry SKU/UPC-shaped ids that resolve in the earlier product/UPC buckets, so the 2,199 customer-scoped xrefs (e.g. {customer_id:'206300', item_id:'1224KP'}) never get a chance — and even when they would, the bucket is customer ∪ __UNIVERSAL__, so a wrong customer still gets the same universal hits.
• The pricing resolver — the heavily customer-keyed lever (auto-price-resolve.ts resolveAndReviewPrices :88-93 → fetchEffectivePriceMap :108 + getCustomerTierPricesByName :109 + resolveCustomerPriceWithCache :119) — is never invoked by the eval at all. The eval only runs the matcher and scores product-id/name; it reads raw eoi.unit_price (:191/:214) but never resolves or scores a price.

So the ~0pp oracle→resolved delta is a measurement artifact: the eval’s data path makes customerId inert and never prices anything. Treat the 38.7% headline as a product-matcher-quality number only; it says nothing about retailer-resolution risk.

Extraction-level wrong-retailer cases exist (24+)

The wrong-retailer harm is real at the extraction layer even though the matcher eval is blind to it: on the broker corpus, the extracted customerName disagrees with the truth WhereFour customer in 24+ orders (broker fan-out — awginc/johare/unfi each resolving to the broker rather than the destination DC). These are exactly the orders ship-to disambiguation fixes; they are invisible in the match aggregate because of the inertness above, not because they don’t happen.

What must change in the eval to measure the real wrong-retailer cost

To make customerId actually change a scored output, one of:

1. Wire + score price (the actual business cost). Call resolveAndReviewPrices with the resolved customerId and compare the resolved customer-tier price against the WhereFour answer’s unit_price. A wrong retailer produces a wrong contract price — this is where wrong-retailer money is lost, and it is currently 100% unexercised.
2. Segment for customer-specific product binding. Construct/select cases where the same their_item_id maps to a different item_id per customer, so the xref bucket’s customer filter changes the predicted product. (CK’s current SKU/UPC-shaped lines resolve before xref ever runs, so today this never triggers.)
3. Until (1) or (2) lands, do not read the ~0pp oracle→resolved delta as evidence that wrong-retailer is cheap. It only means the eval can’t see it yet.