10 — Pack/Variant Matcher (recall-vs-rank, broaden-union)

This closes out the doc-02 / doc-04 thread with a definitive recall-vs-ranking split and a measured strategy bake-off, and specifies the one matcher change to ship. All numbers are the 207-order corpus (conn 4f234677, deterministic n=3, zero variance); the easy-66 figures in doc 02 are superseded.

The recall-vs-ranking split (refined)

Doc 02 said “58% of wrong binds never had the correct product retrieved” — a recall problem — measured against a narrow candidate notion. We re-measured the 380 wrong binds in /tmp/ck-eval-runs/oracle-1.json by replaying the full production candidate union of findProductMatchesBatch (xref→product, product-fuzzy num_typos=1/ per_page=10, upc-filter, idstrip, name-fuzzy num_typos=2/per_page=5) per wrong line and asking whether the truth product (answer sku/upc, joined from the golden via answerId, judged exactly as the scorer’s judgePair) is in the union.

Bucket	n / 380	Share
RANKING failure (truth WAS retrieved, matcher picked wrong)	208	54.7%
RECALL failure (truth NOT retrieved)	172	45.3%

This reverses the prior reading: ranking is now the larger bucket. The shift is methodological — replaying all five production buckets (not a name-only proxy) retrieves more truth products, converting “recall” misses into “rank” misses. Faithfulness check: the production-picked product is reproduced in the replayed union in 379/380 (99.7%) cases; candCount==0 is 0/380 (every wrong line did retrieve candidates — consistent with a bind happening, not an empty result).

Breakdowns:

Pack signal is absent on the extracted line in 358/380 (94%). CK PO lines extract as bare names ("CLASSIC CARAWAY", "CK KIMCHI PICKLE"); the pack/size lives only in the catalog/truth name. A pack-aware reranker can only act on the 22 lines that carry a pack token — it is structurally inert on 94% of wrong binds. This is the root of why pack-rerank keeps measuring neutral: a signal gap, not a weight gap.
By family (predicted-vs-truth base-name Jaccard ≥0.5): same-family 258/380 (67.9%), split 61.6% rank / 38.4% recall; cross-family 122/380 (32.1%), split 40.2% rank / 59.8% recall. Same-family is 76.4% of all rank-misses — the two-series pack/variant confusion (Classic Caraway Kraut 6/16 oz [1616CC] → 9/36 oz [774537]; Pickled Red Onion 6/16 oz Case → Pura Vida 2 Gallon; Kimchi Pickle Chips 12/16 fl oz → Costco 12/3.5 fl oz). Cross-family skews recall (truth genuinely not retrieved).

Recall ceiling. Of 1066 aligned lines: (601 correct + 208 retrievable-but-misranked) / 1066 = 75.9% — corroborating the independent ~77% identifier-bucket coverage. Perfect ranking on already-retrieved candidates would lift match recall 56.4% → ≥75.9% (+208 lines, +19.5pp) with no new candidate generation. The other 172 lines (recall failures) need broader candidate generation to become reachable at all.

Strategy bake-off (measured, 207-order corpus)

Five candidate-generation + pick strategies, replayed through CkMatchScorer (coverage = candidate-only; recall/precision/wrong/unmatched = downstream after pick()). The name-only baseline runs hotter (46.6% wrong) than the production 5-bucket oracle (38.7% — the baseline of record from doc 02), so compare lifts to both.

#	Strategy	Coverage	Recall	Precision	Wrong	Unmatched
1	baseline (name-only)	69.1%	48.4%	53.4%	46.6%	9.4%
2	broadened (full-fields + pack-stripped)	79.0%	52.1%	—	45.1%	5.1%
3	upc/identifier bucket	77.8%	61.4%	66.0%	34.0%	6.9%
4	broaden-union + name-rank	87.7%	64.7%	66.7%	33.3%	3.1%
5	broaden-union + pack-rerank	87.7%	63.9%	—	33.9%	3.5%

Strategy 4 = union(upc-identifier + broadened full-fields + pack-stripped name) with the existing name+rank pick(). Winner on every axis: highest coverage, highest recall, lowest wrong-match, lowest unmatched, no precision tradeoff (precision rises to 66.7%). The entire gain is candidate recall (coverage 69.1% → 87.7%).

Pack-rerank does not earn its place. Strategy 5 is neutral-to-slightly-worse vs 4 (wrong +0.6pp, recall −0.8pp — both inside the ±0.4pp run-to-run noise band). A 3×-heavy pack weight (1.5) still gave wrong 33.7% — no movement, confirming the diagnosis above: the pack signal is absent on 94% of lines, so no weight can act on it. Reranking is a weight knob on a missing feature.

Expected wrong-match reduction (winner, strategy 4):

vs name-only harness baseline: 46.6% → 33.3% (−13.3pp)
vs production 5-bucket oracle (the real baseline): 38.7% → ~33.3% (≈ −5.4pp)
recall 56.4% → ~64.7% end-to-end; unmatched 8.0% → ~3.1%.

The recall-ceiling math says the headroom above broaden-union is the 208 rank-misses — reachable only by a pick-side lever that can discriminate same-family pack variants (the deferred gated-LLM select-or-0 of doc 04 #2), not by more candidate generation.

The matcher change to ship

One change, in findProductMatchesBatch (apps/webapp/src/services/search/typesense-search-service.ts), CK/connection-gated and default-off, wired exactly like the existing packAwareRerank / gs1UpcRecovery gates.

Add broadenUnion?: boolean to MatchGateOptions (the interface at L31), with the same doc-comment convention: _additive candidate generation, OFF by default, gated per-connection via extraConfig.matchingConfig._for CK; other connections unchanged.* Resolve it from connectionextra_config at the call site, same as the others.
Candidate generation (the recall lever). Today the per-item sub-searches are: (a) xref, (b) upc-filter, (c) idstrip exact, (d) product-fuzzy, (e) name-fuzzy. When broadenUnion is on, add two more multi_search sub-searches and merge their hits into the same productResult/dedup path the existing buckets feed:
- full-fields name — query_by over name,description,sku,upc,upc_case (vs the current narrow name query), widening lexical recall for the 172 recall-misses.
- pack-stripped name — the line name with the pack signature removed (parsePackSignature already exists in matching-helpers.ts), so "... 6/16 oz" and "... 2 gallon" retrieve the same family, surfacing the sibling pack that is currently absent. Dedup by erp_product_id against the existing union. This is precisely the harness broaden-union strategy (L103 of run-ck-candidate-experiments.ts), which it already proxies against the real buckets.
Pick stays the existing name+rank pick() — do NOT enable pack-rerank. Leave packAwareRerank as-is (gated, off); the bake-off shows turning it on alongside broaden-union is neutral-to-worse. Ship broaden-union with pack-rerank off.

Net: one new gate flag + two additive sub-searches in the candidate-build block, no change to pick(), no change to any other connection. Additive-only candidate generation cannot regress an existing exact/UPC bind (those buckets still run and still win first).

Held-out / overfit guard

Every number is conn 4f234677 only — CK overfit risk is real and acceptable only because the flag is org/connection-scoped (D11) and cannot touch other customers.

Held-out split before flipping prod. Partition the 207-order corpus by sender domain (the harness already segments johare/fsproduce/sirna/etc.); tune/confirm on a train split and report wrong-match + recall on a held-out domain set. The lift must hold on held-out, not just in-sample. Broaden-union’s gain is candidate recall (a mechanical union, not a fitted parameter), so it should generalize — verify it does.
Per-connection regression gate. Run a non-CK connection through the harness with the flag ON to confirm zero delta (the flag is gated, so this should be a literal no-op; prove it). Coverage and precision are the stable axes; the aligned denominator drifts 1066–1079 across runs due to pre-existing alignment tie-break non-determinism — read deltas, not the third significant figure.
No pack-weight tuning. The pack signal is absent on 94% of lines; resist re-opening the rerank weight — it is measurement noise. The real residual lever is the gated-LLM select-or-0 picker (doc 04 #2) over the broaden-union top-K, which attacks the 208 rank-misses; gate it behind the same overfit discipline when built.

Artifacts

apps/webapp/scripts/ck-wrongbind-recall-vs-rank.ts — replays the 5 production candidate buckets per wrong-bind line, classifies recall vs ranking, breaks down by pack-signal and family.
apps/webapp/scripts/run-ck-candidate-experiments.ts — extended: broaden-union strategy + --rerank name|pack + --packw weight override.
packages/eval/reports/ck-cand-{baseline,broadened,upc,broaden-union}-{name,pack}.json — per-strategy results.
Oracle baseline: /tmp/ck-eval-runs/oracle-1.json; golden: packages/eval/test-data/golden-dataset/cleveland-kitchen-golden-dataset.json; scorer: packages/eval/src/scorers/ck-match-scorer.ts.