12 — Precision & Abstention Architecture (fail-closed: prefer unmatched over wrong)

May contain stale references

This document was audited on 2026-06-22 and contains references that need updating: line-number reference (L931) exceeds file length. Cross-check critical details against the current codebase before relying on them.

This is the implementation design for cutting CK wrong-match by making the matcher refuse to bind lines it can’t resolve, instead of binding the least-bad candidate. It supersedes the ad-hoc “shipped a flag, hope ops flips it” pattern: it specifies the production activation path, a calibrated per-line abstain decision, and a robust per-change validation protocol. Every claim about current behavior is anchored to a file:line verified against the codebase on 2026-06-17 (post-#1731 merge).

Companion docs: 10 (broaden-union recall), 11 (price-tier pack selector), 02 (baseline). Research + external-model basis: deep-research (24/25 claims verified) + Codex consult, 2026-06-17.

1. Goal and the cost asymmetry

CK auto-submit is fail-closed: a confident WRONG bind ships the wrong product; an UNMATCHED line just routes the order to human review. Wrong ≫ unmatched in cost. The goal is to move mass from “confident-wrong” to “unmatched/review” without collapsing auto-submit coverage.

Measured state (207-order eval, history truth, post-#1731 broadenUnion+pricePackSelect): recall 71.1%, precision 73.7%, wrong-among-matched 26.3%, unmatched 3.6%. Of the wrong binds, 93% are flat-0.92 fuzzy (P3) binds (32% wrong-rate); identity/UPC binds are 2.6% wrong; P4 name binds are 1 correct / 12 wrong. Wrong and correct fuzzy binds are not cleanly separable by any single signal we tested (best abstain rule: 42–45% precision).

2. Current pipeline — verified facts the design depends on

These were mapped, not assumed. The design is built on them.

2a. The abstention block is NOT deterministic today — we must add a hard gate (CORRECTED after dual-voice review; the original draft was wrong). Order auto-submit is gated in apps/temporal-worker/workflows/agentic_validation.py:294-295: threshold = flags.get("agentic_confidence_threshold", 0.95); auto_submit = all_valid and confidence >= threshold. But all_valid = sections_valid and items_valid (:276-280) where sections_valid is the LLM validator’s own boolean judgment (Gemini JSON), and the LLM’s confidence rubric scores “one unmatched item” at 0.75, not a hard fail (activities/llm_validation.py:71). The only deterministic item check (filter_invalid_items_activity) verifies that matched items still exist in the ERP — an unmatched line (no erp_item_id) is invisible to it, and for WhereFour filterInvalidItems is a passthrough (wherefour-service.ts:931). A deterministic unmatched-check (validate_items_matched, activities/agentic_validation.py:406) EXISTS but is deprecated and not called by this workflow. The LLM also has a search_products tool (llm_validation.py:108) and can decide an abstained line is “fine enough” and return valid=true + confidence ≥ 0.95. So an order with an unmatched line CAN still auto-submit. Matcher abstention is therefore NOT free fail-closed behavior as the original draft claimed.

The fix (now Phase-0 scope): add a deterministic pre-submit invariant in agentic_validation.py after :280 — has_unmatched = any(not i.get("erp_item_id") for i in input_data.item_match_results); force all_valid = all_valid and not has_unmatched. Only then does “the matcher refuses to bind → order routed to review” hold. The eval must also simulate this gate (see §5) to report wrong-among-auto-submitted at all. The lever stays: make the matcher return no bind on shaky lines — but it requires the deterministic gate to be real first.

2b. Item-level has no confidence threshold; the matcher confidence is a bucket label. findProductMatchesBatch assigns: P1 xref 0.95–0.99, P2 UPC 0.97, P2.5 idstrip 0.97, P3 fuzzy 0.99-exact / min(0.92, 0.5+overlap) when the name-overlap gate fires / ≤0.4 on no-signal near-tie, P4 name 0.8 + jacc*0.1 + … (0.8–0.93) (typesense-search-service.ts :2108-2110, 2179, 2207, 2406-2408, 2460). P3 fuzzy effectively pins at 0.92 — it labels a bucket, it does not score risk. This is the root of the “confidence cliff.”

2c. ACTIVATION PREREQUISITE (ship-blocker). The gated flags are dead code in production. MatchGateOptions declares broadenUnion / gs1UpcRecovery / packAwareRerank / pricePackSelect, but: (i) the ExtraConfig schema (packages/db/.../erp-connections.ts:74-160) has no matchingConfig field; (ii) no production call site reads it — auto-validate.ts:289-294 and auto-validate-with-price-check.ts:139 pass only nameOverlap + onLowRelevance; (iii) the connections PUT handler (api/erp/connections/[id].ts) accepts extraConfig in its Zod schema but drops it when building dbUpdates (lines 183-189) — confirmed decisions.md ship-blocker. Net: #1731’s pricePackSelect and doc-10’s broadenUnion cannot be turned on for CK today. Nothing in this roadmap matters until this is fixed — but the fix reuses the EXISTING org-settings plane, not a new extraConfig field (see Phase 0b / §6; the original draft’s “add matchingConfig to extraConfig” was the wrong plane and contradicted decision D11 + doc 06, which already have a wired, UI-backed organization_settings.fuzzyMatchingConfig).

2d. Use extracted_unit_price, NOT unit_price, for price signals. extractedUnitPrice/extractedQuantity are immutable records of what the LLM extracted; unit_price/quantity are writable and overwritten by the persist activity with ERP-tier prices (extracted_order_items, migration 0062). The price-pack signal must read the customer’s actual ask (extracted_unit_price), or it becomes partly circular (matching a price the system itself wrote). This affects #1731 and the eval (run-ck-match-eval.ts reads eoi.unit_price): re-measure with extracted_unit_price as the first validation task.

2e. Catalog field gaps. On the eval connection 4f234677 (collapsed sandbox), ~55% of products carry a positive list_price; price1..price10 are all 0 for WhereFour; unit_of_measure is declared but not indexed in Typesense (always undefined); there is no stored pack/size and no per-base-unit ($/oz) field anywhere — parsePackSignature is match-time only (typesense.py provenance; matching-helpers.ts:459-484). Per-base-unit normalization must be computed at match time (parse catalog name → size → list_price/size) for v1; a stored field is a later dagster change, not a v1 blocker.

2f. Label inventory (decides calibration design). Golden v1 = 835 entries / 690 POs / 3,759 answer lines; the eval’s extracted-and-aligned subset is 207 orders / ~1,066 lines. 54 distinct product classes, median 21 labels/class, 36 classes ≥10, 42 ≥5, 8 with 1-2, 7 with exactly 1; 50 sender domains. Class-conditional conformal is viable (not per-SKU, which would collapse). Caveat: answer key = WhereFour fulfilled order, so partial fulfillment / substitution can mislabel a system-correct bind as “wrong” — flag answer lines where fulfilled_qty != po_qty as lower-confidence truth.

3. The architecture: retrieve → score → calibrate → abstain

Unanimous across our data, Codex, and the literature (see §7 citations). This replaces “first plausible bucket hit wins” with an explicit risk-scored, calibrated, abstaining pick.

extracted line ──► (A) RETRIEVE top-K candidates   [existing buckets + broadenUnion]
                       │
                       ▼
                   (B) SCORE each candidate         [label-light risk model over features]
                       │
                       ▼
                   (C) CALIBRATE + ABSTAIN          [global Platt threshold @ FMR floor (2a);
                       │                              class-conditional conformal only if needed (2b)]
            ┌──────────┴───────────┐
        accept                    below threshold / ambiguous
        = BIND (auto-eligible)    = NO BIND → unmatched → DETERMINISTIC review gate (§2a)

(A) Retrieve — keep the existing P1/P2/P2.5 identity buckets (already 2.6% wrong; do not touch). For the fuzzy path, use broadenUnion’s widened candidate pool so the right pack sibling is present to be chosen or rejected.

(B) Score — for each fuzzy candidate, a small regularized, label-light risk model (NOT a deep cross-encoder; see §7). Features that generalize (no SKU-identity features, to avoid overfitting 54 classes):

• retrieval bucket + Typesense text_match, rank, and gap to rank-2;
• raw name overlap and family-normalized (pack-stripped) name overlap;
• pack negative-evidence: candidate carries a pack token the line lacks → ambiguity;
• price fit of extracted_unit_price to the candidate’s nearest tier after per-PO markup normalization (see §4), and the price-fit gap to the next same-family pack;
• price-per-base-unit ($/oz) consistency (catalog list_price ÷ parsed size);
• priced-vs-unpriced flag (unpriced = can’t confirm = higher risk);
• family-ambiguity count: how many same-family priced packs are in the pool. Start with an interpretable additive log-likelihood-ratio (Fellegi-Sunter m/u) score or a tiny L2-logistic — both need few labels; calibrate with Platt/sigmoid (NOT isotonic — isotonic needs ≥~1000 points and overfits below), fit out-of-fold with grouped-by-ORDER CV (line-level CV leaks).

(C) Abstain — a class-conditional (pack-family-grouped) split-conformal threshold targeting a false-match-rate floor (precision is a ratio and not a monotone loss, so conformal-risk-control must target FMR/FNR, not “precision” directly). Singleton survivor → bind; empty (novelty) or multi-candidate (genuine ambiguity) → no bind → review. Plus two hard rules that need no model:

• Suppress P4 name-only auto-binds (1 correct / 12 wrong) → unmatched.
• Abstain on unresolved pack-ambiguity: ≥2 same-family priced packs in the pool, no pack token on the line, and price not decisive → refuse.

4. The one genuinely new feature: per-PO price normalization

The price veto was weak (caught ~8 of 270 wrong) because the customer markup over WhereFour list runs 10–30% and varies across POs — that noise swamps the signal. But within a single PO the markup is ~constant. Infer the PO’s markup band from its confident lines (UPC/xref binds, which are 2.6% wrong), then judge each ambiguous line’s extracted_unit_price against tier_price × inferred_markup. This is exactly Amazon’s per-base-unit price-consistency pattern (WWW 2020) applied per-document. It sharpens the price feature precisely where the cross-PO version failed. Measure its marginal contribution as its own A/B (§5).

5. Validation & measurement protocol (robust, per change)

Non-negotiable: every change is a gated flag, default-OFF, byte-identical off-path, and ships only on a measured held-out lift. No change lands on vibes.

Harness & determinism. Use run-ck-match-eval.ts (207-order corpus, conn 4f234677). Each run records a reproducibility snapshot: connection UUID + org, Typesense catalog counts

• maxMaterializedAt, golden-dataset SHA256, git rev/branch, extracted-orders snapshot date. Determinism control: packages/eval/scripts/run-ck-eval-repeats.sh (n=3; read deltas, not the 3rd significant figure — the aligned denominator drifts 1066–1079 on alignment tie-breaks).

Held-out split (new). Partition the 690 POs by sender domain into train (60%) / calibration (20%) / test (20%). Tune and fit conformal on train+cal; report wrong-match and auto-submit-rate on the held-out test domains. A lift must hold out-of-sample, not just in-sample (overfit guard, per D11 — CK-scoped).

Metrics — report the thing that actually ships. Today we report wrong-among-matched. Add and headline:

• wrong-among-AUTO-SUBMITTED (order-level, after the 0.95 gate) — the real customer-facing error rate;
• auto-submit rate (orders auto-submitted / total) — the coverage cost of abstaining;
• review load (lines routed to review) — the ops cost;
• plus existing recall / precision / unmatched. The trade we are tuning is wrong-among-auto-submitted ↓ at an acceptable auto-submit-rate ↓.

Conformal-specific. On held-out test, report empirical false-match-rate vs target α, and coverage; verify class-conditional coverage per pack-family (not just marginal). Recalibrate when the catalog materializes (conformal needs exchangeability; CK churns).

Label-noise control. Stratify verdicts by matchedVia ∈ {sku, upc, name} and flag answer lines with fulfilled_qty != po_qty; report wrong-match with and without those lines so fulfillment-vs-PO disagreement isn’t blamed on the matcher.

Per-change ship gate. A change ships iff: (1) held-out wrong-among-auto-submitted drops, (2) auto-submit-rate stays within an agreed budget, (3) a non-CK connection through the harness with the flag ON shows zero delta (prove the gating), (4) unit tests for the new pure functions pass.

6. Phased plan (revised after review)

Each phase is its own PR, gated + default-OFF, with the §5 protocol. Two gates precede any Phase-2 build (see §8): a business-economics decision (review cost / SLA / auto-submit-rate budget / target false-match-rate α) and eval credibility (price+gate wired so the metric being optimized is the one that ships).

• Phase 0 — Activation + deterministic gate (ship-blocker, do first). Two parts:
  • 0a. Deterministic unmatched-line block. Add the invariant in agentic_validation.py (§2a): any line lacking erp_item_id forces all_valid=false. This is what actually makes abstention fail-closed; without it, dropping a bind to “unmatched” does not guarantee review. Add a gate-simulation stage to run-ck-match-eval.ts so wrong-among-auto-submitted is measurable.
  • 0b. Flag activation via the EXISTING org-settings plane (not a new one). Reuse organization_settings.fuzzyMatchingConfig (organization-settings-service.ts), which is already persisted, already has an admin UI (FuzzyMatchingSection.tsx + endpoint), and already feeds nameOverlap into MatchGateOptions (auto-validate.ts:279). Extend its schema (fuzzy-matching-config.ts) with the matcher flags (+ an optional connectionOverrides for CK’s single connection, per doc 06 / decision D11). Thread them into MatchGateOptions at all three matcher call sites: auto-validate.ts, auto-validate-with-price-check.ts, and the auto-submit path persist-auto-priced-matches.ts (TS-worker) — missing the third means the flag is on for operator review but off for actual auto-submit. Choose Flagsmith for the runtime kill switch (instant, audited; already used for matcher-adjacent gates). Do NOT add a fourth config plane in extraConfig. Separately fix the broken extraConfig PUT deep-merge (it’s needed for ship-to resolution regardless), with a test that discountConfig/database_config survive. Validation: round-trip via the existing UI; non-CK connection byte-identical; pricePackSelect/broadenUnion ON for CK reproduces the (re-confirmed, §2d) eval numbers.
• Phase 1 — Free precision win (model-free). Suppress P4 name-only auto-binds (1 correct / 12 wrong) + abstain on unresolved pack-ambiguity (gated). Make abstention legible (required, not optional): a structured abstainReason enum (p4_suppressed | pack_ambiguity | below_relevance | no_candidate) + the calibrated/relevance score + the rejected candidate names, logged at the matcher, emitted as a per-reason metric, persisted on the line, and rendered in the CSR review UI (ReviewIssuesRail/ReviewErrorDrawer) so the reviewer sees “refused: two same-family packs, price didn’t decide” and the rejected packs pre-load into the override editor. Confirm review_v2_line_override is ON for CK so CSRs can force-bind. Re-run #1731’s A/B using extracted_unit_price (§2d) — thread it through the fuzzy fallback too — to confirm the lift is real and re-confirm the 26.3% baseline. Let it bake in prod for a measurement window.
• Phase 1.5 — Per-PO price normalization (promoted; the one strong new feature). Ship §4 as its own gated A/B ahead of any calibration. Specify the no-anchor fallback: a PO with no confident (UPC/xref) lines has no markup band — fall back to a cross-PO prior or skip the feature for that PO (do not fabricate a band). Model-free, churn-robust (relative within a document), directly attacks the dominant right-family/wrong-pack failure.
• Phase 2a — Single global calibrated threshold (the dial, simple form). Per-line interpretable risk score (§3B, FS-additive or tiny logistic) → single global Platt-calibrated threshold tuned to the chosen false-match-rate budget on held-out domains → accept/abstain. Add --matching-config to the harness; ship the split builder + a single committed calibrate command writing a versioned artifact stamped with the catalog maxMaterializedAt. This is most of the value at a fraction of the complexity.
• Phase 2b — Class-conditional conformal (CONDITIONAL, not committed). Build the pack-family-grouped split-conformal layer only if Phase 2a leaves measurable per-family FMR violations on held-out data, AND the business has set α, AND a recalibration trigger is wired to the dagster CK materialize with a staleness guard (refuse auto-submit if the calibration is older than the live catalog version). Until then, conformal is deferred: exchangeability is continuously violated by catalog churn and the long-tail classes (8 with 1-2 labels, 7 with 1) give no real finite-sample guarantee.
• Phase 3 — Not now. No contrastive embedding / cross-encoder (drops 25–56% F1 on unseen items). Revisit only with a second customer and 10x labels.

7. What NOT to build (and why)

• No heavy cross-encoder / deep contrastive matcher. WDC Products (EDBT 2024, arxiv 2301.09521): a symbolic word-occurrence baseline beats fine-tuned RoBERTa on small/medium matching; deep matchers drop 25–56% F1 on unseen items — fatal for a churning 186-product single-customer catalog.
• No per-SKU conformal thresholds — collapse below ~10 labels/class (NeurIPS 2023, arxiv 2306.09335); use class-conditional/clustered (pack-family) grouping.
• No isotonic calibration on this data size (ICML 2005; scikit-learn) — Platt/sigmoid.
• No LLM judge, no CP+RL adaptive thresholds — overkill at this scale.
• Don’t target “precision” directly in conformal-risk-control — it’s a non-monotone ratio; target false-match-rate / FNR.

8. Risks, caveats, open questions (sharpened after review)

STATUS (2026-06-17): ECONOMICS DECIDED — build unblocked. Owner (David) set the policy: review costs ~$0.10/line (cheap vs ~100–1000× per wrong shipment), so the policy is fail-closed, field-agnostic — auto-submit only when confident on ALL fields (product, customer, price, qty); anything uncertain on any field → review. Hard target: wrong-among-AUTO-SUBMITTED ≤ 2%. Coverage is NOT traded against this target — grow auto-submit rate by improving accuracy (ship-to, per-PO price, recall), not by loosening α. Both tracks pursued: ship-to/customer resolution (doc 09) AND product-line precision (this doc). Flag substrate: organization_settings.fuzzyMatchingConfig + Flagsmith kill switch.

Decisions that gated the build (now RESOLVED):

1. Economics (RESOLVED 2026-06-17). α = ≤2% wrong-among-auto-submitted, field-agnostic abstention, review ≈ $0.10/line. Coverage earned via accuracy, not by loosening α. Implication: the deterministic gate (Phase 0a) + Phase 1 model-free rules are the foundation; the calibrated dial (Phase 2a) is justified IF Phase-1 measured wrong-among-auto-submitted is still > 2%; conformal (2b) only if 2a leaves per-family violations.
2. Eval credibility / the metric is a proxy. Per doc 05, the eval uses an oracle customer (customerId inert, xref bucket fired 0/1134) and never scores price — so wrong-among-matched is decoupled from the dominant production failure, wrong retailer/customer (one wrong customer poisons every line’s match AND price). Ship-to resolution (doc 09, 93.7% top-1) may be the higher-leverage lever. Wire price + customer into the eval (and the deterministic gate, §2a) so the optimized metric is wrong-among-auto-submitted-with-correct-price before committing to Phase 2.

Other risks: Exchangeability — conformal’s guarantee breaks on continuous catalog churn; the recalibration trigger (Phase 2b) needs an owner (dagster materialize) + a staleness guard. Label noise — WhereFour-fulfilled truth ≠ PO-as-written (partial fulfillment / substitution), and it contaminates the calibration target exactly on the hard cases; flag fulfilled_qty != po_qty lines. Measurement validity — the unit_price-vs-extracted_unit_price contamination (§2d) is selective (correlates with match success), so it must be fixed before any price-feature numbers (including #1731’s merged baseline) are trusted. Per-PO anchor cold-start — POs with no UPC/xref line have no markup band (§4 fallback). Platform trap — CK-specific scoring logic (family-grouped conformal without canonical family data, match-time $/oz instead of materialized catalog fields) won’t generalize to a 2nd customer; prefer upgrading shared catalog primitives.

Dismissed-alternative to revisit: a gated LLM select-or-0 picker over the broaden-union top-K (choose among ~5 retrieved siblings or return 0 = abstain) is itself a fail-closed mechanism that needs no calibration pipeline. Deserves a head-to-head vs Phase 2a before the calibration build, not a one-line dismissal.

9. Decision-log additions (proposed; revised after review)

• D16 (corrected): Abstention is NOT free today — add a deterministic unmatched-line block in agentic_validation.py (force all_valid=false on any line missing erp_item_id), plus a gate-simulation stage in the eval. Only then does “matcher refuses to bind → review” hold. (Supersedes the original draft’s claim that the existing LLM-mediated gate suffices.)
• D17 (revised): Flag activation reuses the EXISTING org-scoped organization_settings.fuzzyMatchingConfig plane (per doc 06 / D11) + Flagsmith for the runtime kill switch — NOT a new extraConfig.matchingConfig plane. Thread flags into all three matcher call sites incl. the TS-worker auto-submit path. The broken extraConfig PUT is fixed separately (needed for ship-to), not as the matching substrate.
• D18: Price signals key off immutable extracted_unit_price, never the ERP-rewritten unit_price (the contamination is selective → fix in the eval and re-confirm #1731’s baseline).
• D19 (revised): Calibration ships as a SINGLE GLOBAL Platt-calibrated threshold targeting a false-match-rate budget (Phase 2a) first. Class-conditional split-conformal (Phase 2b) is conditional on 2a leaving per-family FMR violations + α set + a recalibration trigger wired — deferred otherwise (churn breaks exchangeability; long-tail classes give no real guarantee).
• D20: Phase 2 (any calibration build) is gated behind two decisions: the business economics (α / auto-submit-rate / review-cost) and eval credibility (price+customer+gate wired). Phase 1 + Phase 1.5 are unconditional; Phase 2 is explicitly conditional.

GSTACK REVIEW REPORT

/autoplan pipeline, 2026-06-17. Dual voices per phase (Claude independent subagent + Codex), auto-decided via the 6 principles; premises + user challenges held for the human gate. Design phase skipped (no real UI scope). The review found a critical, code-verified flaw in the original draft (the abstention gate was not deterministic) — corrected above before this report.

Review	Trigger	Why	Runs	Status	Findings
CEO Review	/plan-ceo-review	Scope & strategy	1	ISSUES → revised	8 findings; premise + scope challenged
Codex Review	/codex	Independent 2nd opinion	2 (CEO+Eng)	CONFIRMED critical	verified D16 false, config plane, conformal overbuild
Eng Review	/plan-eng-review	Architecture & tests	1	ISSUES → revised	8 findings; 1 critical (D16), file:line verified
Design Review	/plan-design-review	UI/UX	0	SKIPPED	no UI scope (only “admin UI” mention)
DX Review	/plan-devex-review	Ops/dev experience	1	ISSUES → revised	6 findings; config legibility + abstain observability

CEO consensus (Claude + Codex): scope of Phase 0/1 + per-PO price = CONFIRMED sound; Phase 2 conformal = CONFIRMED over-scoped; premise (“prefer unmatched”, target metric) = DISAGREE-with-doc (both challenge); wrong-retailer is the dominant failure the eval can’t see.

Eng consensus (Claude + Codex): abstention-gate determinism = CONFIRMED FALSE (critical); config substrate = CONFIRMED wrong plane (reuse org-settings); 3rd call site missed = CONFIRMED; extracted_unit_price contamination = CONFIRMED; conformal = CONFIRMED overbuild (global threshold first).

DX consensus (Claude): config substrate contradicts D11/doc-06 (critical); no operator UI/kill-switch; abstention invisible to on-call + CSR; calibration loop greenfield + churn footgun unautomated.

Cross-phase themes (flagged in ≥2 phases independently — high-confidence):

• Conformal is over-engineered for this data/scale — CEO + Eng + DX all independently. → resolved: deferred to conditional Phase 2b.
• Config substrate is wrong/fragmented — Eng + DX. → resolved: reuse org-settings + Flagsmith.
• The optimized metric is a proxy (oracle customer, no price) — CEO + Eng. → elevated to a Phase-2 gate (§8).

Decision Audit Trail

#	Phase	Decision	Class	Principle	Rationale
1	Eng	Add deterministic unmatched-line block + eval gate-sim (D16 rewrite)	Mechanical	P1 completeness	Code-verified the LLM gate is soft; abstention thesis requires it
2	Eng/DX	Reuse org-settings + Flagsmith; no new extraConfig plane (D17)	Taste→resolved	P4 DRY	Existing plane wired + has UI; avoids 4th plane; aligns D11/doc-06
3	Eng	Phase 0 covers all 3 call sites incl. persist-auto-priced-matches.ts	Mechanical	P1 completeness	Else flag is off on the actual auto-submit path
4	Eng	Eval reads extracted_unit_price + threads it through fuzzy fallback (D18)	Mechanical	P1 completeness	Verified unit_price is ERP-overwritten; selective contamination
5	CEO/Eng/DX	Conformal → conditional Phase 2b; global Platt threshold = Phase 2a (D19)	Taste→resolved	P3/P5 pragmatic+explicit	5/5 reviewers; churn breaks exchangeability; thin tail
6	CEO	Promote per-PO price normalization to Phase 1.5 + no-anchor fallback	Mechanical	P2 boil-lakes	Best new idea, model-free, churn-robust; was buried
7	DX	Abstention legibility (structured reason + score + metric + CSR UI) required	Mechanical	P1 completeness	Else on-call/CSR can’t answer “why unmatched?“
8	CEO	Slim Phase 0 (defer shadow-repro); Phase 2 explicitly conditional (D20)	Mechanical	P3 pragmatic	Activation PR shouldn’t carry a measurement project

VERDICT: Design REVISED, not yet approved. CEO + Eng + DX cleared after the corrections above (deterministic gate, org-settings substrate, 3 call sites, extracted_unit_price, abstention legibility, per-PO price promoted, conformal deferred). Phases 0–1.5 are implementable and high-ROI. Phase 2 is gated behind two human decisions.

DECISIONS MADE AT THE GATE (2026-06-17, owner):

• Direction → PAUSE for economics. Build does not start until the economics premise below is decided; then re-scope. (Owner chose “decide economics first” over proceeding.) The user-challenge to defer the Phase-2 conformal program is effectively accepted via the pause.
• Flag substrate → RESOLVED: organization_settings.fuzzyMatchingConfig plane + Flagsmith runtime kill switch. No new extraConfig plane. (Aligns D11 / doc 06; see D17.)

ECONOMICS RESOLVED (owner, 2026-06-17) — build unblocked: review ≈ $0.10/line; policy is fail-closed + field-agnostic; α = ≤2% wrong-among-auto-submitted; coverage earned via accuracy, not by loosening α; pursue ship-to AND product-line precision. First PR = Phase 0a (deterministic gate + eval gate-simulation) so the ≤2% target becomes measurable, then Phase 0b activation + Phase 1; Phase 2 conditional on Phase-1 measured wrong-rate > 2%.