08 — Broker Extraction Enhancements (supplier-vs-retailer + structured ship-to)

Thesis: the wrong-retailer failure (doc 05) starts upstream of the resolver. On broker documents the extractor hands the resolver a poisoned customer_name — our own customer’s brand (“Cleveland Kraut”/“Cleveland Kitchen”) or the broker parent (“AWG”) — and throws the one signal that disambiguates the DC (ship-to) into a single free-text blob. Fix extraction so it yields a clean candidate retailer + structured ship-to; the deterministic sender-domain + ship-to resolver (doc 05) maps that to the WhereFour customer. Line-item extraction is already good (~94%) — this doc touches only the customer/retailer + ship-to fields.

⚠ CRITICAL CORRECTION (2026-06-16) — supplier-grabs are an eval-harness artifact

The baseline below (45.8% exact, “24 supplier grabs”) was measured on the broker corpus that was extracted under connection da2f05f8 (Demo ERP) / org eval-service — NOT the CK connection. That config has no CK company_info. CK’s real connection (4f234677, org org_3Disst…) prompt_config already lists Cleveland Kraut/CK/ CLEVKRT as aliases of us and carries a CRITICAL requirement: “Cleveland Kitchen (and aliases …) is the SELLER/VENDOR … customer_name MUST be the BUYER, NOT Cleveland Kitchen.”

Re-extracting the 40 supplier-grab POs under the CK connection (ck-reextract-ckconn.ts) yielded 0/40 supplier-grabs (was 40/40 under Demo config). So:

The “supplier grabbed as customer” failure does NOT exist in CK production. Section 2b’s supplier-vs-retailer prompt rule is essentially already shipped (in the CK overlay/ requirements). Do not treat this as open work.
The offline lab confirmed the inverse: any prompt that names “Cleveland Kraut” as us → 0 supplier-grabs; the verbose label-vocabulary prompt (ck-broker-v1.txt) added nothing over the simple aliased baseline.
The real, remaining extraction gap is DC GRANULARITY. Under CK config: kehe → "KeHE Distributors, LLC" ✓, johare/empire ✓, but awginc → "AWG" (parent, not the division) and unfi → "UNFI"/"UNFI ACCOUNTS PAYABLE" (brand, not the division). The division is encoded in the ship-to (UNFI ships clean 7/7) or, for AWG eBilling docs that have no address at all, only a numeric Whse: code.
The true CK-config baseline must be re-measured under conn 4f234677 — the table below is invalid for production. The valid work items are §2a (structured ship_to) and capturing the AWG Whse: code, plus the resolver keyed on ship-to history (doc 05/06, domain+ship-to → 93.7%) and an AWG warehouse→DC table.

The original (Demo-config) analysis is retained below for the record.

1. Problem (real PDF evidence — measured under Demo config, see correction above)

The just-extracted broker corpus (packages/eval/reports/ck-corpus-expansion.json, 142 broker docs across awginc/kehe/unfi/johare/empirefoods, scored against golden truth) gives the baseline broker customer-extraction accuracy:

	docs	exact-customer correct
All brokers	142	65/142 = 45.8%
awginc.com	33	0/33 (100% wrong)
kehe.com	25	0/25 (100% wrong)
unfi.com	6	0/6 (100% wrong)
empirefoods.com	5	0/5 (100% wrong)
johare.com	73	65/73 (89% — distributor-as-customer; mostly fine)

77/142 are wrong. They fall into three buckets, all customer/ship-to, none line-item:

Supplier grabbed as customer — 24 docs. The extractor returns the CK brand:
- PO 48803 (awginc) truth St. Cloud → extracted CLEVELAND KRAUT
- PO 3468348 (kehe) truth KeHE → extracted CLEVELAND KITCHEN
- PO 4530490 (unfi) truth United Natural Foods - Iowa → extracted CLEVELAND KITCHEN CO. Neither “Cleveland Kraut” nor “Cleveland Kitchen” is a WhereFour customer, so name-only resolution (doc 05) returns nothing or a garbage hit. This is unrecoverable downstream.
Broker parent grabbed instead of the buying division — 39 docs. Right org family, wrong (or missing) DC:
- PO 58751 / 58401 (awginc) truth Gulf Coast → extracted AWG
- PO 47601 (awginc) truth St. Cloud → extracted AWG
- 16 kehe docs truth KeHE → extracted KeHE Distributors, LLC (parent legal name) “AWG” resolves to the broker parent, not the St. Cloud/Gulf Coast/Norfolk DC that is the actual WhereFour customer. The DC is only recoverable from ship-to.
Different retailer name / null — 14 docs. e.g. PO 1204151 truth SpartanNash → CAITO FOODS (the 3PL on the doc); PO 516813 truth Lunds → LUND FOOD HOLDINGS (legal parent); 3 nulls. Some are resolver-fixable name variants; some need ship-to.

Why it happens (current extraction has no notion of broker/sold-to/ship-to):

The full-pass prompt (packages/pdf-shared/pdf_shared/utils/local_prompts.py:22-34, mirrored to Langfuse pdf_extraction_prompt_{env}) gives only “customername = company name” and “exclude _our info”. On a broker PO the most prominent brand IS our customer (Cleveland Kraut), and the buyer is split across a “Vendor”, a “Sold To”/ “Bill To”, and a “Ship To” the prompt never names. A grep for retailer|sold_to|broker|distributor|vendor over the prompt + schema files returns zero hits.
The response schema (packages/pdf-shared/pdf_shared/llm/schemas.py:19-29, Pydantic mirror models.py:84-105) has no supplier/vendor field, no sold-to/bill-to-name, and ship-to is a single free-text string customer_shipping_address — not the structured {descriptive_name, line1, city, state, postal_code} the golden truth uses and the resolver needs.
The fast pre-extraction pass DOES have structured ship_to_* (apps/temporal-worker/activities/gemini.py:252-264) but (a) no supplier/retailer disambiguation in its prompt (gemini.py:266-272), and (b) those fields get flattened into one string and only used if the full pass left shipping empty (workflow/pdf_extraction.py:382-415). The structure is discarded.

Truth ship-to is the discriminator (golden, 190/190 broker entries carry structured ship-to). PO 58751 truth Gulf Coast: shipTo={descriptive_name:"Associated Wholesale Grocers", line1:"63331 Old Military Road", city:"Pearl River", postal_code:"70452"} — the descriptive_name is just “AWG”, but city+zip pins the Pearl River DC. PO 55700 truth Nashville: descriptive_name:"Associated Wholesale Grocers - Nashville", city:"Goodlettsville". Doc 05 measured: domain alone pins 64.8%; domain + ship-to = 100%. Extraction must surface that ship-to in structured form for the resolver to use it.

2. Prompt + schema changes

2a. Schema — additive, optional, backward-compatible

Extend the customer object in GEMINI_RESPONSE_SCHEMA (schemas.py:19-29) and CustomerInfo (models.py:84-105). All new fields optional (not in required), so a config-less org / non-broker doc is byte-for-byte unchanged.

Before (schemas.py:19-29):

"customer": {
    "type": "object",
    "required": ["customer_name"],
    "properties": {
        "customer_name": {"type": "string"},
        "customer_email": {"type": "string"},
        "customer_shipping_address": {"type": "string"},
        "customer_billing_address": {"type": "string"},
        "customer_phone": {"type": "string"},
        "customer_point_of_contact": {"type": "string"},
    },
},

After:

"customer": {
    "type": "object",
    "required": ["customer_name"],
    "properties": {
        "customer_name": {"type": "string"},
        "supplier_name": {"type": "string"},          # the vendor/brand being sold (e.g. "Cleveland Kraut") — captured so it is NOT mistaken for the customer
        "sold_to_name": {"type": "string"},           # the bill-to / sold-to retailer entity
        "customer_email": {"type": "string"},
        "customer_shipping_address": {"type": "string"},   # kept for back-compat
        "ship_to": {                                   # structured, mirrors golden truth + shipping_addresses Typesense collection
            "type": "object",
            "properties": {
                "descriptive_name": {"type": "string"},
                "line1": {"type": "string"},
                "city": {"type": "string"},
                "state": {"type": "string"},
                "postal_code": {"type": "string"},
            },
        },
        "customer_billing_address": {"type": "string"},
        "customer_phone": {"type": "string"},
        "customer_point_of_contact": {"type": "string"},
    },
},

Mirror the same fields onto CustomerInfo (all | None = None). Keep populating customer_shipping_address (join the ship_to parts) so nothing downstream breaks; the resolver reads the new structured ship_to when present.

2b. Prompt — broker disambiguation, CK-scoped via the overlay

The base prompt stays generic. The broker rules live in the CK org overlay ck_dspy_org_overlay (Langfuse), appended only for org_3DisstYyUfNutmFA68xw5hxXfq1 via compose_prompt_with_overlay (langfuse_prompts.py:344-357, CUSTOMER_SLUG_BY_ORG_ID :260). This keeps the change provably isolated to CK (other orgs never see it) and is the safe lever — same principle as doc 06’s org-scoping.

Customer-section change to the base prompt (local_prompts.py:30-33), generic and safe for every org:

Before:

- We are {{company_name}} in {{company_location}} (also known as {{company_aliases}}) — exclude our info from customer fields
- customer_name = company name, customer_point_of_contact = individual person

After:

- We are {{company_name}} in {{company_location}} (also known as {{company_aliases}}). We are the SUPPLIER/VENDOR on this document — never put our name in customer_name; put it in supplier_name.
- customer_name = the BUYER (the sold-to / bill-to company purchasing from us), NOT the supplier and NOT a broker/distributor's parent company.
- If the document distinguishes a parent buyer from a delivery location, capture the parent in sold_to_name and ALWAYS fill the structured ship_to (descriptive_name, line1, city, state, postal_code) — the ship-to destination is what identifies the specific buying division/warehouse.
- customer_point_of_contact = individual person

CK overlay addition (broker-specific, with 2-3 few-shots drawn from the REAL corpus PDFs — ck_dspy_org_overlay):

BROKER / DISTRIBUTOR PURCHASE ORDERS:
These POs come from grocery brokers and distributors (AWG / Associated Wholesale Grocers,
KeHE, UNFI, J&O/Johare, Empire). On these documents:
- "Cleveland Kraut" / "Cleveland Kitchen" / "Cleveland Kitchen Co." is OUR brand (the
  product being bought) — it is the supplier_name, NEVER customer_name.
- The broker's own parent name (e.g. "AWG", "KeHE Distributors, LLC") is NOT the customer.
  The customer is the specific buying DIVISION / DISTRIBUTION CENTER, which you identify
  from the SHIP-TO block. Always fill the structured ship_to.

Examples (from real POs):
1. AWG PO header shows "Vendor: Cleveland Kraut", "AWG", and a Ship To of
   "Associated Wholesale Grocers, 63331 Old Military Road, Pearl River, LA 70452".
   → supplier_name="Cleveland Kraut", customer_name="Gulf Coast" (the AWG Pearl River
   division), ship_to={descriptive_name:"Associated Wholesale Grocers", line1:"63331 Old
   Military Road", city:"Pearl River", state:"LA", postal_code:"70452"}.
2. AWG PO with Ship To "Associated Wholesale Grocers - Nashville, 500 South Cartwright
   Street, Goodlettsville, TN 37072".
   → customer_name="Nashville", ship_to.descriptive_name="Associated Wholesale Grocers -
   Nashville", city="Goodlettsville", postal_code="37072".
3. KeHE PO showing the KeHE logo and "Cleveland Kitchen" line items.
   → supplier_name="Cleveland Kitchen", customer_name="KeHE", and fill ship_to from the
   KeHE DC address on the document.

Mirror the supplier/ship-to instruction into the pre-extraction prompt (gemini.py:266-272) since it anchors customer_name for the merge: add “If this is a broker/distributor PO, customer_name is the buying division shown in the SHIP-TO block, not the brand or the broker parent.”

3. Measurement harness (offline, already built)

The candidate-prompt test loop already exists: packages/eval/scripts/extract-ck-broker-corpus.ts. It pushes every broker-domain golden doc through the real extraction path (POST /api/eval/upload-from-r2 → Temporal → staging DB), reads back the extracted customer, and scores extractedCustomer vs golden _wherefourMatch.customerName per domain — exactly the table in §1. It writes packages/eval/reports/ck-corpus-expansion.json (read-only against golden; only side-effect is extractions on staging).

Run it (token from op, never written to disk):

STAGING_EVAL_TOKEN=$(op item get "Staging Eval Service" --vault ERP-Unlocked \
  --fields label=service_token --reveal) \
  bun run packages/eval/scripts/extract-ck-broker-corpus.ts

(--dry-run / --limit N / --concurrency C supported.)

The A/B loop is: publish a candidate ck_dspy_org_overlay version + schema branch to staging → re-run the harness → diff ck-corpus-expansion.json against this baseline. Score two things the harness must be extended to compute (line-item scoring stays as-is):

Retailer accuracy — does extracted customer resolve to the truth WhereFour customer (after the doc-05 resolver), counting supplier-grab and broker-parent as failures.
Ship-to capture — fraction of broker docs where structured ship_to is non-empty and city+postal_code match the golden shipTo (the resolver’s actual input).

Baseline to beat (this corpus, 142 docs): 45.8% exact customer / 0% on awginc·kehe·unfi·empirefoods / 0% structured ship-to (no ship_to field exists yet). Gate target below.

Determinism caveat from doc 06 applies (Gemini + Typesense nondeterminism); report over repeats, treat sub-~2pp single-run deltas as noise.

4. How it pairs with the resolver (doc 05)

Extraction and resolution split the work cleanly — the prompt fixes what no resolver can, the resolver fixes what no prompt can:

Failure	Fixed by	How
Supplier grabbed (`CLEVELAND KRAUT`) — 24 docs	Prompt	A non-customer brand can never be resolved to a WhereFour customer by name or ship-to. Only the prompt can stop emitting it (→ `supplier_name`). Resolver cannot recover this.
Broker parent vs DC (`AWG`→St.Cloud/Gulf Coast) — 39 docs	Prompt → Resolver	Prompt must surface structured `ship_to`; the resolver maps `domain + ship_to(city,zip)` → the division (doc 05’s 100% domain+ship-to pin). Prompt supplies the signal, resolver makes the pick.
Name variant / legal parent (`LUND FOOD HOLDINGS`→Lunds, `KeHE Distributors, LLC`→KeHE)	Resolver	These are recoverable by name fuzzy-match / sender-domain affinity; the prompt need not be perfect.
Multi-DC under one shared domain	Resolver (needs prompt’s ship-to)	The resolver’s ship-to disambiguation tier (doc 05 §2a) is only usable if extraction emits structured ship-to — which this doc adds.

Net: this doc is the prerequisite that makes doc 05’s ship-to disambiguation tier real. Doc 05 measured domain+ship-to = 100% using ground-truth ship-to; the production number is bounded by ship-to extraction quality, which §3 now measures.

5. Rollout

CK-scoped and safe by construction:

Schema (schemas.py + models.py): additive optional fields, no required change, merge-join ship_to→customer_shipping_address for back-compat. Lands first; no behavior change until the prompt asks for the fields.
Base-prompt supplier/sold-to wording (local_prompts.py) + Langfuse pdf_extraction_prompt_{staging,production}: generic, safe for all orgs. Update Langfuse first (source of truth), then sync the local fallback.
CK overlay broker rules + few-shots: published to ck_dspy_org_overlay, attached only to org_3DisstYyUfNutmFA68xw5hxXfq1. Only CK sees it.
Pre-extraction prompt tweak (gemini.py:266-272) + plumb structured ship_to through the merge (pdf_extraction.py:382-415) instead of flattening.

Eval gate before shipping (run §3 harness on staging):

Broker exact-customer ≥ 75% (from 45.8%), and 0 supplier-grabs on awginc/kehe/unfi (the unrecoverable class) — this is the hard gate.
Structured ship_to populated with matching city+zip on ≥ 85% of broker docs.
End-to-end (with doc-05 resolver, --resolve-customer): broker retailer-resolution accuracy materially up vs the doc-05 baseline; no regression on the easy single-domain accounts (johare distributors, Giant Eagle) and no regression on line-item recall (this change must not touch the ~94% line extraction).
Per doc 06: a CK overlay/schema change must show no regression on a basket of other-org extraction fixtures (the schema is shared) before it ships.

Dead-code note: packages/pdf-shared/pdf_shared/ck_extraction_prompts.py (CK POC with identify_customer/extract_addresses) is not imported anywhere and the temporal-worker match_customer activity (activities/typesense.py:29-43) is a stub (return None) — neither is the place to make this change. The live levers are the pdf-shared prompt/schema (this doc) + the resolver (doc 05).