Design: Quantity-Tier Quote Extraction

Date: 2026-07-14 Branch: docs/quantity-tier-quote-extraction-design Repo: ERP-Unlocked/ordermatic Status: Proposal — awaiting review Mode: prod

Problem Statement

A VMS-issued quote showing one item at several candidate quantities (e.g. 25/50/100/250 at descending unit prices) extracts as four order lines at $0 instead of one line at the selected tier, and additionally fabricates a PO number from the quotation number. This is the only confirmed prompt-resistant extraction failure: it reproduces 100%, and all four controlled prompt variants — including one targeted specifically at quantity tiers — scored identically on it (packages/eval/docs/vms-eval-findings-2026-07-13.md, §2.5 and §3).

Status Quo

Two forcing functions produce the failure:

1. The response schema cannot express “no committed price.” unit_price is in the required list and typed as a non-nullable number in the Gemini line-item schema (packages/pdf-shared/pdf_shared/llm/schemas.py). When a document lists candidate tiers without committing to one, the model must emit some float — hence the $0 fabrications. Same mechanism fabricates the PO: the model needs an order identifier and reaches for the quotation number.
2. The pipeline has no document-type concept at extraction time. Email triage can classify a message as RFQ (no extraction), but any quote PDF that reaches extraction is processed identically to a PO: no document_type on extracted_orders, and neither the Langfuse base prompt nor per-connection prompt_config mentions quotes or price tiers (apps/temporal-worker/activities/gemini.py prompt assembly, packages/pdf_shared/pdf_shared/utils/langfuse_prompts.py).

Line items store a single quantity/unitPrice pair (packages/db/src/schema/pdf-documents.ts); no structure exists for tiered pricing.

Constraints

• Response-schema changes affect every connection, not just VMS — any change must be validated against the full eval suite, not just the adversarial set.
• Extraction behavior changes ship behind a Flagsmith flag, default off (extraction_buyer_seller_guard is the precedent).
• Any new column goes through the Drizzle schema + generated migration.
• Prompt-only fixes are ruled out by the §3 null result — this needs a structural change.

Approaches Considered

Approach A: Nullable unit_price + prompt permission — CHOSEN (phase 1)

Make unit_price nullable in the response schema (["number", "null"], removed from required) and add one base-prompt instruction: leave unit_price null when the document does not commit to a single price. extracted_unit_price is already nullable in the DB, and review UI/evals already handle null prices.

The flag selects between the legacy and the nullable schema+prompt at request-assembly time (same pattern as include_buyer_guard in the prompt builder) — flag off sends today’s exact schema and prompt, so rollback is a flag flip, not a deploy.

Nullability removes the forced fabrication but does not prohibit the model from still emitting 0. Residual guard: a $0 price is only trusted when the document states it (no-charge lines exist legitimately); on documents that phase 2 classifies as quotes, an uncommitted/placeholder $0 is normalized to null and the line is flagged for review — a zero backed by explicit wording on the line (“free”, “no charge”, printed $0.00) keeps its value. The eval tracks fabricated-$0 rate as its own metric either way.

• Pros: removes the forced fabrication for every document class, not just quotes; smallest possible diff; a null price is an honest “unknown” the review UI can surface, where $0 masquerades as data.
• Cons: does not fix the line multiplication (4 lines for 1 item) nor the fabricated PO; a global schema change needs a full regression pass.

Approach B: Quote-vs-PO document distinction — CHOSEN (phase 2)

Have the model emit a document_type (purchase_order | quote | other) in the same extraction response (no extra LLM call), persist it on pdf_documents (Drizzle migration) — classification is per-document and a document can yield multiple extracted_orders rows, so storing it on the orders would allow duplicated or conflicting values; orders read it through their pdf_document_id. Quote-specific handling gated behind a flag that, when off, also selects the legacy response schema and persistence path (not just the routing). Ambiguous or low-confidence classifications default to purchase_order handling — quote mutations apply only on a confident quote. Detected quotes get po_number = null (document number captured as reference, not as PO), tier candidates collapse to one line per document item, and the order is badged as a quote in review instead of masquerading as submittable.

Canonical collapsed line when the document commits to no tier — for a quote offering 25/50/100/250:

{ item_ids: [...], item_name: "<as printed>", quantity: null, unit_price: null }

Item identity is real and persists; quantity and price are genuinely uncommitted, so both stay null rather than picking (fabricating) a tier. Quote lines are never auto-validated or submittable; the reviewer sees the tier table in the document preview. This requires quantity to be nullable end to end: the DB column is NOT NULL today, so phase 2 includes a Drizzle migration for it alongside the extraction-schema and persistence-path changes (same flag).

• Pros: kills the fabricated-PO failure at the root; gives the review flow and evals an explicit signal (today these docs pollute the “unmatched orders” bucket); the classification is nearly free.
• Cons: touches schema + worker + review UI; classification itself can err — needs its own accuracy check on the adversarial set before the routing acts on it.

Approach C: Structured price_tiers extraction — DEFERRED

Extract tier tables as a structured object ([{quantity, unit_price}], jsonb on line items) preserving full fidelity.

• Pros: no information loss; enables tier-aware quoting features later.
• Cons: new schema, UI, and downstream consumers for a document class we currently cannot act on; without a product decision on what to do with tier data, it is speculative. Revisit if sales/quoting wants tier data — A + B do not preclude it.

Recommended Approach

Ship A, then B, each behind its own flag:

1. Phase 1 (A): schema + one prompt line. Verify: the schema no longer forces a price — fabricated-$0 rate on the quantity-tier adversarial docs drops to near zero; no regression on the well-formed golden set. (The hard zero-fabrication guarantee needs phase 2’s classification + guard.)
2. Phase 2 (B): document_type classification persisted and, once its accuracy is confirmed, quote routing (null PO, single line, review badge).
3. C deferred until a concrete consumer for tier data exists.

Open Questions

1. Should detected quotes appear in the normal review queue (badged) or in a separate queue? Affects phase 2 UI scope.
2. Is there near-term product interest in tier data (would pull C forward)?
3. Should extracted_orders carry a denormalized read-model copy of the document’s document_type for query convenience, or is the join through pdf_document_id enough?

Success Criteria

• Phase 1: fabricated-$0 rate on the quantity-tier adversarial docs (packages/eval prompt-analysis harness) drops to near zero; no golden-set regression.
• Phase 2: one line per document item with null quantity/price when no tier is committed, zero fabricated $0 prices, no PO derived from a quotation number.
• No metric regression on eval:vms-ground-truth and the extraction eval over the golden set with flags on.
• Flags off → the request sent to Gemini (schema and prompt) and all persistence behavior are identical to today.

Dependencies

• Flagsmith flags (extraction-guard pattern).
• Drizzle migrations for document_type on pdf_documents and quantity nullability on line items (phase 2).
• Eval reruns require WARP + VMS SQL creds (packages/eval/docs/vms-ground-truth-eval.md).