Customer optimization playbook

This playbook covers how we make a customer install work well after the connection is live. It covers data readiness, ERP sync health, item and customer matching, extraction, validation and pricing, customer-specific rules, measurement, and rollout. Each SOP says when to use it, what to do, how to prove it worked, and what happened when we did it for a real customer.

Use it two ways. When you pick up a customer, read the SOP for the problem in front of you and follow the steps in order. When you join the team, read the lessons learned and anti-patterns sections first. Most of our rules exist because something broke.

Customers are anonymized. Each one keeps the same pseudonym everywhere in this page. ERP names, industries and numbers are real.

Pseudonym	What they are	ERP
Customer A	Food manufacturer selling to grocery chains and distributors	WhereFour
Customer B	Industrial label and marking manufacturer	Eclipse E4
Customer C	Pipe fittings distributor	Prophet21
Customer D	Waterworks distributor, proof of concept	Prophet21
Prospect E	Bag manufacturer, pre-sales proof of concept	Epicor Kinetic
Prospect F	Fastener distributor, proof of concept	Prophet21
Prospect G	Materials manufacturer, demo connection	none

Buyers who send orders to these customers appear as Buyer A1, Buyer A2 and so on, numbered within each customer.

Dates are 2026. When two sources disagree, we use the newest measurement and say so.

How a customer install fits together

An order moves through six stages. Each stage has its own per-customer knobs, and most of our bugs live at the joins between stages.

1. Intake. Email pull, manual upload, SMS or EDI copies arrive. Triage decides whether a message is an order. Per-customer knobs live in the connection’s email_config, including trusted_sender_domains, trusted_sender_patterns and excluded_sender_addresses, and in triage flags such as email_triage_agent_v1.
2. Extraction. The model reads the document and returns header fields and line items. The live prompt comes from Langfuse by the label that matches NODE_ENV. Per-customer prompt text lives in erp_connections.prompt_config (companyAliases, customInstructions) and in per-customer overlay prompts named <slug>_dspy_org_overlay. See packages/pdf-shared/pdf_shared/utils/prompt_dynamic.py and langfuse_prompts.py.
3. Validation of customer and ship-to. Name search, sender, ship-to address and trading-partner hints pick the ERP customer. Knobs are matching flags such as matching_customer_address_disambiguation_v1, matching_shipto_resolver_v1 and matching_customer_hints_v1.
4. Validation of items and price. The matcher binds each line to an ERP product through exact id, UPC, cross-reference, customer history and fuzzy search. Price comes from the document first, then ERP tier data. Knobs live in extra_config.matchingConfig, extra_config.pricingConfig, catalog exclusion rules, and per-lever matching_* kill switches. The schema is packages/contracts/src/connection-config.ts.
5. Review. A CSR fixes what is left. What the CSR sees is controlled by the connection’s extraction_schema override, which is display only.
6. ERP submit. Each ERP write path has its own default-off flag, such as wherefour_order_submission_v1, eclipse_e4_order_submission_v1 and the org-level erp_write_enabled.

Under all of this sits the data pipeline. Dagster and DLT pull ERP data into shared bronze Iceberg tables, dbt builds silver and gold, and Dagster publishes products, customers, ship-tos and cross-references to Typesense. The matcher only sees what reaches Typesense.

Flags are declared in infrastructure/flagsmith/configs/flags.json. Per-customer targeting lives in segments.json and segment-overrides.json in the same folder, and the deploy applies them.

SOPs

SOP-OPT-01: New customer install and data readiness

Use this when a customer has signed and the ERP connection is being set up, or when a proof of concept starts ingesting real documents. It sits on top of SOP-OB-001, which covers the 14 gated install steps.

1. Run the SOP-OB-001 connection test. Nothing else starts until it passes.
2. Run the instance explorer where one exists. Acumatica, Prophet21 and NetSuite have explorers. Dynamics, Odoo and E4 use the manual checklist. Record every non-default finding, because custom fields mean different things per customer. One customer used a UDF for product category and another used the same UDF for supplier code.
3. Set extra_config, email_config and trusted senders. Trusted sender patterns are regular expressions, not shell globs. A catch-all pattern skips LLM triage for everything that matches it. Merge email_config with the jsonb || operator so existing keys survive.
4. Run the full initial sync. Do not invite users until products and customers both show a completed sync.
5. Check the data, not the job status. For each table the matcher needs, count rows in bronze and in Typesense. Check that key fields are populated: zip on ship-tos, UOM fields on products, delete_flag on customers, description and UOM on cross-references.
6. Measure the catalog shape. Record placeholder share, how many variants share one base item, and how many lines carry a printed SKU or UPC versus description only.
7. Ask the customer early for any cross-reference export and item master they have. Treat it as the long pole.
8. Plan history backfill as its own step. A new inbox only looks back 48 hours on first connect, so a baseline needs a one-off pull. The constants are _MAX_LOOKBACK in apps/temporal-worker/activities/gmail.py and _INITIAL_LOOKBACK in microsoft_graph.py.
9. If this is a proof of concept, write the cleanup script before ingesting. It must cover every table and store the POC writes to.

You know it worked when the explorer findings are written down, every matcher input table has nonzero rows with populated key fields, and a known SKU and a known customer come back correctly from search.

Real example

Prospect F’s Prophet21 ship-tos all had an empty zip. 0 of 1,618 Typesense ship-to documents carried one, and a later bronze check found 0 of 128,588 rows across 11 Prophet21 connections. E4 and WhereFour connections had zips. The transformer read postal code from several fields but never fell back to phys_postal_code. We added that fallback and deliberately left out the mailing postal code, because the mailing address is not the ship-to. An empty zip removes the zip gating the matcher uses to tell addresses apart. PR #3578.

Real example

Customer D’s catalog had 320,711 Typesense documents, and 73% of them were one-off placeholder items. One physical fitting mapped to 6 to 15 vendor and coating SKUs. Measuring this first told us a generic text matcher would fail and that the customer’s own item-id grammar was the lever. See SOP-OPT-03.

Real example

Prospect G’s demo cleanup script was copied from an earlier one. Review found it missed live-forwarded PO bodies, email aliases, email pull connections, inbox routing rules and Clerk memberships. The NDA required deleting all of it and purging raw files from object storage. We now write the cleanup list before ingesting.

SOP-OPT-02: ERP data sync health

Use this when matching, pricing or customer search looks stale or empty, when a sync run fails, or after any change to a bronze asset, dbt model or Typesense schema.

1. Judge a sync by rows, not by run status. Group rows by _dlt_load_id and compare counts. Long Dagster syncs can show FAILURE while data still commits, and a scrape can show SUCCESS while writing zero rows.
2. Read the real error chain in the Dagster run log before guessing. Run duration is not a diagnosis.
3. For a Typesense collection, read the live collection schema and the run log for PATCH errors. Every new field on a populated collection must be optional: true with a planned reindex.
4. For Dagster-side flag gates, read the --vars in the run’s own “Running dbt command” log line. A flag showing ON in Flagsmith proves nothing for a Dagster run.
5. For paginated E4 reads, pass an explicit order_expr. A wrong primary key silently skips or duplicates rows.
6. Never hand-roll a bronze write and never use a replace disposition on a shared bronze table. Bronze is multi-tenant and partitioned by connection_id, so replace wipes other customers.
7. After a fix, verify the effect from a separate read: search for the product, count the joined rows, or run the eval.

You know it worked when row counts move as expected, the searched record shows the new value, and a downstream join count is the same or higher than before.

Real example

Customer A’s bronze sync failed all 27 runs from October 5 to 13 on one orders table with “Cannot change column type: total double to string”. Prod and staging tables disagreed on column types, and a staging-only fix hard-coded staging’s types and broke prod. Iceberg cannot change a string column to double in place, so fixed type hints can never fit both. The fix reads the live table’s column types at write time. PR #3880 caused it, PR #4164 fixed it.

Real example

Customer A’s dbt runs ran without catalog exclusion and retired-field vars on all 97 runs after September 8, while Flagsmith returned True for those flags. Dagster run containers had no Flagsmith env, and the SDK silently returned the default. 33 single-container SKUs that should have been excluded stayed searchable. PR #3675.

Real example

Customer B’s E4 order lines carried revision suffixes on item_id, so per-SKU history joins to products matched 26.2% of 41,555 lines. Joining on a stripped base_item_id matched 83.1%. The partial join returned plausible empty answers with no error. PR #3605. Count joined rows before and after any model change.

SOP-OPT-03: Catalog and item matching, cross-references and UoM

Use this when lines come through unmatched, bind to the wrong product, or bind to the right product in the wrong pack size.

1. Check whether matching ran at all. Look at erp_item_id and pricing_writeback_status per line before blaming extraction. Intake matching only runs when the connection’s pricingConfig.autoRepriceOnExtraction is set. Confirm key names against the Zod schema before writing SQL.
2. Split every wrong match into wrong pack versus wrong product, and every miss into recall (right product never retrieved) versus ranking.
3. Measure the ceiling of a lever before building it. Count how many real lines print a live SKU or UPC, and how many are customer code only.
4. Fix the direct-ID path first. If the PO prints an identifier the catalog holds, exact match should win.
5. Use customer history as a filter, not an answer. Restrict to items this customer has bought, then disambiguate by unit price and pack text.
6. Turn on levers per connection, one at a time, each behind its own matching_* kill switch. Some levers are only safe together. Skipping the UPC name check is only safe when the UPC has one hit or pack disambiguation is on.
7. For customers whose item ids encode size, end and type, add a per-connection catalog normalizer chosen from extra_config. Do not add a special case in the matcher.
8. Write cross-references to Postgres manual_cross_references, then let the DLT, dbt and Dagster chain publish them. Anything pushed straight to Typesense is deleted at the next reconcile.
9. Run any xref backfill with --connection-id and --since, review the dry run, then commit. Only active products qualify.
10. For UoM, check that product UOM fields reach Typesense before enabling any conversion gate. A CSR-taught conversion applies only after two confirmations for the same customer, product and unit.

You know it worked when the matching eval on the prod catalog shows higher recall with the same or lower wrong rate, reported separately for wrong pack and wrong product, and run-to-run noise is accounted for.

Real example

Customer A’s matcher recall@1 was 61.3% on 1,096 aligned PO lines, while a direct-ID search returned the right product at rank 1 for 94.7%. 98.2% of lines printed a live SKU or UPC. Only 1.8% were customer code only, which caps what a cross-reference backfill could add. We moved the work from xref backfill to the direct-ID path.

Real example

Customer A’s UOM conversion gate destroyed 576 of 1,048 binds, 55%, on a replay of 1,415 prod lines. Every abstain said no conversion path. All 140 product documents in Typesense had no UOM fields, even though bronze held them for 326 of 326 finished goods. The dbt raw model hard-coded unit_of_measure to NULL. We removed the gate from prod config the same day and fixed the projection chain.

Real example

Customer C’s buyer printed codes with a vendor prefix, while the ERP indexed the bare code. The model emitted the bare code only for some separators, and an unmatched line blocked submit with no override. The fix adds the prefix-stripped code as an exact-only extra candidate. PR #1533, live June 8.

SOP-OPT-04: Customer and ship-to matching

Use this when orders land with no customer, the wrong customer, or the wrong ship-to, or when duplicate ERP accounts exist.

1. Measure how often the customer is filled today and against what truth. Use the account a CSR hand-keyed in the ERP for the same PO as ground truth. CSR picks inside our tool are biased toward the search the CSR used.
2. Use deterministic signals before names. Sender email, ship-to address and EDI trading-partner codes beat name guessing. A sender that maps to more than one customer abstains.
3. Use the PO ship-to to choose among name-search candidates when a company has several ERP accounts.
4. Auto-apply a ship-to only when the top candidate strictly beats the second. Do not add numeric score floors.
5. Exclude deleted customers in gold and reconcile the Typesense customer collection. Upserts never delete. Retired is not deleted, so use the active flag for retired records.
6. Any lookup tool needs a result cap and must refuse when candidates span more than one customer.
7. Leave the field blank when no candidate matches, such as a drop-ship. A blank costs a CSR a search. A wrong customer costs a wrong price tier and a wrong shipment.

You know it worked when precision and coverage on ERP hand-keyed truth both move up, and the number of wrong fills does not rise.

Real example

Customer B had 0 of 912 untouched orders with a customer filled. Name search plus PO ship-to, measured on 133 POs a CSR hand-keyed in E4, went from 96.3% precision and 80.5% coverage with name only to 97.6% and 92.5% with the address rule. Wrong fills went from 4 to 3. The sender-domain method that worked for Customer A filled only 26% for Customer B and broke on forwards. Flag matching_customer_address_disambiguation_v1, PR #3688.

Real example

Customer A’s ship-to resolver was once reported at 93.5% top-1. That number came from a measurement bug fixed in PR #3244, and we treat it as invalid. On the prod-parity sweep it scores 99.3% precision at 91.6% coverage on 154 cases with tie-abstain. Any score floor removed all 35 correct answers from one warehouse seed and zero wrong ones. The one remaining wrong case is a drop-ship no threshold separates. PR #3282.

Real example

Customer C’s order was rejected by Prophet21 for an invalid customer. The matcher picked a customer record deleted since 2022 over its active duplicate, at a hardcoded 0.95 confidence. Products filtered deleted rows at every layer. Customers filtered them at none.

SOP-OPT-05: Extraction tuning and prompt rollout

Use this when lines are missing, merged, or misread, when a field comes back empty, or when you change any extraction prompt.

1. Check the plumbing before the model. Compare items_after_parse with raw items seen, check finish_reason, and check that the attachment was stored. The email.persist_attachments span shows attachments_total of 1 or more with persistable 0 when a PDF was read but not saved.
2. Treat a zero-line result from an email with content as a failure. After repeated identical failures, route to CSR entry.
3. Look at the prompt wording for contradictions. Small models take verbs literally.
4. Edit the prompt in Langfuse as a new version. Run the scope checker (packages/pdf-shared/scripts/check_live_prompt_scope.py) on the live prompts.
5. Measure every extraction path separately: PDF, OCR, tabular and image. A fix on one path can regress another.
6. Roll out by moving the environment label onto the new version. Roll back by moving it back. Never include latest in newLabels, which returns 400.
7. Read back the live prompt by label before calling it live. The repo copy in local_prompts.py is only a fallback and drifts.
8. For per-customer text, use prompt_config or the customer overlay. extraction_schema changes what the CSR sees, not what the model extracts.

You know it worked when per-path row counts match the fixtures, the live label points at the new version, and a single-document re-extraction shows the change.

Real example

Multi-chunk PDF fixtures lost line items. A 10-page fixture scored 53% and the rest about 5%. Our prompt said “Consolidate all items into one order”, and small models applied that to line items, bundling 19 part numbers into one row. We first blamed model capability. A reworded clause took one slice from 23 rows to 210 of 210. Pasting the same wording into the tabular prompt dropped 110 rows to 22, so PDF and OCR carry the full clause and tabular and image only lose the verb. Live as production v24, staging v12, development v47 on September 1. PR #3290.

Real example

Customer A image POs extracted 1 of 8 items. Routing images through the PDF pipeline fixed it, behind image_extraction_via_pdf_pipeline_v1. The first per-connection enablement would have done nothing, because the flag helper never sent the connection id. The fallback result was also cached for 30 days under the flag-on slot. Adversarial review caught both. PR #2439 and PR #3074.

Real example

An email from Buyer A2 produced an order with no lines on three prod runs. Each call took 56 to 62 seconds and returned an empty list with no error. Good runs returned both lines. The retry button was hidden until the item was claimed. A CSR keyed the two lines by hand.

SOP-OPT-06: Validation and pricing rules

Use this when prices are wrong, an ERP rejects or alters a submitted order, or you add a validation rule.

1. The source document price wins. ERP tier price is a non-blocking flag with a one-click apply. A missing price is null, never $0, and the server refuses unpriced submits. This was decided July 2. Whether pricing_source_price_authoritative_v1 is global or per connection is inconsistent in our records, so check the resolved flag for the connection before you claim a rollout.
2. Before trusting a price source table, match it to exact values on a screen the customer showed you and check its row count.
3. Before changing an ERP write payload, probe the behavior on a test job.
4. Treat each submit route as its own contract. A rule that lives in one route is missing from the other.
5. For revision and identity fields, require an exact match or a CSR review. Do not auto-normalize or auto-swap.
6. Merge duplicate SKU lines only when price and UOM match. Otherwise refuse with an error that names the SKU.
7. Put deterministic checks first and spend model calls only on fuzzy questions such as company name.
8. For any line dispute, read what we actually sent. For Prophet21 that is the request payload in integration_audit_trail. For split orders the oracle is orders.status plus erp_order_number, not quote_splits.status.

You know it worked when a test job lands with the expected price and flags in the ERP, and verification sums per item rather than per sent line.

Real example

Customer B’s E4 marked every line we priced as a manual price override. With no sell price, E4 priced from its own schedule correctly, for example 120 units at $3.89 in the 100 to 149 tier. Our saved price was stale or from the wrong quantity break on 12 of 66 lines. We now omit the price only when the CSR did not override it and E4’s tier gives the same total to the cent. Update calls still send the price, because omitting it there silently reprices. PR #3478.

Real example

Prophet21 collapses lines that share an item id, and the later quantity wins. On a demo quote for Customer C, 14 rows went in and 11 lines landed. Quantities of 10 plus 6 became 6, and 4 plus 2 became 2. Our verification compared per line and reported “sent 10, erp 6” when 16 were ordered. Flag p21_merge_duplicate_sku_lines_v1, default off, PR #3686.

Real example

Customer B prints padded revisions such as 000 or AA.0 and stores unpadded values such as 0 or AA. A dry run of a strict gate on 85 bound lines would stop 24: 11 had no PO revision, 10 were padding only, and 3 were real revision changes. Auto-normalizing would have picked the wrong revision for those 3. The decision on October 12 was exact match or CSR review, with a blank PO revision taking E4’s value.

SOP-OPT-07: Customer-specific fields, rules and org flags

Use this when one customer asks for a UI change, a new field, an exclusion rule, or behavior other customers should not get.

1. Before removing or changing a shared UI element, check other customers’ usage in audit logs.
2. Gate the change per org in Flagsmith. Use the org segment for org-scoped features and the connection segment only for connection-scoped ones. Customer A has one of each and they key on different identities.
3. For a new per-line extraction field: add the schema key default hidden, add the column and migration, gate extraction behind a flag on every extraction path, enable per org in segment-overrides.json, fix any downstream dbt or Typesense mapping, and add a test that the flag-off schema is unchanged.
4. Show the field to CSRs only after the extractor fills it, through the connection’s extraction_schema override.
5. For operational rules the ERP does not record, such as retired items or EDI accounts, use a connection rule like catalog_exclusion_rules or a customer-name archive rule. Cross-check name rules against live ERP history before seeding them.
6. Replace hand-kept customer tables with values mined from ERP history, behind a default-off flag, and use the customer’s table as the first check.
7. Keep customer facts in per-customer files and config, never in shared skills or the global prompt.

You know it worked when the customer who asked sees the change, a control org reads the flag as off, and the field fills on real orders before it is shown.

Real example

Removing the row Trash action for Customer B’s request broke Customer A, which had used it 66 times in 60 days. We restored it as the default and put the new action behind the org flag inbox_row_complete_v1.

Real example

A Customer A PO had a blank price column, and the model priced the line from the gross weight column at $16. We added weight, weight unit and measure basis as line fields plus a price-column-only rule, behind extraction_line_measure_v1. The first cut only covered one of the two extraction paths Customer A’s email PDFs use. WhereFour already exposed weight on 198 of 326 finished goods, but our dbt model dropped it. PR #3610.

Real example

Customer B’s EDI clearinghouse kept a 39-row spreadsheet of due days per ship-to, set by hand from 1 to 10. We mined per-ship-to buffers from E4 order history that reproduce the pattern, with a state and region fallback. Flag eclipse_e4_historical_due_days_v1, default off, PR #3440.

SOP-OPT-08: Measuring a baseline and proving improvement

Use this before any matching, extraction, triage or validation change, and before quoting any number to a customer.

1. Write the bar and the stop rule into the plan before running.
2. Pick the harness by the question. Matching uses packages/eval/scripts/run-match-correctness-eval.ts, or the per-customer run-<customer>-match-eval.ts beside it, on the prod connection read-only. Search uses packages/eval/scripts/run-search-id-eval.ts before and after on the affected connection. Validation uses run-auto-validation-eval.ts. Each report should record connection id, dataset and git revision.
3. Run against the prod catalog and config. Staging catalogs and configs drift.
4. Pass eval flags as a shell array. In zsh, a variable holding --a --b is one argument and is silently ignored.
5. Run repeats. Matching runs vary by about 1 to 2 points, so smaller deltas are noise.
6. Check your labels before blaming the model. A synthetic ERP number, a reply thread, or a label sampled only from engine output can produce false misses.
7. Use splits that keep an order’s own lines from teaching the lever it is scored on.
8. Report every rate with its denominator, its coverage and a confidence interval or lower bound.
9. Label every number with environment, scorer version and date. Numbers from different catalogs or scorers are not a trend.
10. Reconcile row counts and sums against the source before any number goes into a customer document.
11. Build a control that changes one variable before proposing an infrastructure fix.

You know it worked when someone else can rerun the eval from the report and get the same number within the noise band.

Real example

Customer A’s matching numbers come from different setups and must not be read as one trend. On the prod catalog with prod levers, October 12, recall went from 79.0% to 89.3% and wrong among matched from 18.5% to 8.9%. On a 199-document set with a fixed scorer, recall was 61.4% and wrong pack 17.0%, after scorer fixes moved wrong pack from 12.4% to 17.8% to 17.0%. An earlier June run on 66 orders gave 60.4% recall and 26.1% wrong. The staging QA catalog had 181 products against 73 in prod, and the same SKU had different product ids per connection.

Real example

A sandbox validation run for Customer A reported 99.8% bucket accuracy and looked green. The same run showed 5.7% auto-submitted and 7.2% price accuracy. Bucket accuracy only checks that a line landed in the right bucket. We now read auto-submit and price first.

Real example

A pricing pull for Customer D through a remote shell mixed shell output into the rows. It returned 436,953 lines against 343,382 that exist, with 40,858 duplicates and about 35,000 missing. The bad pull gave $5.38M in overrides against $5.78M on verified data. The verified pull also found a script dropping 4,084 NULL-price lines because it compared to the string “NULL”.

SOP-OPT-09: Safe rollout and verifying live

Use this for any flag flip, config change, prompt label move or deploy that changes customer behavior.

1. Declare every flag in flags.json before code reads it. An undeclared flag silently returns the caller’s default.
2. Before merging, diff segment-overrides.json between main and staging for every flag in the release. The deploy applies overrides, so a merge is a production change. default_enabled: false does not mean the change ships dark.
3. Confirm the code path sends the identity the segment matches. A connection segment never matches if the code only sends org or user identity.
4. Read the resolved value through the SDK with the full connection UUID and the connection_id trait. Use the server key the deployed service reads. The local .env key points at the Development environment.
5. List segments and priorities that could override your change. A higher-priority segment can silently win.
6. Removing an entry from segment-overrides.json does not turn a flag off. Set it to false and deploy, or remove the override through the admin API.
7. Confirm the running process changed. Check container creation time and the startup log line. A webapp-only PR does not rebuild the Temporal worker.
8. Check the data oracle. Each fix has a row or log line that proves the new path ran, such as an email_events.triage_engine_status value, an E4 routing log line, or an SO<n>-OM number in orders.
9. For ERP writes, keep the write gate default off, add a second check at the route, and run shadow mode before real writes. Shadow paths must be provably write-free.
10. Set kill triggers before turning on auto-submit. The plan uses ERP rejection above 2% or CSR override above 5%.
11. Monitor freshness and liveness, not status fields. Alarm on stale poll times across all connections and on paused connections that still carry an error.

You know it worked when the SDK read, the running build and the data oracle all agree.

Real example

email_triage_agent_v1 was shipped twice and never fired. For Customer B the segment was right but the code never sent the connection_id trait. For Customer A the feature segment was never created, so an audited fix for a paying customer sat unused for about three days. Zero rows with the expected engine status over a real window means the flag is not live, whatever the dashboard says.

Real example

eclipse_e4_fg_base_code_match_v1 shipped September 18 and did nothing until September 22, because it was not declared in flags.json. Customer B lines kept failing with “E4 has no finished good by this code” for four days.

Real example

Customer A’s go-live on September 8 had three locks: an org-level approval-only mode, a connection-level submission flag, and a fail-closed submit override in the database. We checked prerequisites, flipped two identity-level overrides, and confirmed success in the orders table. The first real submits landed about an hour later. Two weeks in, a CSR found that wherefour_delivery_date_v1 had never been enabled for Customer A, so 17 of 23 orders sent the extraction timestamp instead of the PO date. PR #3604.

SOP-OPT-10: Diagnosing a bad order

Use this when a CSR or customer reports a specific order as wrong, missing or stuck.

1. Find which stage failed. Did the email arrive, was it triaged as an order, did extraction return lines, did the attachment persist, did matching run, which submit route did it take, what did the ERP receive?
2. Check intake first if the order is missing. Classify the message by its latest workflow run. Group email_events by day and sender. Most mailbox mail legitimately never becomes an event.
3. If every line is unmatched, check matching before extraction.
4. If the document view says “Couldn’t locate this item on the document”, check whether a PDF was extracted but not stored.
5. If the price is wrong on a correct item, check the unit before rebinding. ERP prices per foot against a PO per roll is a common case.
6. If the ERP has something different, read the outgoing payload and the ERP record, and check whether someone edited the ERP after submit. Check update timestamps before treating current ERP state as submit-time state.
7. For split orders, read orders.submission_error for the real error. Repairs are paired: clearing a split claim without soft-deleting the failed order row blocks retry another way.
8. Check for a parallel manual entry of the same PO before comparing outcomes.
9. Write the finding down with the order, the stage, the cause and the fix, so the next person starts there.

You know it worked when you can name the stage, show the evidence row, and the fixed order or the next similar order comes through correctly.

Real example

Customer C PO with 7 lines landed in Prophet21 as 4. The matcher sent 4 uncatalogued lines to one special-order placeholder at $0.00, and Prophet21 merged them. A slow Prophet21 call then hit the 2-minute activity timeout with no HTTP timeout, the retry’s dedupe said “already exists”, and the app recorded failure while the partial order stayed in the ERP. We removed the blind top-hit prefill and put timeouts under ERP calls.

Real example

Customer B split orders failed and could never be retried. All 6 failed splits between July 16 and September 22 wedged, because the split claimed its order id before the ERP call and never released it. The client also overwrote the real error in 4 of 6 rows. Separately, 8 of 8 fully split-submitted POs sat at awaiting review because the split path never flipped the work item. PR #3642 and PR #3733.

Real example

Customer B’s first submission comparison showed every test job as a duplicate. 30 of 39 test jobs had a hand-keyed twin with the same customer and PO, because production control keyed orders in parallel. The comparison now pairs each job with its twin.

Lessons learned

Intake and email

Mail that staff had read was never ingested. The Gmail poll asked for unread messages only, so 66 of 369 inbound mails were ingested in one week. We now poll by a last_polled_at watermark, PR #1874. The watermark is floored at 48 hours to limit blast radius, so an outage longer than that still loses mail. One did, and left Customer A with zero emails on July 13 against a weekday baseline of 53 to 83. Run the backfill workflow whenever a gap is longer than 48 hours.

Pausing a broken mailbox turned off its alarm. Customer C’s mailbox token failed on August 3 and someone paused it 18 minutes later. The monitor treated paused as intentional. Weekly intake went from about 60 to 3, then 0, then 0, and nothing paged. We now page on a paused or disconnected row that still has last_error, after a 24-hour grace, PR #3126. Disconnect a broken mailbox rather than pausing it.

One long Message-ID stopped intake for every customer. A notification with a 253-character Message-ID overflowed a 255-character workflow id. Temporal retried 742 times in 67 minutes, every later dispatch cycle was skipped, and about 2 hours of ingestion were lost with every dashboard green. We cap external strings before they enter ids, and alarm on stale poll times across all connections.

Per-email triage cannot see a thread. Customer A’s skip rate looked alarming at 93.5% but was mostly correct. Real loss was about 14 POs in 25 days, roughly 16% of volume, spread across about 10 domains. Order changes landed as disconnected skips: 137 update messages across 96 threads, only 3 linked to an ingested order. We shipped an override that processes a fresh, attached, first-in-thread PO that is not on file. We now analyze by thread, not email.

Internal forms polluted the queue. 40 of 167 Customer A “orders” in 30 days were the customer’s own sample request spreadsheets, producing 617 unmatchable items. We added a deterministic skip rule inside the triage activity, behind email_triage_sample_request_form_v1. Rules go in activities, because branching in the workflow body breaks replay.

A sender gate blocked first orders. Customer B skipped a real PO from a known buyer because a sender-history gate needed a prior bound order, and manual uploads never create one. Of 61 unknown-sender skips, 42 had no PO token. We removed the gate in PR #3993 and accepted more review load.

Extraction

Two code paths disagreed about what an attachment is. Extraction accepted any .pdf name, while persistence trusted file extensions only for images. A PDF with an octet-stream type was read but never stored, and every line showed “Couldn’t locate this item”. Logs stayed clean. Keep one file-type rule for both paths.

Addresses were extracted and thrown away. A July audit found extracted ship-to and bill-to never reached extracted_orders, billing was always None in validation, email orders always validated with no shipping address, and the prompts had no address instructions. Address is how we tell duplicate ERP accounts apart, so every downstream lever lost it. We treat address extraction as unmeasured until it is persisted and has fixtures.

Two photos became one order. An SMS intake merged two handwritten photos into one PDF, and the first page’s order disappeared while its customer hint carried into the second. This is still a hypothesis to verify. The candidate fix is one document per photo.

Fractional quantities were rounded down. 16 Customer D lines saved 291.2 FT as 291. The extraction eval caught it at 99.2% against a 100% target.

Matching and catalog

Pack size, not product, is the main error for Customer A. In the June run, 52 of 70 wrong binds were the right product family in the wrong size. Quantity added nothing as a signal. Per-SKU tier price added 8 points and customer pack preference added 4.6 points on the ambiguous lines.

A buyer’s printed SKU can be wrong. Buyer A3 printed Customer A’s 6-count code on a line priced at the 12-count case price. Exact-ID matching trusted the code. A CSR-confirmed customer mapping fixed that buyer. GS1 UPC recovery measured 76.2% correct and 20.0% wrong on Customer A history on October 13, so we do not enable it there. An earlier plan listed it as enabled and helpful. The October measurement is newer and wins.

Generic names are not one item. Customer B’s catalog has 14,324 products, and one generic decal name appeared at 17 different prices. Customer history for Customer B had 0 documents because the gold model read only WhereFour order lines. Turning on the history gate would have done nothing.

Clean-code evals lied for Customer D. On clean item codes matching scored 100% precision and recall. On customer wording it scored 32.7% precision and 19.7% recall, writing a wrong item on 40.5% of lines. Description-only matching wrote 789 wrong items against 345 right. Evaluate in the mode the live button uses.

Customer D’s buyers write the vendor’s id grammar. A grammar parser read 25,413 of 86,236 stocked ids, 29.5%, into 4,531 keys, with 99.1% id and description agreement. Disagreements were catalog errors. PR #3920 shipped the parser and lever. Customer D’s bigger win was stock, measured October 17 on the ERP replica: 20.5% of non-placeholder lines had zero at the selling branch, and 67.5% of those had the same item at another branch.

CSR fixes take hours to reach the matcher. A mapping a CSR confirmed at 20:15 was not live for hours, because it rides on Customer A’s slow ERP sync, about every 8 hours. Tell CSRs about the lag. The recommendation is a separate sync path for confirmed mappings.

Is Customer A matching at intake? Plans and config notes say levers such as customer autobind and 178 cross-references were enabled. A measurement on October 13 found every awaiting-review order had 0 matched lines, because the Python batch matcher is a stub and the connection has no autoRepriceOnExtraction. The October measurement is newer. Check the live connection row before asserting either.

Validation and ERP writes

Deterministic checks beat a model loop. The model-based validator took a p50 of 9.98 seconds over 2.87 rounds and let 9.6% of faulty orders through to auto-submit on a 300-run synthetic benchmark. A deterministic-first engine with a narrow model judge ran at p50 0.12 seconds with 0 misses on held-out seeds. Faults were synthetic, so this does not prove real-world rates. It runs dry for Customer A while auto-submit and ERP write stay off.

Retries created duplicate ERP jobs. A Customer B submit that timed out but landed was retried under a new job number, creating a real duplicate production job. We now claim one number per workflow and resubmit under it.

Kept items must carry their own revision. When a CSR keeps a bare finished good over a PO line that prints another revision, the line must carry the kept item’s revision, not the PO’s. Customer B has 13 pairs of bare and revision codes that are different finished goods.

A gate that exists on one path promises nothing on the other. The catalog check runs only in the single-order submit route. Customer B’s multi-line orders take the split route, which never runs it. A UI warning that implied a block trained CSRs to ignore warnings.

We never recorded what we sent. Customer B drift reports could not prove what was submitted, because integration_audit_trail had zero writers and zero rows. We built an ERP write ledger that stores the outgoing request per submit.

Pricing

A zero was used as a null. Legacy extraction stored missing prices as $0, and a Customer A filter dropped zero-priced products from search, so a product search returned nothing. We nulled 8,916 legacy rows in staging and 2,567 in prod, keeping 2 legitimate $0 lines.

The price matcher hit the record without a price. Customer A’s Buyer A4 PO priced every line at list instead of contract, $30.82 against $35.93 on one item. Each product had two records linked only by name. The matcher hit the catalog record, which had no customer price row. A name bridge recovers the tier price for now. The durable fix is collapsing duplicate records.

Pricing freshness alerts could never fire. Customer A was 83.5 hours stale against a 36-hour SLA with no alert. The SLA registry had one entry, unregistered sources returned “observed” instead of “stale”, and all seven monitor schedules were stopped by default.

Measurement

Inference was approved twice and was wrong twice. A slow sync was blamed on file count, then on snapshots. The query read 45 files, not 2,368, and the proposed library call did not work in the installed version. The same pipeline on local disk took 6.4 minutes against 62.6 on object storage, which isolated metadata I/O. Both wrong fixes had been approved on reasoning alone. Measure before you recommend.

The triage gate fails on the threshold, not just sample size. Pooled Customer A and Customer B labels give 0.952 recall for the live engine against a 0.98 gate. On 228 Customer A records the live engine had recall 179 of 184 but specificity 7 of 44. Labels that sit in “PO handled outside” are real POs, never negatives.

Customer A’s value was accuracy, not speed. Median time to order was 2.9 business hours before go-live and 3.3 after on 55 orders. Customer B’s median was 7.9 business hours with 50% of 4,271 POs matched, and it rose as coverage widened, so it is a floor. Report coverage next to every median.

Go-live coverage depends on how you count. From September 8 to October 6 Customer A had 687 WhereFour orders, 603 by EDI. Of 74 external manual orders, we handled 45, about 61%. The misses were mostly portal notices with no lines, which email cannot solve. 38 EDI orders arrived as PDFs and sat in the queue as noise.

Demand tests come before builds. About 31,000 prod emails over 90 days and 30 meeting notes showed no “same as last time” orders without lines. The team built the feature anyway for a demo and recorded that as an explicit decision.

Open items

The 17% duplicate PO rate, 18 of 108 POs in 30 days, appears in our notes for both Customer A and a separate buyer sample. Verify the source before quoting it as a Customer A number.

Auto-submit readiness for Customer A appears as 2.5%, 3 of 122 orders in a 30-day plan audit, and as 0 of 154 auto-validated in a September 16 success review. These are different windows. Quote each with its window.

A fleet-wide Typesense customer sync outage from mid-July, caused by a non-optional field on a populated collection, has no confirmed resolution in our records. Check the live collection before assuming customer search is current.

Anti-patterns

We stopped trusting run status. Long syncs report FAILURE while committing data, and scrapes report SUCCESS with zero rows. Row counts are the truth.

We stopped treating merged as live. Code, repo config and the Flagsmith dashboard all disagreed with what ran, more than once. We trust the resolved SDK value, the running build and a data row, together.

We stopped writing to Typesense directly. The reconcile deletes anything that is not in gold.

We stopped hand-rolling bronze writes and stopped using replace on shared tables. One rewound the Iceberg pointer and would have restored about 886 wrong codes into the CSR picker. Another wiped other tenants in staging.

We stopped using staging catalogs for matching evals until they are rebuilt from prod rules.

We stopped adding score floors and fixed margins to resolvers. They removed correct answers and kept the wrong one.

We stopped auto-normalizing revisions, prices and customers when the two sides disagree. A CSR review is cheap and reversible. A wrong ERP write is not.

We stopped building per-tenant configuration before measuring a need. Run the search eval on the affected connection first.

We stopped putting prompt changes behind feature flags. A Langfuse label move is instant and reversible.

We stopped caching fallbacks and empty results. A cached failure returns the same broken answer for 30 days.

We stopped mapping several unmatched lines to one placeholder SKU. The ERP merges them.

We stopped repeating “already merged” claims without checking ancestry against the integration branch. A fix described as merged was an open PR, #3741.

We stopped running global sweeps for problems an event can catch. Fixes are event-driven and scoped to one connection.

We stopped treating gbrain design notes from May 2026 as shipped behavior. The unified decision layer and formula pricing module are ideas, not features.

Appendix: source index

These are the places to read more. Sources whose names identify a customer are left out here. Ask the team lead for the customer key to find them in memory, gbrain and the incident folder.

Repo docs and code

• apps/docs/src/content/docs/guides/sop-ob-001-supported-erp.md
• apps/docs/src/content/docs/incidents/2026-07-16-email-pull-connections-stuck-error.md
• apps/docs/src/content/docs/incidents/2026-08-13-email-pull-fleetwide-dispatcher-wedge.md
• docs/plan/validation-agent-optimization-and-auto-submit.md
• packages/eval/README.md and packages/eval/scripts/
• packages/contracts/src/connection-config.ts
• packages/pdf-shared/scripts/check_live_prompt_scope.py
• apps/webapp/src/lib/matching/write-confirmation-xref.ts and uom-learnings.ts
• infrastructure/flagsmith/configs/flags.json, segments.json, segment-overrides.json

Pull requests

• Extraction and prompts: #2439, #3074, #3290, #3610
• Intake and email: #1874, #3126, #3520, #3860, #3874, #3993
• Matching and catalog: #1533, #3151, #3244, #3282, #3577, #3578, #3688, #3920
• Validation, pricing and ERP writes: #2057, #2058, #2060, #3440, #3476, #3478, #3480, #3587, #3642, #3682, #3686, #3733, #4100, #4201
• Data sync and flags: #2514, #2566, #2901, #3604, #3605, #3675, #3880, #4164
• Plans repo: PR 28 catalog normalizer, PR 57 duplicate PO detection

Agent memory notes, kept on the team lead’s machine

• chunked-extraction-line-loss, langfuse-prompt-rollout-mechanism, image-extraction-pdf-reroute, address-extraction-current-state, email-order-document-binding-failures
• email-triage-rollout, email-pull-operations, ordermatic-match-key-vendor-prefix, manual-xref-conversion-factor-their-uom-dataloss, p21-sandbox-connections, p21-duplicate-sku-lines-merge
• shipto-resolver-and-validation, customer-autofill-address-disambiguates, customer-match-ignores-deleteflag
• price-precedence-source-wins, pricing-freshness-monitoring-is-structurally-blind, direct-erp-search-escape-hatch-no-pricing
• jev-validation-engine-bench, agent-shadow-submit-mode, agentic-dry-run-double-gate, keep-bound-means-po-revision-wrong, quote-split-failed-retry-wedge, quote-split-submitted-flip-and-status-lag, reextract-split-quoted-orders-refusal, catalog-revalidation-submit-path-only
• wherefour-orders-prod-staging-schema-split, dlt-hint-dicts-must-not-be-shared, bronze-iceberg-snapshot-pointer-rewind, iceberg-metadata-retention, dagster-ops-runbook, dagster-run-workers-no-flagsmith, edi-flag-wherefour-only-never-landed, fact-order-lines-base-item-id, typesense
• flagsmith-operations, ordermatic-deploy-topology-and-logs, release-and-deploy-triggers, paging-and-incident-triage, per-customer-ui-asks-need-org-flag
• measure-directly-before-recommending-fixes, erp-order-number-not-a-real-order-label, verify-precedent-claims-before-trusting, prefer-event-driven-over-sweeps

gbrain slugs

• docs/onboarding/sop-ob-001-supported-erp
• docs/designs/org-onboarding-wizard
• wiki/personal/reflections/2026-05-05-staging-db-schema-drift-0d5287