Skip to content
Ordermatic Docs

Incident report — Genfit P21 "Special Order" partial submission (2026‑06‑01)

Author: Claude (AI pair) · Date: 2026‑06‑02 Customer: Genfit = General Fittings LLC (genfitllc.com), org org_33vHtt6OUnCFD3QGmZeAJXSYsH7, ERP = Prophet 21 (“Epicor”) Operator: Michele (michele@genfitllc.com) Subject order: PO P41497 from buyer A&H Parts Sale, Inc · extracted_order e9f2159e‑0928‑4189‑9ddb‑a68cea2b63e9 · P21 order 1017435

Scope note: this report doubles as the “where did the AI have to assume?” retrospective requested after I mis‑identified the customer. Every assumption I made — and whether the data later confirmed or refuted it — is in §5. The observability gaps that forced those assumptions are in §6, and concrete tooling fixes in §7.

1. TL;DR

Section titled “1. TL;DR”

Two independent failures stacked:

  1. Matching defect (primary cause of “4 of 7 lines”). The auto‑matcher mapped 4 of the 7 PDF line items to the same PL-08-06 “SPECIAL ORDER” placeholder at $0.00 (the un‑catalogued “OB‑…” parts). Submitting four identical PL-08-06 $0 lines to P21 results in P21 merging/rejecting the duplicates, so the order materialises in Epicor with ~4 distinct lines instead of 7.
  2. Submission‑path cascade (caused the stuck lock + orphan + silent partial). A slow P21 transaction exceeded the 2‑minute Temporal activity timeout (no client‑side HTTP timeout), got abandoned and retried; the retry’s fail‑closed PO dedupe reported “already exists” → the app recorded failure while the partial order stayed orphaned in P21. REJECT_DUPLICATE on the per‑order workflow id then blocked every retry with the misleading “submission already in progress.”

Fix status: #1 (matching) is already fixed on staging by ENG‑280 (#1440) — it removed the dead blind‑top‑hit prefill (performValidationPrefill) and routes through a gated Typesense matcher with the same Jaccard guard; it reaches prod via the next staging→main promotion. This PR carries only #2 (the submission cascade) and makes partial commits visible. The submission files are byte‑identical on main and staging, so this lands via staging.

2. Who/what (corrected)

Section titled “2. Who/what (corrected)”
Value
Ordermatic customer (seller) General Fittings LLC (“Genfit”), Willoughby OH
ERP Prophet 21 (type=prophet21, connection “Genfit Production”, prod only — no sandbox)
Buyer on the PO A&H Parts Sale, Inc (Indianapolis IN)
Operator Michele (michele@genfitllc.com)
PO / order P41497, 7 lines, total $370.64, dated 06/01/2026
App record extracted_order e9f2159e…, orders row c4842336… = quote_pending, erp_order_number=NULL
P21 reality (per Michele) order 1017435 created with 4 of 7 lines; she deleted it

I originally reported the customer as “Encore Industrial.” That was wrong — see §5.

3. What actually happened (data‑driven timeline, UTC)

Section titled “3. What actually happened (data‑driven timeline, UTC)”

From GCP Cloud Run logs (erp-webapp-prod) + the prod Neon DB:

  • 2026‑06‑01 19:16:58 — extracted_order e9f2159e (PO P41497) last updated; 7 lines, 4 already collapsed onto PL-08-06 $0.
  • 19:17–19:31 — retry storm: the same order submitted 8+ times (19:17:49, 19:18:07/15/34/51, 19:21:40, 19:24:01, 19:31:…) plus a ~20‑entry burst at 19:23:5x. Each resubmit hit REJECT_DUPLICATE → 409 “already in progress.”
  • The Temporal worker ran the submission once; P21 created order 1017435 with a subset of lines. The worker’s logs are not shipped anywhere (§6.1), so the exact P21 request/response is unobservable.
  • App never recorded erp_order_number → orders row stuck quote_pending, no error, no verification status. App ↔ ERP desync: the app thinks “not submitted,” P21 has a partial order.
  • Michele found 1017435 (4/7 lines) in Epicor and deleted it.

History check (validates the sequence): prior A&H orders P40901→1017127, P40582→1016964, P40519→1016936 … all quote_generated. 1017435 fits the next slot — consistent with a real-but-unrecorded P21 order.

4. Root cause — corrected

Section titled “4. Root cause — corrected”

4.1 Primary: matching → duplicate $0 special‑order placeholders (CONFIRMED from data)

Section titled “4.1 Primary: matching → duplicate $0 special‑order placeholders (CONFIRMED from data)”

PDF line → extracted/matched line:

# PDF part (qty) Matched to
1 OB 1468‑6 — 3/8×1/4 MALE CONN (10) PL-08-06 “SPECIAL ORDER” $0
2 OB 1468‑6‑8 — CONNECTOR 1/2”→3/8” (10) PL-08-06 “SPECIAL ORDER” $0
3 OB 3220‑12‑8 — BRASS REDUCER (10) PL-08-06 “SPECIAL ORDER” $0
4 C2462‑6 — PUSH IN UNION (20) C2462‑6 ✓ ($2.31)
5 OB 3152‑4 — HEX PLUG (30) PL-08-06 “SPECIAL ORDER” $0
6 1493‑6D — RUBBER HOSE (20) O15535 (fuzzy; different #)
7 207‑8 — FRAME NIPPLE (20) 207‑8 ✓ (desc differs)

4 of 7 items have no catalog match and were all mapped to the single PL-08-06 placeholder at $0. When the order is submitted, P21 collapses the four identical PL-08-06 lines (duplicate oe_order_item_id) into one → the order materialises with 4 distinct lines = “4 of 7.”

CONFIRMED via live P21 (§8): a sandbox submission of 4 identical lines + 2 distinct (6 lines, 3 distinct) came back as 3 lines in P21 with a mismatch — i.e. P21 merged the duplicates. So the mechanism is duplicate‑line merge, not rejection; the $0 price is incidental.

This is a matching defect, upstream of submission. ENG‑280 (#1440, already on staging) prevents it — its gated Typesense matcher leaves these un‑catalogued parts unmatched (Jaccard guard) instead of binding them to a placeholder. This PR’s submission fix additionally makes any partial visible (§4.3).

4.2 Secondary: submission cascade (CONFIRMED from code + corroborated by data)

Section titled “4.2 Secondary: submission cascade (CONFIRMED from code + corroborated by data)”
  • No client‑side timeout on makeP21Request (fetch w/o AbortSignal) + 2‑min Temporal activity timeout (startToCloseTimeout) + 7 retries → slow‑but‑successful P21 write abandoned and retried.
  • Retry’s fail‑closed PO dedupe (PR #1217, 2026‑05‑26) reports “already exists” as failure → orphaned partial order.
  • REJECT_DUPLICATE on workflow id manual-erp-submit-{id} (PR #1213, 2026‑05‑26) blocks every resubmit once the run is terminal → the misleading “already in progress” (matches the retry storm + Michele’s screenshot).
  • A partial commit returns success:true with a mismatch verification, but the order path dropped it → silent partial.

4.3 The fix in this PR — submission cascade only (fix/p21-submission-fix → staging)

Section titled “4.3 The fix in this PR — submission cascade only (fix/p21-submission-fix → staging)”
Defect Change Status
Stuck retry REJECT_DUPLICATE → ALLOW_DUPLICATE_FAILED_ONLY (submit.ts) ✅ tests pass
Abandoned slow write activity timeout 2m→6m + 4‑min AbortSignal on P21 HTTP ✅
Silent partial persist erpVerificationStatus/erpUnverifiedItemIds on the order path (mirrors quote path) ✅ test added (4‑of‑7 mismatch case)
Matching → dup placeholder (§4.1) Fixed on staging by ENG‑280 (#1440) — gated Typesense matcher; not in this PR ✅ separate

Follow‑ups (beyond ENG‑280 + this PR): (a) when an item legitimately needs a special‑order line, sum/dedupe identical placeholder lines before P21 submit so they don’t collapse; (b) make P21 dedupe idempotent (return the existing order # as success on our own retry) to fully prevent orphans; (c) review match quality on items that did match (207‑8, and the operator’s 1493‑6D→O15535).

5. Assumptions I made, and corrections

Section titled “5. Assumptions I made, and corrections”
# I assumed Reality Why I assumed it (the gap)
1 Customer = Encore Industrial Genfit / General Fittings LLC Read the email domain david@encoreindustrial.co in the PDF’s title bar as the customer. It was an inbox label, not the customer. No customer/order id was present in any human‑readable place I had.
2 “4 of 7” = slow‑timeout partial write Matching → 4× PL-08-06 $0 placeholder → P21 collapse The worker logs that would show the P21 line handling aren’t shipped anywhere; I reasoned from code alone. Only the real PDF + DB rows revealed it.
3 Prod DB = Cloud SQL Neon (PROD_DATABASE_URL) GCP had cloudsql.googleapis.com/postgres.log; I inferred runtime DB from infra logs. The app actually uses Neon.
4 DATABASE_URL (.env) = prod It’s dev/eval (2,560 rows eval-service); prod is PROD_DATABASE_URL Two near‑identical env files, both pointing at genfit‑named DBs; nothing labelled which is which.
5 PO number = P41467 P41497 OCR’d from a low‑res email screenshot myself.
6 The logged orderIds were orders.id They’re extracted_orders.id review‑v2 submits with the extracted_order_id; not obvious from logs.
7 App logs would be in BetterStack BetterStack has only DB‑query telemetry; app logs are in GCP Cloud Run; worker logs nowhere §6.
8 gcloud not installed It was installed + authed, just not on the sandbox PATH which gcloud in the tool sandbox returned nothing.

Net: 6 of 8 first‑pass conclusions needed correction once I had real data. Every correction traces to a missing or mislabelled signal, not to reasoning — which is the point of §6.

6. Observability gaps (ranked by triage impact)

Section titled “6. Observability gaps (ranked by triage impact)”
  1. The Temporal worker is a logging black box. P21 createOrder (request, response, order #, which lines P21 accepted/merged/rejected, verification mismatch) runs on the GCE VM erp-staging-temporal-workers, whose stdout is not shipped to Cloud Logging or BetterStack. This is the single biggest gap — the exact failure is unobservable.
  2. BetterStack carries the wrong data. Prod source 1692669 = only service='database' query telemetry (param values not logged → order numbers unsearchable); prod source 1356416 = an unrelated Odoo/Coolify host. No app/order logs.
  3. erp_order_number is never logged, and there’s no record linking the app order to P21 1017435 → the app↔ERP desync is invisible; nobody is paged when an order lands in P21 but the app shows quote_pending.
  4. No correlation IDs across stages. order_id / extracted_order_id / workflow_id / erp_order_number / po_number aren’t consistently attached, so you can’t follow one order across webapp → worker → ERP.
  5. No alerting on submission failures or partial/mismatch submissions. The customer is the monitor.
  6. Matching is unobservable. Nothing records why 4 items became PL-08-06 (no match? low score?) or the per‑line match confidence. Langfuse is wired for extraction (scripts/analyze_langfuse_traces.py) but matching decisions aren’t traced.
  7. No sandbox/staging that mirrors prod. Genfit has a prod‑only P21 connection; test‑play lacks Genfit’s customer/item ids; there’s no read replica or scrubbed dataset. A faithful end‑to‑end repro of this order is therefore impossible without touching prod (§8).
  8. Environment ambiguity. .env DATABASE_URL=dev vs PROD_DATABASE_URL=prod, both named genfit; six .env* variants in the repo. Easy to query the wrong DB (I did).

7. Tooling / package enhancements (actionable)

Section titled “7. Tooling / package enhancements (actionable)”

Logging / tracing

  • Ship structured logs from both apps/webapp and apps/ts-temporal-worker to BetterStack (or one OTel pipeline), with a standard context block on every order event: order_id, extracted_order_id, workflow_id, erp_order_number, connection_id, clerk_org_id, po_number, erp_type.
  • In Prophet21Service.createOrder, log a structured summary: items_sent, items_accepted, lines_merged/rejected, erp_order_number, verification status. (Right now even success only logs orderNumber.)
  • Emit an explicit desync signal: when a submission ends without erp_order_number but a PO dedupe found an existing order, log + alert “possible orphaned ERP order.”

Analytics / alerting

  • PostHog events for the failure modes this incident exposed: order_submission_partial (verification mismatch), order_submission_blocked_workflow_already_started, p21_dedupe_failed, unmatched_items_routed_to_special_order (with count).
  • BetterStack alert + dashboard: submission failures, partials, and “orphaned ERP order” per org.

Matching observability (root‑cause class)

  • Trace matching in Langfuse per PO: input line, candidate matches, scores, final decision (matched / special‑order). Surfacing “4/7 unmatched → PL‑08‑06” at submit time would have flagged this before it reached P21.

Repro / environments

  • A nightly read replica or scrubbed staging DB + a catalog/Typesense mirror so matching‑dependent bugs can be reproduced E2E without prod.
  • Per‑customer P21 test‑play connections seeded with representative customer/item ids, so submission can be exercised against a sandbox.
  • Label DBs unambiguously (genfit_dev vs genfit_prod) and document DATABASE_URL vs PROD_DATABASE_URL in .env.example.
  • Document the gcloud/observability access path (it’s authed as terraform-infra@ordermatic-prod but not on PATH); add the BetterStack/Neon/R2 MCP or CLI shortcuts to the runbook.

Product/data fix for the actual bug

  • Handle unmatched items as distinct special‑order lines (unique keys) or sum identical PL-08-06 placeholders into one line with combined qty before P21 submit — so “7 in → 7 (or N intended) out,” never a silent collapse.

8. Local end‑to‑end test — feasibility & what was run

Section titled “8. Local end‑to‑end test — feasibility & what was run”

Done: retrieved the exact PDF from prod R2 (/tmp/PO-41497.pdf, key Genfit Production-…-PO-41497.pdf) and the real 7‑line extracted payload + order state from the prod Neon DB (read‑only, with approval).

Ran against a real P21 sandbox (“Genfit Playground 2”, connection d4758042). Two live runs via an opt‑in test (prophet21-service.live-e2e.test.ts + vitest.live.config.ts), submitting unapproved quotes through the real Prophet21Service.createOrder:

  1. Exact incident payload (customer 11840, the real 7 lines) → P21 rejected: “A valid customer ID must be entered for this company. Value: 11840.” The sandbox is a different P21 company than prod, so the exact customer/item ids aren’t present — the precise order can’t be replayed there (reinforces §6.7).

  2. Structural repro with sandbox‑valid ids (customer 11039; 4 identical EV6F-108 lines + 2 distinct = 6 lines, 3 distinct; real prices to remove the $0/margin variable) → created order 1013362 and returned: lines_sent: 6, distinct_item_ids: 3, verified_lines_in_p21: 3, verification_status: "mismatch".

    P21 merged the 4 duplicate lines into 1 (6 → 3). This confirms the incident mechanism: PO P41497 sent 7 lines (4× PL-08-06 + 3 distinct); P21 collapsed the 4 duplicate placeholders → 4 lines = “4 of 7.” The run also exercised the fix end‑to‑end: createOrder returned a mismatch verification (3 verified vs 6 sent) — precisely the partial that Defect‑3 of the fix now persists/surfaces instead of dropping.

Conclusion: the “4 of 7” is CONFIRMED to be duplicate‑line collapse driven by the matcher routing 4 unmatched parts to one placeholder SKU — a matching bug surfacing at submit time. ENG‑280 (#1440, on staging) is the durable fix (gated matcher → those parts stay unmatched); the submission fix in this PR makes any residual partial visible and recoverable. Separately verified on the docker stack: the gated matcher leaves all 4 OB‑parts unmatched while exact SKUs (C2462‑6, 207‑8) still match.

Not run (still infra‑blocked): the extraction + matching stages on the raw PDF — extraction is a Python Temporal worker and matching needs Genfit’s prod Typesense catalog; no local/sandbox mirror exists (§6.7). So I confirmed the submission half end‑to‑end against real P21, and the matching half from the persisted prod data.

Sandbox side effects: created unapproved test quotes in the playground (notably order 1013362, PO AI-E2E-…). Harmless, but delete if you keep the sandbox tidy.