Cleveland Kitchen — Unclaimed Complaint Coverage Investigation (2026-08-13)

Context

A PR/coverage sweep over the 2026-08-12/13 window matched Cleveland Kitchen complaints against recently opened/merged PRs and found five items either unsolved or entirely uncovered:

1. WhereFour not picking up updated pricing tiers — touched by merged #2971/#2977, but blocked on a Dagster metadata bug; fix opened as #2988.
2. Spotty inventory ERP match on “01XX” sample SKUs — not covered; #2972 judged pack-size-focused, not a sample-SKU fix.
3. PO confirmations should be filtered for SPS/iTrade customers — not covered.
4. PO revisions should update the existing WhereFour PO, not create a new one — not covered beyond an existing duplicate-order block.
5. Customer-facing testing/training tooling — not covered.

This doc records five independent from-scratch investigations (one per item) against the current codebase and GitHub PR/CI history, run 2026-08-13. Each was told to treat the sweep’s findings as a hypothesis to verify, not a given. No live Dagster run/event-log access or production database access was available at investigation time — anywhere a claim depends on that, it’s called out explicitly as unverified rather than assumed.

One correction surfaces immediately and affects how item 2 should be scoped: Cleveland Kitchen’s ERP is WhereFour, not Prophet21. Confirmed at apps/webapp/src/pages/api/erp/parts/index.ts:611-616 and packages/pricing/src/postgres-effective-price-service.ts:255 (isClevelandKitchenCompany, keyed on WhereFour default_company_id), and stated directly in data-engineering/customers/cleveland-kitchen/README.md. Any fix framed around P21-specific matching code will miss this customer.

At a glance

#	Issue	Status	One-line verdict
1	WF pricing tiers not updating	Fix ready, unmerged	#2988’s fix is correct; blocked on review only (CI green). Real underlying scrape error still unknown until it merges, deploys, and re-runs.
2	01XX / sample-SKU matching	In motion, gate off	Root-cause data plumbing landed today (#2978); the consuming gate is deliberately off for CK. A larger, unowned wrongProduct (19.4%) class exists alongside it.
3	SPS/iTrade confirmation filter	Unaddressed — dead code	A suppression mechanism was already built (customer_email_overrides) but never wired into either of two parallel confirmation-email send paths.
4	WhereFour PO amendment/upsert	Unaddressed — half-built	The block-only guard (#2959, merged 8/10 — likely the triggering incident) leaves clerks stuck. The PATCH capability and revision-matching engine both already exist, unconnected.
5	Customer testing/training tooling	Unaddressed — greenfield	Everything that exists is internal-only or half-built. Nothing here is a matter of “reconnect existing pieces.”

---

1. WhereFour pricing tiers not updating

PR referenced: #2988 (fix-wherefour-pricing-scrape-metadata)

Root cause — confirmed real

• apps/dagster/erp_pipeline/implementations/wherefour_sync_orchestrator.py:1126-1203 (WhereFourSyncOrchestrator.sync_pricing_tiers) captures fetch_started_at = datetime.now(UTC) as a raw datetime (line 1148) and stamps it, unmodified, into the returned summary dict on every path — login-failure (line 1161), scrape-errors (line 1173), and success (line 1195).
• apps/dagster/erp_pipeline/ops/wherefour_pricing_scrape.py:703-716 (_check_scrape_summary) raises Failure(metadata={"summary": summary}) whenever summary.get("success") is False. With a raw datetime inside summary, Dagster’s metadata normalizer can’t serialize it, so raising the Failure itself crashes with DagsterInvalidMetadata, masking whatever real error/pricing_scrape_errors the summary actually carried.
• The existing regression test (apps/dagster/tests/test_wherefour_scrape_step_failure.py) only exercises string-only summaries — it doesn’t include a fetch_started_at datetime, so it passed even with this bug present.
• The bug is the product of combining two independently-correct, independently-merged changes: the Failure(metadata=...) pattern (#2274, merged 2026-07-12) and sync_pricing_tiers stamping a raw datetime into the summary for downstream watermark comparisons (landed with #2971). Neither PR alone was buggy.

Is #2988’s fix correct?

Yes. It adds _serialize_for_metadata(value) — a json.dumps(value, default=_default) / json.loads round-trip where _default converts datetime → isoformat() — and passes a serialized copy to Failure(metadata=...). The original summary dict is untouched, so fetch_started_at stays a real datetime for the callers that need it (wherefour_pricing_tiers_to_rules.py:1157, asserted by test_wherefour_tier_rules.py:333). The real error text (summary.get('error', ...)) still reaches the Failure description, so the run will now fail with a legible message instead of crashing on an unrelated serialization error.

Minor, non-blocking gap: no test directly asserts the datetime→ISO path (flagged by CodeRabbit on the PR too).

Status

• Open, not merged. mergeable_state: blocked — blocked on required review, not CI or conflicts. CI is fully green on head commit c1b43feb (CodeRabbit, Devin Review, Auto Changeset, Branch Protection, Deploy Gate, both Build-and-Deploy workflows, Validate Code, CodeQL).

What’s still unknown

Once #2988 merges and a backfill is re-run, the actual scrape failure underneath is still unverified from this investigation (no live Dagster access). Two plausible causes based on code history, not confirmed:

• Auth/session fragility — this scraper had a subdomain-redirect login break before (wherefour_scrape_login.py, _is_post_login_url), and CK’s WhereFour tenant went through a re-key/migration around 2026-08-05 (per #2971’s changelog entry).
• Selector/page-structure drift — wherefour_pricing_scrape.py:38-46 explicitly documents that selector IDs drift across WhereFour UI versions/tenants as a known failure mode for this scraper.

Residual risk even after this specific bug is fixed

PR #2971’s changelog entry separately flags 26 tier item names with no live catalog row match, disproportionately CK’s core packs — affecting 281 of 437 tier-covered customers. That’s an unrelated, still-open gap that could make CK’s pricing sync look incomplete even once #2988 lands and the scrape succeeds.

Remaining steps

1. Get #2988 reviewed and merged (no code changes needed).
2. Confirm the staging deploy picks it up.
3. Re-run wherefour_pricing_tiers_job for connection 4f234677-7bc5-43c1-9059-bd616b33ef75 and read the now-legible Failure description.
4. Fix whatever that reveals — no such fix exists yet since the error is still unknown.
5. Confirm a full successful backfill through to gold, and separately track the 26-item catalog-match gap from #2971.

---

2. “01XX” sample-SKU matching

PR referenced: #2972 (open, eval-only)

What “01XX” actually means

CK’s own complaint, recorded verbatim in data-engineering/customers/cleveland-kitchen/01_pipeline.py:399-400 (Stage 7): “the inventory ERP match is spotty — picking up sample SKUs (01XX SKUs).” This isn’t an ERP-wide numeric SKU-prefix convention — it’s CK’s shorthand for WhereFour items sold as unit_of_order = 'each' (44 of CK’s 140 finished goods, individual/sample-size units) as opposed to case/pail/container/drum (01_pipeline.py:470). No prefix-stripping logic in the matcher could touch this anyway: the only such helper, stripVendorPrefix (apps/webapp/src/utils/matching-helpers.ts:161-174), matches /^[A-Za-z]{2,}[_/]+(.+)$/ — a leading run of ≥2 letters, structurally unable to match a numeric prefix.

Is this covered?

• #2972 confirmed pack-size-focused and measurement-only — its own description states no production code path changes. It fixed a blind eval scorer and produced a fresh baseline: wrongPack 17.0%, wrongProduct 19.4%, matchRecall 61.9%, matchPrecision 67.8% (n=200 CK docs).
• wrongProduct (binding to a completely different product) is now the larger error class, ahead of wrongPack. Per data-engineering/customers/cleveland-kitchen/README.md:216-217: “wrongProduct (19.4%) has no owner yet. It is now the larger half of the match error and nothing in the current plan addresses it.”
• Two already-merged, production PRs partially address the pack/each-vs-case symptom CK originally named, predating this sweep’s window: #2577 and #2597 (merged 2026-07-24/25), which wired packAwareRerank/broadenUnion/pricePackSelect into findProductMatchesBatch and are confirmed on for CK’s prod connection. They reduced general wrong-match rate in eval (44.0% → 25.3%) but operate on name/price heuristics only — not the WhereFour unit_of_order signal that would directly separate “each” from “case.”

Root cause

The one catalog signal that would directly disambiguate “sample/each” from “case/pail” for the same product (unit_of_order) never reached the matcher until today:

• 01_pipeline.py:463 (Stage 8) confirms unit_of_order is populated 140/140 in WhereFour’s API, but it — along with unit_of_purchase/purchase_conversion/weight — was not projected through raw_products.sql → stg_products.sql → dim_products.sql → Typesense’s PRODUCTS_SCHEMA before today.
• #2978 (fix(dagster): wire WhereFour UOM-conversion fields through the pipeline into Typesense, merged 2026-08-13) plumbs these fields through — the real data prerequisite — but explicitly ships zero behavior change until a reindex + re-measurement happens.
• The consuming gate, matchingConfig.uomConversion (apps/webapp/src/utils/matching-helpers.ts:1052, isUomConversionFlagEnabled in typesense-search-service.ts:356-362), is currently off for CK specifically — deliberately disarmed 2026-08-12 as a fail-safe (isUomConversionFlagEnabled defaults fail-open true, and the five input fields were empty on 140/140 CK products at the time).
• Without that signal, the matcher falls back to name/pack-token heuristics (parsePackSignature/stripPackTokens/packAgreement), but ~72% of CK PO lines carry no pack/size token at all (per #2972’s body) — heuristic disambiguation frequently fails, surfacing as the “spotty” complaint.
• Compounding bug: abstainPackAmbiguity (typesense-search-service.ts:409-428, readPhase1AbstainGates) — which would refuse to auto-bind an ambiguous each/case line and route it to review instead — is silently off for CK despite packSizeAware being on. readPackAwareMatchGates (typesense-search-service.ts:457-483) inherits per-lever settings from the packSizeAware umbrella; readPhase1AbstainGates has no such umbrella fallback and defaults each lever to false unless explicitly set.

What a real fix needs to touch

1. Flip erp_connections.extra_config.matchingConfig.uomConversion back on for CK (connection 9cb6136c-b5b7-48a1-b18d-1e65a2e84521) — only after a Typesense reindex materializes #2978’s newly-piped fields and an A/B bind-rate re-measurement (#2978’s own follow-up note).
2. Fix the readPhase1AbstainGates vs. readPackAwareMatchGates inheritance asymmetry so abstainPackAmbiguity can inherit from packSizeAware, or explicitly enable it for CK.
3. Assign an owner to the wrongProduct (19.4%) class — a distinct, now-larger failure mode nothing current touches.

Other PRs in the window, for completeness

#2974 (open) fixes a measurement-contamination bug that was blocking #2972’s baseline — diagnostic infrastructure, not a matching fix. None of the remaining Aug 11–13 PRs (#2962, #2963, #2970, #2973, #2976, #2979–#2991 — ticketing, dev-hydrate, PDF-review timing, Docker/Infisical, Iceberg pagination, PAN scrub, releases, version bumps) touch the matcher.

---

3. SPS/iTrade PO confirmation filter

Verdict: unaddressed — and the fix already exists as dead code

There is no code anywhere that filters/suppresses confirmation emails by customer order source. Worse than “missing”: a generic per-customer suppression mechanism was built for exactly this purpose and never wired into the live send path.

Two independent, non-communicating send paths

Worth flagging on its own — there are two separate implementations that send these emails, neither aware of the other:

1. apps/ops-mcp/src/tools/send-order-received.ts / send-order-processed.ts — back the MCP ops tools, wired into the dispatcher at apps/ops-mcp/src/mcp.ts:484 / :515. Each does its own local resolveCustomerEmail() (send-order-received.ts:64-90, send-order-processed.ts:59-85) — no suppression check anywhere.
2. apps/ts-temporal-worker/src/workflows/order-confirmation-email.ts (Temporal, on extracted_order.received / wherefour.order_pushed.succeeded) calls activities in apps/ts-temporal-worker/src/activities/order-emails.ts (sendOrderReceivedEmail, sendOrderProcessedEmail, sendOrderShortNoticeEmail), which send directly via Resend with zero DB lookups and no shared code with path 1.

A fix needs to land in both paths (or the two need consolidating first).

How order source is modeled today

• iTrade: not modeled at all. The only repo hit is a customer-name string 'Ahold-Delhaize-iTrade' in an eval exclusion list (packages/eval/scripts/select-eval-set.ts:51) — no concept, enum, or column.
• SPS/EDI: modeled only as a derived customer-level is_edi attribute, sourced by scraping WhereFour order origins (apps/dagster/erp_pipeline/ops/wherefour_order_origins_scrape.py, wherefour_customer_edi_sync.py), surfaced via Typesense. Used in exactly one place — apps/temporal-worker/activities/gemini.py’s _sender_is_known_edi() / _unverified_po_override() — for inbox “lost order” triage, a different subsystem entirely.
• No order_source/channel column exists on orders or extracted_orders. Per apps/docs/src/content/docs/archive/ck-po-channel-inventory.md, SPS Commerce POs arrive today as manually forwarded PDFs via email — indistinguishable from any other emailed PO at the order level. Only the customer carries is_edi, not the order.

The orphaned suppression mechanism

packages/db/src/schema/customer-email-overrides.ts defines customer_email_overrides(clerk_organization_id, account_code, email, enabled), and apps/ops-mcp/src/customer-email-resolution.ts implements a tiered resolveCustomerEmail(account_code, orgId) that treats enabled=false as a suppression toggle, writing an email_resolution_suppressed audit row. This matches the original design doc exactly (apps/docs/src/content/docs/architecture/ck-harness-v2-mcp-substrate.md:533) — but it is never imported or called anywhere except its own test file. The planned set_customer_email_preference MCP tool from that design doc was never built. A second orphaned implementation, apps/ops-mcp/src/tools/email-resolution.ts, is likewise only referenced by its own test. A third orphaned piece, apps/ts-temporal-worker/src/activities/order-confirmation-email-activity.ts, implements the design doc’s planned global kill-switch (ORDERMATIC_OPS_MCP_EMAIL_SEND_ENABLED) — but its sendOrderConfirmationEmail() is never called; the real workflow calls order-emails.ts’s functions instead.

A trap to avoid: don’t key suppression on the derived is_edi flag

A related, separate feature — “Mark as EDI” in the inbox cockpit (apps/webapp/src/lib/inbox/customer-label-rules.ts, #2903/#2898) — is a human-authored, customer-name-keyed rule that auto-archives future inbox work items, used for hiding EDI orders from the internal review queue (not confirmation-email suppression). Its docstring explains a derived is_edi flag was deliberately rejected as the suppression key (“VMS: 13 Lincoln account records… is_edi therefore cannot suppress them”), and a prior derived EDI auto-archive sweep was explicitly retired (e2453cf) in favor of the human-authored rule. Precedent: the confirmation-email fix should pair the flag with an explicit, human-set toggle rather than gating purely on the derived Typesense is_edi attribute.

Recommended approach

Wire the existing customer_email_overrides suppression into both send paths, gated behind a new Flagsmith flag resolved via getFlagForConnection/getFlagForOrganization (packages/feature-flags/src/identity.ts) — mirroring apps/webapp/src/pages/api/email/triage.ts:387-395’s pattern for email_triage_exclusion_rules_v1. Suggested name following the {area}_{feature}_{version} convention: order_confirmation_edi_suppression_v1, default disabled, gating the filter check itself (not the whole send path) so disabling it cleanly reverts to today’s always-send behavior.

---

4. WhereFour PO amendment/upsert

PR referenced: #2959 (merged 2026-08-10, revision-guard.ts)

Verdict: unaddressed, but the pieces exist unconnected

What exists is detection + diff + decision-recording (all read-only), plus a hard block that stops submission rather than performing the amendment.

Two distinct “duplicate-order guards” — don’t conflate them

1. Retry-idempotency guard (apps/webapp/src/services/erp/implementations/wherefour-service.ts:815-860, findOrderByOrderNumber) — gated by erp_submission_idempotency_ref_guard_v1. Only protects against re-POSTing the same submission attempt (network blip / Temporal retry). Unrelated to PO revisions from a new email/document.
2. The guard this complaint is actually about — apps/webapp/src/lib/orders/revision-guard.ts (unresolvedRevisions()), wired into apps/webapp/src/pages/api/erp/orders/[id]/submit.ts:860-904. Shipped in #2959, merged 2026-08-10 — almost certainly the exact Cleveland Kitchen incident behind this complaint. Returns a 422 when a clerk recorded decision revise but patchStatus never reached patch_succeeded.

Confirmed: block-only, exactly as the original sweep said. It prevents Submit from creating a duplicate and tells the clerk to “re-open the revision panel and either apply the revision or choose ‘start clean.’” There is no “apply the revision” action anywhere in the product. A clerk who chooses Revise is currently stuck.

The WhereFour PATCH capability already exists — and is a guaranteed no-op today

WherefourService.updateOrder() (wherefour-service.ts:1260-1386) is a real, tested PATCH /customers/{cid}/orders/{oid} implementation: re-GETs the live order for a lost-update guard, refuses invoiced/shipped orders, builds Rails nested-attributes line items, and is gated by both a caller-supplied options.enableWrite and the connection-level wherefour_order_submission_v1 flag (default off). Every current caller passes it without enableWrite — order.ts:399,506, [id].ts:253,388 (a different feature, legacy order-edit), erp-writes.ts:348 — so it’s a no-op for all current traffic, and none of these are invoked by the revision-decision flow anyway. RevisionPanel.tsx’s “Revise against #X” button only calls recordDecision('revise') → POSTs to /api/erp/orders/revision-decision.ts, whose own docstring states “Read-path only… NO ERP write happens here.” patch_started / patch_succeeded / patch_failed exist only as schema enum values (packages/db/src/schema/order-revisions.ts:33-40) — no application code ever sets them. infrastructure/flagsmith/configs/segment-overrides.json:194-198 says outright for the CK connection: “The revise-in-place WhereFour PATCH is a separate Phase-4 path behind its own ERP-write flag and stays off.”

The matching/diffing machinery also already exists

• Detection: WherefourService.findOrdersByPoNumber() (wherefour-service.ts:1114-1195).
• Scoring: apps/webapp/src/lib/revision/engine.ts — scoreCandidates() (PO 0.4, customer 0.2, ship-to 0.1, same-day 0.05, line-overlap 0.2, total-$ proximity 0.05); tierFor() buckets strong/partial/weak.
• Diffing: diffLines() produces a changed/added/removed/promo/unmatched changeset with net $ delta.
• Endpoint: apps/webapp/src/pages/api/erp/orders/lookup-by-po.ts (flag-gated, read-only, never auto-picks).

This is exactly the “is this a revision” detection an upsert flow needs — it just isn’t connected to an execution step.

What the implementation requires

1. A new endpoint/workflow (e.g. extend ERPOrderUpdateWorkflow) that loads a patchStatus='recorded' revision row, transitions it to patch_started, calls WherefourService.updateOrder(connection, revisesErpOrderNumber, mappedItems, { enableWrite: true, expectedUpdatedAt }), and writes back patch_succeeded/patch_failed/verification_failed.
2. Map the revision engine’s LineMatch/DiffLine output into updateOrder’s items[] shape — mostly plumbing, since WherefourOrderCandidate.lines[].lineId already carries the needed identifier.
3. A UI trigger — RevisionPanel.tsx needs an “Apply” step after recording that calls the new endpoint and surfaces success/failure back to the clerk.
4. No change needed to revision-guard.ts — it already treats patch_succeeded as resolved.

Feature-flag requirement

Reuse wherefour_order_submission_v1 for the write surface itself (already exists, already gates updateOrder) rather than adding a duplicate. Gate the new orchestration separately — suggested name wherefour_order_revision_apply_v1, default off, per-connection via getFlagForConnection, matching how enable_order_revision_match is already scoped to the conn_ck_wherefour segment.

Related, larger, separate effort

debt/backlog/order-lifecycle-expansion-plan.md (approved via /autoplan 2026-06-12, #1615, docs-only) targets a much broader order_events substrate for arbitrary PO-change emails, with its own Phase 2 “assisted apply to WhereFour behind approval+flag.” No order_events table exists yet and nothing indicates this has moved past the planning doc. It’s a parallel, longer-horizon initiative — not a substitute for wiring up the already-built order_revisions/RevisionPanel flow, which is the narrower, immediately-actionable fix for this complaint.

---

5. Customer testing/training tooling

Verdict: confirmed unaddressed, and this is the one genuinely greenfield item

Zero PRs match “test order / sandbox / training / self-service,” zero issues match “Cleveland Kitchen,” searched across the whole repo (not just the two-day window).

What exists today (all internal-only or half-built)

• Internal PoC/demo tooling: scripts/poc-agent/, scripts/poc-setup/, scripts/poc-famous-supply/, scripts/poc-aurora/, scripts/poc-bison-bag/, scripts/poc-core-and-main/ — org/connection/Typesense/Postgres seeders run by engineers with 1Password/Infisical/DATABASE_URL access. Runbook: .agents/skills/seeding-demo-orgs/SKILL.md.
• Voice-call skill (.claude/commands/voice-call.md, scripts/poc-famous-supply/voice-call.sh) — synthesizes a phone call into the Famous Supply demo inbox on staging. Explicitly demo-only by its own docs (“no telephony, no webhook, no diarization, no consent handling”). A companion route (apps/webapp/src/pages/api/voice-calls.ts) is gated by isVoiceCaptureConnection/extraConfig.voiceCapture and is demo-scoped by construction — its own comment warns it would need to move to the production Temporal pipeline before becoming a real ingest route.
• apps/webapp/src/pages/api/work-items/test-order.ts (added Aug 7 2026) — the closest thing to a customer self-service test tool: creates an isTestOrder=true work item, rate-limited 3/hr. But it has no frontend caller anywhere in apps/webapp/src and runs no extraction at all (no extractedOrders/extractedOrderItems rows, no Gemini/Temporal call) — an unwired, empty placeholder.
• A real sandboxed-ERP-write mechanism, but internal-eval-only: scripts/poc-setup/create-ck-sandbox-connection.mjs creates a WhereFour connection at sandbox.wherefour.com, scoped to a separate Cleveland Kitchen SANDBOX Clerk org, honored via submitOverride in wherefour-service.ts:583-634 (reroutes submissions to a single tagged test customer). Driven by packages/eval scripts run by engineers from a separate org CK’s own staff can’t log into. This pattern exists only in the WhereFour adapter — no equivalent in prophet21-service.ts or any other ERP implementation.
• A matching/extraction feedback loop — built but dark: packages/db/src/schema/order-feedback.ts + apps/webapp/src/pages/api/inbox/feedback.ts let a reviewer flag item_match|price|quantity|uom|customer|other as wrong on archive/delete; item_match feedback deactivates the offending manual_cross_references row, and extraction feedback feeds packages/eval/src/datasets/order-feedback-corrections.ts as retraining ground truth. This is real, working correction infrastructure — but it’s gated behind Flagsmith flag inbox-feedback, which does not appear anywhere in infrastructure/flagsmith/configs/flags.json or segment-overrides.json — unregistered and effectively off for everyone. Also purely reactive (fires only on archive/delete of a real production order), with no customer-visible confidence/training-status UI. apps/docs/src/content/docs/design/xref-bootstrap-from-history.md names “spreadsheet-driven xref uploads via UI” as an explicit, deferred, unbuilt “Phase C if a customer requests it” — corroborating that this ask hasn’t been built.

Could the voice-call/demo-inbox tooling extend into a customer-facing sandbox?

Architecturally plausible in one respect — the “flip a per-connection extraConfig bit to unlock a sandbox capability” pattern is proven twice (voiceCapture, submitOverride) — but voice-call only covers the easy half (extraction, stops at pending_review, hardcoded to one org, explicitly disclaims telephony/consent). The harder half — sandboxed ERP writes — lives in a completely separate, unconnected piece (CK’s WhereFour submitOverride). Nothing joins the two today.

What building it would require

1. UI surface — finish (or replace) test-order.ts to actually run the real Temporal extraction workflow against a customer-supplied or synthetic PO, with explicit correct/incorrect affordances and visible confidence/xref-provenance (xref-bootstrap-from-history.md §5) — not just an empty placeholder.
2. Sandboxed ERP writes — generalize submitOverride into a first-class IERPService “test mode” so it works for Prophet21 too, not just WhereFour.
3. Feature-flag gating per AGENTS.md — this touches ERP submission, extraction, and matching, all in-scope categories under the feature-flag policy doc; needs the documented 3-stage rollout (internal org → canary customer → GA), the discipline the currently-dark inbox-feedback flag skipped.
4. Multi-tenant isolation — the clerkOrganizationId/connection_id scoping pattern is already pervasive (cross-org isolation tests exist at apps/ops-mcp/tests/cross-org-isolation.test.ts); open design choice is an in-org test flag (cheap, matches test-order.ts’s direction, but only WhereFour honors it today) vs. a shadow sandbox org per tenant (CK’s existing precedent, higher provisioning cost). Any synthetic/test orders touching shared bronze Iceberg tables need the same quarantine CK’s golden-dataset eval already gets — never merged into a tenant’s real partition.
5. Retraining loop — order_feedback + order-feedback-corrections.ts is the closest scaffold, but produces a dataset for manual internal eval runs, not customer-visible automated retraining. A real “upload your historical POs, see before/after accuracy” feature needs a customer-triggerable bulk-ingest path and a customer-visible report surface — all eval tooling today (packages/eval) is CLI/internal-only.

---

Cross-cutting observations

• Items 3 and 4 are the fastest wins. Both already have a designed, partially-built fix sitting unconnected in the codebase — this is “wire it up behind a flag,” not “design and build from scratch.”
• Item 1 needs a reviewer, not more engineering. #2988 is correct and CI-green; it’s purely blocked on human review.
• Item 2 needs a decision, not code. Someone has to decide when to flip CK’s UOM-conversion gate back on (post-reindex/re-measurement) and who owns the wrongProduct bucket, which is now bigger than the pack-size problem CK originally reported.
• Item 5 is the one real greenfield build — no existing flow to reconnect, and it needs an explicit design decision (in-org test flag vs. shadow sandbox org) before implementation starts.
• Recurring flag-gating pattern worth standardizing on: getFlagForConnection / getFlagForOrganization (packages/feature-flags/src/identity.ts), evaluated through Flagsmith’s segment engine — not locals.isFeatureEnabled, which is session-scoped and unreachable from Temporal workers or the MCP server. apps/webapp/src/pages/api/email/triage.ts:387-395 is a good reference implementation to mirror for any of the above.
• The CK-is-WhereFour-not-P21 correction should propagate to however item 2 gets ticketed going forward, so it isn’t scoped against the wrong ERP integration.

Methodology / limitations

Each item above was investigated independently by a separate research pass against the codebase (erp-unlocked/ordermatic, staging-adjacent state as of 2026-08-13) and GitHub PR history, instructed to verify rather than trust the originating coverage sweep’s claims. No live Dagster run/event-log access and no production database access was available at investigation time; anywhere a claim would normally be confirmed by a live run or query, it’s called out above as unverified rather than presumed true.