Feature Flag Policy: ERP-Write Path Changes

May contain stale references

This document was last spot-checked on 2026-08-04: AGENTS.md now contains the rule quoted in §7 (“Feature Flags Required for ERP-Write Changes”), so that specific staleness noted on 2026-06-22 is resolved. Cross-check critical details against the current codebase before relying on them.

Status: Active
Effective: 2026-06-15
Category: DevOps / Change Management
Scope: All code paths that create, modify, or submit data to external ERP systems

---

1. Scope — What Requires a Feature Flag

Any PR that modifies behavior in the following areas must be gated behind a Flagsmith feature flag before merging to staging:

Area	Examples
ERP order submission	ERPOrderSubmissionWorkflow, IERPService.createOrder(), quote generation, order update
ERP quote/approval	ERPQuoteGenerationWorkflow, ERPQuoteApprovalWorkflow, auto-submit logic
Extraction pipeline	PDFExtractionWorkflow, Gemini prompt changes, pre-extraction, anchor merge, item matching
Agentic validation	AgenticValidationWorkflow, confidence thresholds, auto-price writeback
Email triage	LLM triage prompt changes, trusted-sender logic, filter heuristic changes
Customer/item matching	Typesense matching logic, fuzzy match thresholds, customer resolution

Exceptions:

• Bug fixes to existing flagged code (fix goes behind the same flag)
• Pure refactors with no behavior change (verified by existing tests passing)
• Documentation, test-only, or CI-only changes

2. Staged Rollout Protocol

Every flagged change follows a three-stage rollout:

Stage 1: Internal (1–3 days)

• Flag enabled for internal test organization only (org_internal_test)
• Monitor extraction accuracy, ERP submission success rate, error logs
• Gate: Zero ERP submission failures in 24 hours

Stage 2: Canary Customer (3–5 days)

• Flag enabled for one low-risk customer organization
• Selected customer criteria: low order volume, tolerant of delays, direct communication channel
• Gate: No customer-reported issues in 72 hours, error rate within baseline

Stage 3: General Availability

• Flag enabled for all organizations (set default to true)
• Continue monitoring for 1 week
• Gate: Error rate within ±5% of pre-change baseline

Rollback

• If error rate spikes >10% above baseline at any stage: disable flag immediately
• Flag owner is responsible for monitoring during each stage
• Rollback does not require a deploy — flag change is instant

3. Flag Naming Convention

{area}_{feature}_{version}

Examples:

• erp_submission_batch_dedup_v1
• extraction_anchor_merge_v2
• validation_auto_submit_confidence_v1
• triage_trusted_sender_bypass_v1

4. PR Checklist for ERP-Write Changes

Add this checklist to any PR touching the scoped areas:

### ERP-Write Change Checklist

- [ ] Feature flag created in Flagsmith (name: `___`)
- [ ] Code path gated behind flag (default: disabled)
- [ ] Flag check verified in unit test (both enabled/disabled paths)
- [ ] Rollout plan documented (which orgs, what order, monitoring plan)
- [ ] Rollback tested (flag disabled → old behavior restored)

5. Monitoring During Rollout

During each rollout stage, the flag owner monitors:

Metric	Source	Alert Threshold
ERP submission success rate	Temporal workflow completion	<95% (baseline ~99%)
Extraction accuracy	LangFuse traces	>5% drop from baseline
Stuck orders	stuck-order-monitor.yml	Any new stuck items
Customer complaints	Support channel	Any complaint within 24h of flag change
Error rate (500s)	Cloud Run logs	>2x baseline

6. Flag Lifecycle

1. Create — Before PR merge. Document purpose in Flagsmith description.
2. Stage 1–3 — Follow rollout protocol above.
3. Permanent — After 2 weeks at GA with no issues, flag can be marked permanent.
4. Cleanup — Remove flag checks from code within 30 days of going permanent. Create a cleanup ticket when marking permanent.

7. Enforcement

• PR reviewers should check for flag gating on ERP-write path changes
• The AGENTS.md rule “Feature Flags Required for ERP-Write Changes” reminds AI agents to follow this policy
• Future: CI lint rule to detect ungated changes to scoped files (tracked separately)

8. Auto-Submit Promotion Ladder + Kill-Switch Criteria

Auto-submit (agentic validation writing a real order to the ERP with no human in the loop) is the highest-blast-radius ERP-write path. ERP accept/reject does not catch a wrong customer or wrong SKU the ERP happily accepts — so an org earns each promotion with shadow measurement first, not a flag flip. The AgenticValidationWorkflow spine already supports every mode; this ladder is the discipline loop around it. No promotion happens without the guardrail data.

Write gates are flags; payload rules are connection config

The flags that decide whether a write reaches an ERP stay in Flagsmith and are promoted per connection. For E4 and WhereFour they are eclipse_e4_order_submission_v1, eclipse_e4_order_update_v1, eclipse_e4_tst_weekly_cleanup_v1, wherefour_order_submission_v1 and wherefour_order_revision_apply_v1; other adapters declare their gate in packages/contracts/src/erp.ts. A kill switch must flip without a database write and fail closed when Flagsmith is down.

What a write contains for one customer (E4 payload fields, job numbering, due dates, WhereFour notes and delivery dates) is connection config in extra_config.e4Rules / wherefourRules (packages/utils/src/connection-rules.ts), changed through the internal app’s audited Connection rules editor. Each edit writes a sys_admin_audit_log row and takes effect on the next request or sync. Adding a new rule is still a code change with review; it ships default false.

Scope: promote per connection, not per org

The enable/threshold flags (agentic_validation_enabled, agentic_auto_submit_enabled, agentic_confidence_threshold) are org-scoped, but the ERP blast unit and the connection-scoped gates (agentic_require_identity_grade, agentic_ariba_total_crosscheck) key on the connection. Track promotion per connection (the guardrail dashboard rows are org/connection/day). For a multi-connection org, gate the threshold conservatively — an org-scoped flip enables auto-submit on every connection at once.

agentic_validation_jev_v1 (default off) only selects the validation engine (jev_validate_order instead of llm_validate_order); it does not enable auto-submit, and the guardrails above still apply.

agentic_shadow_submit_v1 (default off; on for VMS and Cleveland Kitchen) only records: every intake agent submit, live or shadow, lands in agent_shadow_submissions beside what the CSR sent. It no longer forces a dry run. The agent submits live through /api/orders/agent-submit only when agentic_auto_submit_enabled and erp_write_enabled are both on, which the route re-reads server-side; otherwise it runs the CSR submit path in shadow.

Never enable agentic_auto_submit_enabled for an org before agentic_validate_and_submit_v1: the race guard that stops a CSR submit from overwriting the agent’s runs only for pinned submits, and only Validate and submit pins a CSR’s submit.

Manual validation from the review page never submits by itself. With agentic_validate_and_submit_v1 on, a clean verdict hands off to the CSR’s own submit, pinned to the validated version.

Measurement surfaces (build these before promoting)

Surface	Where	Reads
Guardrail dashboard	@erp-unlocked/internal → /autosubmit-guardrails (gated by internal_autosubmit_dashboard_v1)	extracted_orders.auto_validation_* per org/connection/day
Shadow-vs-CSR precision	pnpm --filter @repo/eval eval:autosubmit-shadow-comparison (read-only, staging)	dry-run/would-have-submitted decisions joined to the CSR’s final action
Agent shadow report	npx tsx packages/eval/scripts/agent-shadow-compare.ts --org <org> (SELECTs only)	agent_shadow_submissions: agent vs CSR per line, holds + mismatches
Threshold sweep	pnpm --filter @repo/eval eval:autosubmit-threshold-sweep	CK golden set → coverage/precision per θ

The key precision signal is CSR-override-after-submit: the "Line item overridden" log line joined to auto_validation_status='auto_submitted' orders. It is the only thing that catches a wrong order the ERP accepted. (The dashboard uses extracted_orders.is_edited as a DB proxy until that log join is wired.) Do not treat the idempotency key (erp-create:{conn}:{order}) as a correctness guardrail — it only prevents duplicate writes.

⚠️ The proxy is not gating on its own. is_edited is a coarser signal than the real log join (it doesn’t confirm the edit followed the submit, only that the order was edited at some point) and must not be treated as sufficient to pass S1 by itself. Until the exact log join is wired, S1 promotion requires a manual sample-audit of a random slice of that connection’s auto-submitted orders (CSR/eng spot-check against the ERP) alongside the is_edited proxy — do not promote on the automated proxy alone.

The ladder (per connection)

review-only ──▶ shadow (dry_run) ──▶ auto-submit @0.95 ──▶ widen θ ──▶ steady state
              validation ON,        both write flags ON,   0.95→0.90    monitor,
              both write flags OFF  one connection,         step-wise    mark permanent
              measure precision     canary volume           per policy

approval-only (review_approval_only_mode) is a parallel holding pen for orgs whose ERP write path isn’t live yet (VMS / Eclipse E4 — the “Finished” button logs the actor and writes nothing). It graduates to shadow once createOrder is spec-complete for that ERP.

Eligibility to enter the ladder: (1) the ERP write path is real for that ERP; (2) an offline eval passes bar (CK golden / VMS ground-truth); (3) enough order volume for meaningful shadow metrics; (4) a paying/real customer.

Per-connection stages (specialize the §2 three-stage protocol):

📋 Numeric gates below are provisional placeholders, not signed-off thresholds. The precision target, minimum sample/order count per stage, and the exact kill-switch percentage are open business decisions (see the matching-resolution and auto-submit specs’ Open Questions) — do not promote a connection past S0 until an eng/CEO owner has explicitly confirmed the numbers for that connection. Treat every target/θ/percentage in this section as “TBD, propose during S0 sign-off” until replaced with a decided value.

• S0 Shadow (3–5 days, minimum N=50 orders): agentic_validation_enabled ON, agentic_auto_submit_enabled and erp_write_enabled OFF. Collect dry-run decisions; run the shadow-vs-CSR comparison. Gate: shadow precision ≥ target (TBD — propose at sign-off) at θ=0.95, computed over ≥50 dry-run orders (widen the window if volume is lower).
• S1 Canary auto-submit (3–5 days, minimum N=20 auto-submitted orders): flip both write flags ON for the one connection at θ=0.95, low volume. Gate: zero ERP submission failures in 24h, CSR-override-after-submit within target (TBD) over 72h and ≥20 auto-submitted orders, confirmed by the manual sample-audit above (not the is_edited proxy alone).
• S2 Widen threshold: lower agentic_confidence_threshold step-wise (0.95 → 0.90 → …) only while S1 guardrails hold ≥1 week per step.
• S3 Steady state: monitor 1 week; flag can be marked permanent after 2 weeks clean.

Caveat — “one connection” is not a real Flagsmith scope for these flags: agentic_auto_submit_enabled and erp_write_enabled are org-scoped, and Flagsmith segment overrides target an org, not an individual connection (unlike the connection-scoped gates, e.g. agentic_require_identity_grade → conn_ck_wherefour). So S1 above is only truly scoped to “one connection” when the org has only one connection. For a multi-connection org, flipping these two flags ON enables auto-submit for every connection on that org at once — a real S1 canary there requires either restricting promotion to single-connection orgs or building a connection-scoped write gate first.

Never edit the env defaults for the enable/threshold flags — flip per-org/connection via Flagsmith segment overrides (same pattern as agentic_require_identity_grade → conn_ck_wherefour).

Temporary connection-scoped demo exceptions

A connector that has not passed its full write-readiness gate may be enabled only when all of the following are true:

1. the customer has explicitly approved writes to the named non-customer demo connection;
2. a default-off connection-scoped flag isolates that one connection;
3. the allowed operation is narrower than the unproven connector surface and is enforced in both the route and ERP service;
4. the write uses stable target identity, a fresh pre-read/version check, and exact post-write collateral verification;
5. ambiguous outcomes are never blindly retried;
6. a reversible rehearsal and restoration runbook is completed before the demo;
7. documentation continues to report the general readiness gate as not passed.

The Prophet21 quantity-only Genfit Playground path under p21_purchase_order_update_v1 is such an exception. It must not be enabled for ASF, P.F. Markey, or another Prophet21 connection without separate approval and proof.

Kill-switch criteria (flag off = instant, no deploy)

Breach	Action
Post-submit ERP error rate >10% above baseline at any stage	Disable agentic_auto_submit_enabled for that org immediately
CSR-override-after-submit rate breaches target (provisional default 2%, TBD at sign-off)	Drop back to shadow (auto_submit OFF, validation ON) and re-tune θ
Any stuck-order / customer complaint within 24h of a flip	Hold + investigate before advancing
Master kill	erp_write_enabled=false reverts every org to dry-run in one flip

The 2% kill-switch default above is a provisional placeholder picked to be conservative enough to fail closed while the real target is decided — it is not a signed-off number. Confirm or replace it during each connection’s S0 sign-off.

Owner: the flag owner runs the daily guardrail review off the dashboard during each stage and holds the kill-switch.