Feature Flag Policy: ERP-Write Path Changes

May contain stale references

This document was last spot-checked on 2026-08-04: AGENTS.md now contains the rule quoted in §7 (“Feature Flags Required for ERP-Write Changes”), so that specific staleness noted on 2026-06-22 is resolved. Cross-check critical details against the current codebase before relying on them.

Status: Active
Effective: 2026-06-15
Category: DevOps / Change Management
Scope: All code paths that create, modify, or submit data to external ERP systems

---

1. Scope — What Requires a Feature Flag

Any PR that modifies behavior in the following areas must be gated behind a Flagsmith feature flag before merging to staging:

Area	Examples
ERP order submission	ERPOrderSubmissionWorkflow, IERPService.createOrder(), quote generation, order update
ERP quote/approval	ERPQuoteGenerationWorkflow, ERPQuoteApprovalWorkflow, auto-submit logic
Extraction pipeline	PDFExtractionWorkflow, Gemini prompt changes, pre-extraction, anchor merge, item matching
Agentic validation	AgenticValidationWorkflow, confidence thresholds, auto-price writeback
Email triage	LLM triage prompt changes, trusted-sender logic, filter heuristic changes
Customer/item matching	Typesense matching logic, fuzzy match thresholds, customer resolution

Exceptions:

• Bug fixes to existing flagged code (fix goes behind the same flag)
• Pure refactors with no behavior change (verified by existing tests passing)
• Documentation, test-only, or CI-only changes

2. Staged Rollout Protocol

Every flagged change follows a three-stage rollout:

Stage 1: Internal (1–3 days)

• Flag enabled for internal test organization only (org_internal_test)
• Monitor extraction accuracy, ERP submission success rate, error logs
• Gate: Zero ERP submission failures in 24 hours

Stage 2: Canary Customer (3–5 days)

• Flag enabled for one low-risk customer organization
• Selected customer criteria: low order volume, tolerant of delays, direct communication channel
• Gate: No customer-reported issues in 72 hours, error rate within baseline

Stage 3: General Availability

• Flag enabled for all organizations (set default to true)
• Continue monitoring for 1 week
• Gate: Error rate within ±5% of pre-change baseline

Rollback

• If error rate spikes >10% above baseline at any stage: disable flag immediately
• Flag owner is responsible for monitoring during each stage
• Rollback does not require a deploy — flag change is instant

3. Flag Naming Convention

{area}_{feature}_{version}

Examples:

• erp_submission_batch_dedup_v1
• extraction_anchor_merge_v2
• validation_auto_submit_confidence_v1
• triage_trusted_sender_bypass_v1

4. PR Checklist for ERP-Write Changes

Add this checklist to any PR touching the scoped areas:

### ERP-Write Change Checklist

- [ ] Feature flag created in Flagsmith (name: `___`)
- [ ] Code path gated behind flag (default: disabled)
- [ ] Flag check verified in unit test (both enabled/disabled paths)
- [ ] Rollout plan documented (which orgs, what order, monitoring plan)
- [ ] Rollback tested (flag disabled → old behavior restored)

5. Monitoring During Rollout

During each rollout stage, the flag owner monitors:

Metric	Source	Alert Threshold
ERP submission success rate	Temporal workflow completion	<95% (baseline ~99%)
Extraction accuracy	LangFuse traces	>5% drop from baseline
Stuck orders	stuck-order-monitor.yml	Any new stuck items
Customer complaints	Support channel	Any complaint within 24h of flag change
Error rate (500s)	Cloud Run logs	>2x baseline

6. Flag Lifecycle

1. Create — Before PR merge. Document purpose in Flagsmith description.
2. Stage 1–3 — Follow rollout protocol above.
3. Permanent — After 2 weeks at GA with no issues, flag can be marked permanent.
4. Cleanup — Remove flag checks from code within 30 days of going permanent. Create a cleanup ticket when marking permanent.

7. Enforcement

• PR reviewers should check for flag gating on ERP-write path changes
• The AGENTS.md rule “Feature Flags Required for ERP-Write Changes” reminds AI agents to follow this policy
• Future: CI lint rule to detect ungated changes to scoped files (tracked separately)

8. Auto-Submit Promotion Ladder + Kill-Switch Criteria

Auto-submit (agentic validation writing a real order to the ERP with no human in the loop) is the highest-blast-radius ERP-write path. ERP accept/reject does not catch a wrong customer or wrong SKU the ERP happily accepts — so an org earns each promotion with shadow measurement first, not a flag flip. The AgenticValidationWorkflow spine already supports every mode; this ladder is the discipline loop around it. No promotion happens without the guardrail data.

Scope: promote per connection, not per org

The enable/threshold flags (agentic_validation_enabled, agentic_auto_submit_enabled, agentic_confidence_threshold) are org-scoped, but the ERP blast unit and the connection-scoped gates (agentic_require_identity_grade, agentic_ariba_total_crosscheck) key on the connection. Track promotion per connection (the guardrail dashboard rows are org/connection/day). For a multi-connection org, gate the threshold conservatively — an org-scoped flip enables auto-submit on every connection at once.

Measurement surfaces (build these before promoting)

Surface	Where	Reads
Guardrail dashboard	@erp-unlocked/internal → /autosubmit-guardrails (gated by internal_autosubmit_dashboard_v1)	extracted_orders.auto_validation_* per org/connection/day
Shadow-vs-CSR precision	pnpm --filter @repo/eval eval:autosubmit-shadow-comparison (read-only, staging)	dry-run/would-have-submitted decisions joined to the CSR’s final action
Threshold sweep	pnpm --filter @repo/eval eval:autosubmit-threshold-sweep	CK golden set → coverage/precision per θ

The key precision signal is CSR-override-after-submit: the "Line item overridden" log line joined to auto_validation_status='auto_submitted' orders. It is the only thing that catches a wrong order the ERP accepted. (The dashboard uses extracted_orders.is_edited as a DB proxy until that log join is wired.) Do not treat the idempotency key (erp-create:{conn}:{order}) as a correctness guardrail — it only prevents duplicate writes.

⚠️ The proxy is not gating on its own. is_edited is a coarser signal than the real log join (it doesn’t confirm the edit followed the submit, only that the order was edited at some point) and must not be treated as sufficient to pass S1 by itself. Until the exact log join is wired, S1 promotion requires a manual sample-audit of a random slice of that connection’s auto-submitted orders (CSR/eng spot-check against the ERP) alongside the is_edited proxy — do not promote on the automated proxy alone.

The ladder (per connection)

review-only ──▶ shadow (dry_run) ──▶ auto-submit @0.95 ──▶ widen θ ──▶ steady state
              validation ON,        both write flags ON,   0.95→0.90    monitor,
              both write flags OFF  one connection,         step-wise    mark permanent
              measure precision     canary volume           per policy

approval-only (review_approval_only_mode) is a parallel holding pen for orgs whose ERP write path isn’t live yet (VMS / Eclipse E4 — the “Finished” button logs the actor and writes nothing). It graduates to shadow once createOrder is spec-complete for that ERP.

Eligibility to enter the ladder: (1) the ERP write path is real for that ERP; (2) an offline eval passes bar (CK golden / VMS ground-truth); (3) enough order volume for meaningful shadow metrics; (4) a paying/real customer.

Per-connection stages (specialize the §2 three-stage protocol):

📋 Numeric gates below are provisional placeholders, not signed-off thresholds. The precision target, minimum sample/order count per stage, and the exact kill-switch percentage are open business decisions (see the matching-resolution and auto-submit specs’ Open Questions) — do not promote a connection past S0 until an eng/CEO owner has explicitly confirmed the numbers for that connection. Treat every target/θ/percentage in this section as “TBD, propose during S0 sign-off” until replaced with a decided value.

• S0 Shadow (3–5 days, minimum N=50 orders): agentic_validation_enabled ON, agentic_auto_submit_enabled and erp_write_enabled OFF. Collect dry-run decisions; run the shadow-vs-CSR comparison. Gate: shadow precision ≥ target (TBD — propose at sign-off) at θ=0.95, computed over ≥50 dry-run orders (widen the window if volume is lower).
• S1 Canary auto-submit (3–5 days, minimum N=20 auto-submitted orders): flip both write flags ON for the one connection at θ=0.95, low volume. Gate: zero ERP submission failures in 24h, CSR-override-after-submit within target (TBD) over 72h and ≥20 auto-submitted orders, confirmed by the manual sample-audit above (not the is_edited proxy alone).
• S2 Widen threshold: lower agentic_confidence_threshold step-wise (0.95 → 0.90 → …) only while S1 guardrails hold ≥1 week per step.
• S3 Steady state: monitor 1 week; flag can be marked permanent after 2 weeks clean.

Caveat — “one connection” is not a real Flagsmith scope for these flags: agentic_auto_submit_enabled and erp_write_enabled are org-scoped, and Flagsmith segment overrides target an org, not an individual connection (unlike the connection-scoped gates, e.g. agentic_require_identity_grade → conn_ck_wherefour). So S1 above is only truly scoped to “one connection” when the org has only one connection. For a multi-connection org, flipping these two flags ON enables auto-submit for every connection on that org at once — a real S1 canary there requires either restricting promotion to single-connection orgs or building a connection-scoped write gate first.

Never edit the env defaults for the enable/threshold flags — flip per-org/connection via Flagsmith segment overrides (same pattern as agentic_require_identity_grade → conn_ck_wherefour).

Kill-switch criteria (flag off = instant, no deploy)

Breach	Action
Post-submit ERP error rate >10% above baseline at any stage	Disable agentic_auto_submit_enabled for that org immediately
CSR-override-after-submit rate breaches target (provisional default 2%, TBD at sign-off)	Drop back to shadow (auto_submit OFF, validation ON) and re-tune θ
Any stuck-order / customer complaint within 24h of a flip	Hold + investigate before advancing
Master kill	erp_write_enabled=false reverts every org to dry-run in one flip

The 2% kill-switch default above is a provisional placeholder picked to be conservative enough to fail closed while the real target is decided — it is not a signed-off number. Confirm or replace it during each connection’s S0 sign-off.

Owner: the flag owner runs the daily guardrail review off the dashboard during each stage and holds the kill-switch.