Runbook: retire a Flagsmith flag

Who runs this: a human with the Flagsmith admin key. Agents cannot — the secret guard blocks reading FLAGSMITH_API_KEY.

Pending as of 2026-08-10: inbox_auto_archive_edi_customers, recorded in _deprecated_flags by 4562d59f6 but still present in the live instances.

This DELETES, it does not archive

cleanup_flags() calls client.delete_feature(), which issues DELETE /api/v1/projects/{id}/features/{id}/ (infrastructure/flagsmith/scripts/cleanup-flagsmith.py). The feature and its per-environment state, segment overrides and history go with it. Recreating it afterwards gives you a new feature with the same name, not the old one. Treat this as destructive and dry-run every environment.

Why retirement is two steps

Removing a flag from flags.json is not enough, and stopping halfway breaks deploys. The repo config and the live instance are reconciled on every deploy, and a flag declared but disabled returns 500 on each deploy until it is either fully declared or fully gone. So:

1. Move the flag name out of flags and into _deprecated_flags in infrastructure/flagsmith/configs/flags.json — a reviewed code change. Already done for the pending flag.
2. Run cleanup-flagsmith.py per environment so it is actually removed upstream. That is this runbook.

Between the two the flag is listed as deprecated but still present. That state is stable and safe; it just isn’t finished.

Steps

cd infrastructure/flagsmith/scripts

Dry-run before changing anything — but know what it does and does not tell you. --dry-run prints _deprecated_flags straight out of flags.json and exits; it never contacts the instance. So it previews what the config will act on — which catches a name that landed in that list by accident — but it is not a preview of live state, and its output is identical for staging and production. Only the real run reads the instance, and it deletes only features that actually exist there.

python cleanup-flagsmith.py staging --dry-run
python cleanup-flagsmith.py staging

python cleanup-flagsmith.py production --dry-run   # review this output
python cleanup-flagsmith.py production             # interactive confirmation

Read each dry-run and confirm it names the flags you expect and nothing else. The production confirmation prompt is deliberate — do not script around it.

Credentials: staging and production read from 1Password automatically. To override, set FLAGSMITH_API_URL (prod is https://flagsmith.ordermatic.co/api/v1) and FLAGSMITH_API_KEY, or pass --api-url / --api-key. Never write the key to a file or into a command that gets logged.

Verify

Do not use --dry-run for this. It echoes _deprecated_flags from the config, so it keeps listing the flag forever, deleted or not — it can never report “nothing left to remove.”

Re-run the real command instead. It only deletes features that exist upstream, so a second run is a safe no-op and its own count is the signal:

python cleanup-flagsmith.py staging      # expect: Deleted 0 deprecated flags
python cleanup-flagsmith.py production   # expect: Deleted 0 deprecated flags

Deleted 0 means the feature is gone from that environment. Anything higher means the previous run didn’t take. Live flag state drifts from the repo, so trust this and the API over flags.json. A deploy that completes without the flag 500 is the real end-state signal.

Recovery

Deletion is not undone by editing _deprecated_flags alone. cleanup_flags() only creates features listed under flags and deletes names listed under _deprecated_flags, so recovery is:

1. restore the original flag definition under flags in flags.json,
2. remove its name from _deprecated_flags,
3. re-run reconciliation (gh workflow run flagsmith-seed.yml --ref main) or recreate it in the Flagsmith UI.

Either way you get a fresh feature: per-environment values, segment overrides and history are not restored. Re-apply targeting by hand and verify it through the API.