ADR 0001 — Temporal Persistence Target

May contain stale references

This document was audited on 2026-06-22 and contains references that need updating: status field not yet updated post-cutover. Cross-check critical details against the current codebase before relying on them.

Status: Accepted (interim) — review date 30 days post-prod-cutover Date: 2026-04-19 Owner: @dboone31 Supersedes: none Related: GCP migration plan v3

---

Context

The GCP migration ends with Dagster metadata and Temporal persistence both living on a single shared self-host Postgres VM. This is a known single point of failure. Team review (see eng-reviews/gcp-migration-team-synthesis-2026-04-19.md) surfaced a strong recommendation to move Temporal to Cloud SQL HA.

The core observation from review:

WAL/PITR is not HA. It protects RPO, not RTO. A dead VM still pauses Temporal, and your business just proved that “pipeline paused for days” is unacceptable. Temporal is the control plane for long-running execution; don’t put that on a single pet VM if Cloud SQL HA is available.

Temporal’s event-history durability keeps workflow data safe while the database is reachable. It does not keep workflows running while the database is down. Any outage of the persistence layer pauses every in-flight workflow until the DB is back.

Decision

Target state: Temporal persistence runs on Cloud SQL for PostgreSQL, regional HA configuration.

Interim state (this migration): Temporal persistence runs on the same shared self-host Postgres VM as Dagster metadata, with pgBackRest-based PITR (see PR 3).

Migration to the target state is scoped as Milestone M+1, a standalone PR opened the first week prod is stable on the self-host VM.

Why the interim

1. Shipping Cloud SQL HA inside this migration doubles PR 1 scope and delays cutover.
2. The same SPOF has been running on Hetzner shared-postgres for months without incident.
3. A standalone follow-up PR has tighter blast radius and a cleaner rollback path.

Why the target

1. Cloud SQL HA provides ~2 min RTO on failover. Self-host VM failure RTO is measured in hours (diagnose, restore from snapshot or PITR, re-attach workers).
2. Temporal’s durability model only protects in-flight work against data loss, not against unavailability. The control plane pausing is itself the outage.
3. Team consensus — both tech-lead and backend flagged the self-host target as the wrong end state for the long-running workflow engine specifically.

Consequences

Accepted interim risk

• VM failure pauses every Temporal workflow until restore completes. Expected RTO: several hours.
• pgBackRest + PD snapshot + pg_dump give three independent recovery paths but none of them is automated failover.
• Runbook must treat Temporal persistence restore as a rehearsed drill, not an ad-hoc recovery.

Mitigations

• pgBackRest with continuous WAL archive (PR 3) gives PITR to within seconds.
• PD snapshot policy (daily, 14-day retention) as an independent recovery path.
• Quarterly restore drill documented in GCP_RUNBOOK.md.
• Known SPOFs section of the runbook names this SPOF explicitly with a link to this ADR.

Guard-rails so this does not become permanent

• Review date: 30 days post-prod-cutover.
• Milestone M+1 is pre-scoped in the migration plan (PR 7) — not a blank page when we get there.
• Every GCP runbook section that touches the self-host Postgres VM links back to this ADR.
• This ADR’s status stays Accepted (interim) until PR 7 lands, at which point it flips to Resolved with a pointer to the Cloud SQL module.

Milestone M+1 — target scope

Laid out here so the follow-up is concrete:

1. NEW: infrastructure-gcp/terraform/modules/gcp-cloud-sql-temporal/ — regional HA Postgres 16, db-custom-2-7680 initial sizing, automated backups + PITR, private IP via PSA.
2. NEW: Secret Manager secrets for Temporal default + visibility DSNs.
3. EDIT: staging/main.tf, production/main.tf — add the module.
4. EDIT: ansible/playbooks/temporal.yml — template DSNs to Cloud SQL instead of shared VM. Dagster stays on shared VM.
5. Runbook: migration sequence — stop Temporal frontends + workers, dump temporal and temporal_visibility from shared VM at the same cut point, restore into Cloud SQL with matching Temporal major/minor, redeploy Temporal server with new DSNs, validate via DescribeWorkflowExecution + visibility queries, restart workers.
6. Decommission: drop temporal and temporal_visibility from the shared VM only after 30 days stable on Cloud SQL.
7. Deliberate Cloud SQL failover exercise as PR ship criteria — confirms RTO < 2 minutes.
8. Update this ADR → status Resolved. Remove Temporal line from GCP_RUNBOOK.md “Known SPOFs”.

Alternatives considered

A. Temporal on Cloud SQL HA now, inside the migration. Technically correct end state but doubles PR 1 scope, delays cutover, and increases review surface during a period where the team is trying to close a migration already in progress for months. Rejected as scope-creep.

B. Keep Temporal on self-host VM permanently. Rejected. Team consensus that the RTO profile is wrong for the workflow control plane.

C. Interim self-host + explicit follow-up milestone (chosen). Ships the migration quickly, pays down the SPOF as a targeted change, preserves the right end state.