Initiative: Restore `erp-prod-dagster` (Phase 2 prerequisite)

Status: PROPOSED — runbook drafted 2026-05-24, awaiting review Trigger to start: Full Hetzner → GCP cutover for app.erpunlocked.com is now on the active roadmap; erp-prod-dagster must be healthy before prod workloads (WhereFour, dbt, Typesense sync) can be relocated off the Hetzner-staging-against-prod hack documented in project_wherefour_staging_coupled_to_prod. Builds on: PR #836 (which landed the corrected dagster.yml playbook + deploy-dagster-from-1p.sh on staging) and PR #1149 (which retired the gcp-prod branch so prod now deploys from main). Owner: David Last updated: 2026-05-24

Problem Statement

erp-prod-dagster (10.0.0.3, us-east1-b, ordermatic-prod) has been crash-looping for 71 days with every container restart-count above 30,000. Root cause: the /opt/dagster/.env placed by the original Ansible run (2026-03-14) uses DAGSTER_POSTGRES_* env-var names, but the dagster image’s docker-entrypoint.sh and apps/dagster/dagster.prod.yaml both assert DAGSTER_PG_* (plus DAGSTER_CURRENT_IMAGE). The boot loop is:

[BOOT] ERROR: required env vars are unset: DAGSTER_PG_HOSTNAME DAGSTER_PG_PORT
       DAGSTER_PG_USERNAME DAGSTER_PG_PASSWORD DAGSTER_PG_DB_NAME DAGSTER_CURRENT_IMAGE

The fix already exists on origin/staging:

File	Status
`infrastructure-gcp/ansible/playbooks/dagster.yml`	Corrected: writes `DAGSTER_PG_*` names, sets `DAGSTER_CURRENT_IMAGE` per service in compose, defaults image tag to `latest-main` for production per PR #1149, wires Infisical universal-auth bootstrap (PR #739)
`infrastructure-gcp/scripts/deploy-dagster-from-1p.sh`	New: pulls 6 secrets from 1Password, writes to mode-0600 temp file, runs `make ansible-dagster EXTRA_VARS_FILE=...`

Neither is on main. Per PR #1149 prod now deploys from main, so the fix must land via the normal staging→main release flow before execution.

Investigation findings (why the work is smaller than feared)

Initial scoping suggested this required Phase 2 database migration prerequisites: provisioning erp-prod-postgres VM, applying terraform that would have triggered destroy/recreate of erp-prod-temporal (IP 10.0.0.2 → 10.0.0.5). Direct VM inspection showed:

erp-prod-temporal is stateless. Its workflow data lives on Cloud SQL erp-prod-pg (private IP 10.66.0.3) via POSTGRES_SEEDS=10.66.0.3, TLS enabled, user temporal, databases temporal + temporal_visibility. The auto-setup container’s 71-day “Waiting for Temporal server to start…” retry log is misleading; temporal-server is functional and workers are actively processing ScheduledUsageReportingWorkflow every 15 minutes.
Cloud SQL erp-prod-pg is the de-facto Phase 2 backing store already serving production temporal. It has erp_db (empty, awaiting Phase 2 app DB migration), temporal (in use), temporal_visibility (in use). Users: erp_user, temporal, postgres.
erp-prod-dagster itself holds no state. /opt/dagster/dagster_home/ is empty; the VM has never persisted a single Dagster job.
DAGSTER_URL=http://10.0.0.3:3000 is set on erp-webapp-prod Cloud Run but no application code reads it — confirmed by full grep of apps/ and packages/. The env var is vestigial Terraform config; restoring dagster doesn’t unblock any current user-facing functionality.

Recommended path: use existing Cloud SQL, no new VM

The staging-branch terraform additions for module "postgres_vm" are unnecessary in prod. Cloud SQL is the cleaner long-term home for dagster metadata for the same reasons it already holds temporal data: automated backups, PITR, no VM to maintain.

                                                                  ┌────────────────────────┐
                                                                  │ Cloud SQL erp-prod-pg  │
                                                                  │ (10.66.0.3, Postgres15)│
┌────────────────────────────┐                                    │                        │
│ erp-prod-dagster (10.0.0.3)│  ───── TLS, DAGSTER_PG_* env ────▶ │ db: dagster   ◀── NEW  │
│  • dagster-code-server     │                                    │ db: temporal           │
│  • dagster-webserver       │                                    │ db: temporal_visibility│
│  • dagster-daemon          │                                    │ db: erp_db (empty)     │
└────────────────────────────┘                                    │                        │
                                                                  └────────────────────────┘
                                  ▲
                                  │
┌────────────────────────────┐    │                               ┌────────────────────────┐
│ erp-prod-temporal (10.0.0.2)│ ──┘ TLS, POSTGRES_SEEDS env       │ Infisical              │
│  (already healthy)          │                                   │ /runtime/dagster-gcp   │
└────────────────────────────┘                                    │ /runtime/_shared       │
                                                                  └────────────────────────┘
                                                                              ▲
                                            ┌─── universal-auth client ───────┘
                                            │
                                  ┌─────────┴──────────┐
                                  │ dagster.env at boot│
                                  └────────────────────┘

Pre-flight checks (run all before step 1)

source /Volumes/DevStorage/Developer/Dev/erp-unlocked/.env  # for OP_SERVICE_ACCOUNT_TOKEN

# 1. Confirm staging→main release has landed and main has the fix
git fetch origin main
git show origin/main:infrastructure-gcp/ansible/playbooks/dagster.yml | grep -q 'DAGSTER_PG_HOSTNAME' \
  && echo "OK: playbook on main" || echo "BLOCK: staging→main release not yet merged"
git show origin/main:infrastructure-gcp/scripts/deploy-dagster-from-1p.sh >/dev/null 2>&1 \
  && echo "OK: deploy script on main" || echo "BLOCK: script not yet on main"

# 2. Confirm `latest-main` dagster image exists in Artifact Registry
# (cloudbuild.yaml:60 emits ${_REGISTRY}/dagster:latest-${BRANCH_NAME}; main → latest-main ✓)
gcloud artifacts docker images list us-east1-docker.pkg.dev/ordermatic-prod/erp-unlocked/dagster \
  --filter='tags:latest-main' --format='value(tags,createTime)' | head -3
# Capture the immutable digest for pinning during deploy + rollback parity
DAGSTER_IMAGE_DIGEST=$(gcloud artifacts docker images describe \
  us-east1-docker.pkg.dev/ordermatic-prod/erp-unlocked/dagster:latest-main \
  --format='value(image_summary.digest)' --project=ordermatic-prod)
echo "Pinned digest: $DAGSTER_IMAGE_DIGEST"
test -n "$DAGSTER_IMAGE_DIGEST" && echo "OK: digest captured" || echo "BLOCK: tag has no digest"

# 3. Confirm Cloud SQL erp-prod-pg is RUNNABLE, has >= 5 GB headroom, no maintenance window now
gcloud sql instances describe erp-prod-pg --project=ordermatic-prod \
  --format='value(state,settings.tier,settings.dataDiskSizeGb,settings.maintenanceWindow)'
# Live disk usage — Cloud SQL exposes via Cloud Monitoring; alternatively check the instance:
gcloud sql instances describe erp-prod-pg --project=ordermatic-prod \
  --format='value(currentDiskSize)' | awk '{
    cur=$1/1024/1024/1024;
    if (cur > 0) print "Current disk used: " cur " GB";
  }'
# Hard gate: state must be RUNNABLE, no active maintenance, free disk >= 5 GB

# 4. Confirm VM is reachable via IAP
gcloud compute ssh erp-prod-dagster --zone=us-east1-b --project=ordermatic-prod \
  --tunnel-through-iap --command='echo OK' 2>&1 | tail -3

# 5. Confirm ALL 4 1Password items exist BEFORE we start — script fails late otherwise
for item in "GCP Production Dagster PostgreSQL" "Production Database" "Infisical webapp-prod" "Infisical dagster-prod"; do
  if op item get "$item" --vault "ERP-Unlocked" --format=json >/dev/null 2>&1; then
    echo "OK: 1P item present — $item"
  else
    echo "BLOCK: 1P item missing — $item"
  fi
done

# 6. Confirm cloudbuild `latest-main` is plausibly current (built within last 7 days)
# Stale tag = drift; the playbook on main may expect features the image doesn't have.
gcloud artifacts docker images list us-east1-docker.pkg.dev/ordermatic-prod/erp-unlocked/dagster \
  --filter='tags:latest-main AND updateTime >-P7D' --format='value(updateTime)' | head -1

# 7. Capture main HEAD SHA + image digest for the audit trail
MAIN_SHA=$(git rev-parse origin/main)
echo "Deploying playbook=$MAIN_SHA image_digest=$DAGSTER_IMAGE_DIGEST" \
  | tee /tmp/dagster-prod-deploy-pins.txt

All checks must pass before proceeding. If latest-main doesn’t exist or is >7 days old, trigger a fresh build before deploying. If any 1P item is missing, return to step 3 / 4 to create.

Execution steps

Each step lists what changes, the exact command, expected output, and the rollback if something goes wrong.

Step 1 — Wait for staging→main release

Owner: David Action: Trigger or wait for normal staging→main release that includes commits since the last release (likely PR #836 onwards). The release PR will include the two dagster files plus everything else accumulated on staging. Verification: Pre-flight checks 1 + 2 both return OK. Rollback: N/A (no infrastructure change).

Step 2 — Create `dagster` database, user, and grants on Cloud SQL

Owner: Claude (via gcloud sql + psql) Generates: A new strong password for the dagster Cloud SQL user. Captured for step 3.

Postgres 15 changed default behavior: newly created users have no CREATE on the public schema even of their own databases. Dagster’s daemon will fail at schema bootstrap (permission denied for schema public) unless we explicitly grant ownership. The user-creation step alone is not enough.

# 2.1 — Generate password (32 char URL-safe, no quoting headaches)
DAGSTER_DB_PWD=$(python3 -c "import secrets; print(secrets.token_urlsafe(24))")
echo "$DAGSTER_DB_PWD" > /tmp/dagster-db-pwd  # held in memory for step 3 — DO NOT commit

# 2.2 — Create database
gcloud sql databases create dagster --instance=erp-prod-pg --project=ordermatic-prod

# 2.3 — Create login role with password
gcloud sql users create dagster --instance=erp-prod-pg --project=ordermatic-prod \
  --password="$DAGSTER_DB_PWD"

# 2.4 — Grant ownership + schema privileges (CRITICAL — Postgres 15 won't grant these implicitly)
# Connect from the erp-prod-temporal VM since it already has private connectivity to Cloud SQL
# (verified — it talks to 10.66.0.3 for the temporal db). Authenticates as `postgres` superuser.
POSTGRES_PWD=$(op item get "GCP Production Postgres Admin" --vault "ERP-Unlocked" --fields password --reveal 2>/dev/null) \
  || POSTGRES_PWD=$(gcloud secrets versions access latest --secret=erp-prod-pg-postgres-password --project=ordermatic-prod 2>/dev/null) \
  || { echo "BLOCK: no source for the postgres superuser password"; exit 1; }

# Run grants via the temporal VM (already has psql + private DB connectivity)
gcloud compute ssh erp-prod-temporal --zone=us-east1-b --project=ordermatic-prod \
  --tunnel-through-iap --command="
    sudo apt-get install -y postgresql-client 2>&1 | tail -2
    PGPASSWORD='$POSTGRES_PWD' psql 'host=10.66.0.3 port=5432 user=postgres dbname=dagster sslmode=require' <<'SQL'
      ALTER DATABASE dagster OWNER TO dagster;
      GRANT ALL PRIVILEGES ON DATABASE dagster TO dagster;
      GRANT ALL ON SCHEMA public TO dagster;
      ALTER SCHEMA public OWNER TO dagster;
SQL
  "

Verification:

# Connect as the new dagster user and confirm CREATE works
gcloud compute ssh erp-prod-temporal --zone=us-east1-b --project=ordermatic-prod \
  --tunnel-through-iap --command="
    PGPASSWORD='$DAGSTER_DB_PWD' psql 'host=10.66.0.3 port=5432 user=dagster dbname=dagster sslmode=require' \
      -c 'CREATE TABLE _grant_probe(x int); DROP TABLE _grant_probe;'
  "

Pass: no error. Fail: permission denied for schema public → step 2.4 didn’t apply correctly; do not proceed.

Rollback:

# Order matters: revoke before drop to avoid "must be owner" errors
gcloud compute ssh erp-prod-temporal --zone=us-east1-b --project=ordermatic-prod \
  --tunnel-through-iap --command="
    PGPASSWORD='$POSTGRES_PWD' psql 'host=10.66.0.3 port=5432 user=postgres dbname=postgres sslmode=require' \
      -c 'ALTER DATABASE dagster OWNER TO postgres;'
  "
gcloud sql users delete dagster --instance=erp-prod-pg --project=ordermatic-prod
gcloud sql databases delete dagster --instance=erp-prod-pg --project=ordermatic-prod

Step 3 — Create 1Password item `GCP Production Dagster PostgreSQL`

Owner: Claude (via op with OP_SERVICE_ACCOUNT_TOKEN from .env) Vault: ERP-Unlocked (matches what deploy-dagster-from-1p.sh ENV=production reads) Fields:

Field	Value
`server`	`10.66.0.3` (Cloud SQL private IP)
`port`	`5432`
`username`	`dagster`
`database`	`dagster`
`password`	`$DAGSTER_DB_PWD` from step 2

source /Volumes/DevStorage/Developer/Dev/erp-unlocked/.env  # absolute path per feedback memory

op item create --vault "ERP-Unlocked" \
  --category 'Database' \
  --title "GCP Production Dagster PostgreSQL" \
  "server=10.66.0.3" \
  "port=5432" \
  "username=dagster" \
  "database=dagster" \
  "password=$DAGSTER_DB_PWD"

Note: Title Case category names + space-separated flags match the existing convention in infrastructure/docs/1PASSWORD_ITEMS_REFERENCE.md and the items already in this vault.

Verification: op item get "GCP Production Dagster PostgreSQL" --vault "ERP-Unlocked" shows all 5 fields.

Rollback: op item delete "GCP Production Dagster PostgreSQL" --vault "ERP-Unlocked".

Step 4 — Create Infisical universal-auth client + populate `/runtime/dagster-gcp` (prod project)

Owner: Claude where API permits, David for UI-only operations Infisical: https://infisical.ordermatic.co Project: the existing prod project (same project_id as Infisical webapp-prod)

Sub-steps:

Create universal-auth machine identity with scope read /runtime/dagster-gcp/* + read /runtime/_shared/* in the prod environment of the prod project.
Capture client_id + client_secret at creation time (Infisical shows the secret only once).
Populate /runtime/dagster-gcp/ path in the prod environment with prod values. Mirror the staging shape — check /runtime/dagster-gcp/ in staging for the schema. Likely keys: WHEREFOUR_USERNAME, WHEREFOUR_PASSWORD, WHEREFOUR_BASE_URL, R2_*, anything else the dagster pipelines need. Do not copy staging values verbatim — these must be the prod versions.
Save to 1P as Infisical dagster-prod:

op item create --vault "ERP-Unlocked" \
  --category 'Login' \
  --title "Infisical dagster-prod" \
  "client_id=<from step 4.2>" \
  "client_secret=<from step 4.2>"

Verification: From a probe with the new client_id/secret, infisical secrets --path /runtime/dagster-gcp/ --env prod returns expected keys.

Rollback: Revoke the universal-auth client in Infisical UI; op item delete "Infisical dagster-prod".

Known unknown: Infisical API access for machine-identity creation depends on what credentials Claude has via Infisical Server — Master Credentials 1P item. If that item is admin-tier, this step is fully automated. If not, David performs the UI portion and Claude finishes step 4.4 only.

Step 5 — Verify Cloud SQL `dagster` user can connect

This is a probe — proves credentials work before we hand them to the deploy script.

gcloud compute ssh erp-prod-dagster --zone=us-east1-b --project=ordermatic-prod \
  --tunnel-through-iap --command='
    # Pull password from a temp file we scp over, not from shell history
    sudo apt-get install -y postgresql-client 2>&1 | tail -3
    PGPASSWORD="<from step 3>" psql "host=10.66.0.3 port=5432 user=dagster dbname=dagster sslmode=require" \
      -c "SELECT version();"
  '

Expected: PostgreSQL 15.x .... Anything else = blocker; do not proceed.

Rollback: N/A (read-only).

Step 6 — Run the deploy

Owner: Claude Branch: main (after staging→main release; pre-flight check 1 must pass) Cwd: /Volumes/DevStorage/Developer/Dev/erp-unlocked/infrastructure-gcp (not a worktree — the live tree, so make and Makefile-relative paths resolve correctly)

6.0 — Capture deterministic pre-deploy backups (run FIRST)

Without this, the rollback is broken: the playbook overwrites /opt/dagster/.env and /opt/dagster/docker-compose.yml, and any rollback that references a date-suffixed file the playbook created on its own will point at a file whose name we don’t know in advance.

gcloud compute ssh erp-prod-dagster --zone=us-east1-b --project=ordermatic-prod \
  --tunnel-through-iap --command='
    sudo cp /opt/dagster/.env /opt/dagster/.env.bak-pre-restore
    sudo cp /opt/dagster/docker-compose.yml /opt/dagster/docker-compose.yml.bak-pre-restore
    echo "Backups in place:"
    sudo ls -la /opt/dagster/*.bak-pre-restore 2>&1
  '

Rollback can now reference exact paths: /opt/dagster/.env.bak-pre-restore + /opt/dagster/docker-compose.yml.bak-pre-restore. The known broken-pre-deploy state is recoverable as a clean causal baseline (a worse-than-now hybrid is the danger codex flagged).

6.1 — Take the deploy lock and run

source /Volumes/DevStorage/Developer/Dev/erp-unlocked/.env  # for OP_SERVICE_ACCOUNT_TOKEN
cd /Volumes/DevStorage/Developer/Dev/erp-unlocked/infrastructure-gcp
git switch main && git pull --ff-only origin main

# Sanity: confirm we're deploying the SHA captured in pre-flight
test "$(git rev-parse HEAD)" = "$MAIN_SHA" || { echo "BLOCK: main moved since pre-flight"; exit 1; }

# Concurrent-deploy guard. flock blocks; if another operator is mid-deploy, fail
# fast rather than racing on /opt/dagster file writes inside the ansible run.
flock -n /tmp/dagster-prod-deploy.lock -c '
  ENV=production bash scripts/deploy-dagster-from-1p.sh 2>&1 \
    | tee /tmp/dagster-prod-deploy.log
' || { echo "BLOCK: another deploy is in progress (flock /tmp/dagster-prod-deploy.lock)"; exit 1; }

What this does:

Pulls 6 secrets from the 4 1P items
Writes them to a mode-0600 temp file
Runs make ansible-dagster EXTRA_VARS_FILE=<file> which runs the playbook
Playbook: installs Docker if missing, writes new /opt/dagster/.env with DAGSTER_PG_* names, writes new /opt/dagster/docker-compose.yml with DAGSTER_CURRENT_IMAGE per service, pulls latest-main, restarts dagster.service

Expected duration: 3-5 minutes.

Verification (within 5 min of deploy):

gcloud compute ssh erp-prod-dagster --zone=us-east1-b --project=ordermatic-prod \
  --tunnel-through-iap --command='
    sudo docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Image}}"
    echo
    for c in dagster-code-server dagster-webserver dagster-daemon; do
      echo "-- $c restartCount --"
      sudo docker inspect "$c" --format "{{.RestartCount}}"
    done
    echo
    echo "-- .env env-var names (values redacted) --"
    sudo grep -E "^[A-Z_]+=" /opt/dagster/.env | cut -d= -f1
    echo
    echo "-- image digest pulled (should match pre-flight pin) --"
    sudo docker inspect dagster-webserver --format "{{.Image}}"
  '
# Compare last line against pre-flight $DAGSTER_IMAGE_DIGEST. Mismatch = drift; investigate.

Pass criteria: 3 containers Up X seconds (healthy) or Up X minutes, restart counts low (≤2 from initial start), image digest matches pin. Fail criteria: any Restarting, restart count climbing, digest mismatch.

Rollback (uses exact paths from 6.0 — no ambiguity):

gcloud compute ssh erp-prod-dagster --zone=us-east1-b --project=ordermatic-prod \
  --tunnel-through-iap --command='
    set -e
    cd /opt/dagster
    test -f /opt/dagster/.env.bak-pre-restore || { echo "ERROR: no pre-restore backup; cannot rollback cleanly"; exit 1; }
    test -f /opt/dagster/docker-compose.yml.bak-pre-restore || { echo "ERROR: no compose backup"; exit 1; }
    sudo cp /opt/dagster/.env.bak-pre-restore /opt/dagster/.env
    sudo cp /opt/dagster/docker-compose.yml.bak-pre-restore /opt/dagster/docker-compose.yml
    sudo systemctl restart dagster.service
    # Wait for restart to settle, then confirm we are back to the known-bad pre-deploy state
    sleep 10
    sudo docker ps --format "table {{.Names}}\t{{.Status}}"
  '

Rollback returns the system to the broken-but-known pre-deploy state. The 71-day baseline is preserved, no hybrid state introduced. From there, fix the cause (Cloud SQL grants, Infisical client, image tag, etc.) and re-run step 6 cleanly.

Step 7 — Post-cutover validation (three layers: liveness, write-path, secrets-reachability)

Up containers don’t prove dagster is functional. Codex flagged the “deploy succeeds but silently broken” failure mode where webserver registers, /server_info returns 200, but the daemon can’t write run storage or the code-server is missing required prod secrets. Validate all three layers.

7.1 — Liveness probe (webserver + code-server registration)

# IAP tunnel with cleanup trap so it doesn't leak if curl hangs
gcloud compute start-iap-tunnel erp-prod-dagster 3000 \
  --local-host-port=localhost:13000 \
  --zone=us-east1-b --project=ordermatic-prod &
TUNNEL_PID=$!
trap "kill $TUNNEL_PID 2>/dev/null" EXIT
sleep 3
curl -sS --max-time 10 http://localhost:13000/server_info | jq .
kill $TUNNEL_PID; trap - EXIT

Pass: {"dagster_version": "...", ...}. Fail: any 5xx or no JSON → check docker logs dagster-code-server for Infisical bootstrap or import errors.

7.2 — Write-path probe (proves DB grants + migrations worked)

gcloud compute ssh erp-prod-dagster --zone=us-east1-b --project=ordermatic-prod \
  --tunnel-through-iap --command='
    # Dagster bootstraps these tables on first daemon start. Their presence proves
    # the schema was created in our new database with the new user, which
    # in turn proves Step 2.4 grants applied correctly.
    PGPASSWORD="$(sudo grep ^DAGSTER_PG_PASSWORD /opt/dagster/.env | cut -d= -f2-)" \
      psql "host=10.66.0.3 port=5432 user=dagster dbname=dagster sslmode=require" \
      -c "SELECT count(*) FROM information_schema.tables WHERE table_schema = '\''public'\''"
  '

Pass: any positive count (dagster typically creates 20+ tables: runs, event_logs, instigators_data, etc.). Fail: 0 count or permission denied → DB grants didn’t take; rollback step 6 and revisit step 2.4.

7.3 — Infisical bootstrap probe (both paths, not just env var names)

gcloud compute ssh erp-prod-dagster --zone=us-east1-b --project=ordermatic-prod \
  --tunnel-through-iap --command='
    # Confirm BOTH expected Infisical paths injected non-empty values.
    # Codex flagged: checking env var NAMES exist is insufficient — jobs fail later
    # if values are empty or wrong-environment. Spot-check 2 keys per path.
    declare -A expected=(
      ["WHEREFOUR_USERNAME"]="/runtime/dagster-gcp"
      ["DATABASE_URL"]="/runtime/_shared OR env_file"
    )
    for k in "${!expected[@]}"; do
      val=$(sudo docker exec dagster-code-server printenv "$k" 2>/dev/null || echo "")
      if [ -n "$val" ]; then
        echo "OK: $k present from ${expected[$k]} (len=${#val})"
      else
        echo "BLOCK: $k MISSING — Infisical path ${expected[$k]} not injecting"
      fi
    done
  '

Pass: all expected keys present. Fail: any BLOCK → check Infisical client_id/secret scope against /runtime/dagster-gcp and /runtime/_shared. Common cause: client scoped to staging env, not prod.

Out of scope (explicitly NOT in this runbook)

Wiring prod workloads (WhereFour, dbt, Typesense sync) to this dagster. Currently runs on Hetzner staging dagster against prod creds per memory; that move is a separate effort once this dagster is verified healthy.
Phase 2 app DB migration (Neon → Cloud SQL erp_db). erp_db is sitting empty waiting; migrate-app-db.sh exists; not blocking dagster restore.
DAGSTER_URL env var on erp-webapp-prod. Vestigial — no code reads it. Keep as-is for future use or remove in a follow-up Terraform PR.
Retiring Hetzner. Multi-week project. This is a single prerequisite step.

Open decisions before execution

Infisical API access tier. Does the 1P item Infisical Server — Master Credentials permit machine-identity creation via API? If yes, step 4.1-4.2 fully automated. If no, David performs UI portion. This is the only remaining unknown that can’t be answered without trying.
~~/runtime/dagster-gcp prod values.~~ RESOLVED: Populate prod values before step 6, do not boot empty. Code-server imports modules that read these keys at import time; an empty path causes import errors that look like Infisical bootstrap failures (codex review confirmed). Mirror the staging shape and substitute prod credentials per the WhereFour/R2/etc. items already in 1Password.
~~Image tag verification.~~ RESOLVED: cloudbuild.yaml:60 emits ${_REGISTRY}/dagster:latest-${BRANCH_NAME} — main branch builds produce latest-main by construction. Pre-flight check 2 captures the immutable digest at deploy time for rollback parity.

Validation that “fixed adequately” was achieved

After execution, all of these are true:

3 containers on erp-prod-dagster show Up with restart-count ≤ 5 (vs current 30,000+)
.env has DAGSTER_PG_* names; DAGSTER_CURRENT_IMAGE is set in compose per service
Container image digest matches the pin captured in pre-flight (no tag-drift between pull and run)
Dagster code-server is registered with webserver (probe step 7.1 returns JSON)
Dagster schema bootstrapped on Cloud SQL (step 7.2 returns ≥ 20 tables in public schema)
Infisical bootstrap successful (step 7.3 confirms both /runtime/dagster-gcp and /runtime/_shared keys injected with non-empty values)
No outbound 401s in dagster-code-server logs (Artifact Registry auth working via metadata server fallback)
gcloud compute ssh ... docker logs dagster-daemon --since 5m shows no PermissionError on history/ (per the playbook’s DAGSTER_HOME comment — proves we didn’t reintroduce the staging bug)
/tmp/dagster-prod-deploy-pins.txt exists with the deployed SHA + image digest for the audit trail

References

infrastructure-gcp/PHASE2_DATA_MIGRATION_PLAN.md (on staging)
infrastructure-gcp/CUTOVER_RUNBOOK_2026_05_02.md (on staging)
docs/designs/betterstack-to-gcp-logging-cutover.md (style template, on staging)
PR #836 — landed dagster.yml fix on staging
PR #1149 — retired gcp-prod branch; main now is prod source of truth
PR #739 — introduced Infisical universal-auth bootstrap for runtime services
Memory: project_wherefour_staging_coupled_to_prod
Memory: project_gcp_dagster_recovery_2026_05_07 — analogous staging recovery