Coolify (Hetzner) decommission — June 2026
Status: ✅ STOP SWEEP COMPLETE — 12 idle origins stopped + confirmed exited (all reversible). app.erpunlocked.com redirect confirmed edge-served. Infisical (KEEP) intact. Deletion + Hetzner/Neon teardown remain as follow-ups. See Execution log.
Date: 2026-06-09
Author: infra cleanup pass (ENG)
Blocked on: nothing — stop sweep done. (op ✅, gcloud ✅; the CF redirect was confirmed edge-served by a reversible empirical test since the available CF API tokens lacked ruleset-read scope.) Remaining work is the follow-ups (delete, Hetzner/Neon teardown, CI cleanup).
This is the staged-decommission plan for the legacy Coolify control plane on Hetzner
(https://cool.boonetek.com). All compute moved to GCP in the 2026-05 cutover
(infrastructure-gcp/MIGRATION_STATUS.md, infrastructure-gcp/CUTOVER_RUNBOOK_2026_05_02.md).
Coolify still hosts idle origins that cost money (~$344/mo Hetzner+Neon is the target lever).
Method: STOP (reversible) before DELETE. One service at a time. Verify each is idle and has a healthy GCP/edge replacement before acting. Deletion + Hetzner/Neon teardown is an explicit follow-up, not part of this pass.
Execution log
Section titled “Execution log”2026-06-09 — live verification + staged stop sweep. op + gcloud re-authed. Verified every
GCP replacement healthy first: Cloud Run erp-webapp-{prod,staging} + erp-internal-{prod,staging}
READY; VMs erp-{prod,staging}-{dagster,temporal,temporal-workers} + erp-prod-internal-edge
RUNNING; the live prod webapp (100% traffic) is wired to GCP backends
(TEMPORAL_ADDRESS=10.0.0.2:7233, DAGSTER_URL=http://10.0.0.3:3000) — not the Coolify origins.
Then stopped 11 idle resources, one at a time, observing after each. All reversible via
POST /api/v1/{services|applications}/{uuid}/start:
| Service (uuid) | Kind | Before → After | Observe after |
|---|---|---|---|
multiwoven (k0w4g…) | service | degraded:unhealthy → exited | infisical 200; no prod consumer (MULTIWOVEN_API_URL unset, call site stubbed) |
staging-webapp (p848c…) | app | running:healthy → exited | staging.ordermatic.co 302 (GCP) |
staging-pdf-api (j0ok8…) | app | running:healthy → exited | staging 302 |
staging-pdf-worker (f8sso…) | app | running:healthy → exited | staging 302 |
staging-dagster-compose (akcks…) | service | running:healthy → exited | staging 302 |
staging-temporal-compose (gcok8…) | service | running:unknown → exited | staging 302 |
pdf-prod-api (xwcwc…) | app | running:healthy → exited | app.ordermatic.co 302 |
pdf-worker (hwg0w…) | app | running:healthy → exited | app 302 |
dagster-compose prod (ys8sc…) | service | running:healthy → exited | infisical 200 (shares erp-data server) |
temporal-compose prod (qwo4s…) | service | degraded:unhealthy → exited | app 302; temporal.ordermatic.co 302 (GCP) |
internal (xwcgk…) | app | running:healthy → exited | team.ordermatic.co 302 (GCP path alive post-stop) |
webapp (prod, lsk8ocsc4okk0c8wsc404c48) — STOPPED ✅ (reversible), redirect confirmed
edge-served. The available CF API tokens lacked ruleset-read scope (verified active but
“Authentication error” on the http_request_dynamic_redirect ruleset + account Bulk-Redirect
endpoints), so the edge-vs-origin question was settled by a reversible empirical test: stopped
the origin, confirmed it reached exited:unhealthy, then app.erpunlocked.com still returned
301 → app.ordermatic.co (server: cloudflare) with the origin down — proving the redirect is
a Cloudflare edge rule, not origin-served. app.ordermatic.co stayed 302 (GCP) throughout.
Left stopped. Reverse: POST /api/v1/applications/lsk8ocsc4okk0c8wsc404c48/start.
Aside: the first attempt at this test aborted mid-script (
statusis a read-only var in zsh — use a different name); the stop had already gone through, so it was completed manually. No impact —app.erpunlocked.comnever broke because it is edge-served.
Untouched: infisical (KEEP — HTTP 200 throughout); already-exited dagster-agent,
dagster-code-server, staging-internal (delete-only later); the co-hosted non-ERP apps.
Live-status deltas vs the 2026-04-19 snapshot below:
- Already
exited(no stop needed — delete-only later):dagster-code-server+staging-internal(Applications, exited since 2024-11),dagster-agent(Service). pdf-prod-api,pdf-worker,staging-pdf-api,staging-pdf-worker— confirmed present and running (resolves the earlier HOLD: they exist; PDF now runs on the GCP Temporal worker). STOP candidates.temporal-compose(prod) isdegraded:unhealthy;internal+webapparerunning:healthy(last redeployed 2026-06-08).- ⚠️ Non-ERP apps are co-hosted on this Coolify (BooneTek sites, Invoice Trust, ERP SOFT TEST) — excluded from all actions.
Phase 2 — backups complete + server teardown plan (2026-06-09)
Section titled “Phase 2 — backups complete + server teardown plan (2026-06-09)”Postgres backups → R2 (done). Neither shared Postgres had any scheduled backup; both now captured
via pg_dumpall (all DBs + roles/globals, so the infisical role restores cleanly):
| Source (server) | Key DBs included | R2 object (s3://evl/coolify-pg-backups/2026-06-09/) |
|---|---|---|
dagster-postgres (erp-data) | infisical (secrets), dagster_storage, temporal*, multiwoven* | erp-data_dagster-postgres_all.sql.gz (143 MB) |
staging-shared-postgres (erp-failover) | flagsmith, dagster_storage, temporal* | erp-failover_staging-shared-postgres_all.sql.gz (100 MB) |
Restore: gunzip -c <file> | psql -U postgres. Follow-up: enable Coolify scheduled backups → R2
on dagster-postgres so the secrets DB stays protected ongoing (a one-off dump isn’t a substitute).
Hetzner ERP servers (hcloud API; token = 1Password “Hetzner Cloud”):
| erp-web | cpx21 | 11.99 | 115244148 | idle (webapp+internal stopped) | decommissioned (2026-06-09) | | erp-processing | cpx31 | 20.99 | 115244150 | idle (pdf+temporal stopped) | decommissioned (2026-06-09) | | erp-failover | cpx41 | 38.99 | 115244273 | Flagsmith retired (edge API cutover)| decommissioned (2026-07-04) | | erp-data | cpx41 | 38.99 | 115244149 | Infisical + shared PG (idle) | KEEP (minimal secrets box) |
Deleting erp-web, erp-processing, + erp-failover ≈ €72/mo savings (ENG-260). (Cool-2 + the Coolify master
host are separate / non-ERP.)
Teardown method — use terraform, not raw API. These are hcloud_server resources under
module.servers (infrastructure/terraform/environments/production). A raw hcloud-API delete drifts
state → a later terraform apply would recreate them. Clean path: remove the decommissioned servers
from the servers map (+ any LB/firewall/DNS refs) and apply, or terraform destroy -target='module.servers["erp-web"]' …,
after a plan review confirms only those servers are destroyed. ⚠️ On Hetzner only deleting stops
billing (power-off still bills) — irreversible, hence backups-first.
Infisical → GCP Secret Manager (deferred). The lever that eventually retires erp-data. SM =
fully-managed, native Cloud Run/IAM, no self-hosted secrets-backbone SPOF, kills the last Hetzner box —
but it’s a flat key→version store (no Infisical environments/folders/UI) and requires rewiring every
service’s infisical-entrypoint runtime-auth path (high blast radius; it’s the secrets path). Already
hybrid (Cloud Run uses SM secretKeyRef for some secrets). Recommendation: keep Infisical on erp-data
now; do the SM migration as its own project when “zero Hetzner” becomes the goal.
⛔ Hard safety rails
Section titled “⛔ Hard safety rails”- Infisical (
infisical.ordermatic.co, servicetwwkkcscgcsgwoo44koswccg,docker-compose.coolify.infisical.yml) — the secrets backbone. Every GCP service does Infisical universal-auth at container start. DO NOT STOP. Confirmed still served by the Coolifyerp-dataserver: terraform routesinfisical.ordermatic.co→ A record →module.servers["erp-data"].ipv4_address(main.tf:437), and a live probe returnedHTTP/2 200on 2026-06-09. - Out-of-scope upstream/test Services —
Odoo Test Instance,uptime-kuma-cool-head,Docmenso(Documenso),nocodb-evl. Not part of the ERP compute migration; not git-tracked compose. Leave alone in this pass. - Anything still serving live traffic or whose GCP replacement you cannot confirm healthy → HOLD and report, do not stop.
Correction folded in (2026-06-09):
app.erpunlocked.comis no longer a keep. It is now a Cloudflare edge301→app.ordermatic.co(GCP) — verified live (server:cloudflare, origin not contacted). The Hetzner/Coolifywebapporigin behind it takes zero prod traffic and is a STOP candidate, gated on the Cloudflare check in its row below. This supersedes the earlier “app.erpunlocked.com stays on Hetzner” rail from the 2026-05-02 cutover runbook.
Access checkpoint
Section titled “Access checkpoint”Both credential paths needed for live verification + execution are currently expired:
| Need | Tool | State (2026-06-09) | Used for |
|---|---|---|---|
| Coolify API | op → op://ERP-Unlocked/Coolify Master/api_token | ❌ op whoami → account is not signed in | GET /applications + /services for current status; POST .../stop to execute |
| GCP | gcloud (david@ordermatic.co, ordermatic-prod) | ❌ token expired — Reauthentication failed | Confirm Cloud Run + GCE VM replacements healthy before stopping their Coolify twins |
| Cloudflare | dashboard / API (token in op) | ⏳ needs check | Confirm app.erpunlocked.com 301 is an edge Redirect Rule / Bulk Redirect, not origin-served |
To unblock: re-auth op (refresh OP_SERVICE_ACCOUNT_TOKEN) and gcloud auth login. Then this
plan can be executed top-to-bottom. Everything below is derived from repo evidence + unauthenticated
DNS/HTTP/TLS probes — strong enough to classify, not enough to pull the trigger.
⚠️ Do not use the Coolify API token hard-coded in
infrastructure/coolify/scripts/*.py(see Security finding). Use the 1Password value via a re-authedop.
Live inventory
Section titled “Live inventory”Two sources, both dated:
- Coolify Services snapshot — 2026-04-19 (from
debt/projects/coolify-services-to-applications/README.md, a realGET /api/v1/servicescapture). Gives canonical UUID → name → server → compose-file. ⚠️ Predates the 2026-05-02 GCP cutover, so per-service running/exited status is stale — re-pull via API before acting. - Live DNS/HTTP/TLS probes — 2026-06-09 (unauthenticated; the cutover runbook’s own smoke-test method).
Coolify Services (multi-container compose stacks)
Section titled “Coolify Services (multi-container compose stacks)”| UUID | name | server | compose file | 2026-04-19 status |
|---|---|---|---|---|
twwkkcscgcsgwoo44koswccg | infisical | erp-data | docker-compose.coolify.infisical.yml | running |
qwo4ssosgcwwgk48k0g4w80s | temporal-compose (prod) | erp-processing | docker-compose.coolify.temporal.yml | running |
gcok8880csoo00skw80sc0gw | staging-temporal-compose | erp-failover | docker-compose.coolify.temporal.staging.yml | degraded:unhealthy |
ys8sccsc00oco488skskgkcs | dagster-compose (prod) | erp-data | docker-compose.coolify.staging.yml (sic) | running:healthy |
akcks4s880o88kksgkc8g8w0 | staging-dagster-compose | erp-failover | docker-compose.coolify.staging.yml | running:healthy |
k0w4gs00wgow4osw44os8kck | multiwoven | erp-data | docker-compose.coolify.multiwoven.yml | degraded:unhealthy |
qgkk4gw88cw8004skksg084s | dagster-agent | erp-data | (legacy Dagster+ agent, no repo file) | exited |
Coolify Applications (single-image apps)
Section titled “Coolify Applications (single-image apps)”| name | server | env | GHCR image | notes |
|---|---|---|---|---|
| webapp | erp-web | production | …/webapp | app.erpunlocked.com origin (now edge-redirected away) |
| staging-webapp | erp-failover | staging | …/webapp | staging.ordermatic.co origin (now GCP) |
| internal | erp-web | production | …/internal | team.ordermatic.co origin (now GCP tunnel) |
pdf-prod-api * | erp-processing | production | …/pdf-api | dropped from create-applications.sh; PDF moved to Temporal |
pdf-worker * | erp-processing | prod+staging | …/pdf-worker | dropped from create-applications.sh; PDF moved to Temporal |
flagsmith * | erp-data | production | upstream | feature flags — out of scope, verify separately |
* = existence on the live instance is unconfirmed (no longer referenced by any current
CI workflow; UUIDs not in the 2026-04-19 Services snapshot because they are Applications).
Re-pull GET /applications to confirm whether they still exist before stopping.
Origin / traffic map (terraform + live probe, 2026-06-09)
Section titled “Origin / traffic map (terraform + live probe, 2026-06-09)”| Hostname | Live origin | Evidence | Coolify twin |
|---|---|---|---|
app.ordermatic.co | GCP Cloud Run | CNAME ghs.googlehosted.com, server: Google Frontend, 302 /sign-in | webapp (prod) |
staging.ordermatic.co | GCP Cloud Run | CNAME ghs.googlehosted.com, server: Google Frontend, 302 /sign-in | staging-webapp |
app.erpunlocked.com | CF edge 301 → app.ordermatic.co | 301, location: https://app.ordermatic.co/, server: cloudflare, CF IPs, no origin headers | webapp (prod) |
dagster.ordermatic.co | GCP erp-prod-dagster VM (CF Tunnel) | main.tf:419 “Migrated from Hetzner … full GCP cutover” + CF Access gate live | dagster-compose |
temporal.ordermatic.co | GCP erp-prod-temporal VM (CF Tunnel) | main.tf:428 + CF Access gate live | temporal-compose |
team.ordermatic.co | GCP erp-prod-internal-edge → erp-internal-prod Cloud Run (CF Tunnel) | main.tf:450 + CF Access gate live | internal |
infisical.ordermatic.co | erp-data Coolify app (Hetzner) | main.tf:437 A→erp-data + HTTP/2 200 live | infisical (KEEP) |
CI deploy routing (is it still “used”?)
Section titled “CI deploy routing (is it still “used”?)”The strongest “still live” signal is whether a current workflow still deploys to the Coolify twin.
| Service | Current CI deploy target | Coolify deploy status |
|---|---|---|
| webapp | Manual-only, retired. deploy-webapp.yml header: “real webapp deploy path is now GCP Cloud Build → Cloud Run.” Push/PR triggers removed. | Coolify reachable only via workflow_dispatch; no auto deploy |
| dagster | GCP only. deploy-dagster.yml: Coolify deploy jobs removed (chore/retire-dagster-coolify-deploy) — “the Coolify webhook … returned 200 but nothing actually deployed.” | dead |
| temporal | GCP only. deploy-temporal.yml: Coolify deploy jobs removed 2026-05-26; Cloud Build → VMs is authoritative. | dead |
| internal | Coolify, prod — version-aware-build.yml still POSTs COOLIFY_PRODUCTION_WEBHOOK_INTERNAL on push to main. Best-effort (|| true). | still deploying, but traffic goes to GCP (team.ordermatic.co → tunnel → Cloud Run). Origin is idle. |
| pdf-api / pdf-worker | none — no current workflow references their webhooks | dead |
Classification
Section titled “Classification”Legend: KEEP = never touch · STOP = safe to stop this pass (after its gate clears) · HOLD = needs an extra check before it earns STOP.
| # | Service (UUID) | Class | GCP/edge replacement (live) | Pre-stop gate |
|---|---|---|---|---|
| — | infisical (twwkkcscgcsgwoo44koswccg) | KEEP | n/a — this is the backbone | — |
| 1 | dagster-agent (qgkk4gw88cw8004skksg084s) | STOP | n/a — legacy Dagster+ agent, retired (Dagster OSS on GCP VM) | Already exited as of 2026-04-19; confirm still exited, then stop/delete is a no-op safety |
| 2 | multiwoven (k0w4gs00wgow4osw44os8kck) | STOP | none — abandoned reverse-ETL experiment, absent from architecture | Confirm no consumer of multiwoven-* on erp-data; it was degraded:unhealthy anyway |
| 3 | staging-dagster-compose (akcks4s880o88kksgkc8g8w0) | STOP | GCP erp-staging-dagster VM | Confirm GCP staging dagster healthy (gcloud/IAP) |
| 4 | staging-temporal-compose (gcok8880csoo00skw80sc0gw) | STOP | GCP staging temporal VM | Confirm GCP staging temporal healthy; was already degraded |
| 5 | staging-webapp (Application) | STOP | GCP Cloud Run (staging.ordermatic.co live-confirmed) | Confirm no workflow_dispatch deploy in flight |
| 6 | dagster-compose / prod (ys8sccsc00oco488skskgkcs) | STOP | GCP erp-prod-dagster VM (dagster.ordermatic.co tunnel) | Confirm GCP prod dagster healthy; confirm nothing hits :3000 on erp-data |
| 7 | temporal-compose / prod (qwo4ssosgcwwgk48k0g4w80s) | STOP | GCP erp-prod-temporal VM (temporal.ordermatic.co tunnel) | Highest blast radius. Confirm GCP webapp/workers target the GCP temporal, and nothing hits erp-processing:7233. Do during a quiet window |
| 8 | internal (Application, prod) | STOP | GCP erp-prod-internal-edge → erp-internal-prod Cloud Run (team.ordermatic.co tunnel) | Confirm GCP internal edge + Cloud Run healthy (gcloud); note stale CI deploy (cleanup follow-up) |
| 9 | webapp (Application, prod) | STOP | CF edge 301 → app.ordermatic.co (GCP) | Cloudflare gate: confirm the 301 is an edge Redirect Rule / Bulk Redirect (NOT origin-served) and nothing hits the erpunlocked.com origin directly. The rule is not in terraform → it’s a manual dashboard rule; check dashboard/API |
| ? | pdf-prod-api (Application) | HOLD | Python apps/temporal-worker on GCP | Confirm the app still exists on the instance, then it’s a clean STOP |
| ? | pdf-worker (Application) | HOLD | Python apps/temporal-worker on GCP | Same as pdf-api |
| ? | flagsmith (Application) | HOLD | unknown — feature flags may still be live on Hetzner | Out of scope — do not stop without confirming flag service has a replacement |
Staged kill-list (execution order)
Section titled “Staged kill-list (execution order)”Lowest blast radius → highest. STOP only; reversible. Re-pull live status before each step. Run one at a time, observe, then proceed.
# Prereq (after re-auth):export PATH="$HOME/google-cloud-sdk/bin:$PATH"export COOLIFY_URL="https://cool.boonetek.com"export COOLIFY_TOKEN="$(op read 'op://ERP-Unlocked/Coolify Master/api_token')" # re-authed op
# Reversible STOP — Services use /services/{uuid}/stop, Applications use /applications/{uuid}/stop.stop_service() { curl -fsS -X POST -H "Authorization: Bearer $COOLIFY_TOKEN" "$COOLIFY_URL/api/v1/services/$1/stop"; echo; }stop_app() { curl -fsS -X POST -H "Authorization: Bearer $COOLIFY_TOKEN" "$COOLIFY_URL/api/v1/applications/$1/stop"; echo; }# UNDO (reverse a stop):# curl -fsS -X POST -H "Authorization: Bearer $COOLIFY_TOKEN" "$COOLIFY_URL/api/v1/services/$1/start"| Step | Action | Verify-before | Observe-after |
|---|---|---|---|
| 1 | stop_service qgkk4gw88cw8004skksg084s (dagster-agent) | already exited | n/a |
| 2 | stop_service k0w4gs00wgow4osw44os8kck (multiwoven) | no consumers | erp-data still healthy; infisical untouched |
| 3 | stop_service akcks4s880o88kksgkc8g8w0 (staging-dagster) | GCP staging dagster healthy | staging Dagster UI still served via GCP |
| 4 | stop_service gcok8880csoo00skw80sc0gw (staging-temporal) | GCP staging temporal healthy | staging PDF/extract still flows on GCP |
| 5 | stop_app <staging-webapp uuid> | staging.ordermatic.co 200 from GCP | re-curl staging.ordermatic.co |
| 6 | stop_service ys8sccsc00oco488skskgkcs (dagster prod) | GCP prod dagster healthy | dagster.ordermatic.co still loads (GCP) |
| 7 | stop_service qwo4ssosgcwwgk48k0g4w80s (temporal prod) | GCP prod temporal healthy; quiet window | prod order extraction unaffected; temporal.ordermatic.co loads (GCP) |
| 8 | stop_app <internal uuid> | GCP internal-edge + Cloud Run healthy | team.ordermatic.co still loads (GCP) |
| 9 | stop_app <webapp uuid> | Cloudflare redirect-rule confirmed edge-served | app.erpunlocked.com still 301s to app.ordermatic.co with origin down |
UUIDs for the Applications (steps 5, 8, 9) are not in the 2026-04-19 snapshot (that captured
Services only). Resolve them from GET /applications (name → uuid) at execution time.
Security finding
Section titled “Security finding”infrastructure/coolify/scripts/*.py (15 files: fetch-apps.py, start-proxy.py,
delete-stale-envs.py, redeploy-db.py, restart-db.py, trigger-webapp.py,
revert-public-db.py, check-envs.py, delete-proxy.py, get-projects.py, check-db.py,
start-db.py, fix-temporal-redeploy.py, set-db-public.py, fix-temporal-port.py) embed a
live Coolify API token as a hard-coded default (token = os.environ.get("COOLIFY_TOKEN", "<redacted>")).
It is committed and in git history (last touched in d3b1fbd2d).
Action (separate security PR): rotate the Coolify API token, replace each default with a
required-env lookup (the shell scripts already do op read … || ''), and scrub history if the
token cannot be rotated promptly. Do not reuse this token for the decom — use the 1Password value.
Follow-ups (NOT this pass)
Section titled “Follow-ups (NOT this pass)”Only after the corresponding service is stopped and observed stable for a soak period:
- Delete stopped Coolify resources —
DELETE /services|applications/{uuid}with?delete_volumes=false(belt-and-suspenders; the compose stacks declare no local volumes except infisical/multiwoven Redis caches — and infisical is KEEP). - Hetzner + Neon teardown — power down
erp-processing,erp-failover, and (last, carefully) the non-infisical role oferp-data; decommission the Neon instance. ~$344/mo lever.erp-datahosts Infisical — do not destroy that server. - CI / secrets cleanup PR — remove the
internalCoolify deploy step fromversion-aware-build.yml; delete now-dead GH secrets (COOLIFY_PRODUCTION_WEBHOOK_INTERNAL, any residualCOOLIFY_*_WEBHOOK_*/_UUID); keepCOOLIFY_TOKEN/COOLIFY_URLonly if still used (webappworkflow_dispatch, infisical tooling). - Compose-file removal PR — delete
docker-compose.coolify.temporal*.yml,docker-compose.coolify.staging.yml,docker-compose.coolify.multiwoven.ymlonce their Services are deleted. Keepdocker-compose.coolify.infisical.yml. - Terraform reconciliation —
app.erpunlocked.comis served by a manual CF Redirect Rule not tracked ininfrastructure/terraform; theapprecord block (main.tf:397) describes GCP but the erpunlocked apex redirect is dashboard-managed. Codify it (or document the exception) so a futureapplycan’t clobber the redirect.
Appendix — evidence index
Section titled “Appendix — evidence index”infrastructure-gcp/MIGRATION_STATUS.md— two-phase migration; “decommission Hetzner after GCP validated.”infrastructure-gcp/CUTOVER_RUNBOOK_2026_05_02.md— “After 1 week stable: schedule Hetzner Coolify shutdown.”debt/projects/coolify-services-to-applications/README.md— 2026-04-19GET /servicessnapshot (UUIDs).debt/backlog/pdf-worker-and-dagster-cost-optimization.md— PDF → Temporal migration; Dagster OSS on compose..github/workflows/deploy-dagster.yml,deploy-temporal.yml,deploy-webapp.yml,version-aware-build.yml— CI routing.infrastructure/terraform/environments/production/main.tf(DNS records block) — live origin per hostname.- Live probes (2026-06-09):
curl -sI+digagainst the seven hostnames above.