Independent Per-App Releases

May contain stale references

This document was audited on 2026-06-22 and contains references that need updating: marketing is not a Cloudflare Worker; only three CF workers exist. Cross-check critical details against the current codebase before relying on them.

Status: Draft v2 — revised after full-team review (eng, devex, CEO, architect) Author: David (with Claude) Date: 2026-06-08 Reviews: §12 records the review consensus and how each blocker was dispositioned.

---

0. TL;DR (what changed in v2)

Review surfaced three blockers that reshape the plan:

1. webapp is not digest-promotable. Vite inlines PUBLIC_* (Clerk publishable key, PostHog, APP_ENV) into the client bundle at build time. Promoting a staging image digest to prod would ship staging’s Clerk key to prod browsers — a silent auth misroute. internal is clean (no PUBLIC_*). → The two apps need different promotion mechanics.
2. Migrations are welded inside the monolith. cloudbuild.yaml’s migrate-db step is gated on _BUILD_WEBAPP and blocks deploy-webapp. Decomposing the monolith silently stops migrations from running on the normal path. → A standalone migrate job is a prerequisite, not a nicety.
3. Fast-forward staging→main does not fire push: workflows. The promotion FF-push uses the Actions GITHUB_TOKEN, which GitHub deliberately does not let re-trigger workflows. A naive “deploy on push to main” prod path is dead on arrival. → Promotion must be workflow_dispatch or tag-push driven.

The result: split the work into Decision A (do now) and Decision B (post-POC). Decision A = give internal its own deploy path + a standalone migrate job + a written migration rule (~½ day CC). This kills the blast radius that bit us today. Decision B = the full per-app promotion platform (webapp, VM apps, promotion model, GitOps) — revisit after the Smith Corona / Cleveland Kitchen POCs stabilize.

1. Motivation

On 2026-06-08 a one-line ESM-compat fix in packages/db/src/client.ts (PR #1535) was merged to staging. That single shared-package change triggered a monolithic Cloud Build that rebuilt and redeployed all five GCP services — webapp, internal, dagster, temporal-worker, ts-temporal-worker — because packages/db fans out to nearly every app and the Python apps rebuild on any ^packages/ path. A team.ordermatic.co (internal) hotfix dragged the entire fleet through a ~30-min build, and coupled an internal-tool fix to a redeploy of customer-facing webapp — during active POCs.

Two distinct wins are on the table, and review showed they have very different cost/risk:

• Kill the blast radius (an internal hotfix ships without rebuilding the fleet). Cheap, safe, do now.
• Independent prod promotion (hold internal@0.2.13 back while moving webapp). More machinery; not yet a felt pain. Defer.

We also currently build every GCP app twice: the per-app deploy-<app>.yml builds an image to GHCR, then cloudbuild.yaml rebuilds the same app from source and discards the GHCR image. Decomposition ends that waste.

2. Goals / Non-Goals

Goals

• Independent build per app — only the changed app (and its true dependents) build.
• Independent deploy per app to staging and production.
• Keep the shared-Postgres migrations running and ordered ahead of code at all times.
• Preserve fast rollback per app and a legible “what’s in prod” surface.
• Keep a working escape hatch to rebuild/redeploy everything.

Goals (Decision B / later)

• Independent promotion — “promote internal@0.2.13 to prod” without touching other apps.
• One ship idiom for all apps (Cloudflare Workers + GCP) instead of today’s two.

Non-Goals

• Splitting the monorepo into polyrepos. One repo, one PR flow.
• Per-app databases. One shared Postgres (see §5 — the real constraint).
• Changing the feature → staging → main code-integration model (only the deploy/promote semantics on top).
• Splitting the shared temporal VM (see §6.6) — explicitly out of scope for now.

3. Current State (grounded)

Three overlapping release layers, mid-migration:

Layer	Files	Reality
1. Monolith (deploys GCP today)	gcp-deploy.yml → cloudbuild.yaml (44 KB)	One Cloud Build invocation, ~20 steps each self-skipping on coarse _BUILD_X file-diff flags. One trigger, one COMMIT_SHA, one timeout (--timeout=5400s on the submit step — the in-file timeout: is ignored), one concurrency lane for all 5 apps. migrate-db lives here, gated on _BUILD_WEBAPP, and blocks deploy-webapp.
2. Version-aware (Coolify-era)	version-aware-build.yml + detect-versions.yml	Builds images to GHCR, then fires Coolify webhooks that are no-ops for GCP (failures swallowed with || true). Largely vestigial. detect-versions.yml is good and reused below.
3. Per-app (target)	deploy-<app>.yml × 6	Cloudflare Worker apps fully independent; GCP apps build independently → GHCR but deploy is delegated back to the cloudbuild monolith.

Already independent

Four Cloudflare Worker apps — ops-mcp, email-worker, tidycal-hubspot-sync, marketing — ship today via wrangler deploy --env {staging|production}, path-filtered triggers, per-app change detection. deploy-ops-mcp.yml is the full reference (build → staging → prod-dispatch) the GCP apps should converge on; it already documents and works around the GITHUB_TOKEN FF-trigger constraint (see §6.4).

Still coupled (the 5 GCP apps)

• internal (team.ordermatic.co, Cloud Run) — no per-app deploy workflow at all; only the monolith.
• deploy-webapp.yml is RETIRED/workflow_dispatch-only — and it is a Coolify/Hetzner deploy (app.erpunlocked.com) carrying its own migrate-staging/migrate-production jobs. The live GCP webapp (staging.ordermatic.co, app.ordermatic.co) deploys via the monolith → Cloud Run.
• deploy-dagster.yml, deploy-temporal.yml achieve build isolation (own GHCR image) but deploy is still the monolith. Their deploy is IAP-SSH to a shared VM (docker pull + restart), not Cloud Run — and ts-temporal-worker waitFors temporal-worker because both SSH the same VM and race on a shared SSH key. No Cloud Run revisions → different rollback.

The three coupling points

• A. Single deploy transaction. All 5 GCP apps build+deploy in one Cloud Build run. Skip-flags reduce work, not coupling: a broken sibling step, the shared timeout, or a concurrency cancel still blocks an unrelated hotfix.
• B. Gating is file-diff, not version. What deploys is git diff vs the last successful build SHA — not “did this app’s version bump / is this version approved for prod.”
• C. Branch-as-release-pointer. promotion-guard.yml enforces staging → main as the only promotion path, moving the entire tree. No per-app prod ref.

4. The version-pinning question

“Can’t we pin to specific package versions to avoid things?”

Pinning internal workspace packages (consuming @repo/db@2.4.2 instead of workspace:*):

• ✅ Decouples build-time and makes change-detection precise (explicit dep graph vs coarse ^packages/ grep).
• ✅ Fully decouples stateless shared packages (@repo/ui, @repo/utils, @repo/ai).
• ❌ Does NOT decouple shared state. Every app hits the same physical Postgres. If @repo/db@2.5.0 ships a DROP COLUMN, an app pinned to 2.4.2 still breaks against the migrated DB. One schema, not one-per-version. Same for shared Typesense and queues.
• ⚠️ Carrying cost. Re-introduces in-repo diamond dependencies, version bookkeeping, and lets apps run stale shared code.

Conclusion: pinning helps build-graph precision but is not a substitute for expand/contract migration discipline on the shared DB. Don’t pin @repo/db as the primary lever. Solve deploy-independence at the artifact/deploy layer and runtime-coupling with migration discipline (§5). Pinning stays optional.

5. The real constraint: shared database & migration discipline

Independent deploy is plumbing. Independent correctness is the engineering cost.

Today the monolith accidentally hides DB coupling by redeploying every consumer together. The moment internal deploys independently, two code versions run against one schema, so every packages/db migration must be backward/forward compatible — expand/contract:

1. Expand: additive migration (new nullable column / table). Deploy. Old + new code both work.
2. Migrate: backfill + dual-write while both versions run.
3. Contract: ship the destructive migration only after all consumers are on the new version.

⚠️ This is a latent bug today, not just a future risk. internal and ts-temporal-worker both consume @repo/db at runtime, but migrations only run when webapp builds (the _BUILD_WEBAPP gate). An internal-only change that needs a migration today has no path to run it.

Controls (graduated, see §12 for the block-vs-warn debate)

• Standalone migrate job (see §6.3) — the structural home for “migrations deploy ahead of code.”
• Destructive-DDL lint targeting the generated SQL in packages/db/src/migrations/**.sql (NOT schema/** — the DROP lives in the emitted SQL). Flags DROP, ALTER … DROP, SET NOT NULL, ADD COLUMN … NOT NULL without DEFAULT, type-narrowing.
• The real guarantee (Decision B): a Neon-branch compatibility test — apply the new migration to an ephemeral Neon branch, run the previously released image’s DB paths against it. Green = old code + new schema verified. A label only proves “someone thought about it”; this proves the invariant.
• PR prompt, not shared checklist: the lint posts the expand/contract checklist as a PR comment only when a migrations/** change is detected (a soft checkbox in the shared template gets rubber-stamped).
• Also fail on orphan/scaffold schema/*.ts files (drizzle-kit loads all of them — the PR #1430 drift bug).

6. Proposed Design

6.1 Artifact model — with the digest caveat

Backbone: each app produces an immutable image tagged by version + digest, built once, promoted by digest (@sha256:…), never rebuilt per environment. The deploy job must assert the reference is a digest and fail if handed a mutable tag; :latest-<branch> exists for humans only, never as a deploy input.

Caveat — build-time client env breaks cross-env digest promotion:

• internal and the 4 Workers have no PUBLIC_* → genuinely digest-promotable staging→prod. ✅
• webapp inlines PUBLIC_* into the browser bundle at build time (Clerk publishable key, PostHog, APP_ENV). The staging digest carries staging secrets; promoting it to prod is a silent misconfiguration. Two options for webapp (Decision B):
  • (Recommended) Move PUBLIC_* to runtime SSR config — render window.__ENV__ / /config.json from runtime env at request time (Infisical already provides per-env values at container start). Then the same digest runs in both envs and the backbone holds for webapp too.
  • (Fallback) Version-from-SHA promotion — build a prod image from the same git SHA with prod build-args; promote by version, not digest. Keeps source-immutability, loses artifact-immutability.

6.2 Build (per app)

Extend the deploy-<app>.yml pattern to the GCP apps (deploy-dagster.yml is the build template; deploy-ops-mcp.yml the full build→promote reference). Each:

• triggers on push to staging with paths: filter (app dir + its true shared deps),
• builds + pushes the image to GHCR tagged :<version>-<sha> (version from detect-versions.yml),
• is gated by the per-app *-changed output from detect-versions.yml (version-aware, not raw diff),
• reproduces the cloudbuild node-build bundle contract as an acceptance criterion: Infisical entrypoint, otel-init.cjs, dist/, prod-pruned node_modules, and (webapp) the PUBLIC_* build-args. This bundle contract broke prod twice in 2026-05; it must not regress.
• make the GHCR image the real artifact (ends the double-build).

6.3 Deploy + the standalone migration job

Migrations first. Add a dedicated migrate-db.yml (or a reusable job every deploy-<app>.yml needs:): runs pnpm --filter @repo/db db:migrate once per env, triggered when packages/db/src/migrations/** changed, before any app deploy in that push, idempotent (drizzle’s __drizzle_migrations ledger), and a hard gate — failure blocks dependent app deploys. This re-homes the logic currently trapped in cloudbuild.yaml’s _BUILD_WEBAPP-gated migrate-db.

Cloud Run apps (internal, webapp): move gcloud run deploy out of cloudbuild.yaml into each deploy-<app>.yml, deploying the GHCR digest to the -staging service.

VM apps (dagster, temporal-worker, ts-temporal-worker): “deploy” is IAP-SSH docker pull + restart to a shared VM, serialized (shared SSH key). They are not fully independent of each other until the VM is split (§6.6, out of scope). Preserve the single-key prep + image-prune logic.

Demote cloudbuild.yaml to a manual workflow_dispatch “rebuild-all” escape hatch. The escape hatch and per-app builds must share the same build-arg/bundle script (extract it) so a webapp built by either path is bit-identical — otherwise the §6.1 digest caveat reappears intermittently.

6.4 Promotion — triggers and the GITHUB_TOKEN constraint

Hard constraint: the staging→main fast-forward is pushed with the Actions GITHUB_TOKEN, which does not re-trigger push: workflows (GitHub’s recursion guard). So a prod deploy cannot be driven by push: main — it must be workflow_dispatch or tag-push (PAT/App). deploy-ops-mcp.yml already documents and works around exactly this.

Options (Decision B):

• P1 — workflow_dispatch promote (the proven ops-mcp pattern). Re-point the prod service at the staging-tested digest via a manual dispatch (--ref main). Lowest ceremony; immediate; the durable break-glass.
• P2 — release tags. Push release/<app>/<version> → deploys that app only. Immune to the FF-token trap, git-native audit trail, maps to changesets versions. The durable primary once independence matters.
• P3 — GitOps manifest. deploy/{staging,production}.lock.json pins each service → digest; a reconcile workflow (dispatch/tag-triggered, never push:main-token) deploys only changed pins. Heaviest; also the natural home for the consumer-version registry the contract-migration gate needs (§8). Defer.

Recommendation: P1 now → P2 when independent promotion is actually exercised; defer P3. Keep main as the approved-integration ref; a workflow_dispatch/tag is the release. Eventually retrofit the Worker apps onto the same idiom so all 10 apps ship the same way.

6.5 Rollback & observability

• Cloud Run rollback = re-point the service at the previous digest/revision (one command).
• VM rollback is different: re-pull the previous image tag and restart over SSH — no revision history, no traffic split. Write this as an explicit runbook line.
• “What’s in prod” surface — pulled into Decision A: a prod-status.yml workflow_dispatch that reads each Cloud Run service’s live image digest (gcloud run services describe … --format=…) and the Workers’ deployed versions into one step-summary table. ~30 lines; the thing on-call reaches for at 2am. P3 can supersede it later.
• Failure legibility: each deploy-<app>.yml runs a pre-deploy env-var preflight (reuse the pre_flight_check() built after the 2026-02-09 deployment-order incident) and a post-deploy canary that prints one line: app=internal env=prod version=0.2.13 digest=sha256:… health=PASS/FAIL. On FAIL, auto re-point to the prior digest and say which.

6.6 Out of scope (named, not solved)

• Splitting the shared temporal VM so temporal-worker / ts-temporal-worker deploy truly independently. Today they serialize on one box; that coupling survives this plan by design.
• app.erpunlocked.com is already a Cloudflare edge 301 redirect → app.ordermatic.co (GCP) — verified live 2026-06-09 (server: cloudflare, origin not contacted). The Hetzner/Coolify webapp origin behind it takes zero prod traffic and is being retired in the separate Coolify-decommission thread. So the new deploy-webapp.yml (B1) targets Cloud Run only — there is no live Hetzner webapp path to preserve. (The retired deploy-webapp.yml’s Coolify/Hetzner target is dead, not a cutover to plan.)

7. Rollout Plan

Decision A — do now (~½–1 day CC). Kills the blast radius.

Step	Scope
A0	Land the standalone migrate-db job (decoupled from any app build) + the written “migrations deploy ahead, expand/contract only” rule in the PR template. Prerequisite to A1.
A1	Give internal its own deploy-internal.yml: paths-filtered build → GHCR digest → gcloud run deploy to the staging Cloud Run service. internal is digest-clean and internal-only (lowest blast radius), and is the app that triggered this.
A2	Demote cloudbuild.yaml to a workflow_dispatch rebuild-all escape hatch (for everything still on it). Add the minimal prod-status.yml.

Decision B — post-POC (revisit when independent promotion is a felt need)

Phase	Scope
B1	Extend the pattern to webapp — including the PUBLIC_* runtime-config refactor (§6.1) and the Coolify→Cloud Run cutover (§6.6). Highest blast radius; do when POCs are calm.
B2	VM apps (dagster, temporal stack) per-app deploy; version-aware detection for the Python graph (kills the coarse ^packages/ rebuild).
B3	Per-app prod promotion (P1 → P2); destructive-DDL lint to blocking + Neon-branch compat test; consumer-version registry.
B4 (optional)	P3 GitOps manifest + drift detection; unify Workers + GCP onto one ship idiom.

8. Risks & Mitigations

Risk	Mitigation
Decomposing the monolith silently stops migrations	A0 standalone migrate job lands before A1; required gate on every deploy.
webapp digest carries staging Clerk/PostHog into prod	§6.1 — webapp is not digest-promoted until PUBLIC_* is runtime; until then exempt to version-from-SHA.
Prod deploy never fires (FF-token)	§6.4 — promotion is workflow_dispatch/tag, never push:main-token.
Destructive migration breaks an un-redeployed consumer	Expand/contract + lint on migrations/**.sql; Neon-branch compat test (B3).
Version skew across promotion — internal@new + webapp@old on one schema	The contract step must verify all consumers past version V; consumer-version registry (B3/P3) computes that set.
VM rollback assumed Cloud-Run-shaped	§6.5 — explicit VM re-pull-prior-tag runbook.
Escape-hatch vs per-app build diverge → intermittent webapp bundle drift	§6.3 — both paths call one shared build-arg/bundle script.
Promotion sprawl (5 ways to ship)	One mechanism (P1→P2); unify Workers + GCP (B4).
Bundle contract regression (Infisical entrypoint) → prod 500	§6.2 acceptance criterion; it broke prod twice in 2026-05.

9. Alternatives Considered

• Keep the monolith, improve skip-flags. Partially addresses B, not A or C; promotion stays whole-tree.
• Pin all internal packages (the §4 question as primary lever). Build-time win only; ignores shared-DB; adds in-repo diamond cost.
• Polyrepos. Throws away atomic-refactor benefits for a deploy-layer problem.
• Jump straight to P3 GitOps. Heaviest concept to onboard; its reconcile trigger hits the same FF-token wall; over-built for a 10-app fleet + tiny team.

10. Open Questions (post-review)

1. Decision A scope: confirm internal-only first (recommended), or include webapp now despite the digest + Coolify-cutover risk?
2. Migration lint: block destructive DDL from day one, or warn-first? (Eng/architect/devex lean block-with-auditable-override; CEO leans warn-first to protect hotfix speed.)
3. webapp PUBLIC_*: commit to the runtime-config refactor (unlocks true digest promotion) in B1, or accept version-from-SHA for webapp indefinitely?
4. Promotion default: ratify P1 now → P2 later; drop P3 unless a specific pain demands it.

11. The one-sentence ship instruction (answer to old OQ#5)

To be written into docs/branching.md as part of A1 (staging) and B3 (prod). Target wording once promotion lands (P2): _“To ship app X to prod: land it on staging and confirm it’s healthy, then git tag release/X/<version> && git push origin release/X/<version> — that promotes the exact staging-tested image (digest for internal/Workers; prod-built SHA for webapp until its PUBLIC_* is runtime) to prod for app X only; nothing else moves. Rollback: re-tag the prior version. Break-glass full rebuild: run the cloudbuild rebuild-all workflow.”*

12. Review Consensus (full team)

Reviewed by eng-manager, devex, CEO/strategy, and architect lenses (codex external review attempted but the local CLI’s account rejected available models — not run). Strong convergence:

Finding	Raised by	Disposition in v2
webapp not digest-promotable (PUBLIC_* inlined)	architect (BLOCKER)	§6.1 caveat; webapp → Decision B with runtime-config refactor.
Migrations welded in monolith; decomposition stops them	eng, architect (BLOCKER)	A0 standalone migrate job, prerequisite to A1.
FF-main push (GITHUB_TOKEN) doesn’t trigger workflows	devex, architect (BLOCKER)	§6.4 — promotion via dispatch/tag only.
Do the 80/20: internal-only first, defer the rest	CEO (verdict)	Restructured into Decision A vs B.
Phase 0 must precede Phase 1	eng, devex, CEO	A0 is an explicit prerequisite.
**Destructive-DDL lint scoped to schema/** (wrong)**	eng, architect	§5 — retargeted to migrations/**.sql; Neon compat test as the real guarantee.
VM apps ≠ Cloud Run (deploy + rollback)	eng, architect	§6.3/§6.5 split; §6.6 names the shared-VM coupling.
deploy-webapp.yml is a Coolify/Hetzner cutover, not a rebuild	eng	§3 + §6.6 name it explicitly.
Promotion model → P1 now / P2 later, drop P3	devex, architect, CEO	§6.4 recommendation.
Pull “what’s in prod” + preflight forward	devex	§6.5 into Decision A.
Bundle-contract regression risk (Infisical entrypoint)	architect	§6.2 acceptance criterion.
Answer “how do I ship X” in docs/branching.md	devex	§11.

One thing the team affirmed: the §4 conclusion that pinning @repo/db is not a substitute for expand/contract is the correct, non-obvious call — keep it central.