Skip to content
Ordermatic Docs

Build Pipeline Runbook

1. Architecture diagram (ASCII)

Section titled “1. Architecture diagram (ASCII)”
git push → trigger filter → cloudbuild.yaml | cloudbuild-node.yaml | cloudbuild-workers.yaml → node-builder (Node only) → webapp/internal build → push → Cloud Run / GCE VM deploy

How the cloudbuild*.yaml configs relate

Section titled “How the cloudbuild*.yaml configs relate”
Config file Invoked by Scope
cloudbuild.yaml .github/workflows/gcp-deploy.yml on every push to staging/main Primary end-to-end orchestrator: change-detection, migrate-db, node-builder, webapp, internal, temporal-worker, ts-temporal-worker, dagster, and chatwoot builds + Cloud Run / GCE IAP SSH deployments.
cloudbuild-node.yaml GCP Cloud Build path triggers (erp-webapp-*, erp-internal-*) Builds and deploys a single Node/Astro Cloud Run service (_SERVICE=webapp or internal) using the shared node-builder image and BuildKit secret mounts.
cloudbuild-workers.yaml GCP Cloud Build path triggers (erp-workers-*, erp-dagster-*) Builds and pushes Python/worker images (temporal-worker, dagster) and rolls them out to GCE VMs.
cloudbuild-all.yaml Manual trigger (erp-all-staging-manual / CLI) Full rebuild of node-builder, webapp, internal, temporal-worker, and dagster without path-skipping.

2. Trigger routing

Section titled “2. Trigger routing”
File path pattern Trigger file Cloud Build config
apps/webapp/** erp-webapp-staging cloudbuild-node.yaml
apps/internal/** erp-internal-staging cloudbuild-node.yaml
apps/temporal-worker/** erp-workers-staging cloudbuild-workers.yaml
apps/dagster/** erp-dagster-staging cloudbuild-workers.yaml
images/node-builder/** erp-webapp-staging cloudbuild-node.yaml
cloudbuild-*.yaml erp-all-staging-manual cloudbuild-all.yaml

3. Cache invalidation rules

Section titled “3. Cache invalidation rules”
  • pnpm-lock.yaml, pnpm-workspace.yaml, package.json, any workspace package.json → node-builder install layer invalidated
  • apps/webapp/** (excluding package.json) → webapp build layer invalidated, but install layer cached
  • apps/internal/** → internal build invalidated, install cached
  • pyproject.toml, uv.lock, apps/temporal-worker/pyproject.toml, apps/dagster/pyproject.toml → respective Python uv sync invalidated
  • Python source changes → only final source-copy layer invalidates, uv sync cached

4. Expected build times

Section titled “4. Expected build times”
Service Cold build Warm build Notes
Node Builder 8-12 min 2-4 min pnpm install + Docker build
Webapp (cold) 12-18 min 4-8 min Includes node-builder if cache miss
Webapp (warm) 4-6 min 2-3 min Uses cached node-builder
Internal (cold) 10-15 min 3-6 min Includes node-builder if cache miss
Internal (warm) 3-5 min 2-3 min Uses cached node-builder
Temporal Worker 6-10 min 2-4 min uv sync + Docker build
Dagster 8-12 min 3-6 min uv sync + Docker build

5. How to debug a slow build

Section titled “5. How to debug a slow build”
  • Look at ::cache target: echo lines for which cache tag was tried
  • Look for CACHED markers in BuildKit output (--progress=plain makes them visible)
  • Check Artifact Registry for whether the expected cache tag exists
  • Common causes: lockfile changed (cold install), Artifact Registry GC removed the tag, content hash differs from expected

6. How to trigger a full manual rebuild

Section titled “6. How to trigger a full manual rebuild”
  • CLI: gcloud builds submit --config=cloudbuild-all.yaml --region=us-east1 --substitutions=BRANCH_NAME=staging,COMMIT_SHA=$(git rev-parse HEAD) .
  • Console: Cloud Build → Triggers → erp-all-staging-manual → Run

The cloudbuild*.yaml configs pin serviceAccount: cloud-build-runner@.... A build that does NOT name that SA (e.g. a bare --tag build) runs as the default compute SA, which holds no project bindings and 403s reading its own source tarball. See infrastructure/chatwoot/GCP-DEPLOY.md for the --tag invocation.

7. How to add a new service to the pipeline

Section titled “7. How to add a new service to the pipeline”

Step-by-step:

  1. Create apps/<new>/Dockerfile following the multi-stage pattern (consume node-builder if Node, follow temporal-worker pattern if Python)
  2. Add the package.json to pnpm-workspace.yaml (Node) or to root [tool.uv.workspace] (Python)
  3. Run pnpm generate:dockerfile-copies to update images/node-builder/Dockerfile if Node
  4. Decide which cloudbuild-*.yaml it joins (cloudbuild-node.yaml if Node + Cloud Run deploy; cloudbuild-workers.yaml if Python + VM deploy; new file if neither)
  5. Add a new module instance in infrastructure-gcp/terraform/environments/{staging,production}/main.tf with included_files and substitutions
  6. terraform plan && terraform apply
  7. Push to test

8. Failure modes table

Section titled “8. Failure modes table”
Symptom Cause Fix
image not found: node-builder:${COMMIT_SHA} in webapp build node-builder push failed check push-node-builder step logs
cp: cannot stat 'apps/internal/otel-init.cjs' HAS_OTEL not set to false for internal trigger check trigger substitutions
uv sync re-runs despite no Python deps change Wrong Dockerfile layer order; source COPYed before uv sync follow apps/temporal-worker/Dockerfile pattern
pnpm install re-runs despite no lockfile change Workspace package.json was edited; the COPY block invalidated expected; check images/node-builder/Dockerfile
Build fails with “Authentication required” TRIGGER_* secret missing in GCP Secret Manager create via gcloud secrets create erp-prod-trigger-*

9. BuildKit secret pattern for non-PUBLIC envs

Section titled “9. BuildKit secret pattern for non-PUBLIC envs”

When adding a new build-time env that’s NOT a PUBLIC_* Astro var:

  1. Create the secret in GCP Secret Manager: gcloud secrets create erp-prod-<name>
  2. Add to cloudbuild-node.yaml availableSecrets → secretEnv
  3. In the build step, write to a file in /workspace/_secret_<name>, then pass to docker via --secret id=<name>,src=...
  4. In Dockerfile, consume via RUN --mount=type=secret,id=<name> ...
  5. Explicitly enable BuildKit on the Cloud Build step by setting env: ['DOCKER_BUILDKIT=1'] (as cloudbuild-node.yaml does). Cloud Build’s classic builder does not enable BuildKit by default, so any step using RUN --mount=type=secret (or COPY --chmod=) without DOCKER_BUILDKIT=1 will fail to parse the Dockerfile.
  6. NEVER use --build-arg for non-PUBLIC values — they leak to image history

10. Local development

Section titled “10. Local development”
  • Test a Dockerfile change locally: ./scripts/test-cloudbuild-local.sh <target>
  • Regenerate node-builder COPY block: pnpm generate:dockerfile-copies
  • Verify drift: pnpm check:dockerfile-copies