Build Pipeline Runbook

1. Architecture diagram (ASCII)

git push → trigger filter → cloudbuild-node.yaml | cloudbuild-workers.yaml → node-builder (Node only) → webapp/internal build → push → Cloud Run deploy

2. Trigger routing

File path pattern	Trigger file	Cloud Build config
apps/webapp/**	erp-webapp-staging	cloudbuild-node.yaml
apps/internal/**	erp-internal-staging	cloudbuild-node.yaml
apps/temporal-worker/**	erp-workers-staging	cloudbuild-workers.yaml
apps/dagster/**	erp-dagster-staging	cloudbuild-workers.yaml
images/node-builder/**	erp-webapp-staging	cloudbuild-node.yaml
cloudbuild-*.yaml	erp-all-staging-manual	cloudbuild-all.yaml

3. Cache invalidation rules

• pnpm-lock.yaml, pnpm-workspace.yaml, package.json, any workspace package.json → node-builder install layer invalidated
• apps/webapp/** (excluding package.json) → webapp build layer invalidated, but install layer cached
• apps/internal/** → internal build invalidated, install cached
• pyproject.toml, uv.lock, apps/temporal-worker/pyproject.toml, apps/dagster/pyproject.toml → respective Python uv sync invalidated
• Python source changes → only final source-copy layer invalidates, uv sync cached

4. Expected build times

Service	Cold build	Warm build	Notes
Node Builder	8-12 min	2-4 min	pnpm install + Docker build
Webapp (cold)	12-18 min	4-8 min	Includes node-builder if cache miss
Webapp (warm)	4-6 min	2-3 min	Uses cached node-builder
Internal (cold)	10-15 min	3-6 min	Includes node-builder if cache miss
Internal (warm)	3-5 min	2-3 min	Uses cached node-builder
Temporal Worker	6-10 min	2-4 min	uv sync + Docker build
Dagster	8-12 min	3-6 min	uv sync + Docker build

5. How to debug a slow build

• Look at ::cache target: echo lines for which cache tag was tried
• Look for CACHED markers in BuildKit output (--progress=plain makes them visible)
• Check Artifact Registry for whether the expected cache tag exists
• Common causes: lockfile changed (cold install), Artifact Registry GC removed the tag, content hash differs from expected

6. How to trigger a full manual rebuild

• CLI: gcloud builds submit --config=cloudbuild-all.yaml --region=us-east1 --substitutions=BRANCH_NAME=staging,COMMIT_SHA=$(git rev-parse HEAD) .
• Console: Cloud Build → Triggers → erp-all-staging-manual → Run

The cloudbuild*.yaml configs pin serviceAccount: cloud-build-runner@.... A build that does NOT name that SA (e.g. a bare --tag build) runs as the default compute SA, which holds no project bindings and 403s reading its own source tarball. See infrastructure/chatwoot/GCP-DEPLOY.md for the --tag invocation.

7. How to add a new service to the pipeline

Step-by-step:

1. Create apps/<new>/Dockerfile following the multi-stage pattern (consume node-builder if Node, follow temporal-worker pattern if Python)
2. Add the package.json to pnpm-workspace.yaml (Node) or to root [tool.uv.workspace] (Python)
3. Run pnpm generate:dockerfile-copies to update images/node-builder/Dockerfile if Node
4. Decide which cloudbuild-*.yaml it joins (cloudbuild-node.yaml if Node + Cloud Run deploy; cloudbuild-workers.yaml if Python + VM deploy; new file if neither)
5. Add a new module instance in infrastructure-gcp/terraform/environments/{staging,production}/main.tf with included_files and substitutions
6. terraform plan && terraform apply
7. Push to test

8. Failure modes table

Symptom	Cause	Fix
image not found: node-builder:${COMMIT_SHA} in webapp build	node-builder push failed	check push-node-builder step logs
cp: cannot stat 'apps/internal/otel-init.cjs'	HAS_OTEL not set to false for internal trigger	check trigger substitutions
uv sync re-runs despite no Python deps change	Wrong Dockerfile layer order; source COPYed before uv sync	follow apps/temporal-worker/Dockerfile pattern
pnpm install re-runs despite no lockfile change	Workspace package.json was edited; the COPY block invalidated	expected; check images/node-builder/Dockerfile
Build fails with “Authentication required”	TRIGGER_* secret missing in GCP Secret Manager	create via gcloud secrets create erp-prod-trigger-*

9. BuildKit secret pattern for non-PUBLIC envs

When adding a new build-time env that’s NOT a PUBLIC_* Astro var:

1. Create the secret in GCP Secret Manager: gcloud secrets create erp-prod-<name>
2. Add to cloudbuild-node.yaml availableSecrets → secretEnv
3. In the build step, write to a file in /workspace/_secret_<name>, then pass to docker via --secret id=<name>,src=...
4. In Dockerfile, consume via RUN --mount=type=secret,id=<name> ...
5. NEVER use --build-arg for non-PUBLIC values — they leak to image history

10. Local development

• Test a Dockerfile change locally: ./scripts/test-cloudbuild-local.sh <target>
• Regenerate node-builder COPY block: pnpm generate:dockerfile-copies
• Verify drift: pnpm check:dockerfile-copies