Agent Development Lifecycle (ADLC) Guide

1. Overview: The Shift from SDLC to ADLC

In traditional software development, human engineers write code over hours or days, submit a few PRs per week, and wait patiently for serialized CI/CD runs.

In an Agent Development Lifecycle (ADLC), autonomous AI coding agents (Claude Code, Devin, Symphony executors, Antigravity) collaborate alongside humans at machine velocity:

• Dozens of PRs are authored, reviewed, and merged per day.
• Iteration cycles drop from days to minutes.
• Traditional deployment bottlenecks and serial queues quickly back up and halt velocity.

This guide outlines the architecture, guardrails, and tools powering the ADLC at ERP-Unlocked.

---

2. The Bottleneck: The PR #3315 Dilemma

In PR #3315 (“fix(deploy): make GCP releases convergent and diagnosable”), cancel-in-progress in .github/workflows/gcp-deploy.yml was set to false:

concurrency:
  group: gcp-deploy-${{ github.ref_name }}
  cancel-in-progress: false

Why it was disabled

During a previous staging outage, cancelling deploy workflows in GitHub Actions caused cancellation thrash:

1. When GitHub Actions killed a runner mid-flight, underlying Cloud Build jobs or Ansible systemd worker unit migrations were aborted midway through container pulls and unit installs on GCE VMs (erp-staging-temporal, erp-staging-dagster).
2. This left staging VMs in half-converged states with mismatched container tags.
3. Serializing deployments (cancel-in-progress: false) guaranteed that every active run converges to clean completion.

The Tradeoff

While serialization ensured stability, it introduced a new failure mode under agent volume: If 4 agent PRs merge into staging within 20 minutes, each 15-minute deploy queues sequentially. The 4th PR takes 1 hour to reach staging! Furthermore, auto-deploy-main.yml blocks on the exact staging run, stalling production promotions.

---

3. The Solution: The 4 Pillars of the ADLC

┌────────────────────────────────────────────────────────────────────────┐
│                   Agent Development Lifecycle (ADLC)                   │
├──────────────────┬──────────────────┬──────────────────┬───────────────┤
│ 1. Blacksmith    │ 2. Coalescing    │ 3. Decoupled     │ 4. Ephemeral  │
│ NVMe BuildKit    │ Deployment Gate  │ Build vs Deploy  │ PR Previews   │
│ 42s Builds       │ 0s Queue Drain   │ Pure / Atomic    │ Safe Sandbox  │
└──────────────────┴──────────────────┴──────────────────┴───────────────┘

Pillar 1: High-Speed Container Builds with Blacksmith NVMe BuildKit

Standard Google Cloud Build runs on E2_HIGHCPU_8 instances with network-bound layer caching (--cache-from) and GCS rsync pnpm store hydration. Warm builds take 12–18 minutes, and cold builds take up to 57 minutes ($0.19–$0.25 per deploy).

Rather than adding a second vendor bill (such as Depot.dev), ERP-Unlocked leverages Blacksmith’s native BuildKit integration (useblacksmith/setup-docker-builder):

• Persistent NVMe Sticky Disks: Caches Docker layers directly on local high-speed SSDs.
• Empirical Benchmark Results (Workflow run 35648129031):
  • Node.js & TypeScript (tsc --noEmit): 12s on Blacksmith vs 48s on standard runner (4.0x faster).
  • Docker Build (temporal-worker): 42s on Blacksmith vs 159s on standard runner (3.8x faster and 10x faster than Cloud Build).
  • Compute Cost: $0.0097 / run (~1¢).

Using the Composite Action

A reusable composite action is available at .github/actions/blacksmith-build-push:

- name: Build and push container image
  uses: ./.github/actions/blacksmith-build-push
  with:
    file: apps/temporal-worker/Dockerfile
    tags: us-east1-docker.pkg.dev/ordermatic-prod/erp-unlocked/temporal-worker:${{ github.sha }}
    push: true

---

Pillar 2: Deployment Coalescing (Safe Queue Draining)

Rather than abruptly cancelling active builds or blindly running redundant 15-minute builds for superseded commits, .github/workflows/gcp-deploy.yml implements an ADLC Coalescing Gate:

if [ "${{ github.event_name }}" = "push" ]; then
  git fetch origin "${{ github.ref_name }}" --depth=1 2>/dev/null || true
  CURRENT_HEAD=$(git rev-parse "origin/${{ github.ref_name }}" 2>/dev/null || echo "")
  if [ -n "$CURRENT_HEAD" ] && [ "${{ github.sha }}" != "$CURRENT_HEAD" ]; then
    echo "superseded=true" >> "$GITHUB_OUTPUT"
    echo "::notice title=Deployment Coalesced::Commit ${{ github.sha }} has been superseded by $CURRENT_HEAD on ${{ github.ref_name }}."
    exit 0
  fi
fi

How it works in practice:

1. Commit A merges to staging and begins deploying.
2. While Commit A is running, PR B, PR C, and PR D merge to staging.
3. Commit A completes its deployment cleanly (no cancellation thrash).
4. Run B starts -> detects origin/staging is at Commit D -> skips Cloud Build and completes with status success in 3 seconds.
5. Run C starts -> detects origin/staging is at Commit D -> skips Cloud Build and completes in 3 seconds.
6. Run D starts -> confirms it matches origin/staging -> executes deployment!
7. Total queue drain time drops from 45 minutes down to seconds, with zero risk to VM convergence.

Additionally, if all services are unchanged (e.g. documentation, README, or markdown-only commits), Cloud Build submission is skipped (all_unchanged=true), completing the run in seconds.

---

Pillar 3: Decoupling Build from Deploy

In the mature ADLC architecture:

1. Build Phase (Stateless & Cancellable):
   • Builds container images using Depot.dev.
   • Pushes :sha-<commit> images to Google Artifact Registry.
   • 100% safe to cancel on branch updates (cancel-in-progress: true).
2. Deploy Phase (Atomic Convergence):
   • Zero compilation and zero image building.
   • Points Cloud Run to the already-built digest (gcloud run deploy --image=... in ~30s).
   • Runs Drizzle database migrations with idempotency guards.
   • Updates VM systemd unit files atomically (mv .tmp .service && systemctl restart).

---

Pillar 4: Ephemeral PR Preview Environments (Agent Sandboxes)

To prevent agents from competing for shared staging:

• Agents can spin up isolated Cloud Run preview revisions tagged per PR (--tag pr-${PR_NUMBER} --no-traffic).
• The preview URL (https://pr-123---erp-webapp-staging-xxx.a.run.app) serves the PR’s exact build without receiving public staging traffic.
• Agents run automated UI tests (e.g. /recorded-ui-test) and API evaluations directly on the preview URL before marking the PR ready for merge.