Skip to content
Ordermatic Docs

Runbook — Cloud Armor LB edge for the public webapp

Puts an external HTTPS Application Load Balancer + Cloud Armor WAF in front of erp-webapp-prod / erp-webapp-staging, then closes the *.run.app hole. Closes the HIGH finding in docs/security/gcp-cloud-audit.md.

gcloud compute mutations and prod DNS changes are classifier-gated. This runbook gives both the Terraform path (preferred) and the exact gcloud equivalents. A human runs the applies.

Architecture

Section titled “Architecture”
client ──> app.ordermatic.co (Cloudflare DNS-only A record)
        ──> global anycast IP (google_compute_global_address)
        ──> HTTPS proxy (Google-managed cert) ─ HTTP:80 proxy redirects to 443
        ──> backend service  ◀── Cloud Armor (rate-limit + OWASP CRS WAF + L7 DDoS)
        ──> serverless NEG (us-east1)
        ──> Cloud Run service (ingress = internal-and-cloud-load-balancing)

DNS-only (not Cloudflare-proxied) is deliberate: Cloud Armor sees the real client IP, so the rate-limit and WAF rules behave correctly. allUsers invoker stays on the Cloud Run service (the serverless NEG needs it); the ingress restriction is what removes direct public access.

What merging the PR changes by itself

Section titled “What merging the PR changes by itself”

Nothing live. All new behavior is behind variables that default to today’s state:

Variable (stack) Default Cutover value
webapp_ingress (infrastructure-gcp prod/staging) INGRESS_TRAFFIC_ALL INGRESS_TRAFFIC_INTERNAL_LOAD_BALANCER
webapp_lb_armor_preview (infrastructure-gcp prod/staging) true (log-only) false (enforce)
webapp_lb_ip / staging_webapp_lb_ip (infrastructure cloudflare) "" → keeps CNAME → ghs LB IP → A record

module.webapp_lb (the LB + Armor) is created on the first GCP apply — it’s additive and carries no traffic until DNS points at it.

Prerequisites

Section titled “Prerequisites”
  • gcloud authed to ordermatic-prod (gcloud auth application-default login as a project editor, or the GCP SA key the wrapper loads from 1Password).
  • Enable the API once: gcloud services enable compute.googleapis.com --project=ordermatic-prod
  • Terraform via the env wrappers:
    • GCP stack: cd infrastructure-gcp && ENV=production make plan / make apply (set ENV=staging for staging). Pass cutover vars with TF_ENV=production bash scripts/terraform-wrapper.sh apply -var 'webapp_ingress=...'.
    • Cloudflare stack: cd infrastructure && bash scripts/terraform-wrapper.sh plan|apply.
  • Do staging end-to-end first, then production.

Phase 1 — Create the LB + Cloud Armor (preview), no traffic yet

Section titled “Phase 1 — Create the LB + Cloud Armor (preview), no traffic yet”

Ingress stays ALL, DNS stays CNAME → ghs, Armor in preview. Safe.

Terraform (preferred):

Terminal window
cd infrastructure-gcp
ENV=production make apply        # creates module.webapp_lb with defaults
ENV=production make output | grep -E 'webapp_lb_ip|webapp_lb_security_policy'

gcloud equivalent (prod; for staging swap the erp-webapp-prod prefix, service name, and domain):

Terminal window
P=ordermatic-prod; R=us-east1; PFX=erp-webapp-prod; SVC=erp-webapp-prod; DOMAIN=app.ordermatic.co


gcloud compute addresses create $PFX-lb-ip --global --project=$P
gcloud compute network-endpoint-groups create $PFX-neg \
  --region=$R --network-endpoint-type=serverless --cloud-run-service=$SVC --project=$P


# Cloud Armor (rules added in preview; default allow)
gcloud compute security-policies create $PFX-armor --project=$P
gcloud compute security-policies update $PFX-armor --project=$P \
  --enable-layer7-ddos-defense
gcloud compute security-policies rules create 1000 --security-policy=$PFX-armor --project=$P \
  --src-ip-ranges='*' --action=throttle \
  --rate-limit-threshold-count=600 --rate-limit-threshold-interval-sec=60 \
  --conform-action=allow --exceed-action=deny-429 --enforce-on-key=IP --preview
for i in "2000 sqli" "2001 xss" "2002 lfi" "2003 rce" "2004 protocolattack"; do
  set -- $i
  gcloud compute security-policies rules create $1 --security-policy=$PFX-armor --project=$P \
    --expression="evaluatePreconfiguredWaf('$2-v33-stable', {'sensitivity': 1})" \
    --action=deny-403 --preview
done


gcloud compute backend-services create $PFX-backend --global --project=$P \
  --load-balancing-scheme=EXTERNAL_MANAGED --security-policy=$PFX-armor \
  --enable-logging --logging-sample-rate=1.0
gcloud compute backend-services add-backend $PFX-backend --global --project=$P \
  --network-endpoint-group=$PFX-neg --network-endpoint-group-region=$R


gcloud compute url-maps create $PFX-urlmap --default-service=$PFX-backend --global --project=$P
gcloud compute ssl-certificates create $PFX-cert --domains=$DOMAIN --global --project=$P
gcloud compute target-https-proxies create $PFX-https-proxy \
  --url-map=$PFX-urlmap --ssl-certificates=$PFX-cert --global --project=$P
gcloud compute forwarding-rules create $PFX-https-fr --global --project=$P \
  --load-balancing-scheme=EXTERNAL_MANAGED --address=$PFX-lb-ip \
  --target-https-proxy=$PFX-https-proxy --ports=443


# HTTP :80 → HTTPS redirect
gcloud compute url-maps import $PFX-https-redirect --global --project=$P <<'YAML'
name: erp-webapp-prod-https-redirect
defaultUrlRedirect: { httpsRedirect: true, redirectResponseCode: MOVED_PERMANENTLY_DEFAULT, stripQuery: false }
YAML
gcloud compute target-http-proxies create $PFX-http-proxy --url-map=$PFX-https-redirect --global --project=$P
gcloud compute forwarding-rules create $PFX-http-fr --global --project=$P \
  --load-balancing-scheme=EXTERNAL_MANAGED --address=$PFX-lb-ip \
  --target-http-proxy=$PFX-http-proxy --ports=80


LB_IP=$(gcloud compute addresses describe $PFX-lb-ip --global --project=$P --format='value(address)')
echo "LB IP = $LB_IP"

Verify the backend is healthy before touching DNS (Host header → LB IP; expect a TLS error until the cert is ACTIVE, which needs DNS — that’s fine, just confirm the LB answers and routes):

Terminal window
curl -sv -H 'Host: app.ordermatic.co' http://$LB_IP/ 2>&1 | head -20   # 301 → https

Phase 2 — Point DNS at the LB and verify end-to-end

Section titled “Phase 2 — Point DNS at the LB and verify end-to-end”

Cert window (read first). The Google-managed cert only provisions after app.ordermatic.co resolves to the LB IP. Between the DNS flip and the cert going ACTIVE (typically 10–30 min, up to 60), clients that resolve to the LB get a TLS error. Mitigate: lower the record TTL to 60s a day ahead, flip during a low-traffic window, and watch the cert status. For guaranteed zero-downtime, use Certificate Manager DNS-authorization instead — see the appendix.

Terraform:

Terminal window
cd infrastructure
# value from the GCP stack output in Phase 1
bash scripts/terraform-wrapper.sh apply -var "webapp_lb_ip=$LB_IP"          # prod
# bash scripts/terraform-wrapper.sh apply -var "staging_webapp_lb_ip=$LB_IP" # staging

(Persist the value in the env/CI tfvars so the next plain apply doesn’t revert it — the cloudflare-dns module sets allow_overwrite=true.)

Watch the cert and confirm the site serves through the LB:

Terminal window
gcloud compute ssl-certificates describe erp-webapp-prod-cert --global \
  --project=ordermatic-prod --format='value(managed.status, managed.domainStatus)'
# wait for ACTIVE / app.ordermatic.co=ACTIVE
curl -sSI https://app.ordermatic.co/ | head -5     # 200/redirect, served via the LB
dig +short app.ordermatic.co                        # → the LB IP

App is now behind Cloud Armor (still in preview — logging only).

Phase 3 — Review preview logs, then enforce

Section titled “Phase 3 — Review preview logs, then enforce”

Let real traffic flow for a representative window (hours–a day), then check what the rules would have blocked:

Terminal window
gcloud logging read --project=ordermatic-prod --freshness=1d --limit=50 \
  'resource.type="http_load_balancer"
   jsonPayload.enforcedSecurityPolicy.name="erp-webapp-prod-armor" OR
   jsonPayload.previewSecurityPolicy.outcome="DENY"' \
  --format='table(jsonPayload.previewSecurityPolicy.outcome, jsonPayload.statusDetails, httpRequest.requestUrl, httpRequest.remoteIp)'

Tune false positives first: lower waf_sensitivity / drop a noisy rule / raise rate_limit_threshold_count in module.webapp_lb. When the preview hits look like real attacks only, enforce:

Terminal window
cd infrastructure-gcp
ENV=production bash scripts/terraform-wrapper.sh apply -var 'webapp_lb_armor_preview=false'
# gcloud equivalent: for each rule priority, --no-preview, e.g.
#   gcloud compute security-policies rules update 2000 --security-policy=erp-webapp-prod-armor --no-preview --project=ordermatic-prod

Phase 4 — Close the *.run.app hole (ingress)

Section titled “Phase 4 — Close the *.run.app hole (ingress)”

Only after Phase 2 verified the LB serves the site. This makes the Cloud Run service reachable only via the LB + VPC.

Terminal window
# Immediate (and persists — the cloudbuild deploy step runs `gcloud run deploy`
# WITHOUT --ingress, so it preserves this setting):
gcloud run services update erp-webapp-prod --region=us-east1 --project=ordermatic-prod \
  --ingress=internal-and-cloud-load-balancing


# Keep Terraform in agreement so a future infra apply doesn't reopen it:
cd infrastructure-gcp
ENV=production bash scripts/terraform-wrapper.sh apply \
  -var "webapp_ingress=INGRESS_TRAFFIC_INTERNAL_LOAD_BALANCER" -var "webapp_lb_armor_preview=false" \
  -var "webapp_image=$(gcloud run services describe erp-webapp-prod --region=us-east1 \
        --project=ordermatic-prod --format='value(spec.template.spec.containers[0].image)')"

Prefer the gcloud run services update line above — it changes ingress only and is image-safe. The Cloud Run service is TF-defined but its image is CI-deployed (gcloud run deploy in cloudbuild.yaml), so any terraform apply of this stack reconciles the image to var.webapp_image — modules/gcp-cloud-run has no lifecycle { ignore_changes }. The -var webapp_image=… above pins the apply to the live image so the ingress flip doesn’t roll the running revision back; confirm the plan shows only the ingress change.

The stated reason for not ignoring the image used to be that Grafana relied on Terraform to roll out its image. Grafana is gone (see the Grafana teardown PR), so nothing in this stack needs TF-driven image rollout any more and ignore_changes = [template[0].containers[0].image] is now the safer default. Note the recorded image in state is still the cloudrun/container/hello bootstrap placeholder for every service, so an un-pinned apply today would roll production back to a sample container. Worth fixing before the next apply.

Verify the hole is closed and the site still works:

Terminal window
curl -sS -o /dev/null -w '%{http_code}\n' https://erp-webapp-prod-35mielb4ca-ue.a.run.app/   # expect 404 (ingress blocks direct)
curl -sSI https://app.ordermatic.co/ | head -3                                               # still 200 via LB

Phase 5 — Cleanup

Section titled “Phase 5 — Cleanup”

The Cloud Run domain mapping (app.ordermatic.co → service via ghs) is superseded and stops working under LB-only ingress. Remove it:

Terminal window
# Terraform: delete the google_cloud_run_domain_mapping.webapp_app_ordermatic
# resource block (and the staging one), then apply. Or:
gcloud beta run domain-mappings delete --domain=app.ordermatic.co \
  --region=us-east1 --project=ordermatic-prod

Then repeat the whole runbook for staging if you started with prod.

Rollback

Section titled “Rollback”
Phase Rollback
4 (ingress) gcloud run services update erp-webapp-prod --ingress=all (instant); revert webapp_ingress var
3 (enforce) webapp_lb_armor_preview=true apply, or per-rule --preview
2 (DNS) revert webapp_lb_ip to "" → record returns to CNAME → ghs (the domain mapping still exists)
1 (LB) terraform destroy -target=module.webapp_lb (nothing depends on it until Phase 2)

The fastest full revert from any point: set DNS back to CNAME → ghs and ingress back to all. Both are single commands.

Notes

Section titled “Notes”
  • Cost: a global forwarding rule + LB is ≈ $18–25/mo per env plus egress; Cloud Armor is ≈ $5/policy/mo + per-rule + per-request. Two envs ≈ $50–70/mo.
  • Drift: the cloudbuild webapp deploy (cloudbuild.yaml, cloudbuild-all.yaml) runs gcloud run deploy without --ingress, so it preserves the ingress set in Phase 4. Optionally pin --ingress=internal-and-cloud-load-balancing there as a follow-up belt-and-suspenders.

Appendix — zero-downtime cert (Certificate Manager DNS-auth)

Section titled “Appendix — zero-downtime cert (Certificate Manager DNS-auth)”

To avoid the Phase-2 cert window entirely, validate the cert before moving the A record, using a DNS authorization (a CNAME that can be added while DNS still points at ghs):

Terminal window
P=ordermatic-prod; DOMAIN=app.ordermatic.co; PFX=erp-webapp-prod
gcloud services enable certificatemanager.googleapis.com --project=$P
gcloud certificate-manager dns-authorizations create $PFX-dnsauth --domain=$DOMAIN --project=$P
gcloud certificate-manager dns-authorizations describe $PFX-dnsauth --project=$P \
  --format='value(dnsResourceRecord.name, dnsResourceRecord.data)'   # add this CNAME in Cloudflare
gcloud certificate-manager certificates create $PFX-cm-cert --domains=$DOMAIN \
  --dns-authorizations=$PFX-dnsauth --project=$P
gcloud certificate-manager maps create $PFX-certmap --project=$P
gcloud certificate-manager maps entries create $PFX-entry --map=$PFX-certmap \
  --certificates=$PFX-cm-cert --hostname=$DOMAIN --project=$P
# then point the HTTPS proxy at the map instead of --ssl-certificates:
gcloud compute target-https-proxies update $PFX-https-proxy --global --project=$P \
  --certificate-map=$PFX-certmap

Once the cert reports ACTIVE (cert provisions off the CNAME, A record still on ghs), do the Phase-2 A-record flip with zero TLS window. Swap the module’s google_compute_managed_ssl_certificate block for the Certificate Manager resources if you adopt this permanently.