Investigation Report: Temporal Worker Deploy Pin on GCE VMs

Ticket ID: ENG-463
Title: fix: investigate stale-image deploy pin on GCE temporal-worker VMs
Date: June 1, 2026
Status: Closed (Won’t Fix / Already Correct)

---

Executive Summary

The Cloud Build deploy-temporal-workers task (cloudbuild-workers.yaml) executes an SSH command on the GCE VMs to deploy new builds of temporal-worker and ts-temporal-worker. It was hypothesized that the systemd unit files on the VMs might pin a static image SHA, meaning that restarts of the systemd unit would not pick up new changes if the underlying image tags were updated.

An empirical investigation was conducted on both the Staging and Production temporal worker VMs. The findings show that:

1. No static SHA pinning is present in the systemd unit files.
2. The systemd unit files are configured to run containers using the dynamic latest-staging (for staging) and latest-main (for production) tags.
3. The deployment process correctly handles updating the local image tag mapping (:COMMIT_SHA tagged as :latest-${BRANCH_NAME}) prior to restarting the systemd unit.
4. The Ansible playbooks are templated to automatically use environment-appropriate tags (latest-staging / latest-main) instead of hardcoded SHAs.

Accordingly, the VM systemd units and Ansible templates were verified to be functioning exactly as designed. The only change introduced by this PR is a VM-routing update in cloudbuild-workers.yaml so the deploy-temporal-workers step targets erp-staging-temporal-workers for staging and erp-prod-temporal-workers for main (removing the prior skip-on-main guard, which is no longer needed now that both VMs’ systemd units already reference the correct dynamic tags). Aside from that pipeline routing change, no further fixes to the deployment pipeline or VM configuration are required.

---

Detailed Investigation Findings

1. Verification on erp-staging-temporal-workers VM

We successfully established an IAP SSH session to erp-staging-temporal-workers and inspected the systemd unit configurations for both temporal-worker and ts-temporal-worker:

Service definition for temporal-worker:

/etc/systemd/system/temporal-worker.service

[Unit]
Description=Temporal Worker
Requires=docker.service
After=docker.service

[Service]
Type=simple
Restart=always
RestartSec=10
ExecStartPre=-/usr/bin/docker rm -f temporal-worker
ExecStart=/usr/bin/docker run --rm \
  --name temporal-worker \
  --env-file /opt/temporal-worker/.env \
  us-east1-docker.pkg.dev/ordermatic-prod/erp-unlocked/temporal-worker:latest-staging
ExecStop=/usr/bin/docker stop temporal-worker
TimeoutStartSec=120

[Install]
WantedBy=multi-user.target

Service definition for ts-temporal-worker:

/etc/systemd/system/ts-temporal-worker.service

[Unit]
Description=TS Temporal Worker
Requires=docker.service
After=docker.service

[Service]
Type=simple
Restart=always
RestartSec=10
ExecStartPre=-/usr/bin/docker rm -f ts-temporal-worker
ExecStart=/usr/bin/docker run --rm \
  --name ts-temporal-worker \
  --env-file /opt/ts-temporal-worker/.env \
  us-east1-docker.pkg.dev/ordermatic-prod/erp-unlocked/ts-temporal-worker:latest-staging
ExecStop=/usr/bin/docker stop ts-temporal-worker
TimeoutStartSec=120

[Install]
WantedBy=multi-user.target

Running Docker Container Verification:

Inspecting the active containers confirmed they are indeed running using the latest-staging tag as intended:

• /temporal-worker: us-east1-docker.pkg.dev/ordermatic-prod/erp-unlocked/temporal-worker:latest-staging
• /ts-temporal-worker: us-east1-docker.pkg.dev/ordermatic-prod/erp-unlocked/ts-temporal-worker:latest-staging

---

2. Verification on erp-prod-temporal-workers VM

We similarly SSH’d into erp-prod-temporal-workers and inspected the production services:

Service definition for temporal-worker:

/etc/systemd/system/temporal-worker.service

...
ExecStart=/usr/bin/docker run --rm \
  --name temporal-worker \
  --env-file /opt/temporal-worker/.env \
  us-east1-docker.pkg.dev/ordermatic-prod/erp-unlocked/temporal-worker:latest-main
...

Service definition for ts-temporal-worker:

/etc/systemd/system/ts-temporal-worker.service

...
ExecStart=/usr/bin/docker run --rm \
  --name ts-temporal-worker \
  --env-file /opt/ts-temporal-worker/.env \
  us-east1-docker.pkg.dev/ordermatic-prod/erp-unlocked/ts-temporal-worker:latest-main
...

Running Docker Container Verification:

Active containers on the production VM are running:

• /temporal-worker: us-east1-docker.pkg.dev/ordermatic-prod/erp-unlocked/temporal-worker:latest-main
• /ts-temporal-worker: us-east1-docker.pkg.dev/ordermatic-prod/erp-unlocked/ts-temporal-worker:latest-main

---

3. Infrastructure as Code / Configuration Audit

We verified the Ansible playbook infrastructure-gcp/ansible/playbooks/temporal-workers.yml which handles the deployment/provisioning of the temporal-workers service.

Lines 23-24 define:

temporal_worker_image_tag: "{{ (app_environment == 'production') | ternary('latest-main', 'latest-staging') }}"
ts_temporal_worker_image_tag: "{{ (app_environment == 'production') | ternary('latest-main', 'latest-staging') }}"

The systemd files generated by Ansible template are written as:

ExecStart=/usr/bin/docker run --rm \
--name temporal-worker \
--env-file {{ worker_dir }}/.env \
{{ artifact_registry }}/temporal-worker:{{ temporal_worker_image_tag }}

This ensures that any fresh provisioning or reconfiguration automatically uses the appropriate environment-specific dynamic tag.

---

4. Deploy Pipeline Verification

The GCE deploy pipeline defined in cloudbuild-workers.yaml (under the deploy-temporal-workers task) runs:

sudo docker pull ${_REGISTRY}/temporal-worker:${COMMIT_SHA} &&
sudo docker tag ${_REGISTRY}/temporal-worker:${COMMIT_SHA} ${_REGISTRY}/temporal-worker:latest-${BRANCH_NAME} &&
sudo systemctl restart temporal-worker

Because the systemd unit references latest-${BRANCH_NAME}:

1. Cloud Build pulls the precise built image (COMMIT_SHA).
2. Cloud Build tags that image on the VM as latest-staging (or latest-main).
3. Running systemctl restart stops the old container, and starts a new one which pulls/resolves latest-${BRANCH_NAME} locally.
4. Since the local tag has been updated to point to the new COMMIT_SHA image, Docker runs the newly deployed code.
5. docker image prune -af is then executed, which cleans up the untagged COMMIT_SHA images while preserving the tagged ones and keeping disk usage bounded.

---

Conclusion & Recommendations

The hypothesis that the systemd unit files pin a static image SHA is unverified and incorrect. The VMs and deployment pipeline are already configured to use dynamic, branch-specific tags, which are updated locally during each deployment cycle.

The only change this PR introduces is a VM-routing update in cloudbuild-workers.yaml: the deploy-temporal-workers step now routes staging builds to erp-staging-temporal-workers and main builds to erp-prod-temporal-workers, replacing the prior _TEMPORAL_WORKERS_VM substitution and the temporary skip-on-main guard. Aside from that pipeline routing change, the target behavior was already implemented correctly and no further code or pipeline modifications are needed.