Runbook — make `/api/erp/force-sync` work (webapp → self-hosted Dagster)

Status: code shipped; infra steps below are not yet applied (you apply them). Owner: infra/you · Surfaced: 2026-06-13, activating the CK collapse — POST /api/erp/force-sync returns 500 "Failed to schedule ERP sync".

Root cause

triggerDagsterJobFromServer (trigger-job.ts) throws "DAGSTER_GRAPHQL_URL or DAGSTER_CLOUD_URL must be configured" because neither is set in the webapp env — in either environment. The config was never migrated when Dagster moved from Dagster+ (Coolify) to the self-hosted VM. So force-sync (and force-image-sync) have been broken in prod and staging since the GCP cutover; nobody noticed because operators materialize via the Dagster UI directly.

Second wrinkle: the Dagster VM has no public IP — its GraphQL is reachable only through a Cloudflare Access–gated tunnel (dagster.ordermatic.co / dagster-staging.ordermatic.co). So a URL alone isn’t enough: the server-side call needs a CF Access service token, or CF returns its login page and the call fails.

What the code already does (shipped in this PR)

trigger-job.ts now sends CF-Access-Client-Id / CF-Access-Client-Secret headers when DAGSTER_CF_ACCESS_CLIENT_ID + DAGSTER_CF_ACCESS_CLIENT_SECRET are set. It’s a no-op until those env vars exist, so merging is safe. You wire the env + token below.

Steps (do staging first, verify, then prod)

1. Create a CF Access service token

Cloudflare Zero Trust → Access → Service Auth → Create Service Token.

• Name: webapp-dagster-force-sync-staging (make a separate …-prod for prod — token isolation per env).
• Copy the Client ID and Client Secret (secret shown once).

2. Add the token to the Dagster Access policy

Zero Trust → Access → Applications → the dagster-staging app → Policies → add an Include rule, selector Service Auth → . (Repeat on the dagster app for the prod token.) This lets the service token through CF Access for that hostname.

3. Store the token in 1Password + wire it into the secrets-map

Create a 1Password item (vault ERP-Unlocked), e.g. Dagster Force-Sync CF Service Token (staging) with fields client_id, client_secret. Then add to infrastructure/infisical/secrets-map.json under the webapp build/runtime block:

"DAGSTER_GRAPHQL_URL": {
  "production": "https://dagster.ordermatic.co/graphql",
  "staging":    "https://dagster-staging.ordermatic.co/graphql"
},
"DAGSTER_CF_ACCESS_CLIENT_ID": {
  "production": "op://ERP-Unlocked/Dagster Force-Sync CF Service Token (prod)/client_id",
  "staging":    "op://ERP-Unlocked/Dagster Force-Sync CF Service Token (staging)/client_id"
},
"DAGSTER_CF_ACCESS_CLIENT_SECRET": {
  "production": "op://ERP-Unlocked/Dagster Force-Sync CF Service Token (prod)/client_secret",
  "staging":    "op://ERP-Unlocked/Dagster Force-Sync CF Service Token (staging)/client_secret"
}

DAGSTER_GRAPHQL_URL is a plain literal (not a secret); the two CF values resolve from 1Password. Create the op items before the secrets sync runs, or it will fail to resolve.

4. Redeploy the webapp

So it picks up the new env (Infisical → build/runtime). Staging first.

5. Verify

From an admin session on the env, or curl with the operator cookie:

POST /api/erp/force-sync   body {"connectionId":"<conn>","entityType":"products"}
headers: X-Requested-With: XMLHttpRequest
→ expect 200 { runId... }   (was 500 "Failed to schedule ERP sync")

Then confirm a run launched in the Dagster UI for that connection’s partition.

Notes

• Until this is applied, re-materialize manually: authed dagster-staging.ordermatic.co tab → launchPipelineExecution on __ASSET_JOB (location erp_pipeline, repo __repository__) with the asset selection + dagster/partition tag. See project_ck_two_series_fix_2026_06_12 memory for the exact recipe.
• The orchestration sensors that auto-chain bronze→dbt→typesense are fixed separately (default_status=RUNNING) and started on staging.