ops-mcp Debugging Runbook

Operators-only. For end-user docs see apps/ops-mcp/INTEGRATION.md and docs/getting-started/connect-claude-code-to-ops-mcp.md.

Quick triage

# Health + metadata + AS resolution
pnpm mcp:doctor                    # staging
pnpm mcp:doctor --env production

# Full auth probe (provide a token)
pnpm mcp:doctor --token mcps_xxx --env production
MCP_OPS_TOKEN=eyJ... pnpm mcp:doctor --env production

Reading a 401

Every 401 from ops-mcp includes three diagnostic headers:

Header	Purpose
`X-Ordermatic-Request-Id`	Per-request 16-byte hex. Quote this in tickets — it’s the join key into Worker logs.
`X-Ordermatic-Auth-Decline-Reason`	Stable reason code (see table below).
`WWW-Authenticate`	RFC 6750 challenge incl. `resource_metadata` URL for OAuth discovery.

Decline reasons

Reason	Cause	Fix
`missing_header`	No `Authorization` header	Send `Authorization: Bearer <token>`
`empty_token`	Header present but `Bearer` followed by empty string	Same
`unknown_token`	Service token: hash not in DB, OR revoked. OAuth: shouldn’t fire (sig errors first)	Re-issue / re-authenticate
`token_expired`	Service: `expires_at <= now()`. OAuth: JWT `exp` past	Re-issue (service) / Claude Code auto-refreshes (OAuth)
`token_revoked`	OAuth: `revoked:<jti>` sentinel found in KV	Re-authenticate
`signature_invalid`	OAuth: JWT signature didn’t verify against current+previous JWKS	Likely key rotation issue OR forged token. Investigate.
`denylist_check_unavailable`	KV binding missing OR KV read errored	Check Cloudflare KV namespace binding; check status.cloudflare.com
`database_unavailable`	Service path only: Neon connection failed	Check Neon status; check `DATABASE_URL` secret on Worker
`internal_error`	Unexpected — open an incident	Check logs by `request_id`

Inspect a specific request

# From Better Stack / Cloudflare Logs, search for the request_id
# Worker logs include: request_id, user_id, org_id, tool_name, decline_reason

# Local tail (dev only):
cd apps/ops-mcp && pnpm exec wrangler tail --env staging

Common ops actions

Issue a new service token

Webapp UI: /sys-admin/mcp-tokens → New Token
Select org, service name (ops-mcp), env, scopes (mcp:read and/or mcp:write), expiry
Copy the displayed mcps_… token IMMEDIATELY — not stored in plaintext

Revoke a session

-- DB-side revoke (single session)
UPDATE mcp_oauth_sessions
SET revoked_at = now(), revoked_reason = 'admin'
WHERE id = '<session-uuid>';

After the DB write, also write the KV sentinel (otherwise Worker won’t see the revoke until JWKS cache refresh, ~5min). Either:

Trigger the webapp’s revoke endpoint (it does both)
Manually wrangler kv key put --binding MCP_OAUTH_DENYLIST --env <env> revoked:<jti> 1 --ttl 2592000

Rotate JWT signing keys

Generate new ES256 keypair:

openssl ecparam -name prime256v1 -genkey -noout | openssl pkcs8 -topk8 -nocrypt -out new.pem

Set new key as OPS_MCP_JWT_PRIVATE_KEY_CURRENT in Infisical
Move the OLD _CURRENT value to OPS_MCP_JWT_PRIVATE_KEY_PREVIOUS
Redeploy webapp — JWKS now publishes BOTH kids
Wait access_token_ttl + skew (~20 min) — all old tokens expire
Remove _PREVIOUS from Infisical
Redeploy webapp — JWKS publishes only the new kid

Create the KV denylist namespace (initial setup)

cd apps/ops-mcp
pnpm exec wrangler kv namespace create MCP_OAUTH_DENYLIST --env staging
# Copy the returned `id` into wrangler.toml under [[env.staging.kv_namespaces]]
pnpm exec wrangler kv namespace create MCP_OAUTH_DENYLIST --env production
# Same for production block

Break-glass: unbreak a stuck user

If a user’s OAuth session can’t refresh and they can’t re-auth (e.g. Clerk issue mid-incident), issue a short-lived (24h) service token:

/sys-admin/mcp-tokens
  Service name: ops-mcp
  Subject: <user-email>             # for audit
  Scopes: mcp:read mcp:write
  Expires: now + 24h
  Description: "OAuth break-glass for <user> — incident <ticket>"

Hand the raw token to the user with explicit “delete this after the incident” instructions. Log the issuance in the incident ticket.

Common failure investigations

”Every OAuth request 401s with `denylist_check_unavailable`”

Worker can’t reach the KV namespace. Check:

wrangler.toml has the namespace bound under [[env.<env>.kv_namespaces]] with a non-placeholder id
wrangler deploy succeeded after the namespace was created
wrangler kv key list --binding MCP_OAUTH_DENYLIST --env <env> returns (it should be empty if no revokes have happened, but the call shouldn’t error)

“Service tokens work but OAuth tokens 401 with `signature_invalid`”

Key mismatch. Check:

Webapp OPS_MCP_JWT_PRIVATE_KEY_CURRENT is set per env
Webapp MCP_OAUTH_ISSUER env var matches what’s in the AS metadata
Worker WEBAPP_AS_BASE_URL env var matches webapp’s deployed origin
https://app.ordermatic.co/.well-known/jwks.json is reachable from the Worker
JWT’s kid matches one of the JWKS-published keys

”First /mcp call after deploy is slow (>3s)”

Cold-start JWKS fetch. Mitigations:

KV cache (5min) — second isolate boot fast
Bundled fallback JWKS (Eng FIX E5) — TODO if not in current build
Cloud Run min-instances=1 on webapp OAuth route

Logs queries (Better Stack / Cloudflare Logs)

# All 401s in the last hour with their decline reason
service:"ops-mcp" status:401 | group_by(auth_decline_reason)

# Errors for a specific user
service:"ops-mcp" user_id:"user_xxx" status:>=400

# Reused refresh tokens (should be near zero)
service:"webapp" message:"refresh-token reuse detected"

# Clerk webhook → MCP session revocations
service:"webapp" message:"revokeMcpSessions" reason:*

Escalation

Worker auth issues: ping #platform-eng with the request_id
KV outage: check status.cloudflare.com, fallback to issuing service tokens
Clerk outage: webapp OAuth flow blocked; existing tokens still work until expiry
Neon outage: service-token path AND DB-backed tools (audit/email logs) fail