Incident report — email pull connections stuck in status='error' (2026-07-16)

Author: Claude (AI pair) · Date: 2026-07-16 Customers: Genfit = General Fittings LLC (org_33vHtt6OUnCFD3QGmZeAJXSYsH7) · Cleveland Kitchen (org_3FVE59k9pBxf7fJBJKNCAPNEQaW) — both paying Severity: silent multi-day loss of inbound order-email ingestion (third outage of this shape) Status: RESOLVED 2026-07-16 ~16:10 UTC — see §0. Prevention (dispatcher retry + on-call page) ships with this change — see §7.

0. Resolution (2026-07-16, David + Claude)

Both connections were flipped back to active by a team member on the morning of 2026-07-16 (before 15:45 UTC) — under the pre-fix code an error row cannot self-heal, so this was manual. Follow-up: identify who/how for the timeline.
Genfit: fully auto-recovered on the first poll — its gap (from 07-15 02:48Z) fit inside the 48 h watermark lookback. Per-day email_events counts confirmed no hole.
Cleveland Kitchen: had a real hole — zero emails ingested on Mon 07-13 (weekday baseline 53–83) and a partial 07-14, because the restore happened >48 h after the failure and the auto-poll floor clipped the gap. Recovered via the §4 backfill, run 2026-07-16 ~16:05Z from the worker container on erp-prod-temporal (workflow id email-pull-backfill-ck-20260716). Result: {connections: 3, polled: 436, started: 115, skipped_already_seen: 321, errors: 0}. Post-backfill verification: CK 07-13 went 0 → 72 and 07-14 went 54 → 95 — hole closed. The ~115 delayed emails were flagged to the CSR team (days-old POs, triage first).
Hypothesis updates vs §2.3: the Testing-app 7-day expiry theory is ruled out (both grants had survived 14–15 days of refreshes before the incident), and both token refreshes work post-restore, so no re-auth was needed — consistent with both original failures being transient Google errors made permanent by the poll-set bug.
Ops note: the dedicated erp-prod-temporal-workers VM no longer exists — workers were bin-packed onto erp-prod-temporal on 2026-07-05/06. Runbook commands in §3/§4 that reference “the workers VM” mean that host. In-container one-off scripts run via sudo docker exec temporal-worker /usr/local/bin/infisical-entrypoint.sh python /tmp/<script>.py (a bare docker exec lacks the Infisical-injected env).

1. TL;DR

A single failed poll flips an email_pull_connections row to status='error', but the ScheduledEmailPullDispatcher fetched only status='active' rows — so the success path that flips a row back to active could never run again. Any transient provider error (a Gmail 500, a Google token-endpoint blip) therefore killed a connection permanently and silently: no retry, no alert, no page.

Connection	Mailbox	Last good poll (UTC)	Recorded `last_error`
`b173528e-1484-41aa-b231-9a22c8826f5b` (Genfit)	michele@genfitllc.com (gmail)	2026-07-15 02:48	`HTTP 500`
`8dae96ab-b025-473d-b165-90d02f3acd85` (Cleveland Kitchen)	customerservice@clevelandkraut.com (gmail)	2026-07-12 23:45	`GmailAPIError: Token refresh failed … internal_failure`

Fixes in this change: the dispatcher now retries error rows every cycle (transient failures self-heal), the shared list-fetch no longer decrypts tokens (one poisoned row can’t sink the whole dispatch), manual backfills can target a single connection, and the Pipeline Health Monitor pages on-call whenever any connection is in error or has stopped polling (§5).

2. Root cause

2.1 Systemic: transient errors treated as permanent

poll_gmail_for_connection (apps/temporal-worker/activities/gmail.py) marks status='error' on any failure and returns [] instead of raising, so Temporal’s activity retry policy never fires.
fetch_active_connections (activities/email_connections.py) selected WHERE status = 'active' only. An error row left the poll set forever.
The only paths back to active were a successful poll (unreachable), the org-scoped PATCH /api/email-pull/connections/:id, or an OAuth reconnect. Nothing alerted, so nobody knew to do either — for 4 days in Cleveland Kitchen’s case.

2.2 Genfit (`HTTP 500`)

The error string HTTP 500 is produced at the messages.list step (gmail.py, _ucp(..., error_message=f"HTTP {resp.status_code}")), which runs after a successful token refresh. Genfit’s OAuth grant is healthy; one transient Gmail server-side 500 on 2026-07-15 02:48 UTC killed ingestion. Restoring the row to active is sufficient — no re-auth needed.

2.3 Cleveland Kitchen (`Token refresh failed … internal_failure`)

The error comes from _refresh_access_token_for_connection — Google’s token endpoint returned internal_failure, which is a Google-side backend error, not a verdict on the token. Because the connection was never retried, we cannot tell from the data whether the refresh token is still valid. Two hypotheses, distinguishable on the first retry:

Transient Google outage (most likely at face value): the retry refresh succeeds and the connection self-heals. Only the backfill (§4) remains.
Dead grant: the retry fails with invalid_grant — the customer must reconnect OAuth (§3, step 2b). Prime suspect if so: the Google OAuth app still in Testing publishing status (7-day refresh-token expiry — prior root cause of a similar outage).

Explicitly ruled out for both rows: an OAUTH_TOKEN_ENCRYPTION_KEY mismatch between /runtime/webapp and /runtime/temporal-worker. That failure mode surfaces as ValueError: OAUTH_TOKEN_ENCRYPTION_KEY does not match the key used to encrypt this token in last_error — neither row shows it. (Still verify parity before a CK reconnect — §6.1 — because the webapp encrypts the new tokens the worker must decrypt.)

3. Restore runbook

Convention: through the pooler endpoint use SELECTs for inspection (infisical run --env=production --path=/runtime/webapp -- psql "$DATABASE_URL"); never SET default_transaction_read_only. Prefer the app-level write paths below over raw SQL.

Step 0 — inspect current state:

SELECT id, email_address, provider, status, last_polled_at, last_error, updated_at
FROM email_pull_connections
WHERE id IN ('b173528e-1484-41aa-b231-9a22c8826f5b',
             '8dae96ab-b025-473d-b165-90d02f3acd85');

Step 1 — put the rows back in the poll set. Any one of:

(a) Deploy the dispatcher-retry fix (this PR). error rows re-enter the poll set automatically; both connections are retried within 3 minutes of the worker restart. Nothing else to click.
(b) Settings UI (works today, no deploy): as a member of the customer org, open Settings → Email connections, click Pause then Resume on the connection — the PATCH endpoint accepts error → paused → active.

(c) SQL (ops fallback, needs a write-capable role):

UPDATE email_pull_connections
SET status = 'active', updated_at = now()
WHERE id IN ('b173528e-1484-41aa-b231-9a22c8826f5b',
             '8dae96ab-b025-473d-b165-90d02f3acd85')
  AND status = 'error';

Step 2 — watch the first retry (dispatcher fires every 3 min):

# on the prod temporal-workers VM
journalctl -u temporal-worker --since "-15 min" \
  | grep -E 'b173528e|8dae96ab|Credential resolution failed|Gmail API error|gap exceeds'

2a. Poll succeeds → last_polled_at bumps to now, last_error clears, status stays active. Genfit is expected to land here immediately; CK too if the internal_failure was transient.
2b. CK refresh still fails (invalid_grant/internal_failure persists) → the grant is dead. Ask Cleveland Kitchen (customerservice@clevelandkraut.com) to reconnect: Settings → Email connections → Connect Gmail with that account. The callback upserts the row: fresh tokens, status='active', last_polled_at=NULL. Also check §6.2 (OAuth app publishing status) before they reconnect, or the new token dies in 7 days.

4. Backfill the missed mail

The auto-poll watermark is floored at 48 h (_MAX_LOOKBACK, gmail.py) — a stale or NULL watermark is clamped, and older mail is only recovered by an explicit backfill. Dedup is safe throughout: child workflows use REJECT_DUPLICATE ids (gmail-pull-{conn:8}-{message_id}), so already-ingested messages are skipped.

Genfit — last_polled_at = 2026-07-15 02:48Z (epoch 1784083680):

Restored before 2026-07-17 02:48 UTC: the gap is inside the 48 h floor — the normal watermark poll recovers everything. No manual backfill.
Restored later: run the backfill below with after:1784083080 (2026-07-15 02:38Z = watermark − 10 min overlap) and connection id b173528e-1484-41aa-b231-9a22c8826f5b.

Cleveland Kitchen — last_polled_at = 2026-07-12 23:45Z: the ~4-day gap exceeds the 48 h floor, so a manual backfill is required no matter how the connection is restored (a reconnect resets the watermark to NULL, which also only reaches back 48 h). Anchor: after:1783899300 (2026-07-12 23:35Z = watermark − 10 min overlap).

Run on the prod temporal-workers VM (worker env carries the Temporal address/namespace/TLS config), after the connection is back in the poll set:

infisical run --env=production --path=/runtime/temporal-worker -- \
  temporal workflow start \
    --address "$TEMPORAL_ADDRESS" --namespace "$TEMPORAL_NAMESPACE" \
    --task-queue email-processing \
    --type ScheduledEmailPullDispatcher \
    --workflow-id email-pull-backfill-ck-20260716 \
    --input '{"query": "after:1783899300 -in:sent -in:drafts -in:chats",
              "max_results": 100,
              "connection_ids": ["8dae96ab-b025-473d-b165-90d02f3acd85"]}'

connection_ids (added in this PR) targets the one mailbox. Before the fix is deployed, omit it — the explicit query then runs against every active connection, which is safe (dedup) but re-lists other tenants’ recent mail.
Include the -in:sent -in:drafts -in:chats excludes — an explicit query bypasses the watermark builder that normally adds them.
Capacity: pagination caps at 20 pages × max_results (=2000 messages here) per connection. A 4-day CSR mailbox fits comfortably; if in doubt check the result counts.

Verify:

temporal workflow show -w email-pull-backfill-ck-20260716   # → {"connections":1,"polled":N,"started":X,"skipped_already_seen":Y,"errors":0}

and confirm orders/emails appear in the inbox UI for the window 2026-07-12 → now. Repeat with --workflow-id email-pull-backfill-genfit-20260716, the Genfit epoch and connection id if Genfit’s restore slipped past the 48 h deadline.

5. Alerting added (the long-known gap)

The Pipeline Health Monitor (.github/workflows/stuck-order-monitor.yml — GH Actions, every 30 min, read-only prod DB, BetterStack → erp-unlocked-oncall) now has a money-impacting check that pages when any email_pull_connections row is:

status = 'error' persisting past the self-heal window — last successful poll >10 min old (or never polled and >1 h old). Transient blips the dispatcher clears on its next 3-min retry don’t page. The persistence guard is on last_polled_at, deliberately NOT updated_at: failed retries bump updated_at every cycle, so an updated_at age check would never fire for exactly the persistent failures that must page. Or
status = 'active' with last_polled_at older than 30 min (dispatcher/worker down — the failure the DB-status check alone would miss), or
status = 'active', never polled, untouched for 1 h (a reconnect that never started polling).

paused/disconnected are intentional states and don’t page. The page includes each dead mailbox, org, watermark age and last_error, and links this runbook. This monitor was chosen over a Temporal-schedule alert deliberately: the Temporal worker itself is a suspect in this failure class, and the Python worker’s journald logs aren’t shipped to BetterStack, so a log-based alert had nothing to query.

6. Follow-up verifications (do once during recovery)

Encryption-key parity (compare hashes, never print values):

infisical run --env=production --path=/runtime/webapp          -- printenv OAUTH_TOKEN_ENCRYPTION_KEY | sha256sum
infisical run --env=production --path=/runtime/temporal-worker -- printenv OAUTH_TOKEN_ENCRYPTION_KEY | sha256sum

Must match — the webapp encrypts tokens the worker decrypts. A mismatch turns every (re)connect into an instant error row.

Google OAuth app publishing status: the app behind the worker’s GMAIL_CLIENT_ID must be In production, not Testing (Testing refresh tokens expire after 7 days — prior root cause of this outage shape).
Client-credential parity: worker GMAIL_CLIENT_ID/SECRET and webapp GOOGLE_OAUTH_CLIENT_ID/SECRET must belong to the same OAuth app — refresh tokens are bound to the issuing app (see docs/deployment/email-pull-oauth.md).

7. Prevention shipped in this change

Change	File	Effect
Dispatcher polls `status IN ('active','error')`	`activities/email_connections.py`	Transient failures self-heal on the next 3-min cycle; a persistent error keeps failing fast (one token POST/cycle) until re-auth
List-fetch no longer decrypts tokens	`activities/email_connections.py`	One undecryptable row (key mismatch) can no longer abort the dispatch for every tenant
`connection_ids` param on the dispatcher	`workflows/email_pull_dispatcher.py`	Targeted single-mailbox backfills
`email_pull_dead` money-impacting check	`.github/workflows/stuck-order-monitor.yml`	Dead/stalled connections page on-call within 30 min instead of failing silent
Regression tests	`tests/test_email_pull_dead_connection_recovery.py`	Pin the poll-set + no-decrypt semantics