Incident report — email pull wedged fleet-wide by one over-long Message-ID (2026-08-13)

Author: Claude (AI pair) · Date: 2026-08-13 Blast radius: every email_pull_connections row, every tenant — not one mailbox Duration: ~2 h of inbound order-email ingestion loss on 2026-08-13 Severity: silent total ingestion outage — no connection in status='error', every dashboard green Fix: commit 3814721e4 on claude/fix-dispatcher-workflow-id-length (changeset .changeset/email-pull-workflow-id-length-cap.md)

1. TL;DR

The email-pull dispatcher builds each child workflow id as {provider}-pull-{conn_id[:8]}-{dedupe_key}, and since #2839 the dedupe key is the RFC 5322 internet_message_id — a header the sender controls, with no bound on length. Temporal stores current_executions.workflow_id as VARCHAR(255).

A Microsoft OneDrive/SharePoint notification mail (@odspnotify) carrying a 253-char Message-ID composed a 277-char workflow id. The child-execution INSERT failed with pq: value too long for type character varying(255).

Temporal classes that as a retryable persistence error, so:

it retried the child start 742 times in the first 67 minutes;
it never wrote a StartChildWorkflowExecutionFailed event;
the parent stayed blocked on its start_child_workflow await forever.

The dispatcher runs under a fixed workflow id with Temporal’s default SKIP overlap policy. So that one wedged run silently skipped every subsequent 3-minute cycle, for every tenant — not just the mailbox that received the message.

Any sender who can email a monitored mailbox with a Message-ID longer than 231 characters could trigger this.

2. Why nothing alerted

Every signal we watch was healthy-looking:

no connection moved to status='error' — nothing ever failed at the connection level;
the worker process was up and kept serving the pdf-extraction queue the whole time;
the dispatcher schedule kept firing on time; the runs were skipped, not missed;
Temporal wrote no failure event, because the persistence error is classed retryable.

The failure lives between “the schedule fired” and “the poll happened”, which is exactly the gap none of our checks covered. Follow-up: confirm whether the email_pull_dead check in .github/workflows/stuck-order-monitor.yml (added after the 2026-07-16 incident) fired on the status='active' + stale last_polled_at branch, and if not, why — that branch is the one designed to catch precisely this shape.

3. The fix

_child_workflow_id in apps/temporal-worker/workflows/email_pull_dispatcher.py now SHA-256-hashes the dedupe key (prefixing the digest with a sha256- marker) in exactly three cases:

Case	Why
composed id (`len(prefix) + len(key)`) exceeds `WORKFLOW_ID_MAX_LEN` (255)	the outage itself — the 253-char Message-ID was itself under 255; the id it composed was 277
key contains a NUL	the other byte a Postgres text column cannot hold — wedges identically, different error string
key already begins with `sha256-`	otherwise a sender computes the digest of a message they are about to send, claims that id with a short header, and the real message dedupes silently against it — a silent dedupe here drops a real order

The guards are deliberately narrow, and that is load-bearing. Tabs, CR/LF and non-ASCII are all legal in a Postgres text column, so the length/NUL check leaves them alone — the key reaches the id unchanged by this fix. (The pre-existing .strip("<> \t") on the raw internet_message_id still trims angle brackets and outer whitespace, as it always did.) Re-keying an id that already worked re-ingests that message as a duplicate order — worse than the bug being fixed. An over-broad charset guard was written and reverted for exactly this reason; a regression test in apps/temporal-worker/tests/test_email_pull_stable_message_id.py now fails if anyone reintroduces one.

Hashing rather than truncating: sender-controlled keys can share arbitrarily long prefixes, and a collision silently drops an order. The sha256- marker lets an on-call engineer reading the Temporal UI tell a hashed id from a literal Message-ID.

4. Deploying this fix — the terminate step is REQUIRED

Deploying the fix alone does not restore ingestion. The wedged execution must be terminated by hand.

The change is deliberately not patch-gated. The old, unstorable id is recorded in the wedged parent’s StartChildWorkflowExecutionInitiated event — that is what the server was retrying. Replaying such a parent under the new code therefore fails loudly on nondeterminism, which is the intended outcome: a patch gate would instead have the unpatched replay reproduce the unstorable id and re-wedge silently.

This step stays required even after the follow-up fix (#2994) ships. That change bounds the child-start await, so a future wedge is abandoned after 60s and the next 3-minute dispatch recovers on its own. It does nothing for an execution that is already wedged at deploy time: that run is blocked inside the old code, and under the new code it fails on replay rather than resuming. Still terminate it by hand.

Deploy order:

Deploy the worker (see docs/releases/temporal-worker-role-deployment-safety.md for the staging/prod VM + image-tag topology).
Terminate the wedged ScheduledEmailPullDispatcher execution. Until it is terminated, the SKIP overlap policy keeps skipping every new cycle:

First get the actual workflow id, and — critically — confirm it is genuinely wedged before terminating. A healthy dispatcher run completes in seconds (poll all connections, fan out children, commit watermarks), so running_actions is empty almost all the time and briefly non-empty during a normal run. “Same id twice” is not sufficient proof: two checks a few seconds apart can both catch one healthy run mid-flight. Terminating that kills a good run. Require a positive wedge signal:
- the run’s age is far beyond a normal dispatch — a healthy run finishes in seconds, so anything RUNNING for more than ~2 minutes is already abnormal; the real wedge here had been running for hours; and
- a concrete stuck symptom: the parent holds a pending child with an empty run_id (a start_child_workflow the server never created), or the temporal-server log is looping on value too long for type character varying(255) (sudo docker logs temporal-server --since 10m | grep 'value too long').
worker.py:487 sets email-pull-dispatcher-schedule-workflow as the schedule’s base id, but Temporal appends the scheduled time to each run, so the live execution is email-pull-dispatcher-schedule-workflow-<ISO8601> — for this incident, email-pull-dispatcher-schedule-workflow-2026-08-13T15:27:00Z. Terminating the bare base id fails: no execution exists under it. This probe reads the id AND prints the two signals so you can confirm before killing anything (the temporal CLI is not installed on the VM; the worker container has the temporalio client):
Terminal window
```
sudo docker exec temporal-worker python3 -u -c "
import asyncio
from temporalio.client import Client

async def main():
    c = await Client.connect('10.0.0.2:7233', namespace='default')
    d = await c.get_schedule_handle('email-pull-dispatcher-schedule').describe()
    for a in d.info.running_actions:
        h = c.get_workflow_handle(a.workflow_id, run_id=a.first_execution_run_id)
        wd = await h.describe()
        orphans = [pc.workflow_id for pc in wd.raw_description.pending_children
                   if pc.run_id == '']
        print('workflow_id:', a.workflow_id)
        print('  started:', wd.start_time, '(wedged if this is minutes/hours ago)')
        print('  orphaned child starts:', orphans or 'none')
    if not d.info.running_actions:
        print('running_actions empty — nothing to terminate')

asyncio.run(main())
"
```
Only if the age and a stuck symptom both hold, terminate exactly that id. The terminate re-reads the status and refuses unless the run is still RUNNING, so a run that finished on its own between the probe and here is left alone:
Terminal window
```
sudo docker exec temporal-worker python3 -u -c "
import asyncio
from temporalio.client import Client

WF = 'email-pull-dispatcher-schedule-workflow-<ISO8601 from above>'

async def main():
    c = await Client.connect('10.0.0.2:7233', namespace='default')
    h = c.get_workflow_handle(WF)
    wd = await h.describe()
    if wd.status != 1:  # 1 == RUNNING
        print('not RUNNING (status', wd.status, ') — leaving it alone')
        return
    await h.terminate(reason='wedged on unstorable child workflow id (incident 2026-08-13)')
    print('terminated', WF)

asyncio.run(main())
"
```
Confirm the next 3-minute cycle actually runs and last_polled_at bumps across connections.
If the gap exceeded the 48 h _MAX_LOOKBACK floor, run a backfill — procedure in §4 of the 2026-07-16 incident. A ~2 h gap is well inside the floor, so the normal watermark poll recovers it.

Ids that already fit are unchanged byte-for-byte, so nothing healthy re-ingests.

5. Testing this code path

pnpm test:python does not cover apps/temporal-worker — scripts/run-python-tests.sh runs only packages/pdf-shared and apps/dagster. The regression tests for this fix run with:

pnpm --filter @repo/temporal-worker test
# equivalently: cd apps/temporal-worker && uv run --extra dev pytest

(AGENTS.md and apps/docs/.../guides/testing.md were corrected in this change; they previously described pnpm test:python as “All Python tests”.)

6. Follow-ups

Confirm the email_pull_dead monitor’s stale-last_polled_at branch behaviour for this outage shape (§2). A wedge stops last_polled_at from bumping, so it should page.
Consider an explicit alarm on a dispatcher run whose duration exceeds one schedule interval — the direct signal for “SKIP is skipping everything”. Still worth having: #2994 bounds the child-start await specifically, which is where this outage hung, but it is not a general guard on total run duration.
Bound the child-start await so any unresolvable start (not just an unstorable id) is abandoned instead of blocking the fleet’s poll loop — #2994. Note that asyncio.wait_for does not work here: the SDK swallows a cancel delivered to an unresolved child start, so it hangs exactly where the bare await did. It races with asyncio.wait and abandons the pending start.
Audit other workflow ids composed from sender- or vendor-controlled strings against the 255-char ceiling.