Skip to content
Ordermatic Docs

CSR Automation Build Plan

Status: Draft for review · Author: David + Claude · Date: 2026-06-14 Goal: Every in-app complaint becomes a Plain.com thread that gets auto-triaged — simple cases get actionable in-app advice, real bugs become GitHub issues, and the founder is escalated to only when needed. Built entirely on the existing stack; no new services.

0. Review synthesis & revised plan (2026-06-14)

Section titled “0. Review synthesis & revised plan (2026-06-14)”

Three independent reviews (eng/architect, founder/scope, design/DX + trust) converged: the original phasing front-loads infrastructure and rests on two false “already exists” assumptions. Collapse to one shippable increment now; defer the triage agent until ticket volume justifies it. Where §2/§9 below conflict with this section, §0 wins.

Corrections (verified in code)

Section titled “Corrections (verified in code)”
  • ops-mcp investigation tools are mocked, not real. apps/ops-mcp/src/middleware/auth.ts:37 — “TODO: Replace with actual database query”, hardcoded org-123; org:info/ops:status/ops:logs return fabricated strings (mockGetOrgInfo → “Example Corp”). The agent cannot investigate on these until the tools + auth are real.
  • The Temporal worker has no AI path. No @anthropic-ai/sdk/@repo/ai dependency; packages/ai pins a stale @anthropic-ai/sdk@^0.24.3 shaped for landing-page generation. “Claude via existing packages/ai, nothing to provision” is false — needs deps + ANTHROPIC_API_KEY + egress + SDK bump (new wiring, not a new managed service).
  • Intake already creates a Plain thread on every submit (tickets.ts:188-239) — “Plain-primary” is a flip, not a build.
  • Plain SDK migration blast radius is 4 (webapp, ts-temporal-worker, packages/db package.json + the wrapper), not 1.
  • Never persist a PostHog replay sharing access_token in a Plain thread — it becomes a permanent unauthenticated replay link. Store only session_id; mint a short-TTL link on-demand at founder-view time.

Increment 1 — ship now (~1-few days, ≈80% of the value)

Section titled “Increment 1 — ship now (~1-few days, ≈80% of the value)”
  • Flip tickets.ts to Plain-primary; stop auto-creating a GitHub issue on every submit.
  • Attach the PostHog session_id (id only) to the thread; resolve a short-TTL replay link on-demand at view time.
  • Route every thread to the founder with the existing Customer Card; add a Plain notification (email/mobile) for fast response.
  • Migrate the Plain SDK off the deprecated package (4-package scope; keep client.ts signatures stable so callers don’t change).
  • Trust moments (first-class): a quiet capture-consent line under the textarea (“we’ll attach a recording of your recent activity to help debug — what’s included”) + honest success copy.
  • Blocking gate: extract sanitizeErrorContext into a shared module so the snapshot is redacted client-side before transmit; unit-test against a PII corpus.
  • PostHog: add the prod key + getSessionId(); defer replay-masking config until a ticket needs the replay.
  • Short-circuit severity=critical straight to founder at intake (no async triage).

Increment 2 — defer until intake hurts (the triage agent)

Section titled “Increment 2 — defer until intake hurts (the triage agent)”

Build only when reading every ticket is painful (and let that real backlog be the calibration set). Real prerequisites the original Phase 0 hid: real ops-mcp auth + de-mocked tools; worker→Anthropic wiring + the classifier extracted as a pure packages/ai function wired into packages/eval (clone the email-triage scorer); webhook idempotency (dedup table + workflowId = supportTriage-{threadId}-{eventTs}, catch WorkflowExecutionAlreadyStartedError, Claude in activities); create the thread with the snapshot component in one call (avoid the webhook-before-attach race); calibration table + support_triage_decided event from day one; AI replies self-disclose + offer a one-tap “reply ‘human’” escape (bug vs resolved read differently; never claim unverified resolution); founder review via Plain draft reply + ai-approved/edited/rejected labels feeding the calibration set; calibrated auto-send (Trust-or-Escalate α/δ) is the last step.

1. Why now

Section titled “1. Why now”

We are early in PMF and support is the biggest gap. Users need an on-screen way to ask for help; we need that to land somewhere that can triage and respond without the founder reading every message. The CSR automation plan existed (debt/ideas/sales-marketing-automation-vision.md) but predates the Plain + PostHog + ops-mcp integrations we now have. This plan makes it real on what exists today.

2. What already exists (≈70% of the plumbing)

Section titled “2. What already exists (≈70% of the plumbing)”
CapabilityStatusLocation
On-screen help (global floating button)✅ shipped this sessionapps/webapp/src/components/SupportWidget.tsx (now mounted in Layout + CockpitShell)
Support intake API✅ exists (GH-primary, Plain mirror)apps/webapp/src/pages/api/support/tickets.ts
PII/secret sanitizer✅ exists (regex redaction)tickets.ts sanitizeErrorContext()
Plain SDK wrapper (upsert customer, create thread)⚠️ exists, on deprecated SDKapps/webapp/src/lib/plain/client.ts
Plain Customer Card (live Ordermatic context in sidebar)✅ existsapps/webapp/src/pages/api/support/customer-card.ts
Agent investigation toolbox✅ existsapps/ops-mcp tools: get_order_email_log, list_failed_emails, get_audit_log, org:info, ops:status/logs
Async workflow runtime✅ existsapps/ts-temporal-worker
Claude / Anthropic SDK✅ existspackages/ai (landing-page-generator.ts)
Webhook pattern (verify → start workflow)✅ existsapps/webapp/src/pages/api/webhook/inbound-email.ts
PostHog browser SDK✅ initializedpackages/observability/src/posthog-browser.ts

Gaps to close: capture the PostHog session link + state snapshot; flip intake to Plain-primary; the Plain webhook + Claude triage agent; the escalation path; and three PostHog prerequisites (below).

3. Locked decisions (from this session)

Section titled “3. Locked decisions (from this session)”
  1. Capture = lightweight. Ship the PostHog session_id + a redacted state snapshot; link the replay rather than transmitting a heavy screenshot/DOM payload. Optional single screenshot is a later add.
  2. Triage = hybrid. Plain’s built-in AI agent handles generic FAQ deflection in-thread; our Claude agent (packages/ai + ops-mcp) handles app-aware cases, gives in-app advice for simple issues, and escalates to the founder when needed.
  3. Plain is the system of record for every complaint. A GitHub Issue is auto-created only when triage confirms a real bug.
  4. No new services. Reuse Plain, PostHog, Claude/ops-mcp, Temporal, Redis, Postgres, GitHub.

4. Target architecture (end-to-end)

Section titled “4. Target architecture (end-to-end)”
┌─ Browser ─────────────────────────────────────────────┐
│ SupportWidget (global)                                 │
│  on submit: posthog.get_session_id() + redacted        │
│             state snapshot (route, breadcrumbs, last    │
│             error, app version, ids) — redaction runs   │
│             client-side                                 │
└───────────────┬───────────────────────────────────────┘
                │ TLS POST
                ▼
┌─ Astro API: /api/support/tickets (Plain-PRIMARY) ──────┐
│ 1. rate-limit (Redis, exists)                          │
│ 2. upsertPlainCustomer + createPlainThread  ← record   │
│ 3. attach PostHog replay deep-link (sharing            │
│    access_token) + snapshot as a thread component      │
│ 4. NO GitHub issue here                                │
└───────────────┬───────────────────────────────────────┘
                │ Plain fires thread.* webhook
                ▼
┌─ Astro API: /api/webhook/plain (verifyPlainWebhook) ───┐
│ HMAC verify → start Temporal workflow                  │
└───────────────┬───────────────────────────────────────┘
                ▼
┌─ Temporal: supportTriage workflow (ts-temporal-worker) ┐
│ Claude routing classifier → one of:                    │
│  • FAQ-deflectable  → (Plain AI handles, or canned)    │
│  • app-aware/simple → investigate via ops-mcp tools +  │
│      read snapshot/replay → replyToThread w/ advice    │
│  • confirmed bug    → create GH issue + label + note   │
│  • low-confidence / irreversible → escalate to founder │
│      (assign thread + notify), no auto-reply           │
│ All replies posted as a scoped Plain MACHINE USER      │
└────────────────────────────────────────────────────────┘

The founder sees full context via the existing Customer Card whenever a thread is escalated.

5. Prerequisites (Phase 0 — must land before triage works)

Section titled “5. Prerequisites (Phase 0 — must land before triage works)”
  • P0-a · PostHog prod key. Prod emits zero browser events today (PUBLIC_POSTHOG_KEY missing in Infisical prod /build/webapp; staging has it). Add the secret + force a webapp rebuild. Without this there is no replay to link in prod.
  • P0-b · Enable session replay + masking. Replay is not configured in posthog-browser.ts init(). Turn it on with masking: maskAllInputs (default true — keep), add maskTextSelector/maskTextFn for app-specific PII (order numbers, pricing), and maskCapturedNetworkRequestFn for API redaction. Footgun: a custom network mask replaces PostHog’s deny-list, and the project’s server-side replay config can override SDK defaults — verify both. (Sources: PostHog privacy docs.)
  • P0-c · Expose the session id. The wrapper only exports init/identify/group/capture/reset. Add getSessionId() (and a getReplayUrl() helper). Call posthog.get_session_id() post-init only (e.g. in useEffect) — it returns undefined before init. (Source: PostHog sessions docs.)
  • P0-d · Plain machine user + webhook secret. Create a scoped machine user (thread:reply permission) for the triage agent; create the webhook signing secret; turn on Plain’s built-in AI agent for FAQ deflection (it is optional and coexists with our agent). (Sources: Plain “Bring Your Own Agent” blog + agents docs.)
  • P0-e · Migrate off the deprecated Plain SDK. @team-plain/typescript-sdk@^5.11.0 was archived 2026-06-10 and split into @team-plain/graphql + @team-plain/webhooks + @team-plain/ui-components. Migrate lib/plain/client.ts to @team-plain/graphql; use @team-plain/webhooks’ verifyPlainWebhook for the new endpoint. Same PlainClient({ apiKey }) shape. (Sources: GitHub archive notice + Plain changelog.)

6. Phased delivery

Section titled “6. Phased delivery”

Phase 1 — Capture + Plain-primary intake

Section titled “Phase 1 — Capture + Plain-primary intake”
  • Extend SupportWidget submit to include { posthogSessionId, snapshot } where snapshot = { route, recentActions (breadcrumbs), lastError, appVersion, orgId, userId }, redacted client-side (extend the existing sanitizer; reuse its patterns).
  • Flip /api/support/tickets to Plain-primary: create the thread as the record, attach a PostHog replay deep-link (generated server-side via the sharing access_token endpoint) + the snapshot as a Plain thread component/timeline entry. Stop creating a GitHub issue on every submission.
  • Keep Redis rate-limiting and the server-side sanitizer as defense-in-depth.

Phase 2 — Webhook + triage agent (shadow mode)

Section titled “Phase 2 — Webhook + triage agent (shadow mode)”
  • Add apps/webapp/src/pages/api/webhook/plain.ts: verify with verifyPlainWebhook (HMAC-SHA256, Plain-Request-Signature), then start the Temporal workflow. Mirror inbound-email.ts.
  • Add supportTriage workflow in apps/ts-temporal-worker + activities that (a) call the new packages/ai triage module (Claude routing classifier), (b) invoke ops-mcp read tools to investigate, (c) post back to Plain.
  • Start in shadow / suggest-only mode: the agent posts its proposed reply + reasoning as an internal note on the thread for the founder to approve/send. No customer-facing auto-reply yet. This generates the labeled data we need for calibration.
  • On classified “confirmed bug”: auto-create the internal GitHub issue, label the thread, link both ways.

Phase 3 — Calibrated auto-send

Section titled “Phase 3 — Calibrated auto-send”
  • Collect founder approve/edit/reject decisions from Phase 2 into a small Postgres table (the calibration set).
  • Set the escalate-vs-auto-reply gate with a calibrated confidence threshold (α/δ per the “Trust or Escalate” framework), using a composite signal (classifier confidence + tool-evidence + FAQ match), not raw model probability.
  • Flip high-confidence FAQ/simple cases to auto-send; everything below threshold stays shadow/escalated. Re-calibrate on drift.

Phase 4 — Optional screenshot + polish

Section titled “Phase 4 — Optional screenshot + polish”
  • Add a single client screenshot (start with native capture; html2canvas is unreliable for full fidelity) as a Plain attachment, masked. Richer snapshot. Address the dark-cockpit focus-ring follow-up from the design review.

7. Agent guardrails

Section titled “7. Agent guardrails”
  • Safe-action allowlist (auto, no approval): read-only investigation (get_order_email_log, list_failed_emails, get_audit_log, org:info, ops:status/logs), and Plain actions (replyToThread, add label/note, markThreadAsTodo/Done, assign).
  • Mandatory human approval (irreversible boundary): anything that mutates customer/order data or sends outbound email (e.g. resending order emails, order edits). The agent may recommend these; it never executes them autonomously. (Anthropic checkpoint guidance.)
  • Escalate, don’t guess: on low composite confidence, unresolvable-in-app, or an ambiguous request, assign to the founder and stop — no confidently-wrong auto-reply. (Anthropic trustworthy-agents + Trust-or-Escalate.)

8. Security / SOC2 (mid-Drata)

Section titled “8. Security / SOC2 (mid-Drata)”
  • Data minimization: ship only session_id + a minimized, client-redacted snapshot. Link the replay via a short-lived sharing access_token, never raw recording bytes.
  • Redaction in depth: client-side widget redaction → PostHog replay masking → server-side sanitizeErrorContext.
  • In transit / at rest: TLS in transit (existing); at rest the data lives in Plain/PostHog/Postgres (existing controls).
  • Drata: routing session identifiers (+ optional screenshot) into Plain is a new data flow — document the snapshot’s data-minimization standard and review whether it needs a Drata control / DPA addendum before auto-send.

9. Risks & caveats

Section titled “9. Risks & caveats”
  • Plain SDK churn (high): deprecated package in-repo; migrate in Phase 0 or all new code diverges.
  • PostHog footguns: get_session_id() undefined pre-init; custom network mask replaces the deny-list; server-side replay config can override SDK masking. Verify all three.
  • Calibration cold-start: the confidence guarantee assumes a calibration set matching live traffic — hence shadow-mode-first to collect it; degrades under drift.
  • “No new services” verified: triage runs on existing Temporal worker; webhook on existing webapp; Claude via existing packages/ai. Nothing new to provision.

10. Open decisions (recommendations)

Section titled “10. Open decisions (recommendations)”
  1. Plain SDK migration scope — Recommend: fold into Phase 0 (all new Plain code targets the new packages; the old wrapper would otherwise conflict). Alternative: separate follow-up ticket.
  2. Auto-send posture — Recommend: shadow/suggest-mode first, then calibrate to auto-send in Phase 3.
  3. Triage runtime — Recommend: Temporal workflow in ts-temporal-worker (matches the webapp’s background-job convention). Alternative: a CF Worker alongside ops-mcp.
  4. Screenshot — Recommend: defer to Phase 4; the PostHog replay covers most triage needs.

11. Citations (verified, 3-vote)

Section titled “11. Citations (verified, 3-vote)”