apps/internal → Internal Ops Console: Single-Pane-of-Glass Revamp

May contain stale references

This document was audited on 2026-06-22 and contains references that need updating: apps list is stale (email-api, fakerp, fakerp-site missing). Cross-check critical details against the current codebase before relying on them.

Status: Draft for /autoplan review (CEO + Design + Eng + DX) Author: David Boone (research-assisted; 13-agent cross-repo deep-dive, 2026-06-18) Date: 2026-06-18 Host app: apps/internal (Astro SSR + React islands, team.ordermatic.co, Cloudflare-Access @ordermatic.co gate, prod/staging cookie toggle) Upstream: erp-self-serve-onboarding.md · erp-onboarding-internal-spike-plan.md · ops-mcp-onboarding-assistive-layer-design.md

---

0. Decisions locked at intake (owner)

Three structural forks were resolved by the owner before authoring, because they shape the entire IA:

#	Decision	Choice	Consequence
L1	App boundary (internal vs webapp /admin)	Clean split	apps/internal = cross-customer operator console (CF-Access, multi-env). apps/webapp /admin+/sys-admin = single-org self-service (Clerk). Duplicated surfaces get one canonical home; the loser side is deleted.
L2	Primary persona	Operator-first + CSR read panes	Build the cross-customer operator console + per-customer single pane. Surface CSR-relevant state (failed emails, order status, short-supply) as read-only panes. CSR actions (send email, triage) stay in ops-mcp + Plain. No new RBAC roles.
L3	Trigger posture	Read-only + deeplink triggers	Embed read-only health everywhere. Every deploy/sync trigger is a deeplink to GitHub/Dagster. Prod promotion stays a deliberate GitHub action with its existing smoke gates. No infra write-buttons in internal. ERP connection credential config is not an infra trigger and stays in scope (server-side, Infisical/Nango).

A fourth (telemetry embed depth) is defaulted by the author and surfaced at the final gate: embed only cheap, API-portable health cards behind a URL-builder abstraction so the paused BetterStack→GCP cutover cannot strand the UI.

---

0.5 What changed after /autoplan review (read this first)

The plan was reviewed by 4 independent Claude voices + 4 Codex passes (CEO/Design/Eng/DX). Diagnosis confirmed; Phase 1 is unanimously ship-it. The one human gate — UC-1 — was resolved by the owner: ADOPT THE REFRAME. The original “hand-build all 5 phases in Astro, lead with a cross-customer aggregate board” framing is superseded by:

• L4 (owner, post-review): build-vs-buy + right-size. Ship Phase 1 standalone now. Run a 1-day Retool/Grafana/Metabase spike and buy the commodity read boards (order/health/audit — read-only over Postgres + a few APIs). Hand-build in Astro only the Ordermatic-native, CF-Access-gated, write-bearing views (per-org config/override, connection lifecycle, matching review). Reframe Phase 2 to the owner’s literal #1 question — an order/incident lookup (“where is this order stuck”) — and defer the cross-customer aggregate. Gate Phases 3-5 on a concrete trigger and re-justify each.

Revised phase map (governs §5-§9 below)

Phase	Status	Shape
1 — Fix what’s broken	SHIP NOW (standalone, ~1-2 wk)	All confirmed-bug fixes in §5 plus the cross-cutting Phase-1 items: thread getDb(env, locals.environment) everywhere + banned-import { db } lint + persistent prod/staging banner (T-D); replace db.transaction() on neon-http; repoint the OrganizationDetailTabs billing link before deleting billing-summary.astro; resolve the rbac.ts(bookkeeper)↔middleware(admin) default mismatch; remove the dead /notifications nav; relabel the fake Temporal tile → “Failed PDF extractions (24h)”; delete the internal dup shims + repoint nav.
1.5 — Spike + foundations	NEXT (~1-2 days)	Build-vs-buy spike: prototype the order + health boards in Retool/Grafana/Metabase against the read replica; decide buy-vs-build per surface. Ship the foundations needed either way: the T-A degraded-card state contract { ok | empty | stale | auth-missing | auth-failed | unreachable | error } (+ problem/cause/fix/who-fixes copy), the shared <EmbedCard> scaffold (T-A2), and the env-aware deeplink builder consolidating drill-urls.ts.
2 — Order/incident lookup (BUILD)	reframed	Hand-build (CF-gated, native, reuses in-repo logic): a global order/incident lookup by order#/PO/correlationId → per-org pre-filtered sub-route → integration-audit drill (server-side redaction of requestPayload/responsePayload, unattributed-failures view for null-org rows, (status,created_at)/(org_id,status,created_at) indices). Order board itself = single CTE / read-model anchored on work_items, ≤2 statements, p95-measured (T-F). Cross-customer health aggregate → BUY (Retool) or DEFER per the spike.
3 — External health (GATED)	split 3a/3b, non-blocking	Buy the generic infra board (Grafana/Retool) for commodity signal; build only Ordermatic-specific cards. 3a: GCP Cloud Run/Logging + GitHub Actions (P0). 3b: Dagster/Temporal/Langfuse/BetterStack. Each card behind a flag that degrades to a deeplink chip when its credential is absent + a “is-configured?” indicator. Gate on a trigger (e.g. ≥3 live customers, or owner opens ≥N external tabs ≥M×/day, measured).
4 — Config station (BUILD, GATED)	per L4	Hand-build (write-bearing, CF-gated): per-org Settings tab (billing-override = panel-only, API+audit already ship; fuzzy = adapt the Clerk-bound webapp section, not a drop-in), matching review as a sub-route /orgs/[slug]/matching.
5 — Onboarding/connection lifecycle (BUILD, GATED)	per L4	Hand-build: connection edit/rotate/delete + real Nango/MSSQL probes, consolidated provisioning, completing ENG-383.

IA correction (T-B): /orgs/[slug] stays a hub; heavy workflows become sub-routes (/orgs/[slug]/{orders,matching,email,settings,connections}), grouped Operations vs Configuration, lazy-loaded with per-pane error boundaries — not 12 flat tabs.

---

1. Problem

apps/internal is meant to be the internal dev/support/ops console. Today it fails the owner on three fronts, all confirmed by reading the code:

1. It’s broken. The migration from apps/webapp (PR #444 “Sys-Admin Migration”) was left half-done:

   • 5 of the 6 dashboard quick-action cards in index.astro link to /sys-admin/config|billing-summary|organizations|audit|pipeline-analysis|observability — no /sys-admin/* page routes exist in this app, so those five 404 (the 6th, Feature Flags, already points at the working /flags). The real pages live at the un-prefixed paths.
   • The Sidebar /notifications link is dead (no notifications.astro; only api/notifications/unread-count.ts + a badge component).
   • ~5 pages are dead shells whose API endpoints were left behind in apps/webapp: cache.astro (968 lines, every /api/cache/*, /api/erp/*, /api/maintenance/* fetch 404s), observability.astro (an OTel self-test that literally says “go check BetterStack”), and the write halves of pipeline-analysis.astro and email-aliases.astro (/api/admin/generate-pipeline-report, /api/admin/generate-aliases don’t exist here).
   • 3 duplicate route pairs: orgs↔organizations, flags↔feature-flags, billing↔billing-summary (the latter renders the same backend through two different BillingSummary components).
   • Real correctness bugs: customer-health/[orgId]/optimize/[jobId]/discard.ts + promote.ts query WHERE org_id=… but the column is clerk_organization_id → 500 on every call; the MCP-token create modal never collects/sends the required orgId → 400 on every UI create; services/reaper.ts “retries” promotions via Math.random()>0.2 instead of the real updatePromptLabel that exists; the Temporal-failures tile is a fake DB proxy counting failed pdfDocuments (its own footnote admits it); getDagsterSyncStatus returns a hardcoded 'synced'.

2. It doesn’t pull the operational record. The richest reads live in ops-mcp (agent-only) or aren’t surfaced anywhere as a human UI. orders, work_items, email_events, integration_audit_trail, extracted_order_items + cross_references, price_decisions all sit in our own Postgres that internal already queries via getDb(env), yet there is no order-pipeline board, no matching-review, no CSR email-ops view, no integration-audit viewer. The operator’s #1 question — “where is this order stuck and why?” — has no answer in the app.

3. It’s a weak config/onboarding station. The rich per-org organization_settings (8 JSON blobs incl. fuzzyMatchingConfig, syncConfig, alertingConfig, businessRulesConfig) is editable only by customers in the webapp. The /orgs/[slug] billing tab is a “coming soon” stub even though the internal override.ts (GET+PUT) + AdminService already implement the override writes with audit logging (so only the panel is missing). The prompt panel’s action buttons are dead. ERP connections are create+test only (no edit/rotate/delete; Nango-managed connections return fabricated success; the MSSQL test leg is intentionally deferred — it returns a non-gating “pending”, not a real failure).

Net: the owner still opens 6 external UIs (BetterStack, Langfuse, GCP Cloud Logging, GitHub Actions, Dagster, Temporal) to know what’s going on, because internal embeds none of them.

2. Goal

Turn apps/internal into the single pane of glass for the Ordermatic operator: it embeds health + recent signal + the operational record from our own data, owns per-customer config and onboarding end-to-end, and deeplinks the rich external timelines — collapsing the 6-tab habit into one console that answers “what needs attention now” and “where is this order/customer/sync stuck.”

---

3. Operating model & principles

1. Internal data = embed-first; external telemetry = deeplink-first. Our own Postgres (orders, work_items, email_events, integration_audit_trail, cross_references, audit_logs, organization_settings) is embedded as first-class read surfaces — internal already queries that DB. Third-party stores (BetterStack/Langfuse/Dagster/Temporal/GCP) get thin health/summary embeds + deeplinks, behind a small URL-builder abstraction so a backend cutover doesn’t strand the UI.
2. Health card + recent-signal + deeplink is the dominant pattern for all 6 external surfaces. Embed a small bounded health card (revision/uptime/run-status/freshness/accuracy) + a recent-signal strip (error tail / failing runs / recent traces / failed workflows); deeplink the rich interactive timeline. The embed handles the 90% glance; the deeplink handles the deep dive.
3. Operator-first; CSR state is read, CSR actions stay in ops-mcp/Plain (L2). Don’t rebuild send/triage/ticketing that already exists.
4. Two-pillar config model: global config (Flagsmith rollout %, system_config) stays in its existing, well-built, audited pages; per-org config (matching/sync/alerting settings, billing overrides, connections) consolidates into the per-org single pane. Don’t merge the two pillars.
5. The per-org detail page /orgs/[slug] is the single-pane spine. OrganizationDetailTabs already aggregates 7 tabs (overview, pipeline, prompt, billing, members, activity, connections). Fold order-pipeline, matching-review, email-ops, a Settings tab, and real connection lifecycle into it — and fill its stubs rather than building parallel pages.
6. Read-only beyond our DB (L3): no infra write-buttons; deeplink triggers; never expose tokens to the browser.
7. Auth-to-external-systems is solved server-side, per surface, once — each embed has exactly one credential/policy prerequisite (see §7).

---

4. What already exists (don’t rebuild)

Internal already owns the commercial/health slice well and these stay:

Surface	Where	State
Org list + detail (Clerk)	organizations.astro, orgs/[slug], OrganizationDetailTabs	Works
Billing summary (DB aggregation)	billing.astro, /api/billing-summary	Works
Feature flags (Flagsmith Admin API)	flags.astro, FeatureFlagDashboard, /api/admin/feature-flags	Works
System config	config.astro, /api/config/[category]	Works
User search + impersonation	users.astro, /api/users, /api/sys-admin/impersonate	Works
Audit log	audit.astro, /api/audit	Works (extend, §6 P2)
MCP token mgmt	mcp-tokens.astro, /api/mcp-tokens/*	Works (bugs to fix, §5)
Customer-health engine	services/org-health.ts (computeOrgHealth/getDriftSignal/MRR), revenue-leaks-utils.ts	Computes server-side; under-surfaced
Optimization-runner	@repo/optimization-runner + /api/customer-health/[orgId]/optimize	Works (route uses the package; local copy is dead, §5)

ERP connection onboarding is already in flight — the current ticket ENG-383 / PR #1663 shipped a Connections tab + Nango WhereFour flow into internal, and the reviewed spike plan already decided the direction: extract the tested webapp connection logic into a shared @repo/integrations module, mount in internal behind RBAC. This revamp completes that (edit/rotate/delete + real health probes), it does not restart it.

Langfuse is already hybrid (LangfusePanel embeds a prompt card + builds deeplink chips). The premise “Langfuse is deeplink-only” is wrong; we extend it (§6 P3), not start it.

---

5. Phase 1 — Fix what’s broken + IA cleanup (P0)

Goal: make every working page reachable and kill dead links/duplicates so the console is trustworthy before adding to it. Almost all small, UI-only, low-risk. This is the highest value-to-effort phase.

Item	Fix	Effort
Dashboard cards 404	Rewrite all index.astro hrefs /sys-admin/* → real routes (/config, /billing, /organizations, /audit, /pipeline-analysis, and the new health board). UI-only.	S
Dead /notifications	Build notifications.astro (API + badge already exist and imply intent) OR remove the nav entry. Repoint Sidebar /orgs→/organizations to drop the 301 hop.	S
discard.ts/promote.ts 500	org_id → clerk_organization_id in both raw queries (confirmed against schema + migrations 0063/0067); source prompt version from reportJson (no job.version column).	S
MCP create modal 400	Add an org selector + JSON-correct body (orgId); COUNT(*) for pagination total; drop the redundant plaintext accessToken column (hashedToken already exists and is what auth uses) + null-backfill. NB the '${token.id}' onclick is normal Astro templating, not a bug (Codex caught this).	S
db.transaction() on neon-http	connections/index.ts, connections/finalize.ts, and mcp-tokens/[id]/rotate.ts all call db.transaction(), unsupported on the neon-http driver (PR #1595 proved it 500s; metrics.ts documents it). Replace with sequential statements or a single CTE. This is a latent 500 in the already-shipped connection write path, not a new risk.	S–M
reaper.ts random-number mock	Wire the real updatePromptLabel from lib/langfuse/client.ts (the reaper “retries” promotions with a pseudo-random simulate); find/add the scheduler that actually runs runReapers (none found).	S
Duplicate pairs	Delete billing-summary.astro (keep Sidebar-linked /billing, standardize on one BillingSummary). Keep orgs/feature-flags/organizations/[orgId] as harmless 301 aliases or delete; repoint nav to canonical targets either way. Document the decision.	S
Dead-shell endpoints	Per page: port the route handler from webapp or delete. Recommend: delete observability.astro (rebuilt as the GCP card, §6 P3) and the cache.astro shell unless internal still needs that tooling; port generate-pipeline-report + generate-aliases to unblock pipeline-analysis/email-aliases; add the Redis-DEL /api/admin/unblock-ip route or remove the rate-limiting Unblock button.	S–M
OrganizationPromptSection dead state	API hardcodes optimizationJobs: [] → permanently “No runs yet” + dead buttons + a link to a missing /admin/override-cooldown. Feed real optimization-run history or present read-only and remove dead links.	M
Orphaned-but-working pages	After link/endpoint fixes, surface mcp-tokens, config, audit in the Sidebar/dashboard so they’re discoverable (today reachable only by typed URL).	S
Dead local optimization-runner.ts	Delete the internal copy (the API route uses the @repo/optimization-runner package; the local spawnOptimization is an orphaned parallel signature that will rot).	S
email-aliases raw IDs	Resolve clerkOrganizationId → org name from Clerk.	S

---

6. Phases 2–5 — Embeds, external health, config, onboarding

Phase 2 — Highest-value embeds: the operator single pane (P0)

The order-pipeline view and cross-customer health board don’t exist anywhere today.

• Order-pipeline status board (new page + per-org tab). Per-order lifecycle stage (received→extracted→validated→submitted→processed), status, PO/customer, ERP order #, last error; filter status=error/quote_failed. Direct @repo/db Drizzle reads (extractedOrders, workItems, orders, integrationAuditTrail) via existing getDb(env); org-scoped indexes exist (validate heavy joins on neon-http). The spine of the operator console. Effort L.
• Cross-customer health board = the new index, replacing the all-404 dashboard. Orgs by health status, at-risk MRR, 24h failed-extraction count, pending short-supply, drift signals. Built from org-health.ts (fetchOrgHealthMetricsByOrgIds + getDriftSignal) + revenue-leaks-utils — all already computed in-repo. Effort M.
• ERP integration-audit viewer (integration_audit_trail: status=FAILURE/RETRY, endpoint, errorMessage, payloads, durationMs), linkable from the order board. The authoritative “why did submission fail” log, completely unsurfaced today. Effort M.
• Enrich per-org overview “recent submissions” to join extracted_orders (PO/customer/ERP#) instead of bare PDF filenames. Effort S.
• Short-supply / work-item triage queue read pane (work_items where status in (new,in_progress) + assignee + age). The query already exists in ops-mcp/list-pending-short-supply.ts; embed the read, actions stay in ops-mcp. Effort S.

Phase 3 — External health cards + recent-signal (the 6 UIs) (P0/P1)

Uniform hybrid verdict. Each is a small bounded embed + deeplink; each has one server-side auth prerequisite (§7). All deeplinks go through a single URL-builder module.

Surface	Embed	Deeplink	Priority
GCP Cloud Run / Logging	Per-service revision/deploy-time/traffic%/crash-loop flag for the 4 services (webapp/internal × staging/prod) + tail of last ~20 severity≥ERROR /1h	Logs Explorer (pre-filtered query)	P0 — the one signal BetterStack can’t show (container dies before OTel inits; internal ships no OTel)
GitHub Actions	Last GCP-deploy run per env (status/actor/commit/duration, in_progress=deploying) + recent failing runs (24–48h) + staging-ahead-of-main promotion state	Run page / per-job logs	P0
Dagster	runsOrError grouped by job+connection_id + per-asset freshness vs 36h SLA + sensor/schedule RUNNING/STOPPED	Run timeline / asset history	P1
Temporal	Counts Running / Failed(24h) / Stuck(>30m) per env+task-queue + lookup-by-order-id status (replaces the fake DB-proxy tile)	{workflowId}/{runId}/history	P1
Langfuse	Make accuracy org-scoped (today a global last-100 avg) + last ~10 extraction traces + 7d ERROR strip; keep prompt card	Trace timeline / evals / prompts	P1
BetterStack	is-prod-up uptime pill + request-volume/latency sparkline + per-service top-error-pattern card	Logs Explorer / dashboards (passing correlationId)	P2 (mid-migration → stay deeplink-heavy)

Phase 4 — Config station: own per-org config + matching review (P1)

End the scatter to the customer webapp and DB scripts.

• Per-org Settings tab on /orgs/[slug]: read + operator-override of organization_settings — fuzzyMatchingConfig first (prefixRules/thresholds/nameOverlap; the deliberate precision/abstention substrate), then sync/alerting/businessRules. Adapt the webapp FuzzyMatchingSection.tsx + organization-settings-service.ts — these are Clerk-context-bound, not a drop-in; extract into @repo/ui / off Clerk context first. Effort L.
• Per-org billing-override panel replacing the “coming soon” stub. The internal override.ts (GET+PUT) + AdminService.updateOrganizationOverride/addFreeTrialLineItems already exist and write with audit logging — only the React panel remains. Cheapest config win. Effort M (panel-only).
• Extraction & matching review: per-line matchConfidence/matchedProductName/erpPriceSource/extracted-vs-resolved price/validationError + xref mappings (manual/erp_cross_references source/confidence/verifiedBy) + price_decisions. Where wrong-match/wrong-price tickets get diagnosed. Effort L.
• CSR email-ops read panes: failed email_events (processingError/filterReason/triageReason), per-order email log (received/processed_email_sent + BCC), email_pull_connections health. Reuse the ops-mcp query logic against @repo/db; actions stay in ops-mcp. Effort M.
• Reconcile the 3 parallel per-org feature-flag mechanisms (Flagsmith segments vs organization_overrides.featureFlags vs organization_settings.featureFlags) — pick one source of truth, document. Effort M.

Phase 5 — Onboarding station + connection lifecycle (P1)

The “weak onboarding station” fix, completing ENG-383.

• ERP connection edit/rotate/delete + JSONB-config editors + real Nango/MSSQL health probes (today create+test only; Nango health faked; MSSQL leg hardcoded fail). Uses @repo/integrations CredentialService, testERPConnection/testEclipseE4Soap, Nango connection-get (NANGO_SECRET_KEY present). Effort L.
• Consolidate the split provision-wizard (ProvisionStep1/2 in internal vs a webapp /orgs/new placeholder) into one onboarding flow wired to Nango + the connections endpoints. Effort M.
• Connection sync-cursor health (erp_sync_cursors) + email-pull/forwarding-setup status on the connections tab. Effort M.

---

7. Auth prerequisites (the per-embed gating dependency)

Each external embed needs exactly one server-side credential/policy solved once; tokens never reach the browser:

• GCP — grant the internal service account roles/run.viewer + roles/logging.viewer via ADC (2-line terraform). Cloud Run Admin API v2 + Cloud Logging entries:list (@google-cloud/run, @google-cloud/logging).
• GitHub — fine-grained token: actions:read + contents:read for the board (no write needed under L3). Octokit actions.listWorkflowRuns. Derive deploy state from gcp-deploy.yml runs — the Deployments API is useless (stale Vercel entries).
• Dagster — CF Access service-token Include policy + set DAGSTER_GRAPHQL_URL / CF_ACCESS_CLIENT_ID / CF_ACCESS_CLIENT_SECRET (currently unset → the existing write path is broken). GraphQL runsOrError/assetNodes/sensorsOrError.
• Temporal — @temporalio/client Visibility API (countWorkflowExecutions/workflow.list); per-order via getHandle(id).describe() (already used in /api/temporal/workflow/[id].ts). Org-scoping needs a Search Attribute or workflowId-pattern derivation.
• Langfuse — langfuse SDK (already a dep): traceList({userId:orgId}) + scoreGet; worker traces are org-tagged (user_id=organization_id). Extend the existing /api/organizations/[orgId]/langfuse route.
• BetterStack — telemetry_query (GROUP BY service/_pattern) + Uptime REST (/api/v2/monitors, monitor 3353405); cred via 1Password “Production Observability”.

---

8. Architecture

apps/internal (Astro SSR + React islands · CF-Access @ordermatic.co · prod/staging cookie)
│
├─ NEW index = Cross-customer health board ──────── org-health.ts + revenue-leaks (our DB)
│
├─ /orgs/[slug]  (OrganizationDetailTabs — THE single-pane spine)
│     ├─ overview (enrich) · pipeline · prompt(fix) · members · activity
│     ├─ + Order pipeline      ┐
│     ├─ + Matching review     │  EMBED  ← direct @repo/db reads via getDb(env)
│     ├─ + Email-ops (read)    │         (orders, work_items, extracted_order_items,
│     ├─ + Settings (override) │          cross_references, integration_audit_trail,
│     ├─ + Billing (fill stub) ┘          organization_settings, email_events)
│     └─ + Connections (lifecycle: edit/rotate/delete + real probe) ← @repo/integrations
│
├─ Infra board ── HYBRID health cards (server-side fetch) + DEEPLINKS:
│     GCP Run/Logs · GitHub Actions · Dagster · Temporal · Langfuse · BetterStack
│     └─ all deeplinks via one url-builder module (swaps at BetterStack→GCP cutover)
│
└─ Global config (unchanged): Flagsmith flags · system_config · MCP tokens · audit

• Server-side fetch pattern: every external embed is fetched in the Astro SSR layer (or a thin /api route) using a server-only credential; the browser only ever sees rendered health + a deeplink URL.
• URL-builder abstraction: one module owns every external deeplink so the BetterStack→GCP cutover is a one-file change.
• No new infra: all internal embeds reuse getDb(env); external embeds add only server-side API clients + the §7 grants.

9. Scope & non-goals

In scope: Phases 1–5 above; the clean-split deletions (L1); read-only external embeds + deeplinks (L3); per-org config consolidation; completing connection lifecycle/onboarding.

Non-goals (named):

• CSR roles/ticketing/send-actions — stay in ops-mcp + Plain (L2). Internal shows CSR state read-only.
• Infra write-buttons / prod promotion from internal (L3) — deeplink only.
• Webapp /admin single-org self-service — out of this app’s scope (L1); we only delete what internal canonically owns.
• Embedding full external log/trace search — deeplink (BetterStack/GCP Logs Explorer, Langfuse timeline, Dagster event log, Temporal history, GH run logs).
• Rich BetterStack embeds — mid-migration; cheap portable cards only.
• New ERP connector read/write depth (SuiteQL/SuiteTalk/BC/Epicor) — separate track per erp-self-serve-onboarding.md.

10. Risks & open questions

1. neon-http heavy joins — the order-pipeline board joins 4 tables; validate query cost on the serverless driver (PR #1595 already had to drop db.transaction on neon-http). Mitigation: org-scoped, indexed, paginated reads; consider a read view.
2. External-cred sprawl — 6 surfaces × server-side creds. Mitigation: one secrets-loading convention (Infisical entrypoint), per-surface client module, no secret in logs (assert in tests).
3. prod/staging toggle correctness — every new embed/read must honor the erp-internal-env cookie (right DB + right external endpoint). Mitigation: thread env through every fetch; test both.
4. Telemetry embed durability (defaulted, gate item) — BetterStack→GCP cutover is paused/trigger-gated on the Smith Corona POC; richer BetterStack embeds risk rework. Default = portable health cards behind the URL-builder only.
5. Clean-split deletions — deleting duplicated surfaces from webapp /admin touches another app; sequence carefully and confirm nothing customer-facing depends on them before removal.
6. Coarse internal RBAC — every @ordermatic.co user defaults to admin; the real boundary is CF Access. New read panes are fine; the (few) write paths (settings override, billing override, connection edit) get an explicit confirm step + audit.

11. Test & dogfood plan

• Unit: the org_id→clerk_organization_id fix (regression test the failing query); MCP-modal body schema; reaper updatePromptLabel wiring; url-builder per surface; settings/billing-override authz + audit-write assertions; no-secret-in-logs assertion for every external client.
• Integration: order-pipeline board query against a seeded multi-stage dataset (incl. error/stuck rows); each external health card against a mocked client (success + auth-fail + empty); env-toggle correctness (staging vs prod) on at least the order board + one external card.
• Dogfood: operator opens the new index → spots an at-risk org → opens /orgs/[slug] → sees a stuck order on the pipeline tab → opens the integration-audit row → reads the ERP failure → checks the Dagster freshness card → deeplinks to the run. Zero external tabs for the glance; deeplink only for the deep dive.
• Rollback: Phase 1 is additive/corrective (link + bug fixes). Each embed is an independent pane behind nothing customer-facing; disable the pane to revert.

---

/autoplan REVIEW REPORT

Mode: SELECTIVE EXPANSION · Voices: Claude subagents (CEO/Design/Eng/DX, independent) + Codex gpt (codex-cli 0.139.0) per lens — true dual-voice. · UI scope: yes · DX scope: yes · Date: 2026-06-18.

Both models read the actual code. Every factual bug claim in §1 was independently verified by both (the 5/6 /sys-admin 404s, the org_id/clerk_organization_id 500, the promote.ts job.version, the Math-random reaper, the fake getDagsterSyncStatus/Temporal tiles, the dead-shell endpoints, the db.transaction()-on-neon-http hazard). Codex refuted exactly one claim (the '${token.id}' onclick is normal Astro templating) — corrected above.

Premise gate (locked at intake by owner)

L1 clean split · L2 operator-first + CSR-read · L3 read-only triggers — confirmed, stand. Recorded dissent (Codex CEO): challenges whether internal should be a 4th surface vs putting operator context into webapp /inbox + Plain where work already happens, and whether read-only preserves a “handoff tax.” Claude CEO accepted L1/L2/L3 as sound. Not a both-models user-challenge → owner’s locks stand; dissent surfaced at the gate for awareness.

Consensus tables (Claude voice × Codex voice)

CEO

Dimension	Claude	Codex	Consensus
Premises valid (it’s broken)	sound	confirmed	CONFIRMED — diagnosis is right
Right problem	concern	critical (reframe)	DISAGREE→reframe — both: cross-customer framing is premature at ~1 live customer
Scope calibration	concern	critical (“scope control is fiction”)	CONFIRMED concern — a quarter-plus arc sold as one doc
Alternatives explored	gap (no build-vs-buy)	high (buy the commodity board)	CONFIRMED gap → User Challenge UC-1
Competitive risk	sound	—	CONFIRMED (internal tooling, low external risk; opportunity-cost is the lever)
6-month trajectory	concern	”Phase 3 looks naive in 6mo”	CONFIRMED — regret clusters in Phases 3 + L1 deletions

Design

Dimension	Claude	Codex	Consensus
Information hierarchy	sound	High (index/infra split muddy)	CONFIRMED-with-refinement (index = triage queue incl. active incidents; infra board = diagnostics)
Missing states	gap (critical)	High (state model unspecified)	CONFIRMED CRITICAL → theme T-A
External-API-down vs empty	critical	Medium	CONFIRMED CRITICAL — a down source must not render healthy → T-A
User journey / specificity	concern	Medium (order status model)	CONFIRMED — needs UI spec + canonical row-state
DS alignment + a11y	concern + gap	(Low)	CONFIRMED — extend @repo/ui, --om-* tokens, no color-only status
Spine shape (tabs)	High (mobile-hostile)	High (junk-drawer)	CONFIRMED → theme T-B (sub-routes, not more tabs)

Eng

Dimension	Claude	Codex	Consensus
Architecture (spine)	concern (overload)	finding #1 (overload)	CONFIRMED → T-B
Test coverage	gap	(implied)	CONFIRMED — add island-matrix, env-thread, redaction tests
Performance (neon-http)	concern (real)	#2/#3 (indices missing + db.transaction() used today)	CONFIRMED → T-F (single CTE/read-model on work_items; replace db.transaction())
Security	gap critical (payload redaction; plaintext token)	#5 (error contract)	CONFIRMED CRITICAL → E1 redaction + E6 token
Error paths	concern	#5 (Langfuse swallows→lies)	CONFIRMED → T-A contract
Deployment risk	concern (delete collision, creds)	#4 (webapp richer; fuzzy Clerk-bound; delete-then-port regresses)	CONFIRMED → alias-first, webapp deletes staged last

DX

Dimension	Claude	Codex	Consensus
Time-to-answer	sound	#1 (multi-hop; need incident lookup)	CONFIRMED-with-gap — add a global order/incident-lookup spine
Auth-setup friction	gap	#2 (not operator-visible)	CONFIRMED → setup/“is-configured?” indicator
Error messaging	gap	#3 (non-actionable)	CONFIRMED CRITICAL → T-A (problem+cause+fix+who-fixes)
Impl consistency	sound (if scaffold)	#4 (no shared card schema)	CONFIRMED → T-A2 shared <EmbedCard> scaffold before Phase 3
Discoverability	concern	#5 (overloaded)	CONFIRMED → reconcile the two nav systems
Cutover path	sound	—	CONFIRMED — but make the url-builder env-aware too (T-D)

Cross-phase themes (flagged independently in 2+ phases → high-confidence)

• T-A — Silent-failure / degraded-card contract (CRITICAL; CEO-adjacent, Design D1/D2, Eng E8/#5, DX DX-1/#3). A down or un-provisioned external source must never render as healthy/empty. Mandate a shared state machine { ok | empty(reachable, no data) | stale(>SLA) | auth-missing | auth-failed | unreachable | error }, each with problem + cause + fix + who-can-fix copy and a last-fetched stamp, asserted in tests. This is the single highest-leverage fix; it is what makes the console trustworthy.
• T-A2 — Shared <EmbedCard> scaffold + env-aware deeplink builder, before any Phase-3 surface (Design, DX-2, Codex DX #4). One component owns the T-A state machine + header/freshness/deeplink slot; consolidate drill-urls.ts. “New surface = one fetcher + one config object.”
• T-B — Don’t overload OrganizationDetailTabs; sub-routes not more tabs (Design D3/Codex, Eng E3/#1, DX/#5). Keep /orgs/[slug] as a hub; split heavy workflows to /orgs/[slug]/{orders,matching,email,settings,connections}; group Operations vs Configuration; lazy-load islands + per-pane error boundary.
• T-C — Build-vs-buy + Phase-3-is-platform-work (CEO-1/Codex, Eng E2, Codex DX #4). The commodity read boards are the textbook Retool/Grafana/Metabase use case → User Challenge UC-1; the 6 external embeds are 4 net-new clients + 4 cred grants, not “no new infra” → split Phase 3 into 3a (GCP+GitHub) / 3b (rest), make it non-blocking for Phases 1/2/4/5, each card behind a flag that degrades to a deeplink chip when its cred is absent.
• T-D — Env-awareness leaks (Eng E7, DX DX-3). getDb(env, locals.environment) everywhere (3 existing pages read raw prod db); thread environment through the deeplink builder; banned-import { db } lint; persistent prod/staging banner.
• T-E — Nullable org column hides the rows operators hunt (Design D4, DX DX-4, Codex Eng #2). integration_audit_trail.organizationId + extractedOrders.clerkOrganizationId are nullable/needs_backfill; org-scoped reads silently drop pre-attribution failures. Add an unattributed-failures view (org IS NULL) on the index + a per-org “view unattributed” affordance; add (status, created_at) / (organization_id, status, created_at) indices before the audit viewer.

Decision Audit Trail

#	Phase	Decision	Class	Principle	Note
D1	Eng	T-A degraded-card state machine + copy contract, asserted in tests	Mechanical	P1	The product IS the error states
D2	DX	T-A2 shared <EmbedCard> + env-aware deeplink builder before Phase 3	Mechanical	P5	One scaffold, six surfaces
D3	Eng/Design	T-B: heavy workflows → sub-routes; lazy-load + per-tab error boundary; Operations vs Configuration grouping	Mechanical	P5	Cohesion over god-component
D4	Eng	E1 server-side payload redaction (recursive key-denylist + truncate + gated full view + test)	Mechanical	P1	Least-exposure; creds/PII in JSONB
D5	Eng	E6 drop redundant plaintext accessToken col (+null-backfill); E-new replace db.transaction() on neon-http	Mechanical	P1	Latent 500 in shipped path
D6	Eng	T-F: order board = single CTE / read-model anchored on work_items; ≤2 statements; p95 before committing L	Mechanical	P1	Query count = latency budget
D7	Eng/Design	T-E unattributed-failures view + composite indices	Mechanical	P1	Don’t drop the hunted rows
D8	Eng/DX	T-D getDb(env) everywhere + env deeplinks + banned-import lint + env banner; fix 3 raw-db pages in Phase 1	Mechanical	P5	Wrong-env answer looks correct
D9	Eng/CEO	T-C: split Phase 3 → 3a (GCP+GitHub P0) / 3b (rest); non-blocking for 1/2/4/5; flag-degrade to deeplink chip	Mechanical	P3	Don’t gate cheap DB work on cred-heavy work
D10	DX	Setup/“is-configured?” indicator per external surface + missing-grant CTA	Mechanical	P1	Friction invisible until a card is empty
D11	DX/Eng	Add a global order/incident-lookup spine (by order#/PO/correlationId) as a first-class entry	Mechanical	P1	The owner’s literal #1 question
D12	Design	UI spec (columns/sort/filters/states) + canonical row-state for the 3 spine surfaces before eng	Mechanical	P1	A design doc must be buildable
D13	Design	Desktop-first declared; remove half-built mobile fallback on triage panes	Taste→auto	P5	Posture is a decision, not inertia (TD5)
D14	Design/DX	DS: --om-* tokens, extract Table/Badge/StatusPill/EmptyState/Skeleton into @repo/ui; a11y (no color-only status, focus-trap modals not alert())	Mechanical	P5	11 panes must read as one console
D15	CEO/Eng	Duplicate routes: delete internal shims + repoint nav now; webapp clean-split deletions staged LAST behind grep+log-soak (alias-first)	Taste→auto	P2/P6	Deletion across an app boundary is one-way (TD2)
D16	Eng	E5 repoint OrganizationDetailTabs billing link before deleting billing-summary.astro (one PR)	Mechanical	P5	Atomic refactor
D17	CEO/Design	BetterStack: uptime pill + deeplink only; drop richer embeds until cutover fires	Taste→auto	P3	Don’t build on a substrate you’re replacing (TD1)
D18	Eng	Relabel the fake Temporal tile → “Failed PDF extractions (24h)” now; real Temporal embed in 3b	Taste→auto	P5	Kill the lie cheaply (TD3)
D19	DX	Remove dead /notifications nav now; build the page only when the reaper emits notifications	Taste→auto	P3/P5	Avoid an always-empty “healthy” page (TD4)
D20	DX/CEO	Pin each new pane’s role gate explicitly; resolve rbac.ts(bookkeeper) vs middleware(admin) default mismatch in Phase 1; writes call isInternalAdminRole + audit	Mechanical	P1	One meaning for “default role”
UC-1	CEO	Build-vs-buy + scope/sequencing reframe	User Challenge	—	Owner decides at gate

Required artifacts (condensed)

Architecture (Eng): the /orgs/[slug] spine + getDb(env) injection is sound, but OrganizationDetailTabs is one 200-line client:load island (one fetch per tab on mount, only overview on mobile). Folding 5 heavy panes in = a brittle island. → hub + lazy sub-routes (T-B). External embeds must be client:visible islands fed by thin /api routes (not blocking SSR) with hard per-fetch timeouts (E8).

Test diagram (Eng): new codepaths → coverage. order-board query → integration test asserting ≤2 statements against a seeded multi-stage+stuck+null-org dataset (gap today); each external card → mocked client across the full T-A state set incl. 429/timeout (gap); getDb(env) threading → banned-import lint + an env-toggle test (gap); audit payload → redaction unit test asserting denylisted keys never serialize (gap); org_id→clerk_organization_id → regression test (gap).

Failure-modes registry: ① external API down renders healthy → T-A (CRITICAL). ② un-timeout’d SSR fetch hangs the page → islands+timeout. ③ org-scoped read drops null-org failures → T-E. ④ wrong-env answer looks correct → T-D. ⑤ raw ERP payloads leak creds/PII to browser → E1. ⑥ db.transaction() 500s on neon-http → replace. ⑦ deleting billing-summary.astro 404s the live stub link → repoint first.

DX journey + TTHW: today, “where is this order stuck” → no surface → operator drops to raw SQL / the customer webapp (TTHW effectively unbounded). Target: global lookup or index → pre-filtered per-org sub-route → integration-audit row → ERP failure → deeplink, under 5 clicks, provided the degraded-card contract (T-A) holds — otherwise an empty card reads as “healthy” and the operator reverts to 6 tabs (the exact outcome the revamp must prevent).

NOT in scope (confirmed/added)

CSR roles/ticketing/send-actions (ops-mcp + Plain, L2) · infra write-buttons / prod promotion (L3) · webapp /admin single-org self-service (L1) · full external log/trace search (deeplink) · rich BetterStack embeds (mid-cutover) · new ERP connector read/write depth · [added] the cross-customer health aggregate board is deferred behind UC-1 until customer count makes it non-trivial; the cheap order/incident-lookup spine is the Phase-2 entry instead.

Taste decisions (auto-decided; recommendations stand unless overridden)

TD1 telemetry depth → portable cards only (default). TD2 dup routes → internal delete now / webapp delete last, alias-first. TD3 Temporal tile → relabel now. TD4 notifications → remove dead nav now. TD5 mobile → desktop-first, remove broken fallback. TD6 matching-review → sub-route (folds into T-B).

User Challenge (the one human gate beyond the locked premises)

UC-1 — Build-vs-buy + right-size the scope. → RESOLVED 2026-06-18: owner ADOPTED THE REFRAME (recorded as L4 in §0.5; governs the revised phase map). Original framing superseded. Both models, independently:

• You said: turn apps/internal into the single pane of glass by hand-building boards/cards/viewers in Astro across 5 phases, framed as one initiative.
• Both models recommend: ship Phase 1 now as a standalone deliverable; run a 1-day Retool/Grafana/Metabase spike on the order/health boards and buy the commodity read surfaces, hand-building in Astro only the Ordermatic-native, CF-Access-gated, write-bearing views (config/override, connection lifecycle, matching review); reframe Phase 2 around the owner’s literal #1 question (“where is this order stuck” — an order/incident lookup) and defer the cross-customer aggregate until customer count grows; gate Phases 2-5 behind a concrete trigger and re-justify each.
• Why: the commodity boards are the textbook internal-tool-builder use case; at ~1 fully-live customer a cross-customer aggregate is near-empty; Phase 3 is platform work (4 clients + 4 cred grants), not “no new infra”; every week here is a week off the Q2C/matching/onboarding product that wins customers.
• What we might be missing: one codebase + one CF-Access gate + reuse of in-repo logic is a real engineering value; the owner may have near-term customer-count growth or a deliberate “own the operator surface” bet that the models can’t see.
• If we’re wrong (you build it all in Astro anyway): weeks of build + permanent maintenance for surfaces a tool would cover in days, plus the neon-http join-cost risk a managed connector sidesteps — opportunity cost, not a crash.
• Default = your original direction. The models must earn the change.

End of /autoplan review report.