Ordermatic Metrics Plan

Getting dialed in: one north star, measured weekly, with the leading indicators that move it.

Status: Proposal / v1. Owner: David. Companion to the operational dashboards in docs/observability/. This is a business + product metrics plan — the reliability telemetry already in place is the floor it builds on, not the subject.

Inspiration: Tom Blomfield (YC), B2B Startup Metrics — https://youtu.be/_mKeVGSqQac. The doctrine below is lifted from that talk and applied to Ordermatic’s actual data.

Baseline: reflects the analytics surface on staging as of this writing — the PostHog taxonomy grew substantially in the last ~110 commits (org-group attribution, order_extracted, customer_in_uat, the client-side review-flow funnel, session replay), and the product went multi-ERP. Section 6 catalogs the events this plan actually keys on.

---

1. Why this doc exists

We measured a lot and knew almost nothing. Not a coverage problem — a focus problem. When this plan was written, six layers (OTel/SigNoz → Better Stack, PostHog server and client, Postgres aggregation services, the dbt gold layer, the on-prem “CUI firewall” telemetry, and the offline eval harness) each measured overlapping things — extraction success was computed three different ways — with no reconciliation and no single source of truth. There was no committed north star anywhere in the repo, no weekly growth-rate ritual, and no unit economics at all.

This document is the fix, and it is now largely implemented: one north star with one formula (§3), a metric tree whose inputs a team can actually move this week (§4), a leading-vs-lagging split so reviews manage drivers instead of spectating results (§5), the event taxonomy it keys on (§6), an honest built/open status (§7), and the cadence that makes any of it matter (§8).

Read §7 before proposing work. Read §8 before concluding the work is done — a dashboard nobody reviews weekly is an artifact, not a metrics practice.

---

2. The doctrine we’re adopting

From the talk, the rules we’re committing to:

1. Pick ONE key metric, and make it revenue. Teams optimize whatever you put on the wall (Goodhart’s law). Celebrate a proxy — order volume, GTV — and you get more of the proxy, not more of the business.
2. Measure it every week and plot the growth rate. YC’s bar: 5–7%/wk is good, 10%/wk is exceptional. Growth compounds — 5%/wk ≈ 12x/yr.
3. Know if you’re default alive. Net burn and runway, monthly. Does revenue at the current growth rate reach breakeven before the cash runs out?
4. Protect gross margin. Real SaaS is 80–90%+. Revenue that carries heavy variable cost (here: LLM inference + human review) is not software revenue. Measure it or fool yourself.
5. Separate leading from lagging. Revenue lags — you can’t act on it directly. Instrument the leading indicators that predict it and that a team can move this week.
6. Define everything precisely. A fuzzy metric is a gamed metric. Every number below has one formula and one source of truth (Appendix A).
7. Don’t lie to yourself. Metrics drive decisions and keep us in control — not board optics. Kill vanity metrics (Section 7), and always exclude internal/test accounts (is_test_account / isInternalEmail) — otherwise our own dogfooding inflates every number.

---

3. The North Star

Net Revenue — tracked weekly, growth rate on the wall.

Definition. Net recurring + usage revenue recognized in the period, net of credits and detected under-billing:

Net Revenue = Allocated subscription MRR (status active/past_due, live that week)
            + Usage revenue (Σ usage_events.calculated_charge,
                             status ∈ {pending, reported, invoiced})
            − open revenue_leaks (under-charges we haven't recovered)

(credits: none to subtract — free-trial consumption is already netted out of
 calculated_charge per order)

Source of truth: subscriptions, invoices, usage_events, revenue_leaks (Appendix A).

Why revenue, not “orders processed” or “line items”? Volume is a means. Optimize line-item count and we’ll happily onboard a customer who floods us with tiny, low-margin orders and call it a win. Revenue forces the whole funnel — acquisition, activation, adoption, expansion, retention, and price integrity — to actually pay off.

The trap: usage_events.order_total_amount is the GTV of orders flowing through Ordermatic, not our revenue. It’s a big satisfying number that rises with our customers’ business — exactly the proxy Blomfield warns against. Our revenue is calculated_charge. GTV is context, never the target.

Two surfaces, one definition. Revenue is computed in the warehouse (gold/fact_net_revenue, rendered at /metrics) and emitted per order as order_billed, so PostHog can slice it by the same org-group / ERP / cohort dimensions as the funnel. The two must agree: order_billed carries calculated_charge and billing_status, the same fields the model aggregates. If they ever diverge, that is a bug, not a nuance.

⚠️ billing_status includes pending on purpose. trackOrderBilled only ever emits pending; the transitions to reported / invoiced are plain DB updates with no emit. Filtering to reported+invoiced — which an earlier version of the board did — makes every revenue tile dead by construction.

Cadence & target.

Horizon	What we plot	Target
Weekly	Net new revenue, week over week	Growth rate ≥ 5%/wk (early); ≥ 10% exceptional
Monthly	Net Revenue (MRR run-rate) + NRR	Set each planning cycle; enterprise deals lump — read the trend

Enterprise B2B revenue is lumpy — one ERP connection going live can swamp a week. Track weekly, judge on a trailing-4-week and monthly trend. The weekly graph catches stalls early; it isn’t for celebrating spikes.

---

4. The metric tree

                          NET REVENUE (north star, weekly)
                                     │
   ┌──────────────┬──────────────────┼──────────────────┬──────────────────┐
   │              │                  │                  │                  │
 A. Growth     B. Retention      C. Unit           D. Product         E. Quality &
  & Revenue     & Cohorts        Economics         Activation &        Reliability
                                (Default Alive)    Engagement

The load-bearing chain:

More connections go live (D) → they route a higher share of their orders through us (D, “Ordermatic %”) → more billable line items (A) → at healthy gross margin because most orders are touchless (C, E) → and existing accounts expand faster than they churn (B) → Net Revenue compounds.

---

A. Growth & Revenue (the north star, decomposed)

Metric	Definition	Source	Lead/Lag	Target
Net Revenue	See §3	usage_events, subscriptions, invoices	Lag	Growth ≥ 5%/wk
New revenue	MRR from orgs that went live this period	subscriptions + first usage_events; customer_went_live	Lag	—
Expansion revenue	Δ from existing orgs processing more	usage_events by clerk_organization_id	Lag	> churn
Contraction / churned revenue	Lost MRR from downgrades + churned orgs	erp_connections.stage='churned', subscriptions.status	Lag	↓
New logos live	Count of customer_went_live	PostHog customer_went_live	Lead	—

B. Retention & Cohorts

The truest signal of product-market fit. The recent lifecycle work means the raw signals now exist (they didn’t at v0) — the job is to assemble them, not to instrument from scratch.

Metric	Definition	Source	Target
Net Revenue Retention (NRR)	(start MRR + expansion − contraction − churn) / start MRR, by monthly cohort	usage_events/invoices by org & month; PostHog org-group analytics	> 110%
Gross Revenue Retention (GRR)	NRR without expansion	same	> 90%
Logo retention / churn	orgs by stage transition; churned = stage='churned'	erp_connections.stage (onboarding/uat/live/churned)	churn < 5%/yr
Dormancy (early-warning)	live orgs with 0 billable orders trailing-7d	usage_daily_summary.orderCount	alert on hit
Cohort usage curve	line-items/org/week by go-live cohort	usage_daily_summary, PostHog cohorts	flatten high, not decay

stage='churned' records the fact but not the reason. Add a churn-reason so retention analysis can act (Section 8). Dormancy is the leading edge: usage-based customers go quiet before they cancel.

C. Unit Economics — are we default alive?

The biggest gap. For an LLM product, gross margin is a metric, not an assumption.

Metric	Definition	Source	Target
Gross margin per order	calculated_charge − COGS_per_order	Revenue: usage_events. COGS: new (below)	Trend toward 80%+
COGS per order	LLM inference + PDF/compute + human-review cost	Langfuse token costs; worker compute; review-effort proxy (§E)	↓ per order over time
Human-review cost/order	review-effort proxy × loaded CSR rate	review-flow events (§E) + work_items awaiting_review	↓ (touchless ↑)
Net burn / runway	opex − net revenue; cash / net burn	Finance	Runway > 12mo
Default alive?	Revenue at current growth reaches breakeven before cash out?	Model from above	Yes
CAC & payback	fully-loaded S&M ÷ new logos; months of gross profit to recover	Finance + customer_went_live	Payback < 12mo
LTV / CAC	(ARPA × gross margin) / churn ÷ CAC	Derived	> 3x

Touchless rate (§E) is the single biggest lever on gross margin. Every order a human touches carries labor cost; every touchless order is ~software-margin. Auto-submit quality is a financial metric, not just a product one — and slice it by erp_type, since a mature P21 connection and a new Acumatica one have very different touch rates.

D. Product — Activation & Engagement (leading indicators of revenue)

Metric	Definition	Source	Lead/Lag	Target
Ordermatic % (adoption)	orders via Ordermatic ÷ total booked in the ERP, per connection — per-ERP (§7)	daily report / ERP order views	Lead	80% → 90%
Activation funnel	customer_in_uat → customer_went_live → first order_processed	PostHog lifecycle events + erp_connections.stage	Lead	↑ conversion, ↓ time
Time-to-value	days signup → live; days_to_go_live	customer_went_live	Lead	↓
Billable line items / wk	volume driver of usage revenue	usage_daily_summary.lineItemCount	Lead	↑
Source mix	email / manual-upload / API	usage_daily_summary (ordersFrom*); new_order_started.source	Context	API/email ↑
Upload abandonment	new_order_upload_abandoned by step	PostHog client event	Lead (friction)	↓
CSR weekly active	CSRs actioning the inbox weekly	order_opened, inbox_filter_changed, work_items	Lead	↑

E. Quality & Reliability (now largely live, thanks to the review-flow events)

The client-side review funnel (C/D/E/F events) turned most of these from offline-only into computable-in-production. The job now is to roll them up.

Metric	Definition	Source	Target
Touchless rate	orders auto-validated clean AND submitted with zero manual searches/edits	derive from items_auto_validated (items_failed=0) + customer_selected.match_type IN ('auto','auto_shipto') + no *_search_performed + order_submit_to_erp_clicked	↑ (drives margin)
Review effort / order	# manual searches + validation errors + dwell (opened→submit)	customer_search_performed, shipping_search_performed, item_validation_error, order_opened→order_submit_to_erp_clicked	↓
Auto-submit escape rate	auto-submitted orders later corrected/rejected	extracted_orders ⋈ order_feedback/order_revisions	< 1%
Live extraction accuracy	field/line accuracy from CSR corrections in prod	order_feedback + extracted-vs-final; order_extracted confidence buckets	≥ eval baseline
Item validation errors	rate by error_type (PRICING_ERROR / NOT_FOUND / …)	item_validation_error	↓
Silent order loss	inbound orders dropped by triage; per-org triage health	email-triage / triage-health (#2470)	→ 0
Support friction	help opens + tickets by context/severity	help_opened, support_ticket_submitted	↓ (churn signal)
Order success rate	order_creation_completed{completed}/started	OTel order_creation_*_total	> 95%
P95 cycle time	received → submitted latency	stage timestamps (email/pdf/orders)	set SLA

---

5. Leading vs. lagging — the one table to internalize

Lagging (results — report them)	Leading (drivers — manage them)
Net Revenue, growth rate	Ordermatic % (adoption), per ERP
NRR / GRR	Activation funnel + time-to-value
Gross margin	Touchless rate ↔ review effort/order
Churned revenue	Dormancy alert; support friction
LTV/CAC	Billable line-items/wk, upload abandonment, CSR weekly-active

If a weekly review lives in the left column, we’re spectators. Spend it on the right — those are the numbers this week’s work can move.

---

6. The current event taxonomy (what this plan keys on)

Anchoring the plan to what is actually emitted today, so definitions above aren’t aspirational.

Server events (@repo/observability/posthog, org-group attributed): order_processed, order_extracted (temporal worker → Python analytics), integration_error, submission_partial_mismatch, workflow_already_started, p21_dedupe_failure, customer_in_uat, customer_went_live, quote_approved, email_received, inbox_item_rejected, $pageview, $feature_flag_called. Person props via identifyCustomer: company_name, erp_type, plan_mrr, environment, is_test_account.

Client events (apps/webapp/src/lib/analytics/events.ts, org auto-attached):

Group	Events	Powers
Review pipeline (C2/C3/C5)	order_opened, order_submit_to_erp_clicked, order_finish_clicked	touchless rate, dwell, CSR active
Customer lookup (D1)	customer_search_performed, customer_selected (auto/auto_shipto/manual), customer_search_failed	review effort, touchless
Shipping lookup (D2)	shipping_search_performed, shipping_selected, shipping_search_failed	review effort
Item validation (D3)	items_auto_validated, item_validation_error	touchless, live quality
Inbox + upload (E1–E3)	inbox_filter_changed, new_order_started, new_order_upload_abandoned	source mix, friction
Support (F1/F2)	help_opened, support_ticket_submitted	support friction, churn signal

Plus PostHog session replay (PII-masked, with support-ticket replay links) for qualitative review.

---

7. Status — what is built, what is left

Kept current deliberately: a gap list that still names shipped work is worse than no list, because it sends people to re-solve solved problems.

Built

Metric / signal	Where it lives
Net Revenue (usage + subscription MRR − open leaks)	gold/fact_net_revenue; /metrics panel
WoW growth rate	/metrics panel, derived from the same net-revenue expression
Subscription MRR	subscriptions.mrr_cents, persisted by the Stripe webhook (subscription-mrr.ts)
NRR / GRR / logo retention by cohort	gold/fact_cohort_retention
Expansion / contraction / churn / new	account_revenue_movement (PostHog), monthly Dagster job
Churn reason	erp_connections.churn_reason
Gross margin per order	gold/fact_gross_margin; /metrics panel; cost feed on a daily schedule
Ordermatic % (per ERP + blended)	gold/fact_erp_adoption → erp_adoption_weekly → two /metrics panels
Touchless rate / review effort	order_reviewed event (touched, manual_search_count, dwell_ms, items_failed)
Auto-submit escape rate	gold/fact_auto_submit_escape_rate; /metrics panel
Live extraction accuracy	gold/fact_extraction_accuracy; /metrics panel
Revenue on the funnel	order_billed event (+ line_item_count / calculated_charge on order_processed)
Activation funnel	customer_in_uat → customer_went_live → order_processed
Product-update email engagement	product_update_recipients / product_update_links; open pixel + click redirect
Two boards	/metrics in the internal console (warehouse-shaped) + the PostHog weekly board (event-shaped)

Still open

1. Dormancy alerting. usage_daily_summary has the data and account_revenue_movement catches monthly churn, but nothing fires when a live org goes quiet for 7 days — the earliest warning we have, and still uninstrumented.
2. Email-thread depth as effort. email_threads and the message-count badge exist; back-and-forth per order is a real friction/COGS signal that nothing rolls up.
3. Human-review cost in COGS. fact_gross_margin counts LLM tokens only. The review-effort proxy (order_reviewed.dwell_ms + manual_search_count) exists but is not priced into margin, so margin currently flatters us on every order a human touched.
4. Worker compute in COGS. Same model, same omission.
5. CAC / payback / LTV:CAC. Needs S&M spend, which lives outside this repo. Finance input, not instrumentation.
6. Net burn / runway / default-alive. Same — needs opex and cash, not code.
7. Credits. fact_net_revenue leaves credits NULL. Free-trial consumption is already netted out of calculated_charge per order, so there is nothing to subtract yet; revisit if we ever issue standalone credit grants.

Billing-model change in flight

We are moving off Stripe metered billing — usage will be computed here and Stripe will handle invoicing only. Two consequences for the definitions above:

• billing_status = 'reported' becomes vestigial. It meant “reported to Stripe as a usage record”. The revenue allowlists (pending, reported, invoiced) already tolerate this: trackOrderBilled only ever emits pending, so including pending is what makes the revenue tiles work at all, and reported simply stops occurring. Nothing needs to change to keep working — but do not “tidy up” by narrowing the allowlist to reported/invoiced, which would silently zero every revenue metric.
• invoices becomes more authoritative for recognized revenue, while usage_events.calculated_charge stays the per-order truth that feeds it. When self-invoicing lands, reconcile fact_net_revenue against invoices rather than replacing the usage side, and keep subscriptions.mrr_cents — MRR is unaffected by how usage is billed.
• stripeSubscriptionItemId / stripeUsageRecordId on usage_events become dead fields. Leave them until the migration is done; they are the audit trail for everything billed to date.

---

8. Cadence, not backlog

The build-out is largely done (§7); what determines whether any of it matters is the ritual.

• Weekly (30 min, same slot). One screen: the Net Revenue growth graph, then the five leading indicators — Ordermatic % (blended + worst ERP), activation funnel, touchless rate, billable line items, and support friction. Every number shows WoW change. Read left-to-right once, then spend the time on the right column.
• Check freshness before believing a panel. The board’s freshness row exists because two panels read scheduled feeds; a stale LLM-cost feed makes gross margin look better than it is. A number nobody sanity-checks is worse than a missing one.
• Monthly. NRR/GRR by cohort, gross margin, account_revenue_movement (who expanded, who contracted, who churned and why), CAC payback once finance data exists.
• Quarterly. Re-read §7. Move what shipped into Built, and delete anything nobody has looked at in a quarter — an unread metric is a maintenance cost, not an asset.

Success criterion, unchanged: anyone can answer “what’s our weekly revenue growth rate, and which two leading indicators are we moving to raise it?” from one screen, with definitions nobody disputes and internal accounts excluded.

---

Appendix A — Metric definitions & sources of truth

Metric	Formula	Source (table.column / event)
Net Revenue	subscription MRR + Σ usage calculated_charge (reported/invoiced) − credits − open leaks	subscriptions; usage_events.calculated_charge,.status; revenue_leaks
Usage revenue	Σ calculated_charge where status ∈ {reported, invoiced}	usage_events
GTV (context, not revenue)	Σ order_total_amount	usage_events.order_total_amount
Ordermatic % (per ERP)	orders via Ordermatic ÷ total booked, per connection	ERP order view (P21: p21_view_oe_hdr,entered_by); approval-only: finished ÷ inbound
Billable line items	Σ line_item_count (excl. free-trial)	usage_events; usage_daily_summary.billableLineItems
Touchless rate	auto-validated clean + submitted, zero manual touch	items_auto_validated(failed=0) + customer_selected.match_type IN ('auto','auto_shipto') + no *_search_performed + order_submit_to_erp_clicked
Review effort / order	manual searches + validation errors + dwell	customer_search_performed,shipping_search_performed,item_validation_error, order_opened→order_submit_to_erp_clicked
Escape rate	auto-submitted orders later corrected ÷ auto-submitted	extracted_orders ⋈ order_feedback/order_revisions
Live extraction accuracy	1 − (corrected fields ÷ extracted fields)	order_feedback, order_revisions; order_extracted
NRR (cohort)	(start MRR + expansion − contraction − churn) ÷ start MRR	usage_events/invoices by clerk_organization_id, month
Logo churn	orgs entering stage='churned' ÷ live orgs	erp_connections.stage
Dormancy	live orgs with 0 orders trailing-7d	usage_daily_summary.orderCount
Activation funnel	uat → live → first order conversion + time	customer_in_uat, customer_went_live, order_processed; erp_connections.stage
Order success rate	order_creation_completed{completed} ÷ order_creation_started	OTel order_creation_*_total
Gross margin/order	calculated_charge − COGS_per_order	usage_events + new COGS model

Hygiene: every business metric filters is_test_account = false and environment='production'.

Appendix B — Existing measurement surface (what we build on)

• Product analytics (server): packages/observability/src/posthog.ts (+ temporal-worker Python analytics for order_extracted); org-group attribution; is_test_account flagging.
• Product analytics (client): apps/webapp/src/lib/analytics/events.ts, posthog-browser.ts; session replay (PII-masked).
• Operational telemetry: packages/observability/ (OTel + Pino → GCP Cloud Logging / Better Stack), apps/webapp/src/lib/observability/node-metrics.ts; /observability/observability-index/.
• On-prem telemetry: packages/observability/src/onprem-telemetry.ts (content-free “CUI firewall” allowlist).
• Funnel / lifecycle: email_events, email_threads, pdf_documents + extracted_orders, orders, work_items, erp_connections.stage, pipeline_reports; packages/db/src/services/metrics.ts.
• Financial: usage_events, usage_daily_summary, subscriptions, invoices, billing, revenue_leaks, erp_pricing (gold price resolver); daily order report + end-of-month billing.
• Warehouse: dbt gold star schema in apps/dagster/dbt/models/ (dim_*, fact_order_lines, fact_ar_invoices, fact_ar_aging, fact_price_tier_variance).
• Quality (offline): packages/eval scorers + Langfuse; order_feedback for prod ground truth.
• Ops dashboards: apps/internal/src/data/boards/order-ops.json (rendered at /metrics in the internal console); tier3_order_metrics.