Initiative: BetterStack → Cloud Logging cutover (post-GCP migration)

Status: PROPOSED — future work, not actively executing Trigger to start: full GCP migration of app.erpunlocked.com complete AND Smith Corona POC stable in production Builds on: PR #839 (OTel trace correlation on PostHog events) — the cross-tool incident-response flow that PR enables works with any OTel-compatible destination, including Cloud Trace + Cloud Logging Owner: David Last updated: 2026-05-08

Problem Statement

Once the app is fully on GCP, BetterStack/Logtail’s role narrows to features GCP Cloud Operations covers natively at lower or zero incremental cost:

App log ingest — covered by Cloud Logging (50 GB/month + 30-day retention free; $0.50/GB beyond).
Uptime monitoring — covered by Cloud Monitoring uptime checks.
Trace correlation with logs — Cloud Logging auto-links log records that carry the logging.googleapis.com/trace field to Cloud Trace spans. Our Pino logHook already injects trace_id / span_id (see otel-config.ts:53-60).

The only BetterStack-only capabilities are status pages and a few uptime-flow conveniences. Neither is a customer commitment today.

Continuing to pay for BetterStack post-migration is unjustified vendor cost AND adds a surface engineers have to remember exists during incident response. PR #839’s $otel_trace_id stamp on PostHog events becomes a Cloud Trace ID for free post-migration, so the cognitive-load gain is real.

Demand Evidence

BetterStack subscription cost: not yet quantified — first task in §The Assignment.
Cloud Logging pricing (verified 2026-05-08 GA): 50 GB/month ingestion free, $0.50/GB after, 30 days retention included.
Google Cloud Telemetry (OTLP) API (the ingest endpoint we’d use): single endpoint at https://telemetry.googleapis.com accepts traces + metrics + logs as standard OTLP. No GCP-specific exporter needed; our existing @opentelemetry/exporter-logs-otlp-http works.
Hetzner→GCP Phase 1 cutover landed 2026-05-02: staging.ordermatic.co and app.ordermatic.co already serve from GCP Cloud Run with Google Trust Services certs. app.erpunlocked.com remains on Hetzner pending Phase 2.
Smith Corona POC: no contractual obligation to BetterStack-specific features (per project_smith_kroy_poc.md).

Status Quo

                ┌─────────────────────────────────────────────┐
                │              ERP Unlocked apps              │
                │  (webapp, ts-temporal-worker, email-api,    │
                │   email-worker, temporal-worker [Python])   │
                └──────────┬──────────────────────────────────┘
                           │
                  ┌────────┴────────┐
                  │   Pino logger    │ packages/observability/src/logger.ts
                  └────────┬────────┘
                           │
              ┌────────────┴────────────────┐
              │                              │
              ▼                              ▼
   ┌────────────────────┐        ┌──────────────────────┐
   │  @logtail/pino     │        │  pino-otel transport │
   │  (BetterStack)     │        │  (when               │
   │                    │        │   OTEL_LOGS_EXPORTER │
   │  LOGTAIL_SOURCE    │        │   = otlp)            │
   │  LOGTAIL_INGEST    │        │                      │
   └────────┬───────────┘        └──────────┬───────────┘
            │                               │
            ▼                               ▼
   ┌────────────────────┐        ┌──────────────────────┐
   │   BetterStack      │        │    OTEL_EXPORTER_    │
   │   Telemetry        │        │    OTLP_ENDPOINT     │
   │                    │        │  (today: where? OQ1) │
   └────────────────────┘        └──────────────────────┘

Two parallel log pipelines today. @logtail/pino is the workhorse (always-on if LOGTAIL_SOURCE_TOKEN + LOGTAIL_INGEST_HOST set). The OTLP logs path is opt-in via OTEL_LOGS_EXPORTER=otlp. Each app has its own pipeline config.

Constraints

Do not disrupt customer ops. Smith Corona POC and other in-flight customers must keep getting logs ingested somewhere queryable continuously.
Reversibility ≤5 min during cutover.
No log loss. Every error message must land in at least one queryable surface throughout the migration.
OTel substrate stays. This is the abstraction we built for; it’s the value of the migration.
Status page question is separable. If we need one for Smith Corona, decide separately — it’s not gated by this cutover.

Premises

We are committed to GCP for app hosting (already mostly true; app.erpunlocked.com is the last holdout).
Cloud Logging UI is good enough for incident response. Validated empirically before cutover (Step 1).
Status pages are not a customer commitment today. (If this changes, hybrid approach in §Approaches.)
The OTel substrate is portable leverage; flipping its endpoint is a configuration change, not a code change.
PR #839’s $otel_trace_id correlation stamp continues to work end-to-end when traces live in Cloud Trace and logs in Cloud Logging — the trace ID is the universal handle.
PostHog’s role is unchanged by GCP migration (it’s product analytics, not observability infra).

Real Scope

Surface today	Replaces with	Effort
Pino → `@logtail/pino` (per-app)	Pino → OTLP logs exporter (single pipeline via existing `OTEL_LOGS_EXPORTER=otlp` flag)	S — config + transport swap in `packages/observability/src/logger.ts`
BetterStack uptime monitoring	Cloud Monitoring uptime checks (per Cloud Run service)	S — terraform/Pulumi or console-driven, no app code
BetterStack alerts (if any wired up)	Cloud Monitoring alerting policies	S-M — depends on how many alerts exist
BetterStack dashboards (if any)	Cloud Logging Logs Explorer + Log-based metrics + Cloud Monitoring dashboards	S — dashboards are mostly query bookmarks
BetterStack status pages	TBD (Statuspage.io / Better Uptime status / build a basic one) — separate decision	Out of scope for this initiative

Approaches Considered

Approach A: Single-pipeline OTLP (kill `@logtail/pino`, route all Pino logs through OTel)

Modify packages/observability/src/logger.ts so the Pino transport array always uses the OTel logs path when OTEL_EXPORTER_OTLP_ENDPOINT is set.
Decommission @logtail/pino, drop LOGTAIL_* env vars from all docker-compose files and Infisical scopes.
Point OTEL_EXPORTER_OTLP_ENDPOINT at https://telemetry.googleapis.com per Cloud Run service.

Effort: ~3-5 days human / ~1-2 hr CC. Risk: Medium (no parallel-run safety net during cutover).

Approach B: Dual-write parallel-run, then retire (RECOMMENDED)

Phased cutover:

Phase 1 (parallel-run, 7-14 days)
  Pino transports: [@logtail/pino, OTLP-to-Cloud-Logging]
  Source of truth for incidents: BetterStack
  Verify in Cloud Logging that all expected log volumes show up

Phase 2 (Cloud Logging primary, 7 days)
  Pino transports: [@logtail/pino (shadow), OTLP-to-Cloud-Logging (primary)]
  Source of truth for incidents: Cloud Logging
  BetterStack runs as instant rollback safety net

Phase 3 (decommission, after 7 days clean)
  Pino transports: [OTLP-to-Cloud-Logging]
  Drop @logtail/pino dep, remove LOGTAIL_* env, cancel BetterStack subscription
  Replace any BetterStack uptime checks with Cloud Monitoring uptime checks

Effort: ~2-3 weeks human (mostly observation time, not active work) / ~3-4 hr CC. Risk: Low. Reversibility: rollback by toggling OTEL_LOGS_EXPORTER env var. ≤5 min effective.

Approach C: Hybrid — keep BetterStack for status pages + uptime, retire log ingest only

Phase 3 of B, but stop short of cancelling BetterStack.
Drop their log-ingest tier (cheaper); keep status pages + uptime.
Useful only if status pages become a hard requirement during the cutover window.

Effort: same as B. Cost: smaller savings (~30-50% of BetterStack bill).

Recommended Approach: B — Dual-write parallel-run, conditional on Step 0

Conditional on: Step 0 (volume + cost model) showing Cloud Logging tier is reasonable and Step 1 (UI parity) showing engineers can actually live in Cloud Logging during incidents.

Why B over A: parallel-run gives a safety net during the most error-prone phase (cutover). The cost is 1-2 weeks of paying both subscriptions; the value is no log loss + reversibility.

Why B over C: if status pages become a need, they can be added separately (BetterStack status page, Statuspage.io, or self-hosted). Coupling the log ingest decision to status page need would slow this down for no reason.

Trigger conditions to start:

app.erpunlocked.com migrated to GCP Cloud Run.
Smith Corona POC has been in production ≥2 weeks without log-related incidents.
Cloud Logging cost projection (Step 0) does not exceed BetterStack savings.

Rollback Plan

During Phase 1-2 (cutover):

Toggle OTEL_LOGS_EXPORTER=otlp → OTEL_LOGS_EXPORTER= (unset) to disable Cloud Logging path. BetterStack continues unchanged.
Or toggle LOGTAIL_SOURCE_TOKEN to disable BetterStack while Cloud Logging continues.
Both controlled via Infisical, runtime config (no redeploy required) — same pattern as PR #839’s FEATURE_FLAGS_PROVIDER design.

After Phase 3 (post-decommission):

Re-add @logtail/pino transport via PR. ~1-2 hr to re-deploy. No data loss for the gap window because Cloud Logging keeps ingesting.

Open Questions — must resolve before Phase 1

What’s our actual log volume? Need to model against Cloud Logging tier pricing. Likely candidates for measurement: apps/webapp (Astro SSR + Hono), apps/ts-temporal-worker, apps/temporal-worker (Python), apps/email-worker (Cloudflare — special: not on Cloud Run, may need separate strategy).
What’s BetterStack actually costing today? Look up the actual subscription, model break-even.
Are there log retention requirements? SOC 2 / customer contracts. Cloud Logging defaults to 30 days; longer retention costs $0.01/GB/mo per archive. May matter for Smith Corona’s data residency review.
Does Cloud Logging UI give us enough power vs BetterStack’s? Saved searches, alerting, log-based metrics, regex filters. Validate during Step 1.
Cloudflare email-worker — runs outside Cloud Run, so the OTel pipe to telemetry.googleapis.com may not work directly. Worker needs to either (a) ship its own logs to Cloud Logging via the Logging API, (b) keep using BetterStack as a separate path, or (c) buffer to a queue Cloud Run reads. Decision deferred until volume + need is known.
Python services — apps/temporal-worker uses Python OTel. Same OTLP endpoint works; verify the Python exporter version supports the same auth path as TS.
Status page need. Smith Corona / other early customers — do we need a public status page in 2026? If yes, decision goes from “cancel BetterStack” to “downgrade BetterStack to status-page-only tier.”
Service account permissions. Cloud Run service accounts need roles/logging.logWriter and (if we want trace correlation in the UI) roles/cloudtrace.agent. Codify in IaC.

Cost (model before committing)

Today (estimated):

BetterStack Telemetry: $X/mo (Step 0 confirms actual)
Cloud Logging: $0 (not in use for app logs yet, only platform-internal)

After B:

BetterStack: $0 (cancelled) or ~30-50% of current (hybrid C)
Cloud Logging: depends on volume, expected to be inside or just-above the 50 GB/mo free tier for current scale

Action: Step 0 produces a 1-paragraph cost projection in this doc.

Success Criteria

Quantitative:

BetterStack/Logtail subscription cancelled (or downgraded if hybrid path triggers).
All app log search performed via Cloud Logging by engineers during incidents.
Uptime alerts fire from Cloud Monitoring.
Zero log loss during the 14-21 day parallel-run window (verified by spot-checking a known event in both surfaces).

Qualitative:

During an incident, an engineer pastes $otel_trace_id from a PostHog error into Cloud Trace, sees the full distributed trace, clicks through to correlated Cloud Logging entries — all from a single GCP console without alt-tabbing to BetterStack.
Adding a new alert takes a Cloud Monitoring console click + IaC edit, not a BetterStack dashboard config.

CI / Env-var changes (will live in cutover PR)

New env per Cloud Run service: OTEL_EXPORTER_OTLP_ENDPOINT=https://telemetry.googleapis.com. Auth via Cloud Run service account ADC.
New OTEL_LOGS_EXPORTER=otlp per service.
After Phase 3: drop LOGTAIL_SOURCE_TOKEN, LOGTAIL_INGEST_HOST from docker-compose files (docker-compose.prod.yml, docker-compose.coolify.*.yml) and Infisical scopes.
IaC: grant roles/logging.logWriter and roles/cloudtrace.agent to Cloud Run service accounts.
Cloud Monitoring uptime check definitions (terraform / console).

The Assignment

Order matters. Each step gates the next.

Step 0 — Volume + cost model.

Pull 30-day log volume from BetterStack (per-source: webapp, workers, email-worker).
Project against Cloud Logging pricing.
Compute net monthly delta and document in this doc.
Decision: if Cloud Logging tier ends up costing more than the BetterStack savings, halt and reconsider (likely means we’d just downgrade BetterStack instead).

Step 1 — Cloud Logging UI parity check.

Spend 1 working day actually using Cloud Logging on staging.ordermatic.co (which is already on GCP) for whatever incidents arise.
Pass criteria: can find errors, filter by service + correlation ID, set up a saved search, build a log-based alert.
Decision: if UI is materially worse, switch to Approach C (hybrid, keep BetterStack for the parts Cloud Logging can’t replace).

Step 2 — Resolve §Open Questions (Cloudflare worker strategy, Python parity, status page need, retention).

Step 3 — Implementation begins (Phase 1 of Rollback Plan: dual-write).

Dependencies

Upstream (must be true before we start):

Hetzner→GCP cutover Phase 2 complete (app.erpunlocked.com on GCP).
Smith Corona POC stable in production ≥2 weeks.
Cloud Run service accounts have IAM-grant capability (i.e., we have permission to add roles).

Downstream (changes this enables):

Single observability surface for ops (Cloud Operations) + single product surface (PostHog).
Operational runbooks updated to point at Cloud Logging URLs.
New-engineer onboarding skips one vendor.

Out of scope (intentional)

Status pages. Decided separately when there’s a customer requirement.
Cloudflare email-worker logs. Cloudflare Workers can’t directly emit OTLP to Cloud Logging; needs its own path (deferred to §OQ5).
Metrics migration. Cloud Monitoring + OTel metrics is a related but separate cutover. Logs come first because they’re the most-touched surface during incidents.
PostHog migration to anything. PR #839 keeps PostHog as the product surface; that’s not on the table.
Langfuse migration to anything. LLM-specific tooling stays.
Flagsmith migration. Tracked separately under the paused observability consolidation review.

PR #839 — adds $otel_trace_id correlation on PostHog events. The trace IDs become Cloud Trace IDs once on GCP, so this initiative inherits that cross-tool correlation for free.
PR #837 — PostHog reverse proxy via b.ordermatic.co. Continues to work unchanged on GCP.
Paused observability tooling consolidation review (memory: project_observability_consolidation_paused.md) — this initiative is the “quick wins” subset of that paused work, isolated to the BetterStack surface only.
Hetzner→GCP cutover Phase 1 (memory: project_hetzner_to_gcp_cutover_2026_05_02.md).