Skip to content
Ordermatic Docs

Observability Overview

Stack

Section titled “Stack”
Layer Tool What it covers
Primary logs GCP Cloud Logging Captures structured stdout JSON automatically from Cloud Run (webapp, internal) and GCE VMs (temporal-worker, ts-temporal-worker, dagster)
Alerting & OTLP export Better Stack Operational alerting (#3335), OTLP trace/log ingest (OTEL_LOGS_EXPORTER=otlp), and optional Logtail Pino export (LOGTAIL_SOURCE_TOKEN, LOGTAIL_INGEST_HOST)
LLM tracing Langfuse Token usage, cost, latency, and quality scores per AI call
Instrumentation OpenTelemetry SDK Spans, trace context propagation, Pino log hook
Package @repo/observability Wraps the OTEL SDK for TypeScript apps (apps/webapp/otel-init.cjs + packages/observability/src/otel-config.ts)
Logger @repo/utils/logger (base) / @repo/observability/logger (OTEL) See Logger Usage Reference

OTLP endpoint: https://in-otel.betterstack.com

Services instrumented

Section titled “Services instrumented”
Service Runtime / Host Notes
webapp Cloud Run (erp-webapp-{staging,prod}) Full OTEL via @repo/observability (_HAS_OTEL=true in Cloud Build); service name erp-unlocked-webapp
internal Cloud Run (erp-internal-{staging,prod}) Runs without OTEL preload (_HAS_OTEL=false); structured stdout JSON captured by GCP Cloud Logging
temporal-worker GCE VM systemd unit (erp-{staging,prod}-temporal) Python worker with opentelemetry-distro instrumentation + Langfuse LLM tracing
ts-temporal-worker GCE VM systemd unit (erp-{staging,prod}-temporal) TypeScript Temporal worker on the same GCE VM; uses @repo/observability
dagster GCE VM (erp-{staging,prod}-dagster) Python data pipeline orchestration (apps/dagster/erp_pipeline/observability.py); stdout to Cloud Logging
Cloudflare Workers Cloudflare Workers (email-worker, ops-mcp, etc.) @repo/utils/logger (lightweight — no Node OTEL runtime in Workers)

Correlation

Section titled “Correlation”

Every request gets a UUID v4 Correlation ID injected at the edge. It propagates via:

  • HTTP header x-correlation-id
  • Pino log field correlationId
  • Temporal workflow payloads and search attributes / tags
  • Dagster run tags (correlation_id)
  • Langfuse session_id
  • OTEL span attribute correlation.id

Find a full request trace in Better Stack or Cloud Logging by filtering on the correlation ID. See Correlation ID Querying for details.

Logger architecture

Section titled “Logger architecture”

Two tiers — pick by consumer, not by preference:

  • @repo/utils/logger — lightweight Pino logger. Use in all shared packages/ and in any app that doesn’t run the Node OTEL SDK (internal, Cloudflare Workers, static sites).
  • @repo/observability/logger — OTEL-enhanced logger. Use in OTel-enabled Node services (webapp, ts-temporal-worker).

Details: Logger Usage Reference · Logging Configuration

LLM observability

Section titled “LLM observability”

LLM calls are traced to Langfuse via @with_langfuse() and @repo/ai, covering both Python Gemini / Vertex AI calls (packages/pdf-shared in temporal-worker for PDF extraction and validation) and TypeScript / Python Anthropic calls (@repo/ai and agentic workflows). Tracks: model, tokens in/out, cost, latency, and quality score.

Setup: Langfuse + Better Stack integration

Setup guides

Section titled “Setup guides”

Required env vars

Section titled “Required env vars”
Terminal window
# OpenTelemetry OTLP export (traces + optional OTLP logs)
OTEL_EXPORTER_OTLP_ENDPOINT=https://in-otel.betterstack.com
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer <betterstack-source-token>
OTEL_LOGS_EXPORTER=otlp                 # optional: enable @opentelemetry/api-logs export
OTEL_SERVICE_NAME=erp-unlocked-webapp   # per-service
OTEL_DEPLOYMENT_ENVIRONMENT=production  # or staging


# Optional: direct Pino -> Better Stack Logtail worker transport
LOGTAIL_SOURCE_TOKEN=<logtail-source-token>
LOGTAIL_INGEST_HOST=in.logs.betterstack.com


# Operational alerting API (Temporal / Dagster alert hooks)
BETTERSTACK_API_TOKEN=<betterstack-api-token>