End-to-End Observability Implementation

Historical Architecture

pdf-api, Celery, pdf-worker, and SigNoz dashboards referenced in this document have been retired (#1189). Current Node.js OpenTelemetry bootstrap lives in apps/webapp/otel-init.cjs and packages/observability/src/otel-config.ts, while Python worker tracing and LLM instrumentation live in apps/temporal-worker/ and packages/pdf-shared/. See Observability Overview for the current stack.

This document describes the implementation of end-to-end distributed tracing across the ERP-Unlocked monorepo, following the auto-instrumentation approach outlined in the observability plan.

🎯 Implementation Overview

The implementation provides reliable, end-to-end distributed tracing from user actions in the webapp through to LLM processing in the pdf-worker, using OpenTelemetry auto-instrumentation and W3C Trace Context propagation.

Key Features

• End-to-end correlation: Single trace from webapp → pdf-api → Celery → pdf-worker → LLM spans
• User continuity: User context propagated via OpenTelemetry Baggage
• LLM observability: Langfuse integration for LLM trace visibility
• Auto-instrumentation: Minimal custom code, maximum coverage
• Security preserved: Service-to-service auth via X-Service-Token maintained

🏗️ Architecture

┌─────────────┐    HTTP + traceparent    ┌─────────────┐    Celery + context    ┌─────────────┐
│   Webapp    │ ────────────────────────► │   pdf-api   │ ────────────────────► │ pdf-worker  │
│ (Astro/Node)│                          │ (FastAPI)   │                       │ (Celery)    │
└─────────────┘                          └─────────────┘                       └─────────────┘
       │                                        │                                       │
       │ Baggage: user.id, session.id          │                                       │
       │                                        │                                       │
       ▼                                        ▼                                       ▼
┌─────────────┐                          ┌─────────────┐                       ┌─────────────┐
│ BetterStack │                          │ BetterStack │                       │ BetterStack │
│  (Traces)   │                          │  (Traces)   │                       │  (Traces)   │
└─────────────┘                          └─────────────┘                       └─────────────┘
                                                                                        │
                                                                                        │
                                                                                        ▼
                                                                               ┌─────────────┐
                                                                               │  Langfuse   │
                                                                               │ (LLM Traces)│
                                                                               └─────────────┘

📦 Implementation Details

1. Webapp (Astro/Node.js)

Dependencies Added

{
  "@opentelemetry/instrumentation-undici": "^0.50.0"
}

Key Changes

otel-init.cjs - Enhanced with Undici instrumentation:

const { UndiciInstrumentation } = require('@opentelemetry/instrumentation-undici');

const sdk = new NodeSDK({
  traceExporter,
  instrumentations: [
    getNodeAutoInstrumentations({
      '@opentelemetry/instrumentation-fs': { enabled: false },
    }),
    new UndiciInstrumentation(), // ← Added for fetch() trace propagation
    new PinoInstrumentation({
      logHook: (span, record, _level) => {
        record['resource.service.name'] = process.env.OTEL_SERVICE_NAME;
      },
    }),
  ],
});

fetchWithTrace.ts - New utility for trace propagation:

export async function fetchWithUserContext(
  url: string,
  init: RequestInit = {},
  userContext?: { userId?: string; sessionId?: string }
) {
  // Creates OpenTelemetry Baggage with user context
  // Injects trace context and baggage into HTTP headers
  // Ensures end-to-end correlation
}

API Routes Updated:

• apps/webapp/src/pages/api/pdf-documents/index.ts
• apps/webapp/src/pages/api/pdf-documents/[id]/process.ts

Both now use fetchWithUserContext() instead of plain fetch() to propagate user context.

Development Setup

{
  "scripts": {
    "dev": "cross-env NODE_OPTIONS=\"--require ./otel-init.cjs\" astro dev"
  }
}

2. pdf-api (FastAPI)

Dependencies Added

dependencies = [
  # Auto-instrumentation packages for full observability coverage
  "opentelemetry-instrumentation-fastapi==0.43b0",
  "opentelemetry-instrumentation-psycopg2==0.43b0",
  "opentelemetry-instrumentation-redis==0.43b0",
  "opentelemetry-instrumentation-celery==0.43b0",
  "opentelemetry-instrumentation-logging==0.43b0",
]

Key Features

• FastAPI auto-instrumentation: Automatic span creation for all routes
• Database tracing: psycopg2 instrumentation for PostgreSQL queries
• Redis tracing: Redis operation visibility
• Celery producer: Automatic context propagation to Celery tasks
• Logging correlation: Trace IDs in structured logs

Environment Configuration

OTEL_RESOURCE_ATTRIBUTES=service.name=pdf-processor-api,service.version=1.0.0,deployment.environment=production
OTEL_EXPORTER_OTLP_ENDPOINT=${OTEL_EXPORTER_OTLP_ENDPOINT}
OTEL_EXPORTER_OTLP_HEADERS=${OTEL_EXPORTER_OTLP_HEADERS}
OTEL_TRACES_EXPORTER=otlp

3. pdf-worker (Celery)

Dependencies Added

dependencies = [
  # Auto-instrumentation packages for full observability coverage
  "opentelemetry-instrumentation-celery==0.43b0",
  "opentelemetry-instrumentation-psycopg2==0.43b0",
  "opentelemetry-instrumentation-redis==0.43b0",
  "opentelemetry-instrumentation-logging==0.43b0",
  # Langfuse integration for LLM observability
  "langfuse>=2.0.0",
  "langfuse-opentelemetry>=1.0.0",
]

Key Features

• Celery auto-instrumentation: Automatic task span creation and context propagation
• LLM observability: Langfuse integration for LLM trace visibility
• Database tracing: psycopg2 instrumentation for PostgreSQL queries
• Redis tracing: Redis operation visibility

Langfuse Integration

langfuse_config.py:

def initialize_langfuse():
    langfuse = Langfuse(
        public_key=os.getenv("LANGFUSE_PUBLIC_KEY"),
        secret_key=os.getenv("LANGFUSE_SECRET_KEY"),
        host=os.getenv("LANGFUSE_HOST", "https://cloud.langfuse.com"),
    )
    LangfuseInstrumentor().instrument()  # Links to OpenTelemetry
    return langfuse

celery_app.py:

# Initialize Langfuse for LLM observability (optional)
try:
    import langfuse_config  # This will initialize Langfuse if configured
except ImportError:
    pass  # Langfuse not available, continue without it

Environment Configuration

OTEL_RESOURCE_ATTRIBUTES=service.name=pdf-processor-worker,service.version=1.0.0,deployment.environment=production
# Langfuse configuration for LLM observability (optional)
LANGFUSE_PUBLIC_KEY=${LANGFUSE_PUBLIC_KEY:-}
LANGFUSE_SECRET_KEY=${LANGFUSE_SECRET_KEY:-}
LANGFUSE_HOST=${LANGFUSE_HOST:-https://cloud.langfuse.com}

🔧 Configuration

Environment Variables

Required

# Better Stack/OpenTelemetry
OTEL_EXPORTER_OTLP_ENDPOINT=https://in-otel.betterstack.com
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer your-better-stack-token"
OTEL_TRACES_EXPORTER=otlp

# Service URLs
PDF_PROCESSOR_URL=http://pdf-api:8000
PDF_PROCESSOR_TOKEN=your-service-token

Optional (Langfuse)

LANGFUSE_PUBLIC_KEY=pk_...
LANGFUSE_SECRET_KEY=sk_...
LANGFUSE_HOST=https://cloud.langfuse.com

Docker Compose

The production docker-compose.prod.yml already includes:

• OpenTelemetry instrumentation via opentelemetry-instrument
• Proper environment variable configuration
• Service-to-service networking

🧪 Validation

Automated Validation Script

Run the validation script to check the implementation:

npx tsx scripts/validate-observability.ts

This script validates:

• Environment variable configuration
• Webapp instrumentation (Undici, trace propagation)
• pdf-api health and connectivity
• Celery worker configuration
• Langfuse integration (if configured)

Manual Validation

1. Upload a test PDF document through the webapp
2. Check Better Stack dashboard for:
   • Single trace spanning webapp → pdf-api → pdf-worker
   • User context in span attributes (user.id, session.id)
   • Celery task spans linked to parent traces
3. Check Langfuse dashboard (if configured) for:
   • LLM traces linked to OpenTelemetry spans
   • End-to-end visibility from user action to LLM generation

📊 Expected Trace Flow

1. User Uploads PDF

webapp: POST /api/pdf-documents
├── http: fetch to pdf-api (with traceparent + baggage)
└── pdf-api: POST /pdf-documents/{id}/process
    ├── db: insert pdf_document
    ├── celery: enqueue process_document task (with context)
    └── pdf-worker: process_document task
        ├── db: update document status
        ├── llm: extract data from PDF (Langfuse span)
        ├── db: insert extracted orders
        └── llm: validate extracted data (Langfuse span)

2. Span Attributes

Webapp spans:

• user.id: User identifier from Clerk
• session.id: Session identifier
• http.url: Target URL
• http.method: HTTP method

pdf-api spans:

• user.id: From Baggage
• session.id: From Baggage
• celery.task_name: Task being enqueued
• db.operation: Database operation type

pdf-worker spans:

• user.id: From Baggage
• session.id: From Baggage
• celery.task_name: Task being executed
• document.id: Document being processed
• llm.model: LLM model used (from Langfuse)

🚀 Deployment

Development

# Start with observability enabled
pnpm --filter webapp dev

Production

# Deploy with docker-compose.prod.yml
docker compose -f docker-compose.prod.yml up -d

The production deployment automatically includes:

• OpenTelemetry instrumentation
• Proper environment configuration
• Service-to-service trace propagation

🔍 Troubleshooting

Common Issues

1. No trace propagation from webapp

   • Check if @opentelemetry/instrumentation-undici is installed
   • Verify otel-init.cjs is being preloaded
   • Ensure fetchWithUserContext is being used

2. Missing Celery spans

   • Verify opentelemetry-instrumentation-celery is installed
   • Check that both producer (pdf-api) and consumer (pdf-worker) have the package
   • Ensure opentelemetry-instrument is used in Docker commands

3. No Langfuse traces

   • Verify Langfuse environment variables are set
   • Check that langfuse-opentelemetry is installed
   • Ensure langfuse_config.py is imported in celery_app.py

4. Missing user context

   • Verify fetchWithUserContext is used instead of plain fetch
   • Check that user context is being passed correctly
   • Ensure Baggage is being created with proper keys

Debug Commands

# Check webapp instrumentation
curl -H "Traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01" \
     http://localhost:4321/api/pdf-documents

# Check pdf-api health
curl http://localhost:8000/health

# Check Celery worker status
docker compose exec pdf-worker celery -A celery_app inspect ping

# Validate observability setup
npx tsx scripts/validate-observability.ts

📈 Monitoring

SigNoz Dashboards

Create dashboards for:

• End-to-end request latency: webapp → pdf-api → pdf-worker
• Error rates by service: Track failures across the pipeline
• User activity: Filter by user.id to see individual user flows
• LLM performance: Track LLM call latency and success rates

Key Metrics

• Trace completion rate: Percentage of traces that complete end-to-end
• User context propagation: Percentage of spans with user.id attribute
• LLM trace linkage: Percentage of LLM calls linked to OpenTelemetry spans
• Service latency: P50, P95, P99 latencies for each service

🔄 Future Enhancements

1. Custom span attributes: Add business-specific attributes (document type, order count, etc.)
2. Error tracking: Enhanced error correlation across services
3. Performance optimization: Sampling strategies for high-volume scenarios
4. Alerting: Set up alerts for trace failures or high latency
5. Custom metrics: Business metrics correlated with traces

---

This implementation provides a solid foundation for end-to-end observability while maintaining the existing security model and minimizing custom code complexity.