ERP-Unlocked Observability Architecture

This document outlines the comprehensive observability strategy implemented across the ERP-Unlocked monorepo, covering logging, metrics, tracing, and LLM observability.

Overview

Our observability architecture follows a multi-layered approach with distinct responsibilities:

Better Stack: Primary logging and application metrics
Langfuse: LLM-specific observability and tracing
OpenTelemetry: Distributed tracing and metrics collection

Architecture Components

1. Better Stack Integration

Purpose: Primary logging destination for all applications Implementation: Structured logging with JSON format

Configuration

# Environment variables
BETTER_STACK_SOURCE_TOKEN=your_token_here
BETTER_STACK_ENDPOINT=https://in.logs.betterstack.com
LOG_LEVEL=info

Key Features

Structured Logging: JSON-formatted logs with consistent fields
Trace Correlation: OpenTelemetry trace IDs embedded in logs
Service Identification: Each service tagged with service.name
Error Tracking: Comprehensive error logging with context

Log Format

{
  "timestamp": "2025-01-24T10:30:00.000Z",
  "level": "info",
  "service": "pdf-worker",
  "message": "Processing document chunk",
  "trace_id": "abc123...",
  "span_id": "def456...",
  "document_id": "uuid-here",
  "chunk_number": 2
}

2. Langfuse LLM Observability

Purpose: Specialized observability for LLM operations (Gemini API) Implementation: Native Langfuse 3.x API with isolated OpenTelemetry tracer

Configuration

# Environment variables
LANGFUSE_PUBLIC_KEY=your_public_key
LANGFUSE_SECRET_KEY=your_secret_key
LANGFUSE_HOST=https://cloud.langfuse.com
LANGFUSE_FLUSH_INTERVAL=10
LANGFUSE_TRACE_TIMEOUT=5

Key Features

Session Tracking: User and session correlation
Cost Tracking: Automatic cost calculation for LLM requests
Token Usage: Input/output token monitoring
Quality Scoring: Response quality assessment
Trace Hierarchy: Parent-child span relationships

Implementation Pattern

# Main task span
span = langfuse_client.start_span(
    name="process_document_task",
    input={"document_id": document_id},
    metadata={"user_id": user_id}
)
span.update_trace(session_id=document_id, user_id=user_id)

# Child generation span
generation = span.start_generation(
    name="process_single_chunk",
    model="gemini-3-flash-preview",
    input={"chunk_index": chunk_index}
)

# Update with results
generation.update(
    output=result,
    usage_details={
        "input": input_tokens,
        "output": output_tokens,
        "total": total_tokens
    }
)
generation.end()
span.end()

3. OpenTelemetry Integration

Purpose: Distributed tracing and metrics collection Implementation: Dual tracer architecture

Tracer Architecture

# Primary tracer (Better Stack)
_primary_tracer_provider = TracerProvider()
tracer = otel_trace.get_tracer(__name__, tracer_provider=_primary_tracer_provider)

# Langfuse tracer (isolated for LLM operations)
_langfuse_tracer_provider = TracerProvider()

Key Features

Distributed Tracing: End-to-end request tracking
Custom Metrics: Application-specific measurements
Span Attributes: Rich metadata for filtering
Error Tracking: Exception and error propagation

Metrics Collection

# Custom metrics
llm_request_duration = meter.create_histogram(
    name="llm_request_duration_ms",
    description="Duration of LLM API requests in milliseconds",
    unit="ms"
)

llm_tokens_total = meter.create_counter(
    name="llm_tokens_total",
    description="Total token consumption",
    unit="tokens"
)

Service-Specific Implementation

PDF Worker Service

Location: apps/pdf-worker/ Observability: Full integration with all components

Key Metrics

Document processing duration
Chunk processing success rates
LLM API call metrics
Database operation timing
Error rates and types

Trace Structure

process_document_task (span)
├── download_document (span)
├── chunk_document (span)
├── process_single_chunk (generation)
│   ├── gemini_api_call (span)
│   └── json_validation (span)
├── merge_results (span)
└── save_to_database (span)

PDF API Service

Location: apps/pdf-api/ Observability: HTTP request tracing and metrics

Key Metrics

API endpoint response times
Request/response sizes
Authentication success rates
Database query performance
Error response codes

Trace Structure

http_request (span)
├── authentication (span)
├── request_validation (span)
├── business_logic (span)
├── database_operations (span)
└── response_serialization (span)

Web Application

Location: apps/webapp/ Observability: Client-side performance and user interactions

Key Metrics

Page load times
API call durations
User interaction tracking
Error rates
Performance metrics

Dashboard Configuration

Better Stack Dashboards

Purpose: Application monitoring and alerting Configuration: Automatic log parsing and visualization

Key Panels

Service health status
Error rates and trends
Performance metrics
User activity patterns
System resource usage

Langfuse Dashboards

Purpose: LLM-specific monitoring Configuration: Langfuse provides built-in dashboard interface

Key Panels

LLM request volume
Cost tracking and trends
Response quality scores
Token usage patterns
Error analysis

OpenTelemetry Integration

Purpose: Distributed tracing and metrics collection Configuration: Direct OpenTelemetry data export

Key Features

Distributed tracing visualization
Custom metrics collection
Performance analysis
Error tracking
Cross-service correlation

Alerting Strategy

Critical Alerts

High Error Rates

- alert: HighErrorRate
  expr: rate(http_requests_total{status="error"}[5m]) > 0.1
  for: 2m
  labels:
    severity: critical

LLM Cost Thresholds

- alert: HighLLMCosts
  expr: sum(rate(llm_request_cost_usd[1h])) * 3600 > 100
  for: 5m
  labels:
    severity: warning

Performance Degradation

- alert: HighLatency
  expr: histogram_quantile(0.95, rate(http_request_duration_ms_bucket[5m])) > 5000
  for: 3m
  labels:
    severity: warning

Warning Alerts

Low Quality Scores

- alert: LowLLMQuality
  expr: avg(llm_response_quality_score) < 0.7
  for: 10m
  labels:
    severity: warning

High Token Usage

- alert: HighTokenUsage
  expr: rate(llm_tokens_total[5m]) > 10000
  for: 5m
  labels:
    severity: warning

Best Practices

1. Structured Logging

Use consistent log formats across all services
Include trace IDs for correlation
Add relevant context to error messages
Use appropriate log levels

2. Metric Naming

Follow OpenTelemetry semantic conventions
Use descriptive names with units
Include relevant labels for filtering
Avoid high-cardinality labels

3. Trace Design

Keep spans focused and meaningful
Include relevant attributes
Maintain proper parent-child relationships
Handle errors gracefully

4. Cost Management

Monitor LLM usage patterns
Set up cost alerts and budgets
Optimize prompt efficiency
Track cost per successful operation

5. Performance Monitoring

Track key performance indicators
Set up latency alerts
Monitor resource usage
Analyze performance trends

Troubleshooting

Common Issues

Missing Traces

Check OpenTelemetry configuration
Verify tracer initialization
Ensure proper span creation
Check for context propagation issues

High Costs

Monitor token usage patterns
Review prompt efficiency
Check for retry loops
Analyze cost per operation

Performance Issues

Check latency percentiles
Monitor resource usage
Analyze error rates
Review database performance

Debug Queries

Find Expensive Operations

topk(10, llm_request_cost_usd) by (operation)

Identify Slow Requests

histogram_quantile(0.95, rate(http_request_duration_ms_bucket[5m])) > 3000

Check Error Patterns

rate(http_requests_total{status="error"}[5m]) / rate(http_requests_total[5m])

Configuration Reference

Environment Variables

Better Stack

BETTER_STACK_SOURCE_TOKEN=your_token
BETTER_STACK_ENDPOINT=https://in.logs.betterstack.com
LOG_LEVEL=info

Langfuse

LANGFUSE_PUBLIC_KEY=your_public_key
LANGFUSE_SECRET_KEY=your_secret_key
LANGFUSE_HOST=https://cloud.langfuse.com
LANGFUSE_FLUSH_INTERVAL=10
LANGFUSE_TRACE_TIMEOUT=5
LANGFUSE_DEBUG_LOGGING=false

OpenTelemetry

OTEL_EXPORTER_OTLP_ENDPOINT=https://your-endpoint.com
OTEL_SERVICE_NAME=your-service
OTEL_RESOURCE_ATTRIBUTES=service.version=1.0.0

Future Enhancements

Planned Improvements

Advanced Analytics: Machine learning-based anomaly detection
Cost Optimization: Automated prompt optimization
Performance Tuning: Dynamic scaling based on metrics
User Experience: Real-time user behavior tracking
Security Monitoring: Threat detection and response

Integration Opportunities

Slack Notifications: Real-time alert delivery
PagerDuty Integration: Incident management
Grafana Dashboards: Advanced visualization
DataDog Integration: Enterprise monitoring
Custom Analytics: Business intelligence integration

References

Last Updated: January 2025 Version: 1.0 Maintainer: ERP-Unlocked Development Team