Better Stack Integration Guide

Overview

This guide shows how to set up observability and operational alerting with Better Stack (formerly Logtail) for ERP-Unlocked. Note that Cloud Run and GCE VM stdout logs land natively in GCP Cloud Logging; Better Stack handles operational alerting (#3335), OTLP trace/log ingestion, and optional @logtail/pino log streaming (see the Cloud Logging cutover).

Environment Variables (Better Stack Standard)

Primary Configuration (Recommended)

Following Better Stack’s official documentation and packages/observability/src/logger.ts, use these environment variables:

# Better Stack OTLP Configuration (Traces & OTLP Logs)
OTEL_EXPORTER_OTLP_ENDPOINT=https://in-otel.betterstack.com
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer <your-better-stack-token>
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=https://in-otel.betterstack.com/v1/logs
OTEL_LOGS_EXPORTER=otlp

# Optional: Pino -> Better Stack Logtail worker transport
LOGTAIL_SOURCE_TOKEN=<your-logtail-source-token>
LOGTAIL_INGEST_HOST=in.logs.betterstack.com

# Service Configuration
OTEL_SERVICE_NAME=erp-unlocked-webapp  # or temporal-worker, ts-temporal-worker, dagster
OTEL_NODE_RESOURCE_DETECTORS=env,host,os

# Development/Debug Settings
OTEL_CONSOLE_EXPORT=false  # Set to true for local development

Setup Instructions

Step 1: Better Stack Account

1. Sign up at https://betterstack.com/
2. Create a new project
3. Go to Settings → API Tokens
4. Create a new token with “Send logs” and “Send traces” permissions

Step 2: Environment Configuration

For Production (.env.production)

# Better Stack OTLP Configuration
OTEL_EXPORTER_OTLP_ENDPOINT=https://in-otel.betterstack.com
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer your-actual-better-stack-token-here
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=https://in-otel.betterstack.com/v1/logs
OTEL_LOGS_EXPORTER=otlp
OTEL_RESOURCE_ATTRIBUTES=service.name=erp-unlocked-webapp,service.version=1.0.0

# Optional: Direct Logtail transport for Pino
LOGTAIL_SOURCE_TOKEN=your-logtail-source-token
LOGTAIL_INGEST_HOST=in.logs.betterstack.com
OTEL_CONSOLE_EXPORT=false

For Development (.env.development)

# Development - Use console / stdout export
OTEL_CONSOLE_EXPORT=true
LOG_LEVEL=debug

# Optional: Test with Better Stack in development
# OTEL_EXPORTER_OTLP_ENDPOINT=https://in-otel.betterstack.com
# OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer your-token
# OTEL_LOGS_EXPORTER=otlp

Node.js Logging with Pino

Implementation

Our packages/observability/src/logger.ts provides:

Features

• OTLP & Logtail Support: Emits logs via @opentelemetry/api-logs when OTEL_LOGS_EXPORTER=otlp, or streams via @logtail/pino alongside stdout when LOGTAIL_SOURCE_TOKEN and LOGTAIL_INGEST_HOST are configured
• Development Mode: Clean stdout / console output for local development
• Production Mode: JSON structured logs sent to stdout (GCP Cloud Logging) and optionally Better Stack
• Trace Correlation: Automatic correlation with OpenTelemetry traces

Usage Examples

import { logger } from '@repo/observability/logger';

// Basic logging
logger.info('User logged in', { userId: '123', method: 'oauth' });
logger.warn('High memory usage detected', { usage: '85%' });
logger.error('Payment failed', error, { orderId: 'order-456' });

// With structured metadata
logger.info('Order processed', {
  orderId: 'order-789',
  customerId: 'customer-123',
  amount: 99.99,
  currency: 'USD',
  processingTime: 1.2,
});

Environment Variables

# Enable OTLP logs in production
OTEL_LOGS_EXPORTER=otlp
OTEL_EXPORTER_OTLP_ENDPOINT=https://in-otel.betterstack.com
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=https://in-otel.betterstack.com/v1/logs
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer your-token
OTEL_RESOURCE_ATTRIBUTES=service.name=erp-unlocked-webapp,service.version=1.0.0

# Or enable @logtail/pino transport
LOGTAIL_SOURCE_TOKEN=your-logtail-token
LOGTAIL_INGEST_HOST=in.logs.betterstack.com

# Development mode
NODE_ENV=development
LOG_LEVEL=debug

Python Logging with OpenTelemetry

Implementation

Our Python services use OpenTelemetry’s logging instrumentation:

Features

• Structured Logging: JSON format for better parsing
• Trace Correlation: Automatic span context injection
• Error Tracking: Comprehensive error logging with stack traces
• Performance Metrics: Request timing and resource usage

Environment Variables

# Python services
OTEL_SERVICE_NAME=temporal-worker  # or webapp, internal, dagster
OTEL_EXPORTER_OTLP_ENDPOINT=https://in-otel.betterstack.com
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer your-token
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=https://in-otel.betterstack.com/v1/logs

Better Stack Features

Logs

• Structured JSON logs with automatic parsing
• Log correlation with traces and metrics
• Real-time search and filtering
• Log retention and archiving
• Alerting based on log patterns

Traces

• Distributed tracing across all services
• Trace correlation with logs and metrics
• Performance insights and bottleneck identification
• Service dependency mapping
• Error tracking and debugging

Metrics

• Custom metrics from your application
• Infrastructure metrics (CPU, memory, etc.)
• Business metrics (orders processed, user activity)
• Alerting and monitoring dashboards

Validation

Automated Validation

Run the validation script to check Better Stack integration:

pnpm tsx scripts/validate-observability.ts

Manual Validation

1. Upload a test PDF document through the webapp
2. Check Better Stack dashboard for:
   • Single trace spanning webapp → Temporal workflow → temporal-worker activities
   • User context in span attributes (user.id, session.id)
   • Temporal activity spans linked to parent workflow traces
3. Check Langfuse dashboard (if configured) for:
   • LLM traces linked to OpenTelemetry spans
   • End-to-end visibility from user action to LLM generation

Better Stack Dashboard

Logs View

• Real-time log streaming from all services
• Structured log parsing with automatic field extraction
• Log correlation with traces and metrics
• Search and filtering by service, level, or custom fields

Traces View

• Service map showing dependencies
• Trace timeline with detailed span information
• Error tracking and performance bottlenecks
• Custom span attributes for business context

Metrics View

• Custom dashboards for business and technical metrics
• Alerting rules for proactive monitoring
• Historical data and trending analysis

Troubleshooting

Common Issues

1. No traces appearing in Better Stack

   • Check that OTEL_EXPORTER_OTLP_ENDPOINT is set to https://in-otel.betterstack.com
   • Verify OTEL_EXPORTER_OTLP_HEADERS contains valid Bearer token
   • Ensure services are sending data (check logs for OpenTelemetry initialization)

2. Logs not appearing

   • Verify OTEL_LOGS_EXPORTER=otlp and OTEL_EXPORTER_OTLP_ENDPOINT are set (or LOGTAIL_SOURCE_TOKEN and LOGTAIL_INGEST_HOST if using @logtail/pino)
   • Ensure Pino or OTel logger initialization succeeded in startup logs

3. High latency or timeouts

   • Check network connectivity to Better Stack endpoints
   • Verify OTLP exporter timeout settings
   • Consider increasing buffer limits for high-volume applications

Debug Mode

Enable debug logging to troubleshoot issues:

# Node.js
DEBUG=otel* node app.js

# Python
OTEL_LOG_LEVEL=debug python app.py

Switching OTLP Endpoints

# Better Stack OTLP configuration
OTEL_EXPORTER_OTLP_ENDPOINT=https://in-otel.betterstack.com
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer your-better-stack-token

The OpenTelemetry instrumentation code in @repo/observability remains the same across environments — only the OTLP endpoint and authorization header change.

Cost Optimization

Better Stack Pricing

• Logs: Pay per GB ingested
• Traces: Included with logs
• Metrics: Included with logs
• Retention: Configurable retention periods

Optimization Tips

1. Structured logging: Use JSON format for better parsing and lower costs
2. Log levels: Use appropriate log levels to reduce noise
3. Sampling: Configure trace sampling for high-volume applications
4. Retention: Set appropriate retention periods for different log types