Logging Configuration

This document describes the logging configuration for the ERP Unlocked application, including OTLP (OpenTelemetry Protocol) export settings and fallback mechanisms.

Overview

The application uses a multi-transport logging system with:

Primary Transport: OTLP export to observability platforms (e.g., SigNoz)
Fallback Transports: File-based and console logging for reliability
Development Transport: Pretty-printed console output for local development

OTLP Configuration

Environment Variables

The OTLP exporter is configured via the following environment variables:

Variable	Default	Description
`OTEL_EXPORTER_OTLP_TIMEOUT`	`10000`	Timeout in milliseconds for OTLP export requests
`OTEL_EXPORTER_OTLP_RETRY_ENABLED`	`true`	Enable/disable retry mechanism for failed exports
`OTEL_EXPORTER_OTLP_RETRY_INITIAL_BACKOFF`	`1000`	Initial backoff delay in milliseconds
`OTEL_EXPORTER_OTLP_RETRY_MAX_BACKOFF`	`10000`	Maximum backoff delay in milliseconds
`OTEL_EXPORTER_OTLP_RETRY_MAX_ATTEMPTS`	`5`	Maximum number of retry attempts
`OTEL_EXPORTER_OTLP_BUFFER_LIMIT`	`1000`	Maximum number of log entries to buffer

Default Configuration

The logger uses sensible defaults to prevent silent log loss:

{
  timeout: 10000,           // 10 second timeout
  retryEnabled: true,       // Always enable retries
  retryInitialBackoff: 1000, // Start with 1 second delay
  retryMaxBackoff: 10000,   // Cap at 10 second delay
  retryMaxAttempts: 5,      // Try up to 5 times
  bufferLimit: 1000         // Buffer up to 1000 entries
}

Retry Strategy

The OTLP exporter uses exponential backoff with jitter:

Initial Delay: 1 second
Maximum Delay: 10 seconds
Retry Attempts: Up to 5 times
Buffer Capacity: 1000 log entries

This ensures logs are not lost during temporary network issues or collector outages.

Fallback Logging

Production Fallback

In production environments, two fallback transports are configured:

File-based Logging:
- Path: /app/logs/fallback.log (configurable via PINO_FALLBACK_LOG_PATH)
- Level: info and above
- Purpose: Persistent storage for logs that couldn’t be sent via OTLP
Console Logging:
- Level: warn and above
- Purpose: Immediate visibility in container logs
- Format: Structured JSON (no colors)

Development Fallback

In development, a single pretty-printed console transport is used:

Format: Human-readable with colors
Level: All levels
Purpose: Easy debugging and development

Container Configuration

Docker Compose Setup

The application is configured to work seamlessly in both development and production containers:

Development (`docker-compose.yml`)

services:
  web:
    volumes:
      - ./logs:/app/logs # Mount logs directory for persistence
    environment:
      - PINO_FALLBACK_LOG_PATH=/app/logs/fallback.log
      - OTEL_EXPORTER_OTLP_TIMEOUT=${OTEL_EXPORTER_OTLP_TIMEOUT:-10000}
      - OTEL_EXPORTER_OTLP_RETRY_ENABLED=${OTEL_EXPORTER_OTLP_RETRY_ENABLED:-true}
      - OTEL_EXPORTER_OTLP_RETRY_INITIAL_BACKOFF=${OTEL_EXPORTER_OTLP_RETRY_INITIAL_BACKOFF:-1000}
      - OTEL_EXPORTER_OTLP_RETRY_MAX_BACKOFF=${OTEL_EXPORTER_OTLP_RETRY_MAX_BACKOFF:-10000}
      - OTEL_EXPORTER_OTLP_RETRY_MAX_ATTEMPTS=${OTEL_EXPORTER_OTLP_RETRY_MAX_ATTEMPTS:-5}
      - OTEL_EXPORTER_OTLP_BUFFER_LIMIT=${OTEL_EXPORTER_OTLP_BUFFER_LIMIT:-1000}

Production (`docker-compose.prod.yml`)

services:
  web:
    volumes:
      - app-logs:/app/logs # Named volume for production persistence
    environment:
      - PINO_FALLBACK_LOG_PATH=/app/logs/fallback.log
      - OTEL_EXPORTER_OTLP_TIMEOUT=${OTEL_EXPORTER_OTLP_TIMEOUT:-10000}
      - OTEL_EXPORTER_OTLP_RETRY_ENABLED=${OTEL_EXPORTER_OTLP_RETRY_ENABLED:-true}
      - OTEL_EXPORTER_OTLP_RETRY_INITIAL_BACKOFF=${OTEL_EXPORTER_OTLP_RETRY_INITIAL_BACKOFF:-1000}
      - OTEL_EXPORTER_OTLP_RETRY_MAX_BACKOFF=${OTEL_EXPORTER_OTLP_RETRY_MAX_BACKOFF:-10000}
      - OTEL_EXPORTER_OTLP_RETRY_MAX_ATTEMPTS=${OTEL_EXPORTER_OTLP_RETRY_MAX_ATTEMPTS:-5}
      - OTEL_EXPORTER_OTLP_BUFFER_LIMIT=${OTEL_EXPORTER_OTLP_BUFFER_LIMIT:-1000}

volumes:
  app-logs:
    driver: local

Dockerfile Configuration

Both development and production Dockerfiles create the logs directory:

# Create logs directory for fallback logging
RUN mkdir -p /app/logs && chmod 755 /app/logs

Configuration Examples

Basic OTLP Setup

# Required for OTLP export
export OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.us.signoz.cloud:443
export OTEL_EXPORTER_OTLP_HEADERS=signoz-ingestion-key=your-key

# Optional: Customize retry behavior
export OTEL_EXPORTER_OTLP_RETRY_MAX_ATTEMPTS=10
export OTEL_EXPORTER_OTLP_BUFFER_LIMIT=2000

Docker Compose Configuration

environment:
  - OTEL_EXPORTER_OTLP_ENDPOINT=${OTEL_EXPORTER_OTLP_ENDPOINT}
  - OTEL_EXPORTER_OTLP_HEADERS=${OTEL_EXPORTER_OTLP_HEADERS}
  - OTEL_EXPORTER_OTLP_TIMEOUT=${OTEL_EXPORTER_OTLP_TIMEOUT:-10000}
  - OTEL_EXPORTER_OTLP_RETRY_ENABLED=${OTEL_EXPORTER_OTLP_RETRY_ENABLED:-true}
  - OTEL_EXPORTER_OTLP_RETRY_INITIAL_BACKOFF=${OTEL_EXPORTER_OTLP_RETRY_INITIAL_BACKOFF:-1000}
  - OTEL_EXPORTER_OTLP_RETRY_MAX_BACKOFF=${OTEL_EXPORTER_OTLP_RETRY_MAX_BACKOFF:-10000}
  - OTEL_EXPORTER_OTLP_RETRY_MAX_ATTEMPTS=${OTEL_EXPORTER_OTLP_RETRY_MAX_ATTEMPTS:-5}
  - OTEL_EXPORTER_OTLP_BUFFER_LIMIT=${OTEL_EXPORTER_OTLP_BUFFER_LIMIT:-1000}
  - PINO_FALLBACK_LOG_PATH=${PINO_FALLBACK_LOG_PATH:-/app/logs/fallback.log}

Testing

Container Testing

Use the provided test script to verify logging configuration in containers:

# Test both development and production containers
./scripts/test-container-logging.sh

This script will:

Test development container logging
Test production container logging
Verify fallback log files are created and populated
Display sample log entries

Manual Testing

Test logging manually in containers:

# Development
docker-compose run --rm web npx tsx -e "
  import { logger } from './src/utils/logger.ts';
  logger.info('Test message');
  logger.warn('Warning message');
  logger.error('Error message');
"

# Production
docker run --rm \
  -e NODE_ENV=production \
  -e PINO_FALLBACK_LOG_PATH=/app/logs/fallback.log \
  -v $(pwd)/logs:/app/logs \
  your-image npx tsx -e "
    import { logger } from './src/utils/logger.ts';
    logger.info('Production test');
    logger.warn('Production warning');
    logger.error('Production error');
  "

Monitoring and Troubleshooting

Check OTLP Connectivity

Monitor the fallback log file for signs of OTLP connectivity issues:

# Check if fallback logs are being written
tail -f logs/fallback.log

# Look for patterns indicating OTLP failures
grep -i "otlp\|opentelemetry\|export" logs/fallback.log

Container Log Monitoring

# View container logs
docker-compose logs web

# Follow container logs in real-time
docker-compose logs -f web

# Check specific container
docker logs <container-id>

Log File Management

The fallback log file should be monitored and rotated:

# Check log file size
ls -lh logs/fallback.log

# Rotate logs if needed
mv logs/fallback.log logs/fallback.log.$(date +%Y%m%d)

Environment Variable Validation

Ensure all required environment variables are set:

# Check OTLP configuration
echo "OTLP Endpoint: $OTEL_EXPORTER_OTLP_ENDPOINT"
echo "OTLP Headers: $OTEL_EXPORTER_OTLP_HEADERS"
echo "Retry Enabled: $OTEL_EXPORTER_OTLP_RETRY_ENABLED"
echo "Buffer Limit: $OTEL_EXPORTER_OTLP_BUFFER_LIMIT"

Best Practices

Always enable retries: Set OTEL_EXPORTER_OTLP_RETRY_ENABLED=true
Use appropriate buffer limits: Balance memory usage with log retention
Monitor fallback logs: Check for patterns indicating OTLP issues
Set reasonable timeouts: 10-30 seconds for most environments
Configure log rotation: Prevent disk space issues from fallback logs
Use volume mounts: Ensure logs persist across container restarts
Test in containers: Verify logging works in both development and production environments