Environment Variables Setup for Observability

This document outlines the environment variables that need to be configured in each environment (development, staging, production) for the observability implementation to work correctly.

Overview

The observability implementation requires configuration for:

OpenTelemetry OTLP Exporter - For sending traces to Better Stack
Dagster OpenTelemetry - For Dagster pipeline observability
Webapp / Temporal → Dagster Integration (optional) - For programmatic Dagster job triggering

Required Variables by Service

1. Shared OpenTelemetry Configuration (All Services)

These variables should be set in the root .env file or in docker-compose.yml:

# OpenTelemetry OTLP Configuration (Better Stack)
OTEL_EXPORTER_OTLP_ENDPOINT=https://in-otel.betterstack.com
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer YOUR_BETTER_STACK_TOKEN
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://in-otel.betterstack.com
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=https://in-otel.betterstack.com/v1/logs
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

# Service Identification (per-service)
OTEL_SERVICE_NAME=webapp  # or internal, temporal-worker, ts-temporal-worker, dagster
OTEL_SERVICE_VERSION=1.0.0
OTEL_DEPLOYMENT_ENVIRONMENT=development  # or staging, production

Note: These are already configured if you’re using the existing observability setup. Verify they’re present in your environment.

2. Dagster-Specific Configuration

Dagster needs these variables to send traces to Better Stack. Add them to apps/dagster/.env or your Dagster deployment environment:

# OpenTelemetry for Dagster
OTEL_EXPORTER_OTLP_ENDPOINT=https://in-otel.betterstack.com
OTEL_EXPORTER_OTLP_HEADERS=api-key=YOUR_BETTER_STACK_TOKEN
# OR (Better Stack format)
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer YOUR_BETTER_STACK_TOKEN

# Optional: Service identification
OTEL_SERVICE_NAME=dagster
OTEL_SERVICE_VERSION=1.0.0

Important:

Dagster’s observability.py reads OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS
If OTEL_EXPORTER_OTLP_HEADERS contains api-key=, it extracts just the token
Otherwise, it uses the entire value as the API key
If these variables are not set, Dagster will skip OpenTelemetry initialization (graceful degradation)

3. Webapp / Temporal → Dagster Integration (Optional)

If you want to trigger Dagster jobs programmatically from the webapp or Temporal worker (against the self-hosted Dagster GCE VM behind Cloudflare Access), add these to the relevant service environment (see Webapp → Dagster Force Sync):

# Self-hosted Dagster GraphQL API behind Cloudflare Access
DAGSTER_GRAPHQL_URL=https://dagster.ordermatic.co/graphql
DAGSTER_CF_ACCESS_CLIENT_ID=your_cf_access_client_id
DAGSTER_CF_ACCESS_CLIENT_SECRET=your_cf_access_client_secret
ENABLE_DAGSTER_SYNC=true

Note: This is optional. Correlation ID propagation still works even without direct Dagster job triggering.

Environment-Specific Configuration

Development

# Root .env or docker-compose.yml
OTEL_EXPORTER_OTLP_ENDPOINT=https://in-otel.betterstack.com
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer YOUR_DEV_TOKEN
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://in-otel.betterstack.com
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=https://in-otel.betterstack.com/v1/logs
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_DEPLOYMENT_ENVIRONMENT=development

# Per-service (in docker-compose.yml or service .env)
# Webapp
OTEL_SERVICE_NAME=webapp
OTEL_SERVICE_VERSION=1.0.0

# Dagster
OTEL_SERVICE_NAME=dagster
OTEL_SERVICE_VERSION=1.0.0
OTEL_EXPORTER_OTLP_ENDPOINT=https://in-otel.betterstack.com
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer YOUR_DEV_TOKEN

Staging

# Same as development but with staging token
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer YOUR_STAGING_TOKEN
OTEL_DEPLOYMENT_ENVIRONMENT=staging

Production

# Same structure but with production token
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer YOUR_PRODUCTION_TOKEN
OTEL_DEPLOYMENT_ENVIRONMENT=production

Docker Compose Configuration

Development (docker-compose.yml)

Dagster already has OTEL variables configured (lines 250-254), but they default to empty if not set in .env:

dagster:
  environment:
    # OpenTelemetry (optional)
    - OTEL_SERVICE_NAME=dagster
    - OTEL_SERVICE_VERSION=1.0.0
    - OTEL_DEPLOYMENT_ENVIRONMENT=development
    - OTEL_EXPORTER_OTLP_ENDPOINT=${OTEL_EXPORTER_OTLP_ENDPOINT:-}
    - OTEL_EXPORTER_OTLP_HEADERS=${OTEL_EXPORTER_OTLP_HEADERS:-}

Action Required: Ensure these variables are set in your root .env file:

OTEL_EXPORTER_OTLP_ENDPOINT=https://in-otel.betterstack.com
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer YOUR_BETTER_STACK_TOKEN

Production (docker-compose.prod.yml)

Status: Both dagster-web and dagster-code-server already have OTEL variables configured:

dagster-web (lines 380-384)
dagster-code-server (lines 293-296)

Action Required: Ensure OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS are set in your production .env file:

OTEL_EXPORTER_OTLP_ENDPOINT=https://in-otel.betterstack.com
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer YOUR_PRODUCTION_TOKEN

The docker-compose files already reference these variables, they just need values.

Verification

Check if Dagster OTEL is configured:

# Start Dagster and look for this message:
# ✅ OpenTelemetry initialized for Dagster

# If you see this instead, OTEL is not configured:
# ℹ️  OpenTelemetry not configured (OTEL_EXPORTER_OTLP_ENDPOINT not set)

Check if correlation IDs are working:

Make a request to the webapp
Check the response headers for x-correlation-id
In Better Stack, search for traces with: correlation.id = "your-correlation-id"
You should see spans from webapp → Temporal and/or Dagster (if triggered)

Check Temporal / Dagster jobs:

Trigger a workflow or Dagster job from the webapp
Check temporal-worker logs or Dagster logs for correlation ID attributes
Verify the correlation ID is passed through the workflow/activity or Dagster job tags

Troubleshooting

Dagster not sending traces

Check environment variables are set:

# In Dagster container/environment
echo $OTEL_EXPORTER_OTLP_ENDPOINT
echo $OTEL_EXPORTER_OTLP_HEADERS

Check Dagster startup logs:
- Should see: ✅ OpenTelemetry initialized for Dagster
- If not, check that variables are set correctly
Verify Better Stack token:
- Token should have format: Bearer YOUR_TOKEN or api-key=YOUR_TOKEN
- Check token has “Send traces” permission in Better Stack

Correlation IDs not propagating

Check webapp middleware:
- Verify x-correlation-id header is in response
- Check middleware logs for correlation ID generation
Check Temporal workflows / Dagster jobs:
- Verify the correlation ID is present in workflow inputs, activity logs, or Dagster tags
- Check that the service initiating the job forwards the correlation context
Check Dagster assets:
- Verify set_correlation_id_attribute() is called in assets
- Check that correlation_id tag is set when triggering Dagster jobs

Quick Setup Checklist

✅ Already Configured

docker-compose.yml: Dagster has OTEL variables (lines 250-254)
docker-compose.prod.yml: dagster-web has OTEL variables (lines 380-384)
Webapp, internal, temporal-worker, ts-temporal-worker, dagster: Already have OTEL config

⚠️ Action Required

Root .env file (ALL environments) - Ensure these are set:
Terminal window
```
OTEL_EXPORTER_OTLP_ENDPOINT=https://in-otel.betterstack.com
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer YOUR_BETTER_STACK_TOKEN
```
Note: Use different tokens for dev/staging/production in Better Stack.
Service env (optional) - For self-hosted Dagster GraphQL integration:
Terminal window
```
DAGSTER_GRAPHQL_URL=https://dagster.ordermatic.co/graphql
DAGSTER_CF_ACCESS_CLIENT_ID=your_cf_access_client_id
DAGSTER_CF_ACCESS_CLIENT_SECRET=your_cf_access_client_secret
```
Note: This is only needed if you want to trigger Dagster jobs programmatically from the webapp or Temporal worker.

Testing

Start services and check Dagster logs:

docker-compose up dagster
# Look for: "✅ OpenTelemetry initialized for Dagster"

Test correlation ID propagation:
- Make a request to webapp
- Check response header: x-correlation-id
- Search in Better Stack: correlation.id = "your-correlation-id"