Querying Traces by Correlation ID in Better Stack

This guide explains how to use correlation IDs to query and debug end-to-end traces across all services in Better Stack.

Overview

Correlation IDs enable unified tracing across the entire ERP-Unlocked platform. A single correlation ID tracks a request from the webapp through Temporal workflows, Dagster pipelines, PDF processing, and all other services.

What is a Correlation ID?

A correlation ID is a UUID v4 generated at the entry point (webapp middleware) that:

Propagates via HTTP headers (x-correlation-id)
Passes through Temporal workflow payloads (correlationId in workflow input)
Included in Dagster run tags (correlation_id)
Used as Langfuse session_id for LLM operations
Set as a span attribute (correlation.id) on all OpenTelemetry spans

Finding a Correlation ID

From Webapp Response Headers

Every HTTP response from the webapp includes the x-correlation-id header:

curl -i https://app.erpunlocked.com/api/erp/connections/123/sync
# Response includes:
# x-correlation-id: 550e8400-e29b-41d4-a716-446655440000

From Application Logs

All services log the correlation ID in structured logs:

{
  "level": "info",
  "message": "Starting ERP sync",
  "correlation_id": "550e8400-e29b-41d4-a716-446655440000",
  "connection_id": "123"
}

From Better Stack UI

Navigate to Traces in Better Stack
Click on any span
Look for the correlation.id attribute in span details

Querying in Better Stack

Method 1: Search by Attribute

Go to Traces in Better Stack
Click the Filter button
Add filter: correlation.id = "550e8400-e29b-41d4-a716-446655440000"
Click Apply

This will show all spans across all services that share this correlation ID.

Method 2: Query Language

Use Better Stack’s query language in the search bar:

correlation.id:"550e8400-e29b-41d4-a716-446655440000"

Method 3: Logs + Traces Correlation

Go to Logs in Better Stack
Search for your correlation ID: correlation_id:550e8400-e29b-41d4-a716-446655440000
Click on a log entry
Use the View Trace link to see the full trace

Expected Trace Structure

When querying by correlation ID, you should see spans from:

webapp - Initial request handling
- HTTP request spans
- Database query spans
- Temporal workflow start spans
temporal-worker (Python) - PDF extraction and order automation
- Temporal activity spans
- Document processing spans
- LLM extraction spans (also visible in Langfuse)
ts-temporal-worker (TypeScript) - Pricing, matching, validation
- Temporal activity spans
- ERP submission spans
dagster - Data pipeline execution
- Asset execution spans (bronze, silver, gold layers)
- dbt transformation spans
- Typesense sync spans

Example Queries

Find All Spans for a Specific Request

correlation.id:"550e8400-e29b-41d4-a716-446655440000"

Find Failed Spans for a Correlation ID

correlation.id:"550e8400-e29b-41d4-a716-446655440000" AND status:error

Find Slow Spans (>5 seconds)

correlation.id:"550e8400-e29b-41d4-a716-446655440000" AND duration:>5000ms

Find Spans from a Specific Service

correlation.id:"550e8400-e29b-41d4-a716-446655440000" AND service.name:"dagster"

Cross-Platform Querying

Langfuse Integration

Langfuse sessions are linked via session_id = correlation_id. To find LLM operations:

In Langfuse, search for session: 550e8400-e29b-41d4-a716-446655440000
This shows all LLM calls (Anthropic Claude API) for that correlation ID
Langfuse traces remain isolated but linked via session_id

Combining Better Stack + Langfuse

For a complete picture:

Better Stack: Query by correlation.id to see all service spans
Langfuse: Query by session_id to see LLM-specific operations
Both use the same correlation ID, enabling cross-platform debugging

Debugging Workflow

Step 1: Identify the Correlation ID

From logs, response headers, or user report.

Step 2: Query Better Stack

correlation.id:"<correlation-id>"

Step 3: Analyze the Trace

Check span durations to find bottlenecks
Look for error spans (red indicators)
Follow the trace flow: webapp → temporal-worker → ts-temporal-worker → dagster

Step 4: Check Langfuse (if LLM involved)

Search by session_id = correlation_id
Review LLM token usage and costs
Check for LLM-specific errors

Step 5: Correlate with Logs

Filter logs by correlation_id
Match log timestamps with span timestamps
Get detailed error messages from logs

Tips

Bookmark Common Queries: Save frequently used correlation ID queries in Better Stack
Set Up Alerts: Create alerts for correlation IDs with errors
Use Trace Comparison: Compare traces with the same correlation ID pattern to identify regressions
Export Traces: Export trace data for offline analysis or sharing with the team

Troubleshooting

Correlation ID Not Found

Check if the request actually went through the webapp (correlation IDs are only generated at entry point)
Verify the correlation ID format (should be UUID v4)
Check if observability is enabled for the service

Missing Spans

Verify OTEL_EXPORTER_OTLP_ENDPOINT is set for the service
Check service logs for OTEL initialization errors
Ensure correlation_id is being propagated (check HTTP headers, task payloads, run tags)

Langfuse Not Linked

Verify correlation_id is being passed to pdf-worker tasks
Check that session_id is set to correlation_id in Langfuse span creation
Review pdf-worker logs for correlation_id extraction

Using Correlation IDs in Code

Importing Utilities

All correlation ID utilities are available from the main observability package:

import {
  getOrCreateCorrelationId,
  extractCorrelationId,
  getCorrelationIdFromContext,
  withCorrelationId,
  setCorrelationIdAttribute,
  CORRELATION_ID_HEADER,
} from '@repo/observability';

Common Patterns

Extract from headers:

const correlationId = extractCorrelationId(request.headers);

Get from current context:

const correlationId = getCorrelationIdFromContext();

Execute with correlation ID:

const result = withCorrelationId(correlationId, () => {
  // Your code here - correlation ID is automatically in OTEL context
  return someOperation();
});

Set on current span:

setCorrelationIdAttribute(correlationId);