Observability Testing Guide

This guide covers the secured observability test endpoint and how to use it from the admin panel.

Overview

The observability test is a diagnostic tool for verifying that your observability stack (OpenTelemetry, trace collection, logging) is working correctly. It generates test logs, creates spans, and allows you to verify that all data is being captured properly in your observability backend (Better Stack, SigNoz, etc.).

Security Implementation

The endpoint is protected by multiple layers:

1. Authentication Required

The endpoint requires an authenticated user
Uses the standard getUserFromRequest() from @repo/auth/server
Returns 401 Unauthorized if the user is not authenticated

2. Environment Flag Control

The endpoint requires OBSERVABILITY_TEST_ENABLED=true to function
Returns 403 Forbidden if the flag is not set or is false
This allows you to easily enable/disable testing without code changes

3. Production Safety

In production environments (NODE_ENV === 'production'), the endpoint is disabled by default
To enable in production, you must explicitly set OBSERVABILITY_TEST_ENABLED=true
This prevents accidental exposure of testing endpoints in production

Usage

From Admin Panel (Recommended)

Navigate to /admin/observability in your webapp
You’ll see the Observability Testing page with:
- A Run Observability Test button
- Information about what’s being tested
- Next steps for verification
- Expected output format
Click the button to execute the test
The response will show:
- Success/failure status
- Test execution details
- Optionally, full response payload

From API (For Automation)

Endpoint: GET /api/observability-test

Requirements:

User must be authenticated (via Clerk session)
OBSERVABILITY_TEST_ENABLED=true in environment

Example:

# Must be run authenticated (browser session or API key auth)
curl -H "Cookie: __session=..." https://your-app.com/api/observability-test

Success Response (200):

{
  "status": "success",
  "message": "Observability test completed successfully",
  "instructions": "Check Better Stack dashboard for logs with trace_id and span_id",
  "details": {
    "timestamp": "2024-11-01T12:34:56.789Z",
    "test_duration_ms": 100,
    "test_id": "otel-test-001"
  }
}

Forbidden Response (403 - not enabled):

{
  "status": "forbidden",
  "message": "Observability test endpoint is not enabled. Set OBSERVABILITY_TEST_ENABLED=true",
  "code": "ENDPOINT_NOT_ENABLED"
}

Unauthorized Response (401 - not authenticated):

{
  "status": "unauthorized",
  "message": "Authentication required to access observability test",
  "code": "AUTH_REQUIRED"
}

Environment Configuration

Development Environment

Add to your .env.development:

OBSERVABILITY_TEST_ENABLED=true

The endpoint will be enabled and accessible at /admin/observability

Staging Environment

Add to .env.staging or deployment secrets:

OBSERVABILITY_TEST_ENABLED=true
NODE_ENV=staging

Production Environment

By default, the endpoint is disabled in production.

To enable it temporarily for testing, add:

OBSERVABILITY_TEST_ENABLED=true
NODE_ENV=production

Best Practice: Set an expiration date for when you’ll disable it again:

# Valid only for debugging a specific issue
OBSERVABILITY_TEST_ENABLED=true
OBSERVABILITY_TEST_EXPIRES_AT=2024-11-15T00:00:00Z

What’s Being Tested

The test endpoint verifies:

Logger Functionality - Structured logging with context injection
Trace Span Creation - OpenTelemetry span generation and context
Multiple Log Levels - Debug, info, warn, error level logging
Structured Metadata - Including timing, IDs, and environment info

Verification Steps

After running the test:

Check your observability backend (Better Stack, SigNoz, Datadog, etc.)
Look for recent logs with the test ID otel-test-001
Verify trace linkage - All logs should have the same trace_id
Check span information - Logs should include span_id fields
Confirm lifecycle - Trace should show complete span from start to end

Security Considerations

What’s NOT Exposed

Raw environment variables (NODE_ENV, service names are not returned)
Database connection details
API keys or credentials
Sensitive configuration

What’s Included

Test execution timestamp
Test duration in milliseconds
Test ID for tracking in logs
Generic instructions

Access Control

The endpoint inherits access control from:

Clerk authentication (must be signed in)
Admin panel access requirements
Environment configuration

Troubleshooting

“Observability test endpoint is not enabled”

Set OBSERVABILITY_TEST_ENABLED=true in your environment
Restart the server if running locally

“Authentication required to access observability test”

You must be signed in with a valid Clerk session
Try accessing /api/auth/me to verify authentication

“Endpoint disabled in production”

You’re in production (NODE_ENV=production)
Set OBSERVABILITY_TEST_ENABLED=true if you need to test
Remember to disable it after testing

No logs appearing in observability backend

Check your OTEL_EXPORTER_OTLP_ENDPOINT configuration
Verify Better Stack API key is correctly set
Check network connectivity to your observability backend
Review application logs for export errors

Implementation Details

Files Changed

apps/webapp/src/pages/api/observability-test.ts - Secured API endpoint
apps/webapp/src/pages/admin/observability.astro - Admin UI page
apps/webapp/src/pages/admin/index.astro - Added admin navigation link

Architecture

Admin User
    ↓
/admin/observability (Astro page)
    ↓
Frontend JavaScript calls
    ↓
/api/observability-test (Secured API endpoint)
    ↓
Authentication Check (getUserFromRequest)
    ↓
Environment Flag Check (OBSERVABILITY_TEST_ENABLED)
    ↓
Generate Test Logs & Spans (withTrace)
    ↓
Export via OpenTelemetry
    ↓
Better Stack Dashboard