Health Endpoint Improvements

Overview

The health endpoints have been updated to provide better monitoring capabilities while maintaining reliable liveness checks. The key improvement is that health checks now return HTTP 200 for expected health outcomes (including degraded services) with detailed status information reflected in the JSON payload. Only unhandled internal errors will result in HTTP 500 responses.

🚨 Problem Solved

Before (Caused Liveness Check Failures)

• Health endpoints returned HTTP 503 when database was unhealthy
• Liveness checks would fail even when the service was running
• Container orchestration (Docker, Kubernetes) would restart healthy services
• Monitoring tools couldn’t distinguish between service down vs database issues

After (Reliable Liveness Checks)

• Health endpoints return HTTP 200 for expected outcomes (including degraded services)
• Database status reflected in payload for detailed monitoring
• Liveness checks pass when service is running (even with database issues)
• Readiness checks can inspect payload for database connectivity
• HTTP 500 only for unhandled internal errors

🔧 Technical Changes

1. HTTP Status Code Changes

// Before: Variable status codes
status: dbHealth.status === 'healthy' ? 200 : 503;

// After: 200 for expected health outcomes (including degraded services)
status: 200; // Return 200 for completed health checks
// HTTP 500 only occurs for unhandled internal errors

2. New Service Status Field

// Added to all health responses
serviceStatus: dbHealth.healthy ? 'healthy' : 'degraded';

3. Consistent Response Format

Both /health and /api/health endpoints now follow the same pattern:

• HTTP 200 returned for expected health outcomes (including degraded subservices)
• HTTP 500 returned for unhandled internal errors or exceptions
• Service status in payload indicates overall health
• Database details provide granular status information

Note for Operators: Check the payload serviceStatus field for granular health information. HTTP 200 indicates the health check completed successfully, while HTTP 500 indicates an unexpected service failure that requires investigation.

📊 Response Examples

Healthy Service (Database Connected)

{
  "status": "ok",
  "timestamp": "2024-01-15T10:30:00.000Z",
  "service": "erp-unlocked-web",
  "uptime": 3600,
  "memory": {
    "used": 128,
    "total": 512
  },
  "database": "connected",
  "databaseStatus": "healthy",
  "databaseMessage": "Database healthy: 45ms response",
  "serviceStatus": "healthy"
}

Degraded Service (Database Issues)

{
  "status": "ok",
  "timestamp": "2024-01-15T10:30:00.000Z",
  "service": "erp-unlocked-web",
  "uptime": 3600,
  "memory": {
    "used": 128,
    "total": 512
  },
  "database": "disconnected",
  "databaseStatus": "timeout",
  "databaseMessage": "Database health check timed out",
  "serviceStatus": "degraded"
}

Service Error (Critical Failure)

{
  "status": "error",
  "timestamp": "2024-01-15T10:30:00.000Z",
  "service": "erp-unlocked-web",
  "error": "Critical service failure",
  "serviceStatus": "degraded"
}

🎯 Use Cases

1. Liveness Checks (Service Running)

# Docker health check
curl -f http://localhost:4321/health
# Returns 200 - health check completed (service is running)
# Note: 200 doesn't guarantee service is healthy, check payload for details

2. Readiness Checks (Inspect Payload)

# Check if service is ready to handle requests
curl http://localhost:4321/health | jq '.serviceStatus'
# Returns "healthy" or "degraded"

3. Monitoring & Alerting

# Monitor database connectivity
curl http://localhost:4321/health | jq '.database'
# Returns "connected" or "disconnected"

# Check overall service health
curl http://localhost:4321/health | jq '.serviceStatus'
# Returns "healthy" or "degraded"

4. Container Orchestration

# Kubernetes liveness probe
livenessProbe:
  httpGet:
    path: /health
    port: 4321
  initialDelaySeconds: 30
  periodSeconds: 10

# Kubernetes readiness probe
readinessProbe:
  httpGet:
    path: /health
    port: 4321
  initialDelaySeconds: 5
  periodSeconds: 5

🔍 Database Status Categories

The health endpoint now provides detailed database status information:

Status Types

1. healthy - Database responding normally (< 5s)
2. warning - Database slow (> 5s but responding)
3. critical - Database connection failed
4. timeout - Database health check timed out
5. unavailable - Database connection unavailable
6. unauthorized - Database authentication failed
7. error - Database query or other error

Error Differentiation

• Timeout errors - Properly identified and marked
• Connection errors - Network, refused, not found issues
• Authentication errors - Auth, permission issues
• Query errors - Syntax, invalid query issues
• Generic errors - Sanitized to avoid sensitive information exposure

🚀 Benefits

1. Reliable Liveness Checks

• No more false failures due to database issues
• Container orchestration works correctly
• Service restarts only when actually needed

2. Better Monitoring

• Granular status information for different failure types
• Service degradation clearly indicated
• Database connectivity separately monitored

3. Improved Debugging

• Clear error categorization for faster troubleshooting
• Sanitized error messages for security
• Consistent response format across endpoints

4. Production Stability

• Health checks won’t crash the service
• Authentication failures isolated from health monitoring
• Service remains accessible even during database issues

📋 Implementation Details

Files Modified

1. apps/webapp/src/pages/health.ts - Main health endpoint
2. apps/webapp/src/pages/api/health.ts - API health endpoint

Key Changes

• HTTP 200 returned for expected health outcomes (including degraded services)
• HTTP 500 returned for unhandled internal errors or exceptions
• serviceStatus field added to all responses
• Database status reflected in payload, not HTTP status
• Consistent error handling across both endpoints

Backward Compatibility

• Existing health checks continue to work
• New fields are additive (non-breaking)
• Same endpoint URLs maintained

🔧 Monitoring Integration

Prometheus/Grafana

// Custom metric for service status
const serviceStatus = response.serviceStatus; // "healthy" or "degraded"
const databaseStatus = response.databaseStatus; // Detailed DB status

Datadog/New Relic

// Service health metric
dogstatsd.gauge('service.health', serviceStatus === 'healthy' ? 1 : 0);
dogstatsd.gauge('database.health', databaseStatus === 'healthy' ? 1 : 0);

Custom Alerting

// Alert on service degradation
if (response.serviceStatus === 'degraded') {
  sendAlert('Service degraded - database issues detected');
}

🎉 Summary

The health endpoints now provide:

• Reliable liveness checks (HTTP 200 for completed health checks, HTTP 500 for unexpected failures)
• Detailed health information in the payload (including serviceStatus field)
• Database status monitoring without affecting service availability
• Better container orchestration support
• Enhanced monitoring capabilities for production environments

Key Insight: HTTP 200 indicates the health check completed successfully, while HTTP 500 indicates an unexpected service failure. Operators should check the payload serviceStatus field for granular health information rather than relying solely on HTTP status codes.

This solves the original production issue where authentication failures were causing health checks to fail, while also improving the overall monitoring and alerting capabilities of your application.