Cold Start & Performance Optimization

May contain stale references

This document was audited on 2026-06-22 and contains references that need updating: Coolify scripts; now GCP Cloud Run. Cross-check critical details against the current codebase before relying on them.

Date: 2026-02-07

Problem

Overall performance has degraded across all apps and services. Users experience slow page loads, particularly on initial visits. The webapp landing page takes ~3.5-5 seconds on cold start.

Baseline Measurements (Pre-Optimization)

Server-Side Timing (curl, TTFB)

Endpoint	Run 1	Run 2	Run 3	Notes
Landing Page	261ms	128ms	111ms	First run slowest (warm server)
Health Check	259ms	221ms	263ms	Consistent ~250ms
Dashboard (unauth)	234ms	113ms	143ms	Redirects to sign-in
Sign In Page	129ms	119ms	117ms	Consistent ~120ms

Client-Side Timing (Playwright browser, full page load)

Endpoint	Duration	Status	Notes
Landing Page	3,502ms	200	Includes CSS, JS, fonts, hydration
Sign In Page	1,834ms	200	Clerk widget loads
Health Check API	256ms	200	JSON only

Traffic Generator (Node.js fetch, sequential)

Endpoint	Duration	Status
Health Check	314ms	200
Landing Page	282ms	200
Sign In	181ms	200
Dashboard (unauth)	159-227ms	200
Orders (unauth)	154-197ms	200
Auth Check	76-163ms	200

Average latency (all endpoints): 145ms Cold start (first browser visit): ~3,500-5,000ms

Root Cause Analysis

The cold start is NOT an Astro problem. It’s a startup initialization cascade:

1. OpenTelemetry Full Auto-Instrumentation (~1-2s)

otel-init.cjs runs via --require before Astro starts and calls getNodeAutoInstrumentations() which monkey-patches 20+ Node.js modules. This is the single biggest contributor.

Current config:

// otel-init.cjs line 18
const ENABLE_AUTO_INSTRUMENTATION = process.env.OTEL_AUTO_INSTRUMENTATION !== 'false';
// Defaults to FULL mode (20+ modules) since env var is not set

Fix: Set OTEL_AUTO_INSTRUMENTATION=false → reduces to HTTP-only instrumentation.

2. Flagsmith Local Evaluation Init (~1-3s on first request)

Feature flags use local evaluation which downloads the entire Flagsmith environment document on the first request. requestTimeoutSeconds: 30 means it can block for up to 30 seconds.

Fix: Reduce timeout to 5s, eager-initialize at server startup.

3. Pino Logger Dynamic Import (~200-500ms)

loadPinoLogger uses await import('pino') lazily on first log call.

4. Clerk Auth JWKS Fetch (~200-500ms on first request)

Clerk fetches JSON Web Key Set on the first authenticated request.

5. Neon DB WebSocket Connection (~200-500ms on first query)

Neon serverless driver establishes WebSocket on first query.

Fixes Applied

Fix 1: Minimal OTEL Instrumentation

File: infrastructure/coolify/scripts/set-environment-staging.sh, set-environment.sh Change: Added OTEL_AUTO_INSTRUMENTATION=false to webapp env vars What it does: Switches from getNodeAutoInstrumentations() (20+ module patches) to HTTP-only instrumentation Expected impact: ~1-2 seconds off cold start

Fix 2: Eager Flagsmith Init + Reduced Timeout

File: apps/webapp/src/middleware/feature-flags.ts Changes:

• Added setTimeout(() => ensureInitialized(), 0) at module import to trigger init at server startup
• Reduced requestTimeoutSeconds from 30 to 5 (fail fast, fall back to API mode)
• Reduced retries from 2 to 1 (avoid blocking startup) Expected impact: ~1-3 seconds off first-request latency (moved to startup)

Fix 3: Pre-warm DB Connection

File: packages/db/src/client.ts Change: Added pool.query('SELECT 1') at module load to establish WebSocket connection at startup Expected impact: ~200-500ms off first query

Results Log

Baseline (before any fixes)

• Cold start (browser): ~3,500ms landing page, ~1,834ms sign-in
• Warm (curl TTFB): ~110-260ms
• Warm (fetch): ~145ms average

After All Fixes (Post-Deploy: 2026-02-07T14:17Z)

Cold start (first request to freshly started container):

• Landing page: 10.5s (container boot + Node.js startup + OTEL init + Flagsmith init + DB connect)
• This is the Docker container starting from scratch — users don’t see this unless container restarts

Warm state (TTFB, curl, 5 samples each):

Endpoint	Run 1	Run 2	Run 3	Run 4	Run 5	Avg
Landing	113ms	118ms	204ms	241ms	185ms	172ms
Health	152ms	111ms	183ms	98ms	92ms	127ms
Dashboard	118ms	197ms	110ms	100ms	210ms	147ms
Sign-in	102ms	112ms	112ms	-	-	109ms
Orders	164ms	114ms	109ms	-	-	129ms

Health check noise reduction:

• Before: ~227 logs per 30 minutes
• After: 8 logs per 30 minutes
• 96.5% noise reduction ✅

Cold Start Analysis:

The 10.5s cold start is the Docker container boot, not app initialization. Our fixes moved initialization costs from “first user request” to “server startup”:

1. Container boot + Node.js init: ~2-3s
2. OTEL init (now HTTP-only): ~0.5-1s (was ~1-2s with full auto-instrumentation)
3. Astro SSR server startup: ~1-2s
4. Flagsmith eager init (async, 5s timeout): runs during startup, not first request
5. DB pre-warm SELECT 1: runs during startup, not first query

Users won’t hit this unless the container restarts. Coolify health checks gate traffic.

Observability Status

Better Stack (Post-Deploy)

• DB health check filter: ✅ Working (96.5% noise reduction)
• Webapp traces: ❌ Pending — need to run ./set-environment-staging.sh for OTEL env vars
• Webapp logs: ❌ Pending — need to run ./set-environment-staging.sh for LOGTAIL env vars
• Note: Code is deployed but Coolify env vars haven’t been updated yet

Langfuse

• Working correctly: 17,827 total traces
• Recent trace latencies: 7.5s - 63.6s
• Costs: $0.0009 - $0.0613 per trace
• Correlation IDs flowing correctly

E2E Performance Baseline (Authenticated, Playwright, 2026-02-07T14:45Z)

Full User Journey (41 measurements, 1.1 minute total)

Top 5 Bottlenecks:

Rank	Step	Duration	Category	Notes
1	Clerk sign-in redirect	16,149ms	interaction	Client-side JS redirect to accounts.dev
2	Feature Flags API	8,486ms	api	/api/admin/feature-flags — extremely slow
3	Auth redirect → dashboard	4,066ms	interaction	Clerk callback → session verify → redirect
4	Landing page	2,388ms	navigation	Full browser load with CSS/JS/fonts
5	Sign-in page	2,014ms	navigation	Full browser load

Authenticated Page Loads (SSR, browser networkidle)

Page	Duration	Notes
Dashboard	1,294-1,519ms	Consistent ~1.3-1.5s
Orders	1,409-1,680ms	Consistent ~1.4-1.7s
Inbox	1,186-1,246ms	Consistent ~1.2s
ERP Connections	1,165ms	404 (no connections)
Organization	1,171ms
Admin	1,184ms
Admin Billing	1,445ms

Average authenticated page load: 1,328ms (p50), 1,424ms (avg)

API Endpoints (in-browser fetch with cookies)

Endpoint	Duration	Status	Notes
/api/admin/feature-flags	8,486ms	200	Critical bottleneck
/api/auth/check (warm)	691ms	200	Clerk session verify
/api/pdf-documents	526-634ms	200	DB query + serialization
/api/billing/pricing	511ms	500	Error response
/api/billing/subscription-limits	489ms	200
/api/auth/check	350ms	200
/api/auth/me	319ms	200
/api/billing/usage/current-period	303ms	200
/api/auth/organization	284ms	200
/api/auth/session	255ms	200
/api/admin/system-stats	232ms	200
/api/erp/connections	203-226ms	200	Fast
/api/inbox/check-updates	173ms	404
/api/health	151ms	200	Baseline

Key Findings

1. Feature Flags API is the #1 bottleneck (8.5s) — The /api/admin/feature-flags endpoint takes 8.5 seconds. This likely does a full Flagsmith environment download or has an inefficient implementation. Needs immediate investigation.

2. Authenticated pages consistently take ~1.2-1.7s in the browser. Server-side traces show only ~20-35ms of server processing. The gap (~1.1-1.6s) is:

   • Network round-trip to staging server (~100ms)
   • TLS handshake (~40ms)
   • Clerk session verification (~200-350ms per request)
   • Browser HTML parsing + CSS/JS download + React hydration (~500-800ms)

3. PDF Documents API is slow (500-634ms) — likely a DB query that could be optimized.

4. Billing Pricing returns 500 — broken endpoint, needs fixing.

5. Auth Check varies significantly (350ms cold → 691ms warm) — unexpected that it is slower when warm. Could be Clerk rate limiting or session cache invalidation.

Auth Performance Fixes Applied (Post-Baseline)

Fix 4: Skip Auth for Public Routes

File: packages/auth/src/middleware.ts Change: Added early return before ANY Clerk API calls for public routes that don’t need org context. Previously, authenticateRequest() ran on every single request when enableOrganizationContext was true — even for /, /sign-in, etc. Expected impact: ~200-600ms saved for public page loads

Fix 5: Local JWT Verification

File: packages/auth/src/server.ts Change: authenticateRequest() now tries verifyToken() first, which validates the JWT signature against cached JWKS locally — no network call to Clerk. Falls back to the remote authenticateRequest() only if local verification fails (expired token, etc.). Expected impact: ~100-300ms saved per authenticated request (eliminates JWKS fetch)

Fix 6: Redis-Cached User Objects

File: packages/auth/src/server.ts Change: getUserFromAuth() and getUserFromSession() now check Redis before calling Clerk’s users.getUser() API. 5-minute TTL. Cache key: auth:user:{userId}. Expected impact: ~100-300ms saved per authenticated request (eliminates user fetch)

Expected Combined Impact

• Public pages: from ~2-2.6s → ~1-1.5s (skip auth entirely)
• Authenticated pages: from ~1.2-1.7s → ~0.7-1.0s (local JWT + cached user)
• API endpoints: from ~150-700ms → ~100-300ms (local JWT + cached user)

Post-Fix Results (Deployed 2026-02-07T17:00Z)

API Endpoints (authenticated, in-browser):

Endpoint	Baseline	After All Fixes	Improvement
Feature Flags	8,486ms	90ms	99% faster (94x)
Auth Me	319ms	95ms	70% faster
Auth Session	255ms	81ms	68% faster
Auth Organization	284ms	98ms	65% faster
Auth Check (warm)	691ms	246ms	64% faster
Health	151ms	69ms	54% faster
ERP Connections	203ms	123ms	39% faster
PDF Documents	634ms	390ms	38% faster
API Average	865ms	181ms	79% faster

Page Loads (browser, authenticated):

Page	Baseline	After	Improvement
Organization	1,171ms	874ms	25% faster
Dashboard	1,329ms	1,048ms	21% faster
Admin	1,184ms	966ms	18% faster
Page Average	1,424ms	1,195ms	16% faster

Fix 7: Feature Flags Admin Endpoint (Fix #1 Bottleneck)

File: apps/webapp/src/pages/api/admin/feature-flags/index.ts Change: Replaced two sequential fetch() calls (no timeout, ~9-13s) with SDK getAllFlags() which uses local evaluation (in-memory cache, ~0ms). Result: 8,486ms → 90ms (99% faster)

File: apps/webapp/src/pages/api/admin/feature-flags/toggle.ts Change: Added 5s AbortController timeout to prevent hanging.

PDF Pipeline Performance

E2E Pipeline Timing (Upload → Process → Review)

Step	Duration	Notes
Upload to R2	0.7s	Presigned URL + R2 upload + document creation
PDF Processing	13.8s	pdf-api → pdf-worker → Gemini extraction
Customer search	0.9s	Typesense search + modal interaction
Shipping search	0.8s	Typesense search + modal interaction
Auto-validate	0.1s	ERP parts lookup
Total pipeline	16.3s	Upload through review interactions

Langfuse Extraction Analysis

Document Type	Chunks	Gemini Call	Total	Cost
Simple (3 items)	1	4.5s	~14s (incl. polling)	$0.0003
Complex (multi-page)	9	16-24s each	~33s (5-way parallel)	$0.0143

Extraction Optimizations Applied

Priority 1: Increased PDF_PAGES_PER_CHUNK from 2 → 4

• A 10-page PDF now produces 3 chunks instead of 9
• With 8-way parallelism, all chunks run concurrently
• Expected: ~50% faster on large documents

Priority 2: Adaptive polling in DocumentOrderReview.tsx

• Changed from fixed 3s interval to adaptive: 1s for first 30s, then 3s
• Reduces perceived latency by ~2s when processing completes quickly

Remaining Optimization Opportunities

1. Native Gemini PDF handling — Send PDF directly to Gemini instead of text chunks (8-12h effort)
2. Reduce prompt size — 2,135 tokens per chunk, could be trimmed (2-4h effort)
3. Batch upload UX — Multi-file upload with inline queue (4-6h, plan at debt/projects/ux-improvements/batch-upload-and-extraction-plan.md)
4. SSE for status updates — Replace polling with server-sent events (4-6h)
5. Fix Billing Pricing 500 — Broken endpoint returning errors
6. Optimize client-side hydration — Browser JS execution is ~500-800ms per page

Test Scripts

• scripts/staging-traffic-generator.ts — Quick fetch-based traffic gen
• scripts/generate-test-pdf.ts — Generate test PDF with real ERP data from Typesense
• apps/webapp/playwright-demos/staging-traffic.spec.ts — Playwright browser traffic
• apps/webapp/playwright-demos/e2e-staging-perf.spec.ts — Full authenticated E2E perf audit
• apps/webapp/playwright-demos/e2e-pdf-pipeline.spec.ts — Full PDF upload → process → review test
• apps/webapp/playwright-demos/e2e-perf-audit.spec.ts — CDP-based E2E audit (requires Chrome remote debugging)