Cleveland Kitchen (CK) Demo Day — Manual Fallback Procedures

May contain stale references

This document was audited on 2026-06-22 and contains references that need updating: fixture PDF paths do not exist in current codebase. Cross-check critical details against the current codebase before relying on them.

Status: ACTIVE Owner: Engineering / Product / Customer Success Target Environment: Staging (https://staging.ordermatic.co) Last Updated: June 2026

This runbook provides step-by-step procedures to verify the health of the email-to-order pipeline prior to the Cleveland Kitchen (CK) demo and describes clear, manual fallback steps for every component of the pipeline.

The email-to-order pipeline relies on the following sequence:

1. Ingestion: Customer/Prospect sends a purchase order (PDF/Excel) to the email inbox (orders@ordermatic.co or orders+ck@ordermatic.co).
2. Polling / Webhook: Gmail API / Cloudflare Email Worker intercepts the email, parses the attachments, and invokes /api/webhook/inbound-email.
3. Extraction: The email worker or the webapp triggers a Temporal workflow PDFExtractionWorkflow that calls Gemini (via LLM/Parse engine) to extract order metadata and line items from the PDF.
4. Matching: The extraction results are ingested and run through the Typesense-backed fuzzy matching engine to match the extracted Customer and ERP Parts with WhereFour data.
5. Review Queue: The document appears in the webapp review queue (https://staging.ordermatic.co/orders?status=pending_review) for approval and final submission to the WhereFour ERP.

In the event of any failure at any of these steps, follow the fallback procedures described in this document to ensure the demo continues flawlessly.

---

🚀 Pre-Demo Health Check (Smoke Tests)

Perform these automated and manual health checks 30 minutes before the demo. These checks are designed to verify the entire pipeline in less than 2 minutes.

⚡ Quick Smoke Test Commands

1. Webapp & API Gateway Health Check (Duration: < 5s)

Verify that the staging server is up, healthy, and all dependent services (Database, Redis, etc.) are responsive.

curl -s -f https://staging.ordermatic.co/api/health | jq .

Expected Response:

{
  "status": "ok",
  "serviceStatus": "healthy",
  "service": "erp-unlocked-web"
}

2. E2E Webapp Smoke Test (Duration: < 15s)

Run the automated Playwright smoke suite to verify public endpoints, routing, and basic responsiveness.

# Navigate to the webapp directory and run the smoke suite
cd apps/webapp
STAGING_URL=https://staging.ordermatic.co pnpm exec playwright test playwright-demos/local-smoke.spec.ts

Expected Output:

  ✅ Landing (/): HTTP 200
  ✅ Sign-in (/sign-in): HTTP 200
  ✅ Health API (/api/health): HTTP 200
  1 passed (12.2s)

3. Dagster Integration Health (Duration: < 5s)

Verify that the Dagster daemon and code-servers are responsive and scheduling is active.

curl -s -f -I https://dagster.ordermatic.co/graphql | grep -i "http"

Expected Response: HTTP/1.1 200 OK or HTTP/2 200 (Verifies the Dagster webserver/GraphQL is active).

4. Email Pipeline Check (Duration: < 60s)

1. Send a standard test PDF purchase order (e.g., apps/webapp/tests/fixtures/ck/sysco-po.pdf) from your email client to orders+ck@ordermatic.co.
2. Open the Staging Inbox and verify that the new email event appears and starts processing within 60 seconds.

---

📊 Monitoring Dashboards

Use these dashboards to monitor pipeline health in real-time during the demo:

Dashboard	URL	Purpose
Staging Webapp	https://staging.ordermatic.co	Main review inbox and order list
Dagster Orchestrator	https://dagster.ordermatic.co	Check status of daily WhereFour scrape jobs (wherefour_pricing_tiers_job, wherefour_custom_line_defs_job) + bronze sync
Sentry (Staging)	https://sentry.io (Filter: project:webapp, env:staging)	Real-time backend/frontend error reporting
Better Stack Logs	https://app.betterstack.com	Structured server-side logs and infrastructure alerts
PostHog Analytics	https://app.posthog.com (Filter: env:staging)	Live telemetry of user interactions & system operations

---

🛠 Manual Fallback Procedures

If any pipeline component fails, follow the corresponding manual workaround below.

1. Ingestion / Gmail API / Polling Failure

Symptom: You send an email with a purchase order, but nothing appears in the Ordermatic inbox after 2 minutes. The Gmail API might be down, or the Cloudflare email worker/poller is unresponsive.

Workaround A: Direct Webapp Drag-and-Drop Ingestion (Highly Recommended)

Bypasses the entire email/polling layer and directly feeds the document into the processing pipeline.

1. Download the customer’s purchase order PDF or Excel sheet to your local computer.
2. Log in to the Staging Dashboard.
3. Drag and drop the file directly into the Upload Purchase Order zone at the top of the dashboard.
4. Click Upload and Process. The document will immediately enter the extraction pipeline.

Workaround B: Manually Trigger the Inbound Email Webhook (Engineering Fallback)

If you must demonstrate the email ingestion path but Gmail polling is broken, you can manually post the raw MIME email to the webhook, simulating the Cloudflare Worker.

1. Download the raw .eml or copy the raw MIME content of the purchase order email.
2. Retrieve the INBOUND_EMAIL_SECRET from 1Password (ERP-Unlocked → Staging Environment Secrets).
3. Generate the HMAC signature of the payload, or run the manual trigger command:

   # Simulates an inbound email event using the staging secret
   curl -X POST https://staging.ordermatic.co/api/webhook/inbound-email \
     -H "Content-Type: application/json" \
     -H "X-Inbound-Email-Signature: <computed_hmac_signature>" \
     -d '{
       "messageId": "msg-demo-manual-fallback-12345",
       "from": "buyer@clevelandkitchen.com",
       "to": ["orders+ck@ordermatic.co"],
       "subject": "Urgent Purchase Order PO-109923",
       "receivedAt": "'$(date -u +"%Y-%m-%dT%H:%M:%SZ")'",
       "aliasId": "alias-ck-staging-123",
       "subaddress": "ck",
       "erpConnectionId": "conn_wherefour_ck_staging",
       "clerkOrganizationId": "org_clerk_ck_staging",
       "attachmentCount": 1,
       "uploadedDocumentIds": ["doc_demo_fallback_pdf_id"],
       "triageDecision": "process_attachments",
       "triageReason": "Valid PO attachment found"
     }'

4. Check the Staging Inbox. The synthetic email event will be processed immediately.

---

2. Extraction Failure (Gemini/Parser/Temporal)

Symptom: The order ingestion succeeded (it shows in the inbox), but its status is “Extraction Failed”, “Low Confidence”, or line items are completely missing or scrambled. This can happen if the LLM provider is down, or if the PDF layout is extremely abnormal.

Workaround: Use Manual Extraction & Verification UI

1. In the Staging Inbox, click on the failed or low-confidence order to open the Order Review & Editor interface.
2. Click the Manual Edit button on the top-right of the order sheet.
3. Use the split-pane viewer to read the original PDF on the left and enter/correct the fields on the right:
   • PO Number: Type the PO number manually.
   • Customer Name: Select from the dropdown.
   • Required Delivery / Ship Date: Manually enter or pick from the calendar.
4. To enter missing line items, click Add Row under the items table:
   • Manually enter the extracted SKU, Quantity, and Price as shown on the PDF.
   • Click Save Draft to preserve your manual entry.

---

3. Matching Failure (Typesense/Fuzzy Match/ERP Connection)

Symptom: Extraction succeeded and line items are present, but the customer name or the products show “Unmatched” or are matched to the wrong ERP Part/Customer in WhereFour. This happens if the customer has updated SKUs that are not yet synchronized or if Typesense matching thresholds missed the match.

Workaround: Direct Matching Overrides in Review UI

1. Click the unmatched order in the Review Queue.
2. Resolve Customer Match:
   • Under the Customer Details section, notice the red “Unmatched” badge.
   • Click the search input box. Type the customer’s legal name or WhereFour account number.
   • Select the correct customer from the dropdown. The system will prompt you to save this as a permanent cross-reference (xref). Select “Yes, Save Matcher”.
3. Resolve Item/SKU Matches:
   • Scroll down to the Line Items table.
   • For each line item with an unmatched status:
     • Click inside the ERP Product / SKU column box.
     • Type the SKU, description, or UPC code of the item to search live from the WhereFour catalog.
     • Select the correct item from the list.
     • (Optional) Adjust the unit price or quantity if any discrepancies exist.
4. Once all customer and item lines are matched (all badges turn green/resolved), click Submit to WhereFour to push the finalized order to the ERP.

---

🔐 Credentials & Test Data

Staging Credentials Location

All active demo accounts and credentials must be fetched from 1Password (ERP-Unlocked vault):

• Staging Admin User: 1Password → CK Demo Account

  • Username: demo+ck@ordermatic.co
  • Password: [Refer to 1Password]

• WhereFour Sandbox API Credentials: 1Password → WhereFour Sandbox Credentials

• Clerk Verification & Sign-in Token Generation: If password login is failing or Clerk auth is sluggish, you can bypass the password form by generating a one-time sign-in link via the Clerk Admin API using the CLI:

  # Requires CLERK_SECRET_KEY in your env
  CLERK_SECRET_KEY=$(op item get "Clerk Secret Key" --vault "ERP-Unlocked" --fields label=password --reveal)
  
  curl -X POST https://api.clerk.com/v1/sign_in_tokens \
    -H "Authorization: Bearer $CLERK_SECRET_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "user_id": "user_2P9v1...",
      "expires_in_seconds": 600
    }'

  Copy the returned url field and paste it directly into the browser to authenticate immediately.

Tested & Verified Demo Files

Do not try to find arbitrary PDFs during the live demo. Use these pre-verified, robust test documents located in the repository:

1. Standard PO (Sysco):
   • File Path: apps/webapp/tests/fixtures/ck/sysco-po.pdf
   • Customer: Sysco Cleveland (WhereFour ID: cust_sysco_cle_99)
   • Items: 4 standard product lines
2. Complex Multi-Page Excel Order (Giant Eagle):
   • File Path: apps/webapp/tests/fixtures/ck/giant-eagle-order.xlsx
   • Customer: Giant Eagle Corp (WhereFour ID: cust_giant_eagle_101)
   • Items: 12 lines requiring contract-pricing resolution

---

📞 Escalation & Contact Path

If fallback procedures fail or you encounter a hard database/infrastructure block during the demo, immediately contact the on-call team:

1. Primary On-Call SRE / Backend Engineer:
   • Slack: #on-call-eng
   • Phone/Pager: [Refer to PagerDuty / Sentry integration]
2. Lead Platform Engineer:
   • David (@david on Slack)
3. Product Manager (Demo Support):
   • CS/PM Lead (@cs-demo-support on Slack)