Skip to content
Ordermatic Docs

Cleveland Kitchen Sandbox Eval Runbook

This runbook outlines the steps to reproduce the Cleveland Kitchen email-order auto-validation eval against the WhereFour sandbox.

1. Setup Sandbox Connection

Section titled “1. Setup Sandbox Connection”

The eval uses an ERP connection pointing to the WhereFour sandbox. This connection ID scopes all mirrored data in Postgres and Typesense.

Terminal window
# This creates/updates the connection and prints the CONNECTION_ID
node scripts/poc-setup/create-ck-sandbox-connection.mjs

Sandbox connection_id: bd0a441f-d749-404c-9928-230cc2b61cb8

2. Seed Sandbox Data (T2a/T2b/T3a/T3b)

Section titled “2. Seed Sandbox Data (T2a/T2b/T3a/T3b)”

The chosen sync procedure is the unified sandbox seeder. It bypasses Dagster to directly read master data from the WhereFour Sandbox API and populates both Typesense (products, cross_references) and Postgres (erp_products, erp_customers, erp_effective_prices) keyed by the connection_id.

Terminal window
export CONNECTION_ID="bd0a441f-d749-404c-9928-230cc2b61cb8"
node scripts/poc-setup/seed-cleveland-kitchen-sandbox.mjs

3. Dataset Remap Step (T4)

Section titled “3. Dataset Remap Step (T4)”

The golden dataset originally contained WhereFour production IDs. We must remap these _wherefourMatch.customerId values to their equivalent WhereFour Sandbox IDs so the eval assertions pass.

Terminal window
cd packages/eval
tsx scripts/remap-ck-golden-customer-ids.ts

This generates cleveland-kitchen-golden-dataset.sandbox.json.

4. Run the Eval (T5)

Section titled “4. Run the Eval (T5)”

With the mirror seeded and the golden dataset mapped to the sandbox, you can execute the auto-validation runner.

Terminal window
cd packages/eval
pnpm run eval:auto-validation

5. Reproducibility-Snapshot Sanity Checks

Section titled “5. Reproducibility-Snapshot Sanity Checks”

To ensure the environment matches the snapshot loaded on 2026-05-29, verify the following:

  • Test artifacts are removed: Verify that ‘ZZ Test Customer A’, ‘ZZ Test Customer B’, ‘ZZ Bool Test’, ‘ZZ Accrual Real’, and ‘ZZ ACME NoRefs’ (Accrual) are soft-deleted (named DELETED_... and block_shipments: true) or absent from customer listings, and that duplicate ‘Net 30’ payment terms are absent.
  • Connection Isolation: Ensure that Postgres erp_products and Typesense searches are strictly scoped with filter_by: connection_id:=bd0a441f-d749-404c-9928-230cc2b61cb8.