Ordermatic Local — Regulated Deployment Architecture

Status: APPROVED FOR IMPLEMENTATION — Supersedes 2026-07-18 draft Decision Date: 2026-08-29 · Revised: 2026-08-30 (OSS-pattern research pass — see §14) Trigger: YBI Commercialization Grant (DGX Spark) + land-and-expand strategy

2026-08-30 changes: removed MinIO, mDNS and the FTS5 virtual table; corrected the Temporal, Typesense and vLLM pins; corrected the signing story (minisign, not cosign, on the client); resolved every §10 open decision; added nine appliance-product concerns as §14. A second pass then consolidated the client surface: deleted the gRPC contract, the API gateway, the WebSocket channel, the direct-Typesense path, the client prompt/agent/tribal-knowledge tables and the Node sidecar, all of which duplicated something apps/webapp already ships (§13).

---

1. Executive Summary

Ordermatic Local is a dual-mode desktop client + on-prem appliance that lets regulated manufacturers (CMMC/CUI) process orders compliantly while escaping the compliance tax on commercial work.

Layer	Cloud SaaS	Ordermatic Local
Order Processing	Cloud Temporal + Claude	DGX Temporal + Local LLM (all orders CUI by default)
Search/Typesense	Multi-tenant Iceberg → dbt → Typesense	Single-tenant Typesense (direct ERP SQL sync)
Procurement/Non-Order	Cloud workers	DGX Temporal workflows (same runtime as orders)
Tribal Knowledge	Not captured	Client-local SQLite (institutional memory)
Updates	Continuous deploy	Minisigned Tauri updates + offline flag document (§11)
Compliance Boundary	Enclave/tenant isolation	Enforced in the client — routes data, never leaks

Core Insight: The desktop client is the compliance boundary. It decides what goes to DGX (orders = CUI) vs what stays local (procurement, prompts, tribal knowledge). The DGX appliance never sees non-order data. The cloud never sees CUI.

---

2. Architecture Decision: Dual-Mode Client + DGX Appliance

2.1 Deployment Model (Resolved)

Question	Answer
On-prem vs cloud enclave?	On-prem appliance (DGX Spark) — grant-funded, contractually required
CUI classification?	All orders = CUI by default — eliminates classification risk
Non-order features?	Client-local only — procurement, vendor tracking, prompt library run in Tauri
Air-gap support?	Yes — client works offline; DGX syncs when tunnel available

2.2 High-Level Topology

flowchart LR
    subgraph SITE["Helix Site (LAN, egress-restricted)"]
        direction TB
        DC["Ordermatic Desktop<br/>(Tauri shell + WebView + SQLite cache)"]
        subgraph APP["DGX Spark Appliance"]
            GW["Caddy (mTLS)<br/>→ apps/webapp container"]
            TEMP["Temporal Server<br/>(PostgreSQL backend)"]
            WRK["Temporal Workers<br/>(PDF extraction, ERP push)"]
            LLM["vLLM / Ollama<br/>(Quantized models)"]
            TS["Typesense<br/>(Items, Customers, Orders)"]
            FS["PDF Store<br/>(bind mount)"]
            PG[("Encrypted PostgreSQL<br/>Temporal + App Data")]
        end
        ERP[("On-Prem ERP<br/>Epicor / P21 / Eclipse")]
        MAIL[("On-Prem Mail<br/>Exchange / IMAP")]
        SCRUB["Telemetry Scrubber<br/>(Allowlist, Fail-Closed)"]
    end
    subgraph OM["Ordermatic Cloud (Control Plane)"]
        CTRL["Signed Releases<br/>Feature Flags<br/>License Status"]
        TEL["Telemetry Sink<br/>(Scrubbed Events Only)"]
    end

    DC -->|Static URL + pinned CA + mTLS| GW
    GW --> TEMP
    TEMP --> WRK
    WRK --> LLM
    WRK --> PG
    WRK --> FS
    WRK -->|Direct SQL| ERP
    WRK -->|IMAP/Graph| MAIL
    WRK --> TS
    DC -->|Search| TS
    DC -->|Local Search| DC
    APP <-->|Cloudflare Tunnel<br/>(Control + Telemetry Only)| CTRL
    DC -->|UI Events| SCRUB
    SCRUB -->|Allowlisted Only| TEL

    classDef data fill:#fde,stroke:#c39
    classDef ctrl fill:#def,stroke:#39c
    classDef client fill:#e8f5e9,stroke:#2e7d32
    class PG,ERP,MAIL,LLM,FS,TS data
    class CTRL,TEL,SCRUB ctrl
    class DC client

Pink = CUI data (never egresses). Green = Client-local data (procurement, tribal knowledge, prompts). Blue = Control/telemetry channel.

---

3. Component 1 — DGX Spark Appliance (Data Plane)

3.1 Service Stack (Docker Compose)

Service	Image	Resources	Purpose
postgres:16.10	Official	4 GB RAM, 2 CPU	Temporal persistence + Drizzle app data
temporalio/admin-tools:1.31.2	Official	512 MB (init)	One-shot schema create/update; runs to completion, never on restart
temporalio/server:1.31.2	Official	3 GB RAM, 2 CPU	Temporal server (no Web UI — see note)
ordermatic/temporal-worker	Custom build	8 GB RAM, 4 CPU	PDF extraction, ERP push, matching, email sync (arm64 build)
vllm/vllm-openai:v0.28.0	Official	16 GB RAM, 4 CPU	Local LLM inference — US/allied open weights (e.g. gpt-oss / Gemma / Nemotron), see note
typesense/typesense:29.1	Official	4 GB RAM, 2 CPU	Search (items, customers, orders) — GPL-3.0, shipped unmodified
ordermatic/webapp	Existing	2 GB RAM, 2 CPU	The app + its API. apps/webapp/Dockerfile, unchanged
caddy:2	Official	256 MB RAM	mTLS termination in front of the webapp (config, not code)
cloudflare/cloudflared:2026.8.1	Official	256 MB RAM	Control tunnel (--no-autoupdate)

Total DGX allocation: ~37 GB RAM, 15 CPU nominal — UNMEASURED, and it is the assumption that decides whether the product exists. 128 GB on GB10 is unified: quantized weights, KV cache, shared_buffers, Typesense’s in-RAM index and the worker all draw from one pool, and GPU allocation is not free RAM. Week-1 spike: boot the real stack on the real Spark and record actuals.

Object store: none. MinIO was removed. Upstream archived the repo in Feb 2026 (last release RELEASE.2025-10-15) — zero future CVE patches on a service holding CUI PDFs — and it is AGPL-3.0, whose §13 network-interaction clause triggers once the webapp proxies it. One customer per box means there is no multi-node durability requirement to buy. PDFs land on a bind mount at /srv/ordermatic/pdfs. Verify before deleting: grep apps/temporal-worker for a live S3 client; if boto/minio calls are load-bearing the fallback is SeaweedFS or Garage, never MinIO.

No auto-setup. temporalio/auto-setup has no build above 1.29.7 — v1.30.1 stripped bash/tctl/curl from the images and auto-setup.sh shells to both — so the old :1.24 pin capped this appliance below supported forever. It also re-runs schema provisioning on every container start, which is the same unattended-migration failure class this spec rates Critical. Schema setup is now an explicit init container, restart: "no", gated by depends_on: {condition: service_completed_successfully}. Vendor setup-postgres.sh and create-namespace.sh into the repo rather than curling them, and fix the upstream typo $MAX_ATTdMPTS → $MAX_ATTEMPTS in create-namespace.sh: under set -eu it kills the retry loop on iteration 1, every container reports green, the default namespace is never created, and zero orders process silently.

No Temporal Web UI. The old table said auto-setup bundled one; it never did (temporalio/ui is a separate image). Do not add it: it ships auth: enabled: false by default and renders workflow inputs and results — which in this appliance are order payloads, i.e. CUI. That is an unauthenticated CUI console plus privileged terminate/signal/reset controls sitting outside the mTLS gateway.

Model weights: NOT Qwen — and licence is the second filter, not the first.

An earlier revision of this spec defaulted to Qwen on licensing grounds (Apache-2.0, no attribution strings, free to quantize and fine-tune). That recommendation was wrong for this customer and is withdrawn.

FY2026 NDAA §1532 bars DoD/DoW contractors from using AI products developed by entities on the Section 1260H Chinese Military Companies list during contract performance, effective 2026-01-17. Alibaba — which develops Qwen — was added to the 1260H list on 2026-06-08. Waivers are case-by-case and require showing “no reasonable alternative exists”. The published analyses do not carve out self-hosted open weights from hosted APIs, so running the weights on our own appliance is not a safe assumption. §1532 separately names DeepSeek/High Flyer and reaches entities domiciled in or controlled by covered nations (PRC, Russia, DPRK, Iran).

Helix is a CMMC/CUI defense-adjacent manufacturer. Even if they hold no prime DoW contract today, shipping PRC-origin weights inside their appliance creates a flow-down problem the moment they do, and “we self-hosted it” is an argument we would be making to an assessor rather than a rule we can point at.

Therefore: the appliance ships US or allied-origin open weights. Candidates, in the order we should evaluate them:

Weights	Origin	Licence	Note
OpenAI gpt-oss	US	Apache-2.0	Cleanest licence + origin combination; already common in defense builds
Google Gemma 3 / 4	US	Gemma Terms of Use (use restrictions)	Multimodal; see §14 for measured accuracy
NVIDIA Nemotron 3	US	NVIDIA Open Model License	Pairs with the DGX; Nano Omni is the only multimodal one
Meta Llama	US	Llama Community License	Attribution + AUP flow-down, but no 1260H issue
Mistral	France (allied)	Apache-2.0 for several	Not a covered nation; confirm per-model licence

Barred outright: Qwen (Alibaba, 1260H), DeepSeek (named in §1532), and other PRC-origin families — GLM/Zhipu, Kimi/Moonshot, Yi, InternLM.

This is a legal determination, not an engineering one. Get counsel to confirm scope before committing, and record the answer here. The engineering consequence is only that the model slot must stay swappable — which it already is.

Every image is pinned to an explicit tag. A stated Critical mitigation of “customer version pin” is not met by a compose file that floats :latest. Pin by tag in compose and record digests in the release manifest — images arrive at Helix as an offline bundle and docker save/load does not carry RepoDigests, so a @sha256: reference in compose forces a registry round-trip that fails on restart. Typesense 29.1 matches what we already run (docker-compose.yml:197); 30.x is a separate ticket because 30.2 publishes two distinct aarch64 builds (30.2-arm64, 30.2-arm64-lg-page16) and the multi-arch manifest resolves to only one — a coin flip on the DGX kernel page size.

Egress: deny by default. The appliance docker network denies outbound traffic except from cloudflared. This is the SC.L2-3.13.1 boundary control of record; it covers services added in weeks 5-16, which a per-service env var never will. Layer the env vars below on top as hygiene — this stack phones home in four places by default:

vllm:
  environment:
    DO_NOT_TRACK: '1' # stops the 600s POST to https://stats.vllm.ai (envs.py falls back to this)
    HF_HUB_OFFLINE: '1' # raises OfflineModeIsEnabled instead of hanging; also ORs into HF_HUB_DISABLE_TELEMETRY
    HF_HUB_CACHE: /models/hf/hub
  volumes: ['/srv/models:/models:ro']
  # HF_HOME must stay WRITABLE — HF_TOKEN_PATH, CHECK_FOR_UPDATE_DONE_PATH and the xet cache
  # all derive from it. Do not point HF_HOME at a :ro mount.
temporal:
  environment:
    TEMPORAL_VERSION_CHECK_DISABLED: '1' # default is a 24h check to temporal.io
cloudflared:
  command: ['tunnel', '--no-autoupdate', 'run'] # default self-updates from github.com every 24h

Our own @repo/observability PostHog client defaults to https://b.ordermatic.co (packages/observability/src/posthog.ts:24) — it must be explicitly disabled on the appliance, since the client-side scrubber is not in that path. Do not write “outbound telemetry disabled” in the SSP until a packet capture is clean.

Per-service log rotation is mandatory. Docker’s default json-file driver rotates only if configured, so a chatty vLLM fills the CUI box’s disk and takes Temporal down with it. Add logging: {driver: local, options: {max-size: 50m, max-file: 3}} to every service.

3.2 Temporal Workflows (Canonical)

Workflow	Trigger	Activities
ProcessOrderWorkflow	Client PDF drop	extract_pdf → match_lines → human_review → push_to_erp
SyncErpItemsWorkflow	Cron (15min) + Manual	fetch_erp_items → transform_items → upsert_typesense
SyncErpCustomersWorkflow	Cron (hourly)	fetch_erp_customers → upsert_typesense
SyncVendorEmailsWorkflow	Cron (5min) + Webhook	fetch_mail → extract_po → create_tracking_workflow
VendorPOTrackingWorkflow	Per vendor PO	monitor_ack → alert_on_sla → auto_followup

Namespace: Single default namespace per appliance. Multi-tenancy not needed (single customer per box).

3.3 Typesense Collections (Single-Tenant Schema)

Import the cloud schema module; do not hand-write collections here. An earlier draft copied the items/customers/orders field lists into this document as JSON. A copied schema drifts from the code on the first field anyone adds, and then appliance search silently ranks differently from cloud search with nothing failing.

The schema of record is apps/webapp/src/services/search/typesense-search-service.ts. The appliance uses it with one deliberate delta, expressed in config rather than a fork:

• embed blocks and vector fields are disabled. Typesense downloads its built-in models from HuggingFace at index time, which the egress-restricted CUI LAN forbids. This is a stated capability delta, not an accident — see §11. Both name_embedding consumers in our codebase are already WhereFour-gated and Helix runs Eclipse, so nothing is lost today.

Relevance parity is an eval gate, not a hope: schema, synonyms and curation ship as config-as-code alongside the service, and packages/eval runs against the appliance before it is handed over.

3.4 ERP Sync Strategy (Direct SQL → Typesense)

apps/temporal-worker/activities/erp_sync.py

@activity.defn
async def sync_erp_items_to_typesense(connection_id: str) -> SyncResult:
    """
    Direct SQL query → Typesense upsert.
    Replaces: DLT → Iceberg → dbt → Typesense (cloud path)
    """
    # 1. Direct SQL (fast, no API limits)
    sql = """
        SELECT item_id, base_item_key, description, uom, unit_price,
               vendor_id, tags, last_modified
        FROM erp_items
        WHERE last_modified > :last_sync
    """
    rows = await erp_db.fetch_all(sql, {"last_sync": last_sync_ts})

    # 2. Transform (same logic as cloud silver transform)
    docs = [transform_item_row(row) for row in rows]

    # 3. Bulk upsert (Typesense import API)
    result = await typesense.collections["items"].documents.import_(
        docs, {"action": "upsert"}
    )

    # 4. Update sync cursor
    await update_sync_cursor(connection_id, "items", max(row.last_modified))

    return SyncResult(synced=len(docs), cursor=max_ts)

Sync frequency: Items (15 min cron), Customers (hourly), Orders (on commit via workflow signal).

---

4. Component 2 — Ordermatic Desktop Client (Tauri)

4.1 Technology Stack

Layer	Choice	Rationale
Framework	Tauri v2 (Rust + WebView2)	~10 MB, OS webview, code-signing, auto-updater
Frontend	apps/webapp, served by the appliance	One codebase, one API, one set of bytes
Local DB	SQLite via tauri-plugin-sql	Migrations (forward-only, see below), JSON
Sidecar	None	Procurement runs as a DGX Temporal workflow
Update	Tauri Updater (minisign) + Authenticode	Signed releases, pinned versions, grace period
Flags	Flagsmith SDK (cached offline)	Staged rollout, license gating
Telemetry	@repo/observability/onprem-telemetry	Allowlist, fail-closed, air-gap buffer

4.2 Client Architecture

┌────────────────────────────────────────────────────────────────┐
│  Tauri App (Rust Core + WebView)                               │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │ COMPLIANCE BOUNDARY (Rust — cannot be bypassed by JS)    │  │
│  │ • RouteOrderToDGX()          → mTLS → DGX Temporal       │  │
│  │ • RouteProcurementLocal()    → local cache only          │  │
│  │ • ClassifyContent()          → Rules (CUI by default)    │  │
│  └──────────────────────────────────────────────────────────┘  │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │ APPLICATION (the webapp, in the WebView)                 │  │
│  │ • Drag-drop PDF      • Review UI      • ERP push status  │  │
│  │ • SLA tracking       • Vendor PO tracking                │  │
│  │ • Tribal-knowledge capture + suggestions                 │  │
│  │ One UI, one API. Order vs procurement is a ROUTING       │  │
│  │ decision in the Rust boundary above — never a second     │  │
│  │ screen, a second data path, or a second runtime.         │  │
│  └──────────────────────────────────────────────────────────┘  │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │ LOCAL CACHE (SQLite)                                     │  │
│  │ • Offline queue         • Flag + license cache           │  │
│  │ • Read cache for suggestions while the appliance is away │  │
│  │ Truth lives on the appliance. This is a cache.           │  │
│  └──────────────────────────────────────────────────────────┘  │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │ PLATFORM SERVICES                                         │  │
│  │ • Auto-update (cosign verify)  • Feature flags (cached)  │  │
│  │ • License enforcement          • Telemetry scrubber      │  │
│  │ • Static-URL enrollment        • Health chip / tray      │  │
│  └──────────────────────────────────────────────────────────┘  │
└────────────────────────────────────────────────────────────────┘

4.3 Local SQLite Schema (Client-Side)

The client SQLite file is a cache and an outbox. It is never a source of truth.

Three tables an earlier draft defined here already exist in packages/db/src/schema and are already wired into the web client. Redefining them on the desktop is how the two clients diverge:

Draft client table	Already exists as	What the existing one has that a fresh SQLite table would not
prompt_templates	extractorOptimizationJobs	Prompt versioning with quality gates (qualityGateMin, qualityActual, gatesPassed)
agent_definitions	—	Nothing: no agents run on the client any more (§5.3)
tribal_knowledge (part_sub)	manualCrossReferences	theirItemId, theirUom, conversionFactor, uomConfirmationCount, confidence, source — the UOM-conversion story a content TEXT column does not have
tribal_knowledge capture	fieldDecisions	actor, actorId, decisionStage, source, documentValue, workflowRunId — the attribution an assessor asks for

So the desktop writes tribal knowledge through the same API routes the web client uses, into the appliance Postgres. What stays on disk in the client is only what must survive the appliance being unreachable:

-- Offline Queue (for air-gap / appliance down)
CREATE TABLE offline_queue (
  id TEXT PRIMARY KEY,
  seq INTEGER NOT NULL,              -- monotonic. created_at alone is NOT an ordering source:
                                     -- it ties within a millisecond AND is wall-clock, so an NTP
                                     -- step reorders the queue (see §14.6).
  operation TEXT NOT NULL,           -- 'submit_order' | 'sync_catalog' | 'telemetry_batch'
  payload JSON NOT NULL,
  created_at INTEGER NOT NULL,
  attempts INTEGER DEFAULT 0,
  last_error TEXT,
  status TEXT CHECK (status IN ('pending', 'processing', 'failed', 'synced'))
);
-- On app start: UPDATE offline_queue SET status='pending' WHERE status='processing'.
-- 'processing' has no owner, so a crash mid-POST strands the row forever otherwise.
--
-- Drain per-destination, NOT global FIFO. PowerSync's blocking head-of-line rule was evaluated and
-- rejected: this queue spans two trust boundaries — submit_order/sync_catalog go to the DGX on the
-- LAN, telemetry_batch goes to the cloud over the tunnel. A global FIFO makes a tunnel outage block
-- order submission to an appliance sitting on the same LAN, which is the exact condition the queue
-- exists to survive.

-- License / Feature Flag Cache
CREATE TABLE license_cache (
  key TEXT PRIMARY KEY,              -- 'license' | 'flag:<flag_key>'
  value JSON NOT NULL,
  expires_at INTEGER,                -- NULL = no expiry
  updated_at INTEGER NOT NULL
);

Client SQLite migrations are forward-only. tauri-plugin-sql’s MigrationList::resolve() silently discards every MigrationKind::Down while the docs present Up/Down as a working choice. Treat migrations as append-only and immutable once shipped. Also: do not set "preload" in tauri.conf.json — without it, migrations run inside commands::load, so a bad migration surfaces as a catchable rejected promise from await Database.load(...) instead of an app that will not launch. Wrap that call in try/catch and show a recoverable error.

Tribal knowledge may hold customer names and part numbers, so this file is CUI-adjacent — see §14.2 (encryption at rest) and §14.5 (retention/purge).

4.4 Client → DGX Communication Protocol

Channel	Protocol	Auth	Purpose
Discovery	Static URL (customer DNS A record)	Pinned root fingerprint	Locate appliance
API	HTTPS — the webapp’s own routes	mTLS + the app’s existing auth	Everything. See §13
Updates	HTTP polling (check-updates.ts)	same	Work-item updates, progress
Control	Cloudflare Tunnel	Cloudflare Access	Updates, flags, license, telemetry

Certificate Management:

• Appliance generates a self-signed CA on first boot and prints its SHA-256 fingerprint to the first-boot console. (This mechanism is asserted in four places in this spec and specified in none — P0 owes the actual implementation, not just the sentence.)
• The installer enters appliance_url and appliance_root_fingerprint once. Both are stored as keys in the existing license_cache table — no new table, no schema change (its key/value JSON columns are already free-form; the 'license' | 'flag:<key>' list is a comment, not a CHECK).
• Client certs issued over the control tunnel. Lifetime 90 days, auto-renewed at 60. On renewal failure the client keeps working until expiry and raises a health-chip warning, then requires admin re-enrollment. (Clock skew turns this into an unhelpful hard failure — see §14.6 on NTP.)
• Client binaries are verified by the Tauri updater with minisign (Ed25519). This is built in and per Tauri’s docs cannot be disabled. Cosign cannot be used here — tauri-plugin-updater 2.10.1 hard-depends on minisign-verify and Update::verify_signature() calls it directly. Cosign applies only to DGX container images at build time and is never invoked in the appliance boot path (cosign verify checks Rekor by default, which is an outbound sigstore.dev dependency on an egress-restricted LAN and publicly discloses that a CUI appliance exists).
• Windows Authenticode is a separate, unbudgeted requirement. Minisign is invisible to Windows; at a CMMC L2 shop running WDAC/AppLocker an unsigned .exe may not execute at all. §4.1 cites code-signing as a Tauri rationale while nothing procures a certificate. Keys must now live in an HSM/hardware token, so CI signing means wiring Azure Trusted Signing or DigiCert KeyLocker — weeks of calendar lead time and non-zero engineering. Start procurement in week 1. Do not use docker trust anywhere: notary.docker.io shuts down 2026-12-08, inside this project window.
• Key custody: losing the updater’s minisign key means never publishing an update to an installed client again. Escrow it in 1Password at generation time.

mDNS was removed. RFC 6762 defines .local as link-local — “meaningful only on the link where they originate” — and mDNS queries go to 224.0.0.251, inside RFC 5771’s Local Network Control Block (“not forwarded off link”). A CMMC/CUI manufacturer runs a segmented network: servers on a server VLAN, CSR workstations on a user VLAN. Working mDNS across that boundary needs a multicast reflector on gear we do not control, and a reflector repeating discovery between the user VLAN and the CUI server VLAN is precisely the change a network engineer will refuse. A DNS A record is a cheaper ticket their own AD team already knows how to file. It also closes a first-contact MITM window: under mDNS + TOFU any host on the CSR VLAN could answer discovery and get its CA permanently and silently pinned — on the boundary this spec designates as the compliance boundary.

---

5. Component 3 — Procurement / Vendor Tracking

5.1 Why Client-Local?

Reason	Detail
Zero CUI risk	Vendor POs ≠ CUI, but we treat them as local anyway
Free trial viable	No DGX resources consumed, no cloud infra cost
Air-gap friendly	Works without tunnel (IMAP direct from client)
Optimization loop	Client sends prompt templates only to cloud → gets optimized prompts back → stores locally

5.2 Data Flow

Site (client + appliance)               Cloud (Control)
─────────────────────────                ──────────────
│                                           │
│  1. User enables "Procurement Tracking"  │
│     (Flagsmith flag, from the offline    │
│      environment document — §11)         │
│                                           │
│  2. Bundled prompts seeded into the      │
│     appliance Postgres, versioned in     │
│     `extractorOptimizationJobs`          │
│                                           │
│  3. User configures IMAP/Graph creds     │
│     (OS keychain on the client, Local    │
│      persistence — §11)                  │
│                                           │
│  4. SyncVendorEmailsWorkflow (DGX)       │
│     → Extracts vendor POs                │
│     → VendorPOTrackingWorkflow           │
│     → Stores in appliance Postgres       │
│                                           │
│  5. User clicks "Optimize Prompts"       │────► POST /api/optimize-prompt
│     Sends: {template, synthetic_examples}│     Returns: {optimized_template}
│     (NO real vendor data)                │
│                                           │
│  6. New version recorded on the          │
│     appliance, behind the same quality   │
│     gate the cloud path uses             │
│                                           │
│  7. Next run uses optimized prompt       │

5.2a Prompt optimization must run on-prem too

The “Optimize Prompts” loop above sends a template to the cloud and gets a better one back. That is fine for procurement prompts, which the client generates from synthetic examples. It is not sufficient for the order-extraction prompt, and the difference is easy to miss.

Extraction prompts are strongly model-dependent — measured, not asserted: optimizing the prompt for one candidate moved it from 78.7% to 90.0% on the same six documents, closing more than half the gap to the cloud baseline. So the appliance will need to re-optimize whenever the model changes or the customer’s document mix shifts.

Two things follow, and both are architectural:

1. The signal that drives optimization is derived from CUI. What makes the loop work is per-field accuracy on the customer’s own purchase orders — “line items 45%, PO number 100%”. That is a measurement over CUI, and it does not leave the site.
2. Therefore the model that AUTHORS candidate prompts must also run on the appliance. An authoring model we can only reach in the cloud makes the loop permanently cloud-dependent, which is the thing this product exists to avoid, and it breaks entirely for an air-gapped customer.

Requirement: the prompt-authoring model is drawn from the same on-prem candidate set as the extraction model. Authoring is a burst workload — one generation, occasionally — not a batch, so a model too slow for extraction can still be perfectly good at authoring. That widens the eligible set rather than narrowing it.

Only the scrubbed, aggregate outcome (a prompt version id and its score) may cross the control tunnel, on the same allowlist terms as §6.2.

5.3 Agent Runtime — there is no sidecar

An earlier draft ran procurement agents in a bundled Node.js sidecar on LangGraph.js, scheduled by node-cron. That is a third agent runtime beside apps/temporal-worker (Python) and apps/ts-temporal-worker, with its own scheduler, its own retry semantics and its own failure modes — guaranteed to diverge, and unowned by the 1 FTE who maintains the other two.

It was also double-specified: §3.2 already puts SyncVendorEmailsWorkflow on the DGX Temporal we ship anyway. Keep that one. Procurement is SyncVendorEmailsWorkflow + VendorPOTrackingWorkflow on the appliance, written the same way as every other workflow in the repo.

The one thing this costs: the free-trial-without-an-appliance story in §5.1. That was already unresolved (§14.8) and it is now a single question — where does the trial backend run — rather than a second runtime maintained forever to answer it.

---

6. Component 4 — Control Plane (Cloudflare Tunnel)

6.1 Tunnel Responsibilities

Direction	Content	Frequency
Cloud → Appliance	Signed Tauri releases (cosign)	On release
Cloud → Appliance	Feature flags (Flagsmith)	Poll (5 min) + push
Cloud → Appliance	License status / entitlements	Poll (hourly)
Cloud → Appliance	Prompt template updates	On optimization
Appliance → Cloud	Scrubbed telemetry (allowlist)	Batch (5 min)
Appliance → Cloud	Health heartbeat	30 sec
Client → Cloud	UI telemetry (scrubbed)	Batch (5 min)
Client → Cloud	Prompt optimization requests	User-initiated

6.2 Telemetry Allowlist (Enforced by Scrubber)

packages/observability/src/onprem-telemetry.ts

export const EVENT_SCHEMA = {
  // Order processing (aggregate only)
  order_extracted: { duration_ms: 'number', line_count_bucket: 'string', model_version: 'string', confidence_bucket: 'string' },
  match_reviewed: { action: 'accept|edit|reject', latency_ms: 'number' },
  erp_push: { status: 'success|failure', duration_ms: 'number', error_code?: 'string' },

  // Procurement (local, aggregate)
  vendor_po_detected: { vendor_id_hash: 'string', line_count: 'number' },
  vendor_po_tracked: { sla_status: 'on_time|at_risk|breached' },

  // Client health
  heartbeat: { app_version: 'string', os: 'string', uptime_s: 'number', llm_warm: 'boolean', dgx_reachable: 'boolean' },
  error: { code: 'string', stack_hash: 'string' },

  // Feature usage (no content)
  feature_used: { feature: 'string', duration_ms: 'number' },
} as const;

// FORBIDDEN (scrubber drops + alerts):
// - PO numbers, customer names, part numbers, prices, quantities
// - Email bodies, file names, free text
// - Any field not in EVENT_SCHEMA

6.3 License Enforcement

// Tauri core (Rust). NOTE: license_cache is a user-writable SQLite file under
// tauri-plugin-sql, so this is a COMMERCIAL control, not a security boundary — the
// old comment "cannot be bypassed by JS" was false. Seat/entitlement enforcement of
// record is the appliance gateway, which holds per-client identity.
pub fn verify_license() -> LicenseStatus {
    let cached = db.get("license_cache", "license")?;
    if cached.expires_at > now() { return cached.value; }

    match tunnel.request("/license/status") {
        Ok(status) => { cache(status, 24h); status }
        // Fixed deadline from the LAST SUCCESSFUL check — never compounding.
        Err(_) if cached.exists && cached.last_success + 7d > now() => cached.value,
        Err(_) => LicenseStatus::Restricted,
    }
}

• Grace period: 7 days from the last successful check. The old code called extend_grace(7d) on every failed poll, which is unbounded compounding grace — a permanently offline box never expires.
• Never gates order processing. Grafana Enterprise and Replicated KOTS independently refuse to stop a customer’s running workload at expiry. Restricted blocks new ERP connections and new client enrollment. It never blocks processing an order, and it never blocks updates — the updater is the flaw-remediation channel (CMMC SI.L2-3.14.1), and gating it converts an unpaid invoice into an unpatchable CUI appliance.
• Removed: the Err(_) => Trial { days_remaining: 14 } fallback. It handed a perpetual renewing free entitlement to any air-gapped install (DELETE FROM license_cache + block the tunnel).
• Rejected: a signed offline license file. It reads well, but the site-binding cannot be issued before the appliance boots, any CA regeneration bricks it permanently with no self-healing, and the control tunnel it claims to remove is still mandatory for client-cert issuance.
• Version pinning: the pin is what the update endpoint serves. Holding a customer at a version means not publishing a manifest for the next one. The Flagsmith client_version_pin flag was removed — the updater installs whatever the endpoint returns, so a client-side flag read by the client-side updater cannot hold it.
• Rollback: none built in. The old “keeps N-1 signed binary” line was invented behavior — no such logic exists in tauri-plugin-updater, and the Windows path spawns NSIS /UPDATE then exit(0). Downgrade requires the Rust version_comparator API and is “not supported via configuration alone.” Recovery = IT re-runs the previous signed installer from the release archive. The client holds no CUI and its SQLite survives a reinstall.

---

7. Component 5 — Tribal Knowledge Capture (Site-Local Moat)

7.1 Capture Points

Trigger	Action	Storage (already exists)
User edits extracted line item	“Save as substitution?” → Yes	manualCrossReferences (with UOM conversion)
Any field correction in review	recorded automatically	fieldDecisions (actor, stage, source)
User flags vendor behavior	“This vendor always short-ships”	tribalNotes — the one net-new table
User notes customer preference	“Customer X wants PO ref on line 1”	tribalNotes
Pricing override approved	“Emergency pricing for customer Y”	tribalNotes

Two of the five capture points are already built and already writing on the web client: manualCrossReferences carries theirItemId/theirUom/conversionFactor/uomConfirmationCount that a free-text note cannot, and fieldDecisions carries the actor/stage/source attribution an assessor asks for. Only the free-text notes are new — one Drizzle table in packages/db, on the appliance, reached through the same API route from both clients.

7.2 Auto-Suggest in Review UI

Suggestions are served by the appliance and rendered by the existing review surface (apps/webapp/src/components/orders/review-v2). There is no client-side query and no second suggestion ranking to keep in sync — a desktop-only SQL path here would mean the two clients ranking substitutions differently for the same item, with nothing failing to tell us.

7.3 Sync (When Tunnel Available)

• Tribal knowledge never leaves the site — it lives in the appliance Postgres, not in the cloud
• Optional: “Share anonymized patterns with Ordermatic” → scrubbed → telemetry
• Team sync: Export/import signed bundle (future)
• Conflict rule, when sync exists: per-field last-writer-wins. updated_at plus a tie-breaking origin_id; the appliance value wins. No CRDT. This appliance is centralized in exactly Figma’s sense — one authority for one customer — and Figma explicitly did not build a CRDT for that topology: “Figma isn’t using true CRDTs though… Since Figma is centralized (our server is the central authority), we can simplify our system.”
• Blocked on a classification decision, not on engineering: §7.1’s capture points write part numbers, customer names and prices — the exact three classes §6.2’s allowlist forbids. Whether tribal knowledge is CUI decides both this sync and whether it may be backed up at all (§14.1).

---

8. Phasing (Grant-Aligned: Aug 2026 – Jan 2027, ~1 FTE)

Phase	Deliverable	Grant Milestone
P0 (Weeks 1-4)	Week-1 spikes (see §14.9) → DGX Appliance Stack (Postgres, Temporal server+admin-tools, vLLM, Typesense, webapp + Caddy, Tunnel, egress-deny, log rotation) + Tauri Skeleton (mTLS, static-URL + fingerprint enrollment, SQLite, Updater)	Appliance boots; client enrolls from a printed fingerprint; schema-constrained “Hello World” workflow
P1 (Weeks 5-8)	Order Processing (PDF extraction workflow, ERP push w/ idempotency, review UI, Typesense sync workflows, offline queue) + appliance update runbook, backup regime, restore drill (§14.1-14.2)	Helix processes real POs end-to-end on DGX; a restore has actually been rehearsed
P2 (Weeks 9-12)	Procurement Module (IMAP ingestion workflow, vendor PO tracking, “Optimize Prompts”) + arm64 worker image + ERP driver decision (§11)	Free beta to Helix + 2 pilot customers — requires the no-hardware demo path, §14.8
P3 (Weeks 13-16)	Tribal Knowledge + Hardening (capture, auto-suggest, license enforcement, telemetry scrubber audit, support bundle, cosign pipeline for images, CA re-enrollment procedure)	Grant deliverable complete; compliant production deploy

Parallel Track (Cloud): Isolated enclave IaC (GCP Assured Workloads) — same client, different backend target.

---

9. Non-Goals (YAGNI)

• ❌ Fleet management console (build at customer #3). Deferring the console is not deferring the updater — day one needs a signed bundle + update.sh + health gate + pg_dump. The console is later a UI over the same bundle URL and the same update_failed event.
• ❌ A second API, transport, schema, or agent runtime for the desktop client. Every one of those is a place the two clients drift apart with nothing failing to tell us. If the desktop needs something the web client cannot do, add it to apps/webapp and let both have it. The Rust compliance boundary is the only thing that legitimately exists on the desktop and nowhere else.
• ❌ Native mobile client
• ❌ Offline order processing (orders = CUI → must hit DGX)
• ❌ Full Dagster/dbt on DGX (analytics via cloud telemetry)
• ❌ Multi-ERP appliance. Helix runs Epicor Eclipse — the SQL-Server-backed distribution ERP packages/erp-sql-py targets. Earlier drafts wrote “Eclipse/Kinetic” as one string; Kinetic is a different Epicor product with a different schema and a REST/OData surface. If Helix is actually on Kinetic, erp-sql-py reuse is zero and §12 is wrong. Confirm by email before P1.
• ❌ Salesforce / CRM integration — out of scope for the appliance, stated here because “multi-ERP appliance” does not say it and Salesforce appears in §1 with no workflow, connector, or risk row.
• ❌ General “any customer” on-prem product (build Helix deployment first)
• ❌ Custom arm64/sm_121 vLLM builds. Use a prebuilt tag; if a model we need is unsupported, change the model or the tag — never build vLLM from source. Do not adopt the community privileged: true throughput trick; --cap-add=IPC_LOCK is the narrow replacement if IPC pinning is ever needed.
• ❌ OTA frameworks (Mender, RAUC, balena, Talos) and container auto-updaters (Watchtower). DGX Spark recovery is a tar.gz to a 16 GB USB stick that “will completely erase the internal SSD”, needs a directly-attached keyboard and display, and re-enables Secure Boot — a failed OS update is a truck roll, not a rollback, and NVIDIA’s own channel is apt + fwupdmgr. Watchtower’s own README says “We do not recommend using Watchtower in a commercial or production environment”; it would pull a moving tag and restart with no dump, no gate and no rollback — i.e. the Critical migration risk, automated. k3s/Fleet trades a compose problem for a Kubernetes one.
• ❌ Mesh VPN (Tailscale / Headscale / WireGuard) between client and appliance. Tailscale’s client is BSD-3 but its coordination server is proprietary SaaS — the one component we would depend on is the one not open-sourced — and Headscale’s maintainers scope it to “self-hosters and hobbyists”. Plain mTLS on one LAN with one appliance is smaller. Do not write a FIPS justification for this rejection: our own mTLS has no CMVP-validated module either (rustls’ cited cert #4816 lists only Amazon Linux / Ubuntu 22.04 on Xeon and Graviton3 — not DGX OS on GB10, and Windows nowhere). This spec contains zero occurrences of “FIPS” and that is deliberate; an unsupportable FIPS claim in a document a CUI assessor reads is the DoJ Civil Cyber-Fraud fact pattern.
• ❌ Components under the Sustainable Use License or BUSL, ever. They forbid the commercial redistribution this business model requires. This bars n8n and NocoDB regardless of how convenient they look for the §5 procurement build.

---

10. Open Decisions (Need Resolution Before P0)

Decision	Options	Recommendation
Model on DGX	vLLM vs Ollama	CLOSED — vLLM (Apache-2.0, correct engine). The “study Ollama’s docs/architecture.md” action is deleted: that file is a 404 and never existed. Salvage one idea from Ollama — a bounded request queue that returns 503 rather than growing (OLLAMA_MAX_QUEUE), so a burst of PDF drops sheds load instead of OOMing. Do not justify vLLM with “fail-closed guided decoding”: xgrammar, its default backend, has 31 unsound schema keywords and silently ignores pattern.
Where does IMAP run?	Client sidecar vs DGX workflow	CLOSED — DGX Temporal workflow (§5.3). The double-spec is resolved in favour of §3.2’s SyncVendorEmailsWorkflow; the client sidecar is deleted rather than made to agree with it, because it was a third agent runtime. Moving it to the cloud ts-temporal-worker stays rejected — it pulls the whole vendor mailbox into non-FedRAMP infra before anything classifies it, and customer POs land in that mailbox as the norm.
Sync protocol	Custom vs ElectricSQL/PowerSync	CLOSED — Custom, but the stated rationale was wrong. “No cloud dependency” is false: Electric self-hosts (electricsql/electric, Apache-2.0, arm64, no phone-home) and an eng reviewer will kill the row on that. The real reason: Electric was rebuilt in July 2024 as a read-path-only sync engine for Postgres and cannot carry client→DGX writes at all. Keep “boundary control”; drop “no cloud dependency”.
Client cert rotation	Auto vs Manual	CLOSED — auto + admin override, now with a stated lifetime (90d/renew at 60, §4.4). Teleport’s phased CA rotation was evaluated and rejected: its dual-root overlap governs which root is trusted, not whether a leaf is valid, so it cannot fix the offline-laptop case; Teleport’s own prescription for that is “let them expire and re-login.”
Base_item_key on DGX	Sync workflow vs ERP view	CLOSED — sync workflow. Unchanged.
Orchestrator (new)	Temporal vs Hatchet/DBOS/River/pg-boss	CLOSED — Temporal, measured not estimated. 26,953 Python LOC, 25 @workflow.defn, 188 @activity.defn, 80 execute_activity sites, 16 production workflow.patched call sites, plus child workflows, signals, wait_condition, activity.heartbeat, and a replay test asserting a literal patch string. Rejected: Hatchet (5-6 self-host services vs our 2-3 — heavier on one box), DBOS (no wire compat, no patched, a rewrite), River (Go-only), pg-boss (Node-only), Procrastinate (no determinism model). Also rejected as the lazier option: temporal server start-dev --db-filename is one binary with SQLite and no schema step, but Temporal documents it as “not intended for production” and that it “skips certain HTTP security checks.”
Client↔appliance transport (new)	mDNS+TOFU vs static URL vs mesh VPN	CLOSED — static URL + pinned root fingerprint, mTLS retained. See §4.4 and §9.
Is tribal knowledge CUI? (new)	—	OPEN — blocks §7.3 sync and §14.1 backup. §7.1 captures part numbers, customer names and prices. One decision unblocks two workstreams.
How many CSRs at Helix? (new)	—	OPEN — one question, large blast radius. The one remaining client-side mechanism, offline_queue, is written for n=1, and “single-writer today” is the unverified assumption that makes the CRDT deferral correct. See §14.7.
Will Helix grant a direct SQL account on production Eclipse? (new)	—	OPEN — §3.4’s entire design rests on it. At a CMMC shop that is a security review, not a config change, and there is no fallback design if they say no.

---

11. Risk Register (Honest)

Risk	Likelihood	Impact	Mitigation
Local LLM quality < cloud	High	Medium	Eval harness on DGX; higher confidence→review threshold. Make the P0 “Hello World” a schema-constrained call, not a plain chat completion — vLLM rejects incompatible structured-output configs at request validation (HTTP 400 on request #1), so one word in the exit criterion catches an entire class of model/flag incompatibility.
Unattended migration bricks appliance	Medium	Critical	Executable sequence, not aspirations. signed bundle → docker compose down → pg_dump -Fc of all three DBs (ordermatic, temporal, temporal_visibility) → up -d --wait → canary ProcessOrder on a fixture PDF → restore on fail. Stop before you dump, not after: a pre-quiesce MVCC snapshot rewinds Temporal state past ERP pushes the ERP already accepted, and replay re-issues them. Pin name: ordermatic at the compose top level — Compose derives the project name from the containing directory, so a releases/<ver>/ layout emits name: "20261201" and volumes: 20261201_pgdata: a second empty stack that boots healthy, passes --wait, and orphans the CUI database. “Canary appliance” was deleted — it means a second ~$4k DGX nobody budgeted at n=1. Severity stays Critical: temporal, temporal_visibility and ordermatic share one postgres:16 container, so a Drizzle migration that takes a heavy lock, exhausts connections or fills the disk takes Temporal down too.
Temporal schema upgrade at a customer site (new)	Medium	Critical	Sequential-only, one minor at a time, no vendor-documented rollback or backup step. The spec named Drizzle as the bricking risk while the migration with no rollback story is Temporal’s. Mitigation: the P1 runbook above, plus a drain-and-recreate escape hatch — this appliance’s workflows are short-lived and Temporal history is not the audit record of record (the Drizzle DB is), which makes the sequential-hop rule moot.
Air-gap breaks Clerk auth	High (if air-gap)	High	Inherit Helix’s assessed IdP over AD/OIDC. Clerk’s ~50s refresh loop against a Clerk-hosted Frontend API logs every CSR out inside a minute on WAN loss — a week-5 discovery. Do not build an appliance-issued session cookie or a local credential store: that moves 03.05.03 (MFA), 03.05.04 (replay resistance) and 03.05.07/03.05.10 (password management) into our assessment boundary as bespoke 1-FTE code, and adds non-CMVP crypto (32 CFR 170.24 subtracts 3 points for non-FIPS-validated encryption). Helix already runs an MFA’d IdP — it is how CSRs log into Windows and Salesforce, and it is reachable from the egress-restricted LAN when our cloud is not. The §4.4 mTLS client cert supplies the possession factor. Steal one constraint from Teleport: bind the identity source to the user record at creation, never per-login by a global config, so a config edit cannot silently downgrade an existing user’s factor requirement. A 9th IdP container (Keycloak/Authentik/Ory/Zitadel) and a local password+TOTP store are both rejected on the same grounds.
Supply chain via auto-update	Low	Critical	minisign on client binaries (built into the Tauri updater, cannot be disabled) + Authenticode for OS trust + digest-pinned appliance images in the release manifest + pin = “we don’t publish”. Rekor dropped — it adds a sigstore.dev egress dependency and publicly discloses image names, tags and build timestamps. Escrow the minisign key in 1Password: losing it means never updating an installed client again.
Typesense OOM on large catalog	Low	Medium	4 GB is fine as specced: §3.3 defines its own three collections with no embed block and no vector field, and both name_embedding consumers in our codebase are WhereFour-gated. But make the delta deliberate: Typesense downloads built-in models from HuggingFace at index time, which the CUI LAN forbids — so §12’s “unchanged” was false and §3.3’s reduced schema must stay a stated capability delta, not an accident. Adopting the cloud PRODUCTS_SCHEMA verbatim would mandate a new outbound HTTPS fetch of a remote ONNX blob into the CUI boundary.
IMAP credentials on client	Medium	High	OS keychain only; never SQLite; rotate via tunnel. Create with Local, not Enterprise, persistence — Windows Credential Manager’s default is “visible to logon sessions for this user on other computers”, so on a domain with roaming profiles the credential silently leaves the machine.
aarch64 ERP driver wall (new)	High	Critical	apps/temporal-worker/Dockerfile:23 hardcodes infisical_..._linux_amd64.tar.gz — the worker image does not build on DGX Spark. Copy the ${ARCH} fix from apps/dagster/Dockerfile:83-95. The harder half: msodbcsql17 is amd64-only, native arm64 gets Driver 18 only, and the E4/P21 connectors ask for “ODBC Driver 17 for SQL Server” by name (apps/dagster/Dockerfile:104-111). Moving to Driver 18 shifts TLS defaults on live ERP connections — not a Dockerfile-only change. This blocks the appliance’s entire ERP sync path and hits every orchestrator equally.
Feature-flag fail-open silently changes ERP writes (new)	High	Critical	Verified in our code, not hypothesised. packages/feature-flags/src/identity.ts returns the caller’s defaultValue on any Flagsmith failure and negative-caches it. The Python path is worse: packages/feature-flags-py/.../client.py wraps provider init in functools.lru_cache(maxsize=1) and swallows the exception, so if Flagsmith(...) construction throws — exactly what happens when enable_local_evaluation=True cannot fetch the environment document on a dropped tunnel — the worker stays on OpenFeature’s no-op provider for its entire life and never retries. Meanwhile apps/webapp/.../erp/orders/[id]/submit.ts gates five distinct ERP-write behaviors on flags. Net: tunnel drops → worker restarts → the appliance silently changes what it writes into Helix’s ERP, with no signal. §12’s “cached offline” was false (TS = 5-min in-process TTL, Python = 60s poll). Fix is already inside the installed dependency: flagsmith’s offline_mode=True + LocalFileHandler evaluates from an environment-document JSON on disk. Ship that JSON in the appliance bundle; the tunnel refreshes it; appliance flag state becomes a versioned artifact you can hand an assessor. ~10 lines. Separately verify whether flagsmith-nodejs has an equivalent — the client half is unresearched.
GPL/AGPL components redistributed in the appliance (new)	Low	Medium	Typesense is GPL-3.0. Ship an OSS attribution manifest + written offer for source with each appliance; never patch Typesense in-tree. Paperwork nobody has written.
Dropped WebSocket loses work-item updates (closed)	—	—	Closed by deleting the WebSocket (§4.4). The desktop polls api/work-items/check-updates.ts, exactly as the web client already does, and a dropped poll is retried rather than resumed. No sync_id needed.
Model-weight origin: NDAA §1532 / 1260H (new)	High	Critical	Alibaba (Qwen) joined the 1260H list 2026-06-08; §1532 bars 1260H-entity AI in DoW contract performance from 2026-01-17, with no published self-hosted carve-out. An earlier draft of this spec defaulted to Qwen. Mitigation: US/allied weights only (§3.1), plus counsel sign-off recorded in the spec.

---

12. Reuse Map (Ponytail — What We Don’t Build)

Need	Reuse
The whole app + its API	apps/webapp, unmodified. It is already output: 'server' with adapter: node({mode:'standalone'}) and already has apps/webapp/Dockerfile. Run that container on the appliance and the desktop client inherits every route, every zod schema, and every auth check for free. This one row is what prevents web/desktop divergence; the rest of this table follows from it.
Work-item leases	api/work-items/[id]/claim.ts + release.ts — multi-user concurrency (§14.7) already exists
Update polling	api/work-items/check-updates.ts — the update channel, already in use by the web client
PDF extraction	apps/temporal-worker — same code, local namespace. The image needs an arm64 build (see §11): the Infisical ${ARCH} fix plus an Eclipse ODBC Driver 18 decision. Not a Dockerfile-only change, and not “the same worker image”.
ERP connectors	packages/integrations + packages/erp-sql-py (direct SQL) — assumes Eclipse, not Kinetic (§9)
Search	apps/webapp/src/services/search/typesense-search-service.ts — webapp-internal, not a package (an earlier draft cited packages/integrations/typesense-search-service, which does not exist). Running the webapp container on the appliance is what makes this reuse free; consuming it from Tauri directly would mean extracting a package first. Re-verify against 29.1; the appliance uses §3.3’s reduced schema (no embed, no vector field)
Eval harness	packages/eval (run on DGX for model cert)
Feature flags	packages/feature-flags — not “cached offline” as shipped (§11). Requires offline_mode=True + LocalFileHandler with the environment document in the appliance bundle
Telemetry scrubber	@repo/observability/onprem-telemetry (built, tested) — covers our client events only. Third-party service logs never leave the appliance at all (§6.1); an OTel Collector redaction filter was evaluated and rejected, because redactionprocessor’s allowed_keys applies only to map bodies and attributes, leaving a plain-string log body filtered by deny-list regex alone
ERP write idempotency	apps/webapp/src/pages/api/erp/orders/[id]/submit.ts:711 — ranks the DB status check primary with REJECT_DUPLICATE as belt-and-suspenders. Port that, do not invent a sequencing protocol. Temporal Workflow Id uniqueness governs starts, not ERP writes: a worker can crash after push_to_erp succeeds and Temporal will retry the activity. Set id_conflict_policy=USE_EXISTING on ProcessOrder (free; sla-clock.ts:42 already does it) and keep policies per-operation — ProcessOrderWorkflow contains human_review, so a blanket REJECT_DUPLICATE burns the id on any rejected-then-corrected re-drop
Tauri updater	Built-in — minisign verification is already mandatory; there is nothing to add. Cosign cannot be wired into it (§4.4)
Cloud control plane	Existing Cloudflare Tunnel + Flagsmith + Infisical

Net new build: Tauri shell (Rust boundary + WebView + updater), DGX docker-compose, Typesense sync workflows, a tribalNotes table, License enforcement (Rust), and the appliance operations we owe as a product: update/backup/restore runbook (§14.1-14.2), support bundle (§14.3), day-0 installer (§14.4).

Deleted from this list by the 2026-08-30 consolidation pass, because each already exists and each was a place the desktop and web clients would drift apart: the API gateway, the gRPC contract, the WebSocket channel, the direct-Typesense search path, the client prompt/agent registry, the client-side tribal-knowledge store and its suggestion query, and the Node/LangGraph.js sidecar.

License rule (standing): every component entering the appliance image records its SPDX identifier and is permissive (MIT/BSD/Apache-2.0), or — for GPL-family — ships unmodified as a separate container with a written offer for source in the appliance OSS manifest. Verified clean: Flagsmith BSD-3-Clause, vLLM/cloudflared Apache-2.0, Temporal MIT, Postgres PostgreSQL-license. Model weights carry a SECOND filter beyond licence — origin (§3.1). Verified problematic: MinIO AGPL-3.0 (removed), Typesense GPL-3.0 (kept, unmodified). Barred outright: SUL and BUSL (§9).

---

13. Client ↔ Appliance API Contract

There is no second API. The desktop client calls the same HTTP routes the web client calls, served by the same apps/webapp container running on the appliance. This is the single most important anti-divergence decision in this document, and it is a deletion rather than a design.

An earlier draft defined a gRPC OrderService with SubmitOrder, GetWorkItem, SubmitReview, SearchItems, SearchCustomers and WatchWorkItems. Every one of those already exists:

Draft RPC	Existing route
SubmitOrder	api/erp/orders/[id]/submit.ts
GetWorkItem	api/work-items/[id]/index.ts
SubmitReview	api/erp/orders/[id]/approve.ts
SearchItems	services/search/typesense-search-service.ts
SearchCustomers	same
WatchWorkItems	api/work-items/check-updates.ts (polling, in use)
(unlisted)	api/work-items/[id]/claim.ts / release.ts — the multi-user lease §14.7 wanted

Why not adopt gRPC across the board instead

The reverse move — put both clients on gRPC — unifies the contract too, so it deserved a real answer rather than a preference. It loses on our specifics:

• Browsers cannot speak gRPC without a proxy (HTTP/2 trailers). “Across the board” therefore means adding Envoy or an equivalent to the cloud stack — reintroducing exactly the custom gateway service this pass deleted from the appliance.
• The routes are the auth boundary, not thin RPC handlers. apps/webapp/src/lib/api/base-handler.ts is 923 lines and the surrounding auth layer is 1,125; it does Clerk auth, org scoping, erpConnections resolution, CSRF, and error shaping that deliberately avoids leaking schema and the calling org’s Clerk ID. All of that becomes interceptors we write and test again, across 254 non-test routes, during a live AUTH_ENFORCEMENT_MODE migration.
• Our only cross-language hop already has an RPC layer. App→worker is Temporal, which owns that wire. gRPC would sit beside it, not replace it.
• Streaming and performance are not our constraints. We poll and it works; SSE is native to Astro if push is ever needed. The bottleneck is LLM inference and ERP SQL.

What we adopt instead: the contract, not the transport

The real thing gRPC sells is one schema both clients import. We can have that for nearly nothing, because the seam already exists:

• BaseAPIHandler.validateRequest<T>(schema: ZodSchema<T>, …) already takes a zod schema
• 107 of 254 non-test routes already use zod
• packages/contracts already depends on zod-to-json-schema

Decision: request/response zod schemas move out of individual routes into @repo/contracts, and both the web client and the Tauri client import the inferred types. Same compile-time guarantee, zero transport change, zero proxy, zero rewrite. If a non-TS consumer ever needs a machine-readable contract, zod-to-json-schema makes that a script rather than an architecture.

If a genuinely new service is ever built greenfield, use Connect (connectrpc) rather than classic gRPC — it speaks plain HTTP/1.1 + JSON and needs no proxy. That is a choice for a new service, never a migration for these 254 routes.

---

14. Appliance-Product Concerns (net-new — a cloud product never needed these)

Everything above was written for a system we operate. §14 is the set an appliance owes because a customer operates it, on one box, that we cannot log into. None of it appeared in the 2026-08-29 draft.

14.1 Backup and restore of the CUI database

One box, no HA, no replica. With MinIO removed, every order PDF is on a bind mount and every order row is in one postgres:16 container. Disk dies in week 6 → Helix’s order history is gone.

The pg_dump in the update runbook is a pre-upgrade dump, not a backup regime. Required: nightly pg_dump -Fc of all three DBs plus the PDF mount, encrypted (MP.L2-3.8.9 requires confidentiality of backup CUI), written to a customer-provided SMB/NAS path — the DGX’s own disk is not a backup — with a stated retention. The lazy implementation is pg_dump -Fc | age -r <key>; pgBackRest is the fallback if PITR is ever required (note ~90% of its docs assume a standby we do not have; repo-cipher-type=aes-256-cbc is the single-host config to copy).

The work is the restore drill, not the tool. P1 exit criterion: a restore has actually been rehearsed, timed, and written down.

14.2 Encryption at rest and secure decommissioning

MP.L2-3.8.3 / SC.L2-3.13.16, and the appliance is a $4k box that physically leaves the building at end of contract. The hard decision nobody has made: LUKS conflicts with unattended reboot in a locked comms closet — someone must supply the key.

• Recommended: clevis + tang (network-bound disk encryption). A tang server on the customer’s LAN unlocks at boot with no passphrase and no key held by us: clevis luks bind -d /dev/nvmeXn1pY tang '{"url":"http://tang.helix.local"}'.
• Alternative: systemd-cryptenroll TPM2+PCR — unverified whether GB10/DGX OS exposes a usable TPM. Measure on the unit.
• Decommissioning: nvme sanitize / nvme format -s2. Confirm whether the Spark’s SSD is removable. If it is soldered, “return the hardware” and “sanitize the media” are in direct conflict, and the only workable answer is encrypt-then-destroy-key — which requires this section to have been done on day 0, not at end of contract.

14.3 Support and diagnostics without SSH

The largest operational hole. 1 FTE must debug a failed ProcessOrder inside an egress-restricted enclave they cannot log into, and §6.2’s allowlist correctly forbids free text, file names and error bodies — so a failure yields erp_push {status:"failure", error_code} and nothing else. No stack trace will ever leave that site.

Build a customer-mediated diagnostic bundle: an ordermatic support-bundle command a Helix admin runs, reviews, and hands over out-of-band. Copy the shape from replicatedhq/troubleshoot — its collect/analyze spec, host collectors, and above all its redactors, which apply regex and literal redaction before the tarball is written. That is exactly the primitive an allowlist scrubber cannot supply for free-text logs. k3s-collect is the reference for a single-node bundle.

Rejected: a cosign-signed “egress proof suite” whose test asserts a non-zero exit from curl — postgres:16 and typesense do not ship curl, so exit 127 scores PASS on a fully open network. A signed false attestation handed to an assessor is worse than no artifact. Prove egress with a packet capture instead.

14.4 Day-0 install by a non-engineer

P0’s exit criterion says the appliance boots; nobody says who plugs it in. The first-boot CA fingerprint, egress deny rules, ERP SQL credentials, the model volume, the cloudflared token and the NTP server all need a config-entry surface, and the person using it is a Helix IT generalist on a Saturday. Study balenaOS’s config.json-injected-at-provision and the Umbrel/CasaOS first-boot wizard on a headless box. Unverified and load-bearing: whether DGX Spark has a headless first-boot path at all — recovery demonstrably requires a directly-attached keyboard and display, and if setup does too, install is a physical-console job.

14.5 CUI retention and purge — including Temporal history

With MinIO gone, order PDFs sit on a bind mount with no lifecycle. Unanswered: how long orders and PDFs are kept, who may delete them, and what “delete” means on the filesystem.

And the one nobody noticed: Temporal workflow inputs and results are the order payloads, so workflow history is a second, undeclared CUI store with its own namespace retention (default 30d — verify --retention at namespace create). “Purge this customer’s order” is a false claim until history deletion is in the same procedure. This belongs beside §6.2’s allowlist, and it is the same classification decision that unblocks §7.3 sync and tribal-knowledge backup.

14.6 Timekeeping

Zero mentions of NTP in the original. Four distinct bites:

1. §4.4’s 90-day certs auto-renewed at 60 — a skewed client clock turns that into an unhelpful hard failure.
2. offline_queue.created_at is wall-clock; an NTP step reorders the queue and moves Temporal timers and cron workflows.
3. AU.L2-3.3.7 requires clocks synchronized to an authoritative source for audit records. Scored control, currently unwritten.
4. Deny-by-default egress (§3.1) blocks pool.ntp.org — the appliance must point at the customer’s internal NTP/AD DC.

Total work: one chrony.conf line (server <dc> iburst / makestep 1.0 3), one client-side w32tm domain-hierarchy note, one install-runbook line. Current coverage: zero.

14.7 Multi-desktop concurrency

§2.1 says one customer per box and never says how many CSRs. Helix has more than one. Every client-side mechanism here — offline_queue above all — is written for n=1, and “single-writer today” is the unverified assumption that makes the CRDT deferral correct. Two CSRs opening the same work item is covered for the duplicate ERP write (§12 idempotency) but not for the UX (silent last-write-wins on the edit) or for seat licensing.

Already built. apps/webapp/src/pages/api/work-items/[id]/claim.ts and release.ts are the lease, and the web client already uses them. The desktop gets multi-user concurrency by calling the same routes — no new schema, no new transport, no new UX. This entry stays only to record the open headcount question, which still decides the CRDT deferral in §10.

14.8 The demo / sales / trial path when there is no DGX

P2’s grant milestone is “free beta to Helix + 2 pilot customers”, at least two of whom own no appliance. “Dual-mode client” is in the title and §2.2’s topology never draws the cloud-backed mode. The lazy answer is that appliance_url points at our cloud — but nobody has checked that the client’s cert-pinning path and a Clerk-authenticated cloud mode can coexist in one binary. That is the “air-gap breaks Clerk” risk seen from the other end, and it is a P0 design constraint currently scheduled to be discovered in P2. Compounding it: the appliance stack cannot be demoed on an x86 laptop (vLLM+CUDA at these sizes) and the arm64 worker-image problem means no single image serves both. No demo environment is budgeted anywhere.

14.9 Hardware failure, DR, and the week-1 spike list

“The DGX dies” is not in the risk register. SSD, PSU, GB10, or someone unplugs it — MTTR equals NVIDIA RMA time for a 2026-launch product, which is unknown and is the first question Helix’s IT director will ask. This needs a written commercial answer, not engineering: do we hold a spare Spark, does the customer buy two, and what is the documented fallback (CSRs key orders manually into Eclipse — the pre-Ordermatic workflow, which is at least honest). Note this event is identical to the CA-loss event: a replacement box breaks every client cert simultaneously. The re-enrollment procedure therefore belongs in the P1 update runbook, not in P3 — we ship to a CUI customer in P1.

Week-1 spikes (before any P0 build):

1. Boot vLLM on the actual Spark and measure throughput. The NGC claim that nvcr.io/nvidia/vllm carries sm_121 kernels is false — both 26.05-py3 and 26.08-py3 arm64 configs list TORCH_CUDA_ARCH_LIST=8.0 8.6 9.0 10.0 11.0 12.0+PTX, no 12.1; the real delta is a PTX JIT fallback, and the upstream issue that would add 12.1 to published targets is still open. If PTX JIT for 12.1 is broken or slow, discovering that in week 4 leaves 12 weeks. Discover it in week 1.
2. Measure the §3.1 resource table on real hardware. It has never been measured.
3. Build ordermatic/temporal-worker for arm64 and find out what the ODBC Driver 17→18 move actually costs.
4. Start Windows Authenticode procurement (calendar-bound, §4.4).
5. Email Helix: Eclipse or Kinetic? How many CSRs? Will they grant a read account on production?

---

15. Research Provenance

The 2026-08-30 revision came from a fan-out study of open-source systems solving the same problems, with every proposed change adversarially reviewed before adoption. Two notes for anyone re-reading the original recommendation list:

• At least one input to the source research was model-generated rather than fetched. ollama/docs/architecture.md is a 404 that never existed. evil-martians/crdt-sqlite — the repo and the hyphenated org — are both 404 (the real org is evilmartians); the nearest real project is vlcn-io/cr-sqlite. Replicache is Rocicorp’s, unrelated to Linear, and is in maintenance mode recommending migration to Zero; Linear actually uses IndexedDB + MobX with an OT-inspired, explicitly-not-CRDT transaction model. Casecommons/pg_search is a Rails ActiveRecord query builder, not a search engine (the Postgres extension of that name is ParadeDB’s, AGPL-3.0 — a worse position than Typesense’s GPL-3.0 here, because the webapp squarely triggers the network clause).
• The three “study these specifically” recommendations were all rejected on evidence — Tailscale (proprietary coordination server, §9), Ollama (decision already closed to vLLM, §10), ElectricSQL (read-path-only since July 2024, §10). The genuinely borrowed mechanisms came from elsewhere: Temporal’s own self-host compose, Replicated’s troubleshoot redactors, Grafana Enterprise and KOTS on license degradation, Figma on centralized-topology conflict resolution, and Linear’s monotonic resume cursor.

---

End of Spec — Ready for P0 Implementation