07 — Dagster / Pipeline Work
The fidelity levers all bottom out in Dagster/dbt, and today they’re half-wired. This is the pipeline work to support the program, org-scoped, in sequence.
Sequencing principle (critical)
Section titled “Sequencing principle (critical)”The dbt collapse must own erp_product_id/sku stability before any sku-keyed
overlay (UPC override) or item-id-keyed xref runs — otherwise overrides no-op and mined
xrefs bind to collapsed-away ids. Order per sync:
- bronze WhereFour → 2. dbt
stg_productscollapse settles canonical sku/erp_product_id (wherefour_collapse_products) → 3. UPC overlay (DB override + GS1 gap-fill) → 4. xref union → 5. Typesense sync → 6. coverage/disagreement audit.
Step 0 (prereq) — land the GS1 gap-fill
Section titled “Step 0 (prereq) — land the GS1 gap-fill”gs1_upc12_candidates / _gtin_check / derive_each_case + the override > gs1 > raw
precedence currently live only on the main branch (claude/source-of-truth-pr1-…),
not this worktree. Rebase/land it first (doc 03).
Step 1 — DB-backed UPC override + CSV upload UI
Section titled “Step 1 — DB-backed UPC override + CSV upload UI”Add a Drizzle table upc_overrides (org_id/connection_id, sku, upc_each, upc_case,
source, uploaded_by, uploaded_at, active), modeled on manual-cross-references.ts:29
(org-scoped, source-tagged). The dagster sync (assets/typesense.py
load_upc_override_map) reads the DB table instead of the git CSV. Internal-app
upload UI validates GS1 on upload + shows a coverage diff (doc 03). Migrate the existing
92-row CSV into the table as the seed.
Step 2 — name_embedding coverage monitoring
Section titled “Step 2 — name_embedding coverage monitoring”The vector lever depends on embeddings being populated and is currently unmonitored.
Add a Dagster asset_check / typesense_embedding_audit asset (partitioned by
connection_id) that, after typesense_products materializes, asserts the products
collection has name_embedding populated for the connection (count embedded vs total)
and alerts on drift. (The cluster has it today; this guards against a sync that drops it.)
Step 3 — materialize the deterministic customer-part# xref (BIGGEST gap)
Section titled “Step 3 — materialize the deterministic customer-part# xref (BIGGEST gap)”manual_cross_references is referenced nowhere in apps/dagster/dbt/, and the
existing history-mining side-channel pushes to Typesense but gets clobbered every
sync. Promote backfill-xrefs-from-history logic into a Dagster asset
(gold/manual_cross_references_mined, partitioned by connection) that mines the non-UPC
tokens in extracted_order_items.item_ids[] keyed per (connection, erp_customer_id)
→ erp_item_id, and wire the dbt/Typesense union so xrefs survive the sync and feed
the Priority-1 exact-xref bucket. Fix the item_id-namespace self-gate so mined xrefs
bind. (This is the durable no-UPC moat — doc 04 #3.)
Step 4 — per-org matching audit asset
Section titled “Step 4 — per-org matching audit asset”Add a matching_audit asset (partitioned by connection_id, runs after the Typesense
sync) computing per-org coverage: SKU / UPC (each+case) / xref (deterministic vs lexical)
/ embedding coverage, plus the UPC API↔master disagreements and the UNRESOLVED
worklist (doc 03). Surfaces gaps automatically instead of silent staleness.
Guardrails before building
Section titled “Guardrails before building”grepmigrations >0047 to confirmupc_overrides/matching_coverage_audittables don’t already exist.- Probe real
extracted_order_items.item_ids[]values to set the their-code tokenizer for the xref miner (avoid mining UPCs as “part numbers”). - Respect the collapse → overlay → xref → sync ordering above.