PDF Extraction: Dedup, Bounding Boxes, and Eval (2025)

A concise summary of work on duplicate line items, bounding box assignment, dedup strategies, eval tooling, defaults, and CI—what we learned, how we learned it, and best practices.

What We Set Out to Fix

After merging the Mistral OCR pipeline, we saw:

Duplicate line items — same PO line appearing more than once, or distinct lines merged.
Wrong bounding boxes — multiple items sharing the same bbox, or bboxes not matching document order.

We needed to find the cause, fix it, and make the behavior measurable and configurable.

How We Found the Cause (No Mistral Regresson)

Approach: Inspect git history and code paths for dedup and bbox logic instead of assuming the Mistral merge introduced the bug.

Findings:

Mistral merge did not change dedup or bbox logic — It only added pipeline/fallback and task_metadata; no regression there.
Bbox matcher — Always used the first PDF match per part number. When the same part number appeared on multiple lines, they all got the same bbox.
Dedup key — item_ids|qty|price correctly merged overlap duplicates across chunks but also merged distinct PO lines that shared that key (e.g. same product, qty, price on different lines).
Sorting — Mistral text path often doesn’t emit reliable position; sort-by-position then didn’t match document order, which affected both consolidation and bbox matching.

Best practice: Before blaming a recent change, confirm via history and code that the suspected area actually changed and that the symptom is explained by that logic.

Fixes Implemented

1. Consumed-Match Tracking (Bounding Boxes)

File: packages/pdf-shared/pdf_shared/utils/pdf_coordinates.py

Idea: Treat (page, x, y) as a “location.” For each item, pick the first unclaimed match (by page then y). Once a match is used, mark that location consumed so the next item with the same part number gets a different bbox.

Mechanics:

Helpers: _build_search_texts(item), _match_location_key(bbox) for consistent location keys.
In match_items: maintain a set of claimed (page_number, rounded_x, rounded_y); for each item, take the first match whose location is not in that set, then add it to the set.

Best practice: When matching N items to M locations with possible duplicate keys, track consumed locations so one-to-one assignment is stable and document-order aware.

2. Chunk-Aware Dedup

File: apps/pdf-worker/tasks.py (consolidate_single_order)

Idea: Only merge duplicates when the same key appears in different chunks (real cross-chunk overlap). Same chunk + same key → keep as separate lines.

Mechanics:

When building all_orders, tag each item with _source_chunk (chunk index).
Dedup: merge only if key is same and _source_chunk differs; same chunk + same key → keep both.
Strip _source_chunk after consolidation.

Best practice: For chunked extraction, make dedup “chunk-aware” so overlap at chunk boundaries is merged but true duplicate lines within a chunk are not collapsed.

3. Pre-Sort Before Bbox Matching

File: apps/pdf-worker/tasks.py

Idea: Run bbox matching on a stable, document-order-like sort so the consumed-match logic assigns bboxes in reading order.

Mechanics: Before extract_bounding_boxes_for_order, sort items by (position, item_ids) on both the main path and the cached-result path.

4. Dedup and Pipeline Overrides

Idea: Let eval (and future callers) choose pipeline and dedup without env or feature flags.

Mechanics:

Worker: task_metadata["dedup_strategy"] ("chunk-aware" | "llm-reconciliation") and task_metadata["pipeline"] ("mistral-ocr" | "gemini-only") override env/flag when present.
pdf-api: Reads pipeline and dedup_strategy from the process request body and passes them into task metadata.
Webapp eval: Form field dedupStrategy (and pipeline) forwarded to pdf-api and into the worker.

Best practice: Prefer explicit request-level overrides for behavior that varies by run (e.g. eval); use env/flag for global defaults.

Eval: Bounding Boxes and Dedup Comparison

Goal: Score extraction quality including bbox correctness and compare pipelines and dedup strategies.

Changes:

Status API — Include boundingBox in response items; eval schema accepts it.
Scoring — scoreBoundingBoxes(items) → coverage (40%) + uniqueness (60%); scorePdfExtraction adds bbox to overall (10% weight) and exposes bboxCoverage, bboxUniqueness, bbox.
Comparison script — run-comparison-eval.ts: env DEDUP = chunk-aware | llm-reconciliation | both; test matrix = pipeline × dedup; report has dynamic pipeline column and BBox column; averages by pipeline label.
Schema robustness — Orders: poNumber/customerName nullable with default '' so LLM reconciliation runs that return null don’t break parsing.

Best practice: Add metrics (e.g. bbox coverage and uniqueness) and parameterized runs (pipeline × dedup) so you can compare approaches on the same corpus and spot regressions.

Speed, Cost, and Model Choices

What we clarified:

Extraction model: Default gemini-3-flash-preview (env/Flagsmith pdf_extraction_model).
Reconciliation model: Default gemini-2.0-flash (env/Flagsmith pdf_reconciliation_model) when using two-pass LLM dedup.
Eval “cost”: Estimated from page counts and constants (tokens/page, $/M tokens), not real API token usage.
Eval “time”: Wall-clock from upload to completion (full pipeline), not just one model call.

Takeaway: Chunk-aware is faster and cheaper than LLM reconciliation (no extra Gemini call) and in our evals gave much better accuracy; LLM reconciliation tended to over-merge line items and sometimes nulled PO/customer.

Best practice: Document which models and pricing assumptions are used; consider logging real tokens/latency (e.g. Langfuse) for production cost and speed analysis.

Defaults: Accuracy Over Cost

Decision: Default to mistral-ocr pipeline and chunk-aware dedup so customers get the best accuracy; cost savings are secondary.

Implementation:

Worker pipeline: When task_metadata["pipeline"] is absent, env is unset, and feature flag is not set, default to mistral-ocr: features.is_enabled("use_mistral_ocr", default=True).
Worker dedup: When task_metadata["dedup_strategy"] is absent, use chunk-aware only; removed env/flag fallback for “use two-pass,” so LLM reconciliation is opt-in via request only.

Best practice: Encode product preference (e.g. “accuracy over cost”) in code defaults and document them; keep overrides explicit (request or env/flag) for power users and eval.

CI: Lint Must Pass on Committed Code

What happened: GitHub Actions “Run Python linting (safe fixes only)” failed because ruff reformatted packages/pdf-shared/pdf_shared/utils/pdf_coordinates.py (multi-line signatures and long strings); CI fails if any file changes after ruff check --fix and ruff format.

Fix: Run ruff check --fix and ruff format locally on the same paths CI uses, then commit the formatted file.

Best practice: Run the same lint/format commands locally that CI runs (e.g. ruff format for Python); treat “lint made changes” as “commit the formatted result” so CI stays green.

Summary Table

Area	What we learned / did	Best practice
Root cause	Mistral merge didn’t break dedup/bbox; bbox “first match” and dedup key semantics did	Verify with history and code before blaming a merge
Bboxes	Consumed-match tracking gives one bbox per item and document-order alignment	Track consumed locations when matching items to positions
Dedup	Chunk-aware dedup merges only across chunks; same-chunk same-key stays as separate lines	Make dedup chunk-aware for chunked extraction
Eval	Bbox metrics + pipeline × dedup matrix give comparable, reproducible results	Parameterize eval and add metrics you care about
Defaults	mistral-ocr + chunk-aware is the default for all users (pdf-api and worker set it explicitly); USE_MISTRAL_OCR=false to disable	Encode product defaults in code; override explicitly
CI	Ruff reformat counts as “code changed”; CI fails until formatted code is committed	Run same format/lint locally and commit the result

This document reflects the work and learnings from the PDF extraction, dedup, bbox, eval, and CI fixes done in this thread.

What we learned (pipeline comparison + OCR parity)

After wiring the worker to actually run the Mistral OCR path, running comparison evals in Docker, and fixing OCR prompt parity, we learned:

1. Worker was not running Mistral OCR

The “mistral-ocr” pipeline label was only trace metadata. The worker always used the chunked Gemini path. We added a real Mistral branch: full PDF → Mistral OCR → full markdown → one Gemini call (no chunking). Chunking is only needed for the gemini-only path.

2. Chunk-aware strongly beats LLM-reconciliation

With DEDUP=both (6 PDFs × 2 pipelines × 2 dedup strategies = 24 runs):

Pipeline + dedup	Avg accuracy	Avg time	Est. cost/PDF
mistral-ocr + chunk-aware	99.8%	92.0s	$0.0320
gemini-only + chunk-aware	98.6%	93.3s	$0.0192
mistral-ocr + llm-reconciliation	68.3%	96.1s	$0.0320
gemini-only + llm-reconciliation	65.8%	110.4s	$0.0192

LLM-reconciliation often over-merged or dropped items (item/itemCount scores in the 37–43% range) and sometimes zeroed PO or customer. Default to chunk-aware; use LLM-reconciliation only if you have a specific need and accept the accuracy hit.

3. Best accuracy: Mistral OCR + chunk-aware

In the DEDUP=both run, mistral-ocr + chunk-aware had the highest average overall accuracy (99.8%) and 100% on part numbers, quantity, item count, PO, and customer. So for “do everything we can for accuracy,” the current best choice is Mistral OCR pipeline with chunk-aware dedup (and the OCR prompt parity fix below).

4. OCR prompt was handicapped

The OCR extraction prompt was called with no company_info or custom_instructions (unlike the PDF path, which gets prompt_config from the worker). So it used Langfuse defaults and missed company context and custom notes. We fixed it: the worker now passes prompt_config in task_metadata for the Mistral path, and pdf-shared extractors build company_info and custom_instructions from it and call get_ocr_extraction_prompt_with_langfuse(company_info=..., custom_instructions=...). The OCR prompt now gets the same variables (company_name, company_location, company_aliases, additional_notes) as the PDF prompt.

5. Eval in Docker with token on the CLI

Run the stack with the eval token, then run the comparison with the same token (env var or command-line argument):
EVAL_SERVICE_TOKEN=your-token docker compose up -d redis pdf-api pdf-worker webapp
then either
EVAL_SERVICE_TOKEN=your-token pnpm --filter @repo/eval eval:compare
or
pnpm --filter @repo/eval eval:compare --token=your-token
(or with DEDUP=both / PIPELINE=both). Results go to packages/eval-results-*.json.

Step-by-step: Running the comparison eval again

Use this when you want to re-run the pipeline × dedup comparison (e.g. after model or code changes).

1. Prerequisites

Test PDFs in the repo at examples/. Filenames must match the keys in packages/eval/test-data/synthetic/ground-truth.json (e.g. test-po-1page.pdf, test-po-4page.pdf). If examples/ is missing or a file is missing, the script will fail when loading that PDF.
Services running (so the webapp can process PDFs end-to-end):
- Webapp — pnpm --filter webapp dev (port 4321)
- pdf-api — e.g. cd apps/pdf-api && uv run uvicorn main:app --reload (see apps/pdf-api README for port)
- pdf-worker — Celery worker connected to the same broker as pdf-api
- Database and Redis — as required by webapp and pdf-api

1b. Running with Docker (recommended)

Run the full stack in Docker, then run the comparison eval on the host with the same eval service token passed on the command line.

1. Start the stack with the eval token set (from repo root):

# Set the eval service token and start services (webapp, pdf-api, pdf-worker, redis).
# Use any non-empty secret; the same value must be passed when running the eval script.
# For Mistral OCR pipeline: set GOOGLE_CLOUD_PROJECT (and optionally GOOGLE_CLOUD_REGION);
# ensure gcloud ADC is available (e.g. gcloud auth application-default login) so the worker can call Vertex.
EVAL_SERVICE_TOKEN=your-eval-secret-token docker compose up -d redis pdf-api pdf-worker webapp

# With Vertex for Mistral OCR:
EVAL_SERVICE_TOKEN=your-eval-secret-token GOOGLE_CLOUD_PROJECT=your-project docker compose up -d redis pdf-api pdf-worker webapp

The webapp container receives EVAL_SERVICE_TOKEN from this environment variable (see docker-compose.yml). If you omit it, the webapp uses EVAL_SERVICE_TOKEN from apps/webapp/.env if present.

2. Wait for services to be ready (e.g. webapp on http://localhost:4321, pdf-api and pdf-worker healthy). Optionally:

docker compose ps
curl -s -o /dev/null -w "%{http_code}" http://localhost:4321

3. Run the comparison eval with the same token on the command line:

# Full comparison: both pipelines, worker default dedup
EVAL_SERVICE_TOKEN=your-eval-secret-token pnpm --filter @repo/eval eval:compare

# Both pipelines and both dedup strategies (full matrix)
EVAL_SERVICE_TOKEN=your-eval-secret-token PIPELINE=both DEDUP=both pnpm --filter @repo/eval eval:compare

# Quick run: 2 PDFs only
EVAL_SERVICE_TOKEN=your-eval-secret-token MAX_ITEMS=2 pnpm --filter @repo/eval eval:compare

4. Compare results: The script prints a summary table and writes JSON files to packages/eval/eval-results-gemini-only.json and packages/eval/eval-results-mistral-ocr.json (or with RESULTS_SUFFIX if set). Use the console output and those files to compare mistral-ocr vs gemini-only and chunk-aware vs llm-reconciliation.

5. Stop the stack when done:

docker compose down

2. Environment variables

From the repo root, use at least:

EVAL_SERVICE_TOKEN — Required. Must match the webapp’s eval service token (in apps/webapp/.env as EVAL_SERVICE_TOKEN). The webapp uses this to authorize /api/eval/upload and /api/eval/status/[id].
WEBAPP_URL — Optional. Defaults to http://localhost:4321. Set if the webapp is elsewhere (e.g. staging).
PIPELINE — Optional. both (default), mistral-ocr, gemini-only, or compare (only re-analyze existing results, no new runs).
DEDUP — Optional. chunk-aware, llm-reconciliation, or both. Unset = use worker default (chunk-aware) for all runs.
MAX_ITEMS — Optional. Cap the number of PDFs (e.g. MAX_ITEMS=2 for a quick run).
RESULTS_SUFFIX — Optional. Appended to the results JSON filename for multiple runs.
LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY — Optional. If set, results are sent to Langfuse for experiment tracking.

Load apps/webapp/.env (or your env) so EVAL_SERVICE_TOKEN is set, or export it in the shell before the command.

3. Run the comparison

From the repo root:

# Use the webapp’s eval token (from apps/webapp/.env)
export EVAL_SERVICE_TOKEN="your-eval-service-token"

# Full comparison: both pipelines, worker default dedup
pnpm --filter @repo/eval eval:compare

# Same but compare dedup strategies (chunk-aware vs llm-reconciliation)
DEDUP=both pnpm --filter @repo/eval eval:compare

# Only Mistral OCR pipeline, chunk-aware dedup only
PIPELINE=mistral-ocr DEDUP=chunk-aware pnpm --filter @repo/eval eval:compare

# Quick run: 2 PDFs only
MAX_ITEMS=2 pnpm --filter @repo/eval eval:compare

# Re-analyze existing results file only (no uploads)
PIPELINE=compare pnpm --filter @repo/eval eval:compare

The script loads ground truth from packages/eval/test-data/synthetic/ground-truth.json, loads each PDF from examples/<filename>, uploads to the webapp eval endpoint, polls until processing is complete, then scores and prints the report. Results are written to a JSON file in the repo root (name includes timestamp and RESULTS_SUFFIX).

4. What you get

Console: Per-PDF and per-pipeline (and per-dedup) scores, extraction time, estimated cost, and bbox metrics; then a summary table and averages.
JSON file: Full results for each run (scores, duration, pipeline, dedup strategy, etc.) for later analysis or PIPELINE=compare re-runs.

5. Troubleshooting

Issue	What to check
`EVAL_SERVICE_TOKEN is required`	Set `EVAL_SERVICE_TOKEN` (e.g. from `apps/webapp/.env`) before running.
`401 Unauthorized` on upload	Token in env must exactly match webapp’s `EVAL_SERVICE_TOKEN`.
`Failed to load ground truth`	Ensure `packages/eval/test-data/synthetic/ground-truth.json` exists and is valid JSON matching the schema (filename → entry with `filename`, `pages`, `items`, etc.).
`loadPdfLocally` / file not found	Put the test PDFs in `examples/` at the repo root; filenames must match ground truth keys (e.g. `test-po-1page.pdf`).
Processing never completes / timeouts	Ensure pdf-api and pdf-worker are running and connected to the same broker and DB; check worker logs for errors.
Low scores on one pipeline or dedup	Use the JSON output and optional Langfuse traces to inspect which items or fields differ from ground truth.

Chunking and Mistral OCR: Do We Still Need the Same Level of Chunking?

Short answer: No. With the Mistral OCR pipeline, the document is sent to Gemini as one markdown payload (full-doc), so chunking is no longer needed for extraction. Chunking remains only for the gemini-only path (PDF chunks uploaded to the File API).

Why chunking existed (Gemini-only path)

PDFs were split into page-range chunks (e.g. 2–4 pages per chunk with overlap) and each chunk was uploaded to the Gemini File API as a separate PDF.
Chunking was required to stay within context/window limits and to parallelize work.
That led to overlap duplicates and the need for chunk-aware dedup and careful bbox matching.

With Mistral OCR

Mistral OCR runs on the full PDF and returns a single markdown string (tables, headings, text).
That full markdown is sent to Gemini in one request as text (no File API, no PDF chunks).
So there is effectively one logical “chunk” per document: no cross-chunk overlap, no chunk-aware dedup needed for the OCR path (dedup logic still runs but there’s only one source chunk).
Bounding boxes are still matched against the original PDF after extraction (same as today).

Worker change (Mistral path)

When pipeline is mistral-ocr (default), the worker now:
- Runs Mistral OCR on the full PDF.
- Sends the entire markdown to Gemini in one extraction call.
- Produces a single “chunk” result; downstream consolidation and bbox matching are unchanged.
Chunking (split_pdf_into_chunks, overlap, parallel chunk processing) is used only on the gemini-only path.

When might we need to “chunk” markdown?

Only if a document is so large that the markdown length exceeds the model’s text context limit (e.g. 1M tokens). Then you could split the markdown by page or by token count and run 1–2 extraction calls, then merge. For typical POs, full-doc markdown fits in one call.

Summary

Path	Chunking	What Gemini sees
mistral-ocr	None (full doc → one MD → one call)	One markdown string
gemini-only	PDF split into page-range chunks	Multiple PDF chunk uploads

So we do not need the same level of chunking for the Mistral OCR pipeline; the MD can be pushed as one chunk (one request) to Gemini. Speed and accuracy improve by avoiding chunk boundaries, overlap duplicates, and multi-call merge/dedup on the OCR path.

OCR prompt parity: The OCR extraction prompt now receives the same variables as the PDF prompt (company_name, company_location, company_aliases, additional_notes from prompt_config). The worker passes prompt_config in task_metadata when invoking the Mistral path; pdf-shared extractors build company_info and custom_instructions from it and call get_ocr_extraction_prompt_with_langfuse(company_info=..., custom_instructions=...) so the OCR path is not handicapped by a simplified prompt.

Recommendation: Rerun the pipeline comparison eval after enabling the worker’s Mistral OCR path to confirm findings (accuracy, speed, bbox) on the same corpus:

# From repo root; requires EVAL_SERVICE_TOKEN in .env, webapp + pdf-worker running
pnpm --filter @repo/eval eval:compare

# Quick smoke test (2 PDFs only)
MAX_ITEMS=2 pnpm --filter @repo/eval eval:compare

# Both pipelines + both dedup strategies (full matrix)
PIPELINE=both DEDUP=both pnpm --filter @repo/eval eval:compare