Eval Pipeline Analysis: Temporal Gemini-Only Failures & Recommendations

Date: 2026-02-17
Sources: Langfuse traces, eval results, debug logs, codebase comparison

---

1. Temporal Gemini-Only: Root Cause Hypotheses

Hypothesis A: Wrong Document / Hallucination (High Confidence)

Trace 7beda9d0 (test-purchase-order.pdf, temporal gemini-only):

Field	Expected (ground truth)	Extracted
customer_name	West Branch Napa Auto Truck and Trailer	Precision Parts Inc.
po_number	PO-2026-PERF-TEST	2026-01-26
item_names	(3 items)	Brass Fitting, Stainless Valve, End Cap, Gasket Set

The model returned a completely different order — wrong customer, wrong PO (extracted a date as PO), wrong items. This suggests:

1. Chunk confusion: When processing a single-page PDF, the model may be inferring or hallucinating content instead of strictly extracting from the visible pages.
2. Template / prior bias: The model may be filling in plausible-looking data (e.g., “Precision Parts Inc.”) rather than verbatim extraction.
3. Prompt mismatch: Temporal uses get_pdf_extraction_prompt_from_config; for eval, prompt_config may be minimal. Pdf-worker uses get_pdf_extraction_prompt which may inject different context (company_info, schema hints).

Hypothesis B: Missing _source_chunk for Consolidation (Medium Confidence)

Code difference: Pdf-worker tags each item with itm["_source_chunk"] = chunk_index before consolidation. Temporal does not set _source_chunk.

• Consolidation uses _source_chunk to decide: same chunk → treat as within-chunk duplicate; different chunks → merge.
• Without _source_chunk, both values are None, so the merge branch is always taken. That should still deduplicate correctly.
• Conclusion: Unlikely to cause ~10% scores on its own, but should be fixed for parity.

Hypothesis C: Chunk Context in Prompt (Medium Confidence)

Pdf-worker: Uses the same prompt for all chunks; no explicit “this is chunk N of M” or page range in the prompt.

Temporal: Same — no chunk context. But when the model sees a 2-page slice with no header/PO (e.g., pages 2–3 of a 12-page doc), it may:

• Invent header/PO from context or prior
• Return sparse/empty items
• Confuse which order the items belong to

Adding chunk context (e.g., “This is pages 2–3 of a 12-page document. Extract only items visible on these pages. Use ‘Unknown’ for customer/PO if not on these pages.”) could improve behavior.

Hypothesis D: Different Chunk Order / Result Ordering (Low–Medium Confidence)

Pdf-worker processes chunks with as_completed; temporal uses asyncio.gather, so chunk results can arrive out of order. We assign document_order_position = chunks[i][0] + 1 based on index i, so order is preserved. Conclusion: Ordering logic looks correct.

Hypothesis E: process_with_partial_success Filtering (Low Confidence)

Temporal passes consolidated orders through process_with_partial_success, which can drop invalid items. If temporal’s raw extraction has more invalid items (e.g., wrong types, missing required fields), more gets filtered — but that would be a symptom, not the main cause.

---

2. Mistral OCR 10-Page Failure (72.3%)

Mistral OCR does not chunk for extraction. Flow:

1. Full PDF → single Mistral OCR call → full markdown
2. Full markdown → single Gemini call for extraction

So the 10-page failure is not due to chunk order. Plausible causes:

Hypothesis F: Long-Context / Truncation (High Confidence)

• 10 pages of markdown can be large; Gemini may hit context limits or truncate.
• Mistral OCR markdown for tables can be verbose; key rows might be cut or deprioritized.

Hypothesis G: Document-Specific Layout (Medium Confidence)

• The 10-page PDF may have a layout that Mistral OCR parses poorly (e.g., complex tables, merged cells).
• Gemini then gets messy markdown and extracts incorrectly.

Hypothesis H: PO / Header Position

• If the PO number or header is on a later page (e.g., page 5), and markdown order is odd, the model might miss it. Trace showed po_number: 0 for mistral-ocr 10-page — consistent with the model not finding the PO.

---

3. Recommended Fixes

3.1 Temporal Gemini-Only (Priority: High)

1. Add _source_chunk to items (parity with pdf-worker)

   for o in res:
       o["document_order_position"] = chunks[i][0] + 1
       for itm in o.get("order_items", []):
           itm["_source_chunk"] = i

2. Add chunk-context to prompt for chunked extraction

   chunk_prompt = f"""{prompt_text}
   
   CHUNK CONTEXT: This is chunk {chunk_idx + 1} of {len(chunks)} (pages {start_page + 1}-{end_page + 1} of {total_pages}).
   Extract only items visible on these pages. If customer/PO/date are not on these pages, omit them or use empty — they will be merged from other chunks."""

3. Instrument per-chunk LLM calls in Langfuse so we can see inputs/outputs per chunk and verify behavior.

4. Align prompt source: Ensure temporal uses the same prompt path as pdf-worker for eval (e.g., via get_pdf_extraction_prompt or equivalent).

5. Validate single-pass path: Trace 7beda9d0 shows wrong extraction for a 1-page PDF; add logging for raw model output vs. parsed result to detect hallucinations.

3.2 Mistral OCR 10-Page (Priority: Medium)

1. Chunk the markdown for long docs (e.g., by table or by character count) so Gemini never sees more than a few pages of markdown at once.

2. Use a two-pass strategy:

   • Pass 1: Extract header/PO from first 1–2 pages only.
   • Pass 2: Extract line items from full or chunked markdown, using header from Pass 1.

3. Add explicit “PO extraction” instruction to the Mistral OCR prompt for long docs, e.g., “Identify and extract the PO number from the document header, even if it appears on a later page.”

3.3 Prompt Improvements (Both Pipelines)

1. Stricter verbatim rule: “Extract exactly what appears in the document. Do not infer, guess, or fill in missing fields with plausible values.”

2. Part number vs. date rule: “Part numbers and SKUs are never dates. If a value looks like a date (e.g. YYYY-MM-DD), it is not a part number. Put it in order_date or omit.”

3. Chunk awareness for gemini-only: “If this appears to be a partial document, extract only what is visible. Do not invent headers or items from other pages.”

---

4. Celery vs Temporal Alignment (2026-02-17)

Why Celery Gemini-Only Succeeds (99.6% on 10-page)

1. Chunk size: Celery uses PDF_PAGES_PER_CHUNK=8 (pdf_shared default). A 10-page doc = 2 chunks: pages 1–8 and 8–10. The first chunk contains the header (PO, customer) on page 1.

2. No chunk context: Celery passes the same prompt to every chunk. No “omit customer/PO if not on these pages.” The model extracts what it sees; consolidation merges.

3. Direct URL path: When use_direct_url_extraction is on, Celery sends the full PDF via presigned URL—no chunking. Single Gemini call sees the whole document.

Why Temporal Failed (11.6% on 10-page)

1. Chunk size: Temporal used 2 pages per chunk → 6 chunks. Chunk 3–6 don’t see the header; the model hallucinated (“Acme Corporation”, “GF-123456”).

2. Chunk context: Temporal added “omit customer/PO if not on these pages.” That encouraged omission/invention and may have worsened confusion.

3. Extra prompt suffix: _FILE_API_PROMPT_SUFFIX with strict rules—Celery does not add this.

Alignment Implemented

1. Chunk params: MAX_PAGES_PER_CHUNK and OVERLAP_PAGES from get_settings() (8 pp, 1 overlap), matching pdf-worker.
2. Prompt: Use raw prompt_text only—no chunk context, no _FILE_API_PROMPT_SUFFIX.
3. _source_chunk: Already in place for consolidate dedup.
4. Consolidation alignment: pdf-worker now uses pdf_shared.utils.consolidation.consolidate_single_order (single source of truth). Orders are sorted by document_order_position before merging so the header chunk (earliest pages) is always processed first. Celery adds document_order_position when extending chunk results; Temporal already did.

Langfuse Traces

To debug per-chunk extraction, open your Langfuse project and filter traces by:

• metadata.source = "temporal-worker"
• metadata.pipeline = "gemini-only"
• Session/trace ID = pdf_id (document UUID).

Temporal creates a top-level span pdf-extraction (in activities/multiformat.py’s extract_text_with_gemini_multiformat) with Gemini generation spans nested under it.

---

5. Next Steps

1. Re-run eval on 10-page doc to verify alignment improvements.
2. Add Langfuse spans for per-chunk inspection if issues persist.
3. Investigate test-purchase-order ground truth vs. actual PDF content if single-page failures continue.