Gemini Batch API — Opportunities & Integration Plan

Overview

The Gemini Batch API processes requests asynchronously at 50% cost savings and bypasses per-minute rate limits. Trade-off: 15-60 minute latency (vs ~5s realtime).

Validated: Both thinking_budget=0 and thinking_budget=1024 work in batch mode. Two test jobs completed successfully in ~19 minutes with correct extraction results.

API: client.batches.create(model=..., src=[InlinedRequest(...)]) using google-genai SDK.

Opportunity 1: Nighttime Email Batching (HIGH value)

Concept: Emails arriving 9pm-5am (customer local time) are non-urgent. Instead of processing each immediately (~$0.02-0.05/email for Gemini extraction), batch them and process at 5am for 50% savings.

Requirements:

Customer timezone field (or infer from ship-to address)
Queue mechanism: mark emails as pending_batch instead of triggering immediate workflow
Scheduled job at 5am per timezone that collects pending emails and submits batch
Callback handler to process batch results when complete

Where to implement:

apps/temporal-worker/workflows/email_processing.py — add batch routing decision
apps/temporal-worker/activities/gemini.py — add batch submission activity
New scheduled workflow: BatchEmailProcessingWorkflow (runs at 5am per TZ)

Estimated savings: 30-50% of email processing costs (assuming ~60% of emails arrive outside business hours).

Opportunity 2: Bulk Reprocessing (HIGH value)

Concept: When we update prompts, models, or extraction logic, we often need to reprocess hundreds of documents. Currently this hammers rate limits and costs full price.

Where it exists:

apps/webapp/src/pages/api/admin/reprocess.ts — admin reprocessing endpoint
apps/temporal-worker/workflows/batch_extraction.py — already fans out child workflows

Implementation: Add a --batch flag to reprocessing that collects all PDFs, submits as a single Gemini batch job, and processes results when complete.

Estimated savings: 50% cost reduction on reprocessing runs + no rate limit issues.

Opportunity 3: Weekly CI Evals (MEDIUM value)

Concept: The weekly eval suite (packages/eval) runs dozens of extraction calls. These aren’t latency-sensitive — batch them for cost savings.

Where it exists:

packages/eval/scripts/ — eval runner scripts
.github/workflows/ — CI workflow definitions

Implementation: Add --batch mode to eval runners. Submit all test documents as a batch, poll for completion, then score results.

Estimated savings: 50% on eval API costs, plus bypasses rate limits for large eval runs.

Opportunity 4: DSPy Optimization (MEDIUM value, HIGH complexity)

Concept: DSPy prompt optimization makes hundreds of Gemini calls to test prompt variations. These are embarrassingly parallel and not latency-sensitive.

Where it exists:

apps/temporal-worker/tests/dspy_*.py — DSPy optimization scripts

Implementation: Custom DSPy LM adapter that batches requests. Complex because DSPy expects synchronous call-response, but could work with a batch-then-poll wrapper.

Estimated savings: 50% on optimization runs (which can cost $5-20 per run).

Opportunity 5: Nightly Extraction Quality Audit (NEW, LOW complexity)

Concept: Every night, re-extract a random sample of the day’s documents and compare against the original extraction. Flag quality degradation automatically.

Implementation: Scheduled workflow collects sample, submits batch, compares results to stored extractions, alerts on drift.

Implementation Priority

#	Opportunity	Value	Complexity	Do First?
1	Nighttime email batching	HIGH	MEDIUM	Yes
2	Bulk reprocessing	HIGH	LOW	Yes
3	Weekly CI evals	MEDIUM	LOW	Yes
4	DSPy optimization	MEDIUM	HIGH	Later
5	Quality audit	LOW	LOW	Later

Technical Notes

Batch API uses InlinedRequest (not BatchJobSource) for inline data
Maximum batch size: 100 requests per batch job
Results available via job.dest.inlined_responses[i].response
Both thinking_config and response_schema work in batch mode
Job states: PENDING → RUNNING → SUCCEEDED/FAILED/CANCELLED
Typical completion: 15-30 minutes for small batches