PDF Processing Workflow

This document describes the complete workflow for processing PDF documents using AI-powered extraction.

Overview

The PDF processing system handles the complete lifecycle from document upload through AI extraction to ERP integration. It supports large documents through chunking and provides comprehensive error handling and recovery.

Workflow Diagram

flowchart TD
    A[PDF Upload] --> B[Store in R2]
    B --> C[Create Document Record]
    C --> D[Queue Processing Task]
    D --> E[Download from R2]
    E --> F{Document Size Check}

    F -->|Small| G[Process Entire PDF]
    F -->|Large| H[Split into Chunks]

    H --> I[Process Chunk 1]
    H --> J[Process Chunk 2]
    H --> K[Process Chunk N]

    I --> L[Gemini AI Extraction]
    J --> L
    K --> L
    G --> L

    L --> M[JSON Validation]
    M --> N{Valid JSON?}

    N -->|No| O[JSON Recovery]
    O --> P{Recovery Success?}
    P -->|No| Q[Mark as Failed]
    P -->|Yes| R[Consolidate Results]

    N -->|Yes| R
    R --> S[Normalize Data]
    S --> T[Merge Duplicates]
    T --> U[Save to Database]
    U --> V[Update Status: Completed]

    Q --> W[Log Error Details]

    style A fill:#e1f5fe
    style V fill:#c8e6c9
    style Q fill:#ffcdd2
    style L fill:#fff3e0

Processing Steps

1. Document Upload

User uploads PDF through web interface
File stored in Cloudflare R2 storage
Document metadata created in database
Status set to ‘pending’

2. Task Queuing

Celery task queued for background processing
Task includes document ID and processing options
Worker picks up task from Redis queue

3. Document Preparation

PDF downloaded from R2 storage
Document size and page count analyzed
Decision made on chunking strategy

4. Chunking Strategy

graph LR
    A[PDF Document] --> B{Page Count > 2?}
    B -->|No| C[Process as Single Unit]
    B -->|Yes| D[Split into 2-Page Chunks]
    D --> E[Chunk 1: Pages 1-2]
    D --> F[Chunk 2: Pages 2-3]
    D --> G[Chunk N: Pages N-1,N]

    style D fill:#fff3e0
    style E fill:#e8f5e8
    style F fill:#e8f5e8
    style G fill:#e8f5e8

5. AI Processing

Each chunk processed by Gemini AI
Structured prompt for order extraction
JSON response expected with order data
Token usage and costs tracked

6. Response Validation

JSON structure validation
Schema compliance checking
Data completeness verification
Error recovery for malformed responses

7. Data Consolidation

Multiple chunks merged into single order
Duplicate items identified and merged
Customer information consolidated
Line items normalized

8. Database Storage

Extracted orders saved to database
Line items created with relationships
Document status updated to ‘completed’
Processing metrics recorded

Error Handling

Retry Strategy

graph TD
    A[Processing Failure] --> B{Retry Count < 3?}
    B -->|Yes| C[Wait with Exponential Backoff]
    C --> D[Retry Processing]
    D --> E{Success?}
    E -->|No| B
    E -->|Yes| F[Mark as Completed]
    B -->|No| G[Mark as Failed]
    G --> H[Log Error Details]

    style F fill:#c8e6c9
    style G fill:#ffcdd2

Error Categories

Network Errors: API timeouts, connection failures
AI Errors: Model failures, rate limiting
Validation Errors: Invalid JSON, schema violations
Storage Errors: R2 access failures, database errors

Performance Monitoring

Metrics Collected

Processing duration per document
Token usage and API costs
Success/failure rates
Queue depth and processing lag
Error categorization and frequency

Quality Scoring

Response completeness (0-1 scale)
JSON validity and structure
Data extraction accuracy
Processing efficiency

Configuration

Chunking Parameters

MAX_PAGES_PER_CHUNK: 2 pages
OVERLAP_PAGES: 1 page overlap
MAX_RETRIES: 3 attempts
RETRY_DELAY: Exponential backoff

AI Model Settings

Model: Gemini 1.5 Pro
Temperature: 0.1 (low for consistency)
Max tokens: 8192
Response format: JSON

Integration Points

Database Schema

pdf_documents: Document metadata and status
extracted_orders: Order-level information
extracted_order_items: Line item details

External Services

Cloudflare R2: PDF storage and retrieval
Google Gemini: AI-powered extraction
Redis: Task queuing and caching
PostgreSQL: Data persistence

Observability

OpenTelemetry tracing
Structured logging
Cost tracking
Performance metrics