Vertex AI & Mistral OCR (Model Garden) Setup

This guide covers rebuilding the monorepo Docker images and configuring Google Cloud so the Mistral OCR model is available via Vertex AI Model Garden for PDF extraction (feature flag: use_mistral_ocr).

1. Rebuild all Docker images

From the repo root:

# Rebuild images (uses cache)
./scripts/rebuild-docker-and-setup-vertex.sh

# Rebuild from scratch (no cache)
./scripts/rebuild-docker-and-setup-vertex.sh --no-cache

Or manually:

docker compose down
docker compose build webapp pdf-api pdf-worker dagster
docker compose up -d redis pdf-api pdf-worker webapp dagster

2. gcloud CLI setup for Vertex AI and Mistral OCR

Prerequisites

Google Cloud SDK (gcloud) installed and in PATH
A Google Cloud project with billing enabled

Step 1: Set project and enable Vertex AI API

# Set active project
export GOOGLE_CLOUD_PROJECT=your-project-id
gcloud config set project $GOOGLE_CLOUD_PROJECT

# Enable Vertex AI API (required for Model Garden and Mistral OCR)
gcloud services enable aiplatform.googleapis.com --project=$GOOGLE_CLOUD_PROJECT

Step 2: Enable Mistral OCR in Model Garden

Model Garden models must be enabled once per project in the Google Cloud Console. There is no gcloud command for this step.

Open the Mistral OCR model card:
- Direct link: Mistral OCR (25.05) in Model Garden
Ensure the project selector at the top is set to your project ($GOOGLE_CLOUD_PROJECT).
Click Enable.

After this, the model ID mistral-ocr-2505 is available in supported regions (us-central1, europe-west4).

Step 3: IAM for service accounts (optional, for Docker/production)

If pdf-api or pdf-worker run with a service account (e.g. in Docker or Cloud Run), grant Vertex AI access:

# Replace with your service account email
SA_EMAIL="your-sa@your-project.iam.gserviceaccount.com"

gcloud projects add-iam-policy-binding $GOOGLE_CLOUD_PROJECT \
  --member="serviceAccount:$SA_EMAIL" \
  --role="roles/aiplatform.user"

For local Docker, you can instead use Application Default Credentials by mounting your user credentials or a key file and setting GOOGLE_APPLICATION_CREDENTIALS (see Environment variables below).

Step 4: Verify with gcloud (optional)

# Print an access token (used by our app when using ADC)
gcloud auth print-access-token

If you use a service account key file, ensure the key is valid and has roles/aiplatform.user (or equivalent) on the project.

3. Environment variables

Used by packages/pdf-shared when the use_mistral_ocr feature flag is on (default: on).

Variable	Required	Description
`GOOGLE_CLOUD_PROJECT` or `GOOGLE_CLOUD_PROJECT_ID`	Yes	GCP project ID where Vertex AI and Mistral OCR are enabled.
`GOOGLE_CLOUD_REGION`	No	Region for Vertex AI. Default: `us-central1`. Failover to `europe-west4` is automatic.
`GOOGLE_APPLICATION_CREDENTIALS`	No*	Path to a service account JSON key file inside the container (set automatically by Compose when using `GCP_SERVICE_ACCOUNT_KEY_FILE`).
`GCP_SERVICE_ACCOUNT_KEY_FILE`	No*	Host path to a service account JSON key file. When set, Compose mounts it for pdf-worker so credentials never expire. Recommended for local and production.
`GOOGLE_APPLICATION_CREDENTIALS_JSON`	No*	Base64-encoded service account JSON key (for Coolify/CI). Decoded to file at container startup.
`USE_MISTRAL_OCR`	No	Kill switch. Set to `false` to force Gemini-only pipeline. Default: unset (uses Flagsmith flag).

* For production (Coolify), set GOOGLE_APPLICATION_CREDENTIALS_JSON in 1Password (“Google Cloud” item, service_account_key_json field). The Dockerfile CMD decodes it at startup. For local Docker, use a service account key file (see below) so credentials don’t expire.

Production limits

Limit	Value	Behavior when exceeded
QPM (queries per minute)	30 per region	429 error, retried with exponential backoff (3 attempts)
Pages per request	30	Documents >30 pages fall back to Gemini pipeline
Max request size (unary)	10 MB	Documents >10 MB fall back to Gemini pipeline
Region failover	us-central1 -> europe-west4	Automatic on any region failure

Where to set them

Local / host: In .env or apps/pdf-api/.env and apps/pdf-worker/.env.
Docker Compose: The same env files are loaded; you can also set GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_REGION in docker-compose.yml (they are already passed through for pdf-api and pdf-worker). For credentials in Docker, mount the key file and set GOOGLE_APPLICATION_CREDENTIALS to the path inside the container.
Coolify (staging/prod): Set via 1Password “Google Cloud” item. The Coolify scripts (set-environment.sh / set-environment-staging.sh) pull project_id and service_account_key_json automatically.

Example .env (do not commit real values):

GOOGLE_CLOUD_PROJECT=your-project-id
GOOGLE_CLOUD_REGION=us-central1
# Production reliability: service account key path on your machine (Compose mounts it into pdf-worker).
# GCP_SERVICE_ACCOUNT_KEY_FILE=/absolute/path/to/your-sa-key.json
# Or relative to repo root, e.g.:
# GCP_SERVICE_ACCOUNT_KEY_FILE=./apps/pdf-worker/secrets/gcp-sa.json

Production reliability: use a service account key (no expiry)

User ADC (gcloud auth application-default login) and short-lived tokens expire or require re-auth. For production-style reliability locally, use a service account JSON key. The key itself is long-lived; the client library refreshes short-lived tokens automatically.

1. Create a service account and key (one-time):

export GOOGLE_CLOUD_PROJECT=your-project-id
gcloud config set project $GOOGLE_CLOUD_PROJECT

# Create a service account for Vertex / Mistral OCR
gcloud iam service-accounts create vertex-mistral-ocr \
  --display-name="Vertex Mistral OCR (local/dev)"

SA_EMAIL="vertex-mistral-ocr@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com"

# Grant Vertex AI User so it can call Mistral OCR
gcloud projects add-iam-policy-binding $GOOGLE_CLOUD_PROJECT \
  --member="serviceAccount:$SA_EMAIL" \
  --role="roles/aiplatform.user"

# Download a JSON key (store somewhere safe; do not commit)
mkdir -p apps/pdf-worker/secrets
gcloud iam service-accounts keys create apps/pdf-worker/secrets/gcp-sa.json \
  --iam-account=$SA_EMAIL

The folder apps/pdf-worker/secrets/ is in .gitignore; the key file will not be committed.

2. Point Docker Compose at the key:

In your .env (repo root):

# Host path to the key (absolute or relative to repo root)
GCP_SERVICE_ACCOUNT_KEY_FILE=./apps/pdf-worker/secrets/gcp-sa.json

If you put the key elsewhere, use the absolute path, e.g.:

GCP_SERVICE_ACCOUNT_KEY_FILE=/Users/you/.config/gcp/vertex-mistral-ocr.json

3. Restart pdf-worker so the mount and env are applied:

docker compose up -d pdf-worker

After this, Mistral OCR uses the service account key; no token refresh or gcloud auth is needed.

4. One-command rebuild + gcloud setup

From the repo root:

./scripts/rebuild-docker-and-setup-vertex.sh --no-cache --vertex

This will:

Stop containers and rebuild webapp, pdf-api, pdf-worker, and dagster (with --no-cache).
Set the gcloud project from GOOGLE_CLOUD_PROJECT or current gcloud config.
Enable the Vertex AI API.
Print the Model Garden link and remind you to click Enable for Mistral OCR and set env vars.

To run only the gcloud/Vertex setup (no Docker rebuild):

./scripts/rebuild-docker-and-setup-vertex.sh --vertex-only

5. Mistral OCR usage in the app

Model ID: mistral-ocr-2505
Feature flag: use_mistral_ocr (Flagsmith; default on)
Code: packages/pdf-shared/pdf_shared/llm/mistral_ocr.py and extractors.py

When the flag is on and GOOGLE_CLOUD_PROJECT is set, PDF extraction uses the Vertex AI :rawPredict endpoint with document_url (data URI). See debt/backlog/mistral-ocr-vertex-ai-integration.md for architecture and quotas (e.g. 30 QPM, 30 pages per request).

6. Test the full workflow with eval

Run a quick pipeline comparison eval to confirm Mistral OCR on Vertex end-to-end:

1. Start the stack with Vertex and eval token (from repo root):

# Ensure gcloud ADC exists: gcloud auth application-default login
EVAL_SERVICE_TOKEN=your-eval-secret-token \
GOOGLE_CLOUD_PROJECT=your-project-id \
GOOGLE_CLOUD_REGION=us-central1 \
docker compose up -d redis pdf-api pdf-worker webapp

2. Quick Mistral OCR–only eval (2 PDFs):

EVAL_SERVICE_TOKEN=your-eval-secret-token \
PIPELINE=mistral-ocr \
MAX_ITEMS=2 \
pnpm --filter @repo/eval eval:compare

Results are printed to the console and written to packages/eval-results-mistral-ocr.json (or with timestamp/suffix). For full comparison (both pipelines, more PDFs), omit PIPELINE and MAX_ITEMS or set PIPELINE=both.