Troubleshooting: Gopher

Every Gopher failure surfaces as a structured error — in the review companion as the assistant bubble, in API responses as a JSON error object, and in worker logs as a GopherError payload:

Field	Meaning
problem	What went wrong, in one sentence. This is what the UI renders.
cause	The underlying exception message. Operator-facing, not shown by default.
remediation	The next thing to try. Also rendered in the UI.
docs_url	Points here.
trace_id	UUID to grep for in webapp and temporal-worker logs.

Start from the problem text below. If none of it matches, take the trace_id and search the webapp logs (Astro SSR route) and the temporal-worker logs — every failure path logs the full GopherError JSON with it.

Quick triage

Symptom	Most likely cause	Jump to
Gopher UI missing; chat API returns 404	gopher_companion_v1 off for both user and org	Feature flags
“Unsupported LLM provider configured”	Unsupported GOPHER_LLM_PROVIDER	LLM provider errors
“LLM provider credentials missing”	Missing provider API key on the worker	LLM provider errors
“LLM provider timed out” / “Failed to communicate with LLM provider”	Provider timeout, bad key, or network	LLM provider errors
“Gopher could not open a conversation thread” / “could not persist”	Postgres unreachable from the webapp	Persistence failures
“Gopher hit bedrock while excavating”	Agent run failed inside the workflow	Excavation failures
Chat request hangs, then times out	No worker polling the gopher-interactive queue	Temporal worker
Gopher answers “no data” for a tenant that has data	Gold models not materialized for that connection	Empty or wrong answers
First query fails with S3/R2 or extension_directory errors	Missing R2 credentials or broken DuckDB bootstrap	Warehouse session errors
Answers are slow, or a review chip runs the full agent	Cold session, many rounds, or fast path fell back	Slow answers

Feature flags

Gopher is invisible unless the signed-in user or the org is opted in. With gopher_companion_v1 off at both levels, the companion never mounts and POST /api/gopher/chat plus both /api/gopher/threads routes return a bare 404 Not found — deliberately indistinguishable from a route that does not exist.

• Enable gopher_companion_v1 for the organization in Flagsmith. It is required by the other two flags’ surfaces.
• To turn Gopher on for one person (for example Ordermatic staff testing inside a customer org) without the rest of that org, enable gopher_companion_v1 and gopher_inbox_mentions_v1 on their Flagsmith identity (their email). A user-level flag only counts in the org they are signed into, and it does not reach /gopher/audits or the scheduled audits, which stay org-only. Gopher’s replies to their @gopher mentions are internal notes, so everyone in that org can read them.
• gopher_inbox_mentions_v1 gates @gopher mentions in work-item discussions; without it a mention stays plain text and nothing runs.
• gopher_scheduled_audits gates the scheduled audit fan-out and the /gopher/audits page; a connection whose org is not flagged in is skipped.

A 404 from the chat API for a user who should have access is almost always the flag, not a routing bug.

LLM provider errors

Raised by packages/gopher-py/gopher/llm.py before or during a planner call. The problem/remediation pair tells you which one you hit:

• “Unsupported LLM provider configured” — GOPHER_LLM_PROVIDER is set to something outside gemini, openai, anthropic, vllm, mock. Fix the value on the temporal-worker environment.
• “LLM provider credentials missing” — the selected provider needs a key that is not set: GEMINI_API_KEY (or GOOGLE_API_KEY) for gemini, OPENAI_API_KEY for openai, ANTHROPIC_API_KEY for anthropic. Gopher refuses to fall back to the mock planner outside demo mode, so a missing key raises rather than silently answering with canned text. Set the key, or set GOPHER_LLM_PROVIDER=mock / GOPHER_DEPLOYMENT_MODE=demo for an offline run.
• “LLM provider timed out” — the planner call exceeded GOPHER_LLM_TIMEOUT_SECONDS (default 60s). Retry; if the model is legitimately slow, raise the timeout.
• “Failed to communicate with LLM provider” — the provider rejected the call or was unreachable. Verify the API key (wrong, expired, or lacking model access), the provider base URL (GOPHER_LLM_BASE_URL, or GOPHER_VLLM_BASE_URL for the on-prem appliance, default http://vllm:8000/v1), and egress from the worker.
• “LLM provider returned an incomplete response” — the model hit its output-token limit mid-answer. Narrow the question or raise the model’s output limit.
• “Malformed conversation history” — a persisted turn has a role outside system, user, assistant, tool. This indicates corrupt thread history, not user error; inspect the thread’s rows in gopher_messages.

Provider and model defaults come from GOPHER_DEPLOYMENT_MODE: cloud → gemini, local → vllm, demo → mock. There is no shared cloud default model — each provider’s default lives in CLOUD_DEFAULT_MODELS in packages/gopher-py/gopher/config.py.

Persistence failures

“Gopher could not open a conversation thread” or “Gopher could not persist your message” come from the Astro route in apps/webapp/src/pages/api/gopher/chat.ts when the gopher_threads / gopher_messages write fails. Retry the request; if it persists, check Postgres connectivity from the webapp (connection string, pool exhaustion, replica lag if reads go to a replica the write has not reached).

“Gopher produced an answer but could not persist it” means the agent succeeded and only the write-back failed — reload the thread; if the answer is missing, ask again. The same persistence contract is enforced worker-side by gopher_audit_persistence.py for scheduled audits, so audit findings that cannot be written fail loudly the same way rather than vanishing.

Excavation failures

“Gopher hit bedrock while excavating” is the workflow-level failure: the agent itself raised mid-run. The cause carries the underlying exception (SQL validation rejection, tool error, LLM failure). Take the trace_id and search the temporal-worker logs for the workflow on the gopher-interactive task queue. Common underlying causes:

• Generated SQL rejected by the validator — the model produced something the sqlglot read-only validator blocks (mutation, multi-statement, blocked function, filesystem access). Rephrasing the question usually gets a valid query; a recurring rejection on a reasonable question is a prompt or few-shot problem, not user error.
• A live ERP tool call failed — the tenant’s ERP connection is down or its credentials rotated. Check the connection’s status on the integrations page; live reads are also capped at 5 calls per turn, so a question that needs many live lookups can exhaust the limiter.
• “Gopher exceeded maximum tool iterations” — the agent hit its tool budget (GOPHER_TOOL_TIME_BUDGET_SECONDS, default 90, or GOPHER_MAX_ITERATIONS, default 25) and then failed to answer from what it had found. Interactive surfaces get a lower cap than the env values: the review companion 6 rounds and 30s, inbox mentions and Chief of Staff 10 rounds and 60s (SURFACE_TOOL_CAPS in packages/gopher-py/gopher/config.py). Raising the env values does not lift these caps; they only ever lower them. In chat and inbox mentions, hitting the budget alone does not show this error; the agent answers with what it found. Narrow the question.
• “LLM provider timed out” after tool rounds — if a data tool already returned results (a table listing or a failed tool call does not count), a planner timeout no longer discards them: Gopher answers from what it found and marks the answer as stopped early. You only see the timeout when that answer call fails or returns nothing.
• “Gopher customer_dormancy audit stopped early” (or price_leakage) — a scheduled audit used its whole tool budget (600 seconds or 25 rounds), or a planner call failed partway through, before finishing. Audits treat that as a failure and persist nothing, so an unfinished scan is never reported as clean. Re-run the audit; if it keeps stopping early, narrow its query.
• “Gopher finished without an answer” — the run completed but produced no final assistant text; retry, and check the worker logs for the underlying planner step.

Temporal worker

A chat request that hangs until the route gives up almost always means no worker is polling the gopher-interactive task queue (GOPHER_TASK_QUEUE overrides the name). Confirm the temporal-worker deployment includes that queue — it is separate from pdf-extraction on purpose, so a worker started only for extraction will never pick up Gopher workflows. Worker-side concurrency is tuned by GOPHER_MAX_CONCURRENT_ACTIVITIES and GOPHER_MAX_CONCURRENT_WORKFLOWS.

For scheduled audits that never produce findings, check the ScheduledGopherAuditDispatcher schedule in the Temporal UI and whether the org has gopher_scheduled_audits enabled.

Warehouse session errors

Each Gopher session is an in-memory DuckDB database over the gold Parquet layer, built for one ERP connection and kept warm for GOPHER_SESSION_TTL_SECONDS (default 300) so later runs on that connection skip the 5 to 14 second build. Set it to 0 to build a fresh session for every run. Each run gets its own cursor on the shared session. Failures at session setup:

• Missing R2 credentials in cloud mode — raises at session start rather than serving fixtures. Verify the R2 environment on the worker (R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY, and R2_ENDPOINT_URL or R2_ENDPOINT). Gopher loads them as a DuckDB secret, not s3_* settings, so current_setting('s3_access_key_id') is empty by design.
• SET GLOBAL extension_directory failure — the baked DuckDB extension directory (/tmp/duckdb-extensions in the worker image) is missing or corrupt. Rebuild the worker image; the bake step replays the bootstrap with suppression removed precisely so a bad bake fails the build instead of the first query.
• Bucket LIST fails — propagates as an error rather than reporting an empty warehouse. Check R2 credentials and bucket name. An unreadable warehouse must never look like an empty one.
• Out-of-memory errors, or queries slow only while others run: the DuckDB caps, GOPHER_SESSION_MEMORY_LIMIT (default 1GB) and GOPHER_SESSION_THREADS (default 4), belong to the warm session, so every concurrent run on that connection shares them. Raise them on the temporal-worker environment if one busy connection needs more.
• Every query on a connection fails after a DuckDB fatal, internal or connection error: query_gold_warehouse evicts the session on those errors, so the next run builds a new one. If a different error keeps a session broken, set GOPHER_SESSION_TTL_SECONDS=0 or restart the worker to drop the cache.

Slow answers

Each agent-loop run logs gopher run finished: rounds=N elapsed=Xs setup=Ys on the temporal-worker, and the activity result carries rounds.

• High setup: the run built a session instead of reusing a warm one. Expect it on the first question for a connection, and after GOPHER_SESSION_TTL_SECONDS lapses.
• High rounds: the planner spent many round trips. Interactive surfaces stop at their cap (6 on the review companion, 10 on inbox mentions and Chief of Staff) and answer from what they found.
• A review chip took as long as a typed question: the chip fast path did not answer. It needs gopher_chip_fast_path_v1 on for the user or org and a document that resolved to an order with line items. With no matched ERP customer, the past-prices, our-pricing and credit chips answer at once that the customer must be matched first. When it gives up, the worker logs quick action <id> fell back: with the reason (facts incomplete, an empty required table, a failed fixed lookup, no model answer, or an unexpected error) and the agent loop answers instead. Facts are incomplete for an order with more than 40 lines. The stock chip also falls back when no line matches an inventory row.

Empty or wrong answers

Gopher answers “no data” when the gold model has no Parquet for the tenant’s connection_id — this is by design (an unmaterialized model becomes an empty typed table). If a tenant with data gets empty answers:

1. Confirm which connection the thread is bound to — tenants often have multiple active connections, and the gold layer is partitioned by connection_id.
2. Materialize the gold dbt models for that connection and let the next sync land. A warm session keeps the model list it was built with, so a model that syncs while it is warm shows up only after GOPHER_SESSION_TTL_SECONDS (default 300s).
3. Remember the per-model caveats: fact_customer_item_history is derived from WhereFour order lines and is empty for every other ERP; fact_ar_invoices holds open invoices only, but fact_ar_invoice_lines holds every billed line, paid or open, on Prophet21 connections and is empty on other ERPs, so open-A/R line questions must join it to fact_ar_invoices; dim_orders.total_amount is NULL for some connections — order value comes from SUM(fact_order_lines.extended_price).

In demo mode Gopher answers from five unfiltered fixture tables, not tenant data — an answer that looks plausible but matches nothing in the ERP usually means the deployment is in demo mode.

Volume report is empty or missing a tool. Check that gopher_volume_report_v1 is on for the org, that the thread is a Chief of Staff thread (the two report tools exist only there), and that the date range is YYYY-MM-DD. The billing statement ignores partial months and excludes events with no billing period, which it reports in events_without_period.

Reason codes

Tools return one of these codes in reason_code. The UI picks its state from the code, never from the message text. The meanings follow packages/gopher-py/gopher/tools/reasons.py.

Code	What it means	What to do
customer_not_matched	This customer is not matched to an ERP account yet.	Match the customer to an ERP account, then ask again.
no_order_history	No orders for this customer in the synced history.	Check that the order sync runs for this connection.
item_not_in_history	This item does not appear in the synced order history.	Check the item code. A new item has no history until it is ordered.
no_stock_record	No stock record for this item in the synced data.	Check that the inventory sync runs for this connection.
uom_unverified	The unit of measure could not be verified against the ERP.	Check the unit of measure on the item before you rely on quantities.
no_quote_history	No quotes for this customer in the synced history.	Check the quote sync for this connection if quotes should exist.
minimum_not_tracked	Order minimums are not tracked for this customer.	Nothing to fix. Gopher cannot answer minimum questions here.
staleness_not_tracked	Data freshness is not tracked for this source.	Nothing to fix. Gopher cannot say how fresh this source is.
no_erp_link	This record is not linked to an ERP order.	Check whether the order was submitted. An order that was not submitted has no ERP link.
connection_ambiguous	More than one ERP connection matches, so no answer was given.	Set a default connection for the organization, or open the thread from the right connection.
sync_overdue	The last sync is overdue, so this data may be stale.	Check the sync job for this connection. Answers may be stale until it runs.
agentic_validation_disabled	Agentic validation is turned off for this connection.	Turn on gopher_agentic_validation_v1 for the organization if validation should run here. Validation never writes to the ERP.
already_submitted	This order has already been submitted to the ERP.	Check the ERP order instead. Gopher does not change submitted orders.
work_item_archived	This work item is archived.	Open the work item from the archived view. Gopher does not answer for archived items.
erp_unavailable	The ERP could not be reached, so live data is unavailable.	Check the ERP connection and retry.
not_found	Nothing matching that request was found.	Check the order number, customer or item name, and try again with the exact identifier.
quota_exceeded	The live ERP call limit for this turn has been reached.	Ask a narrower question, or ask again in a new turn.
invalid_argument	A required value was missing from the request.	Name the missing value in the question, such as the order number or the date range.
report_unavailable	The volume or usage report could not be read.	Check the webapp route and the gopher_volume_report_v1 flag for the org, then retry.

Where things live

Piece	Location
Chat / threads API	apps/webapp/src/pages/api/gopher/
Review companion UI	apps/webapp/src/components/gopher/GopherReviewCompanion.tsx
Inbox @gopher mentions	apps/webapp/src/lib/gopher/inbox-discussion.ts
Agent runtime	packages/gopher-py/gopher/
Workflows & activities	apps/temporal-worker/workflows/gopher_research.py, apps/temporal-worker/activities/
Error contract	packages/contracts/src/gopher.ts (GopherErrorSchema)
Architecture deep-dive	Gopher Research Agent Architecture