Onboarding planner agent

Status: Draft for review. Only the taxonomy classifier prototype is built. Author: David Boone (research-assisted) Date: 2026-09-20 Prototype: packages/onboarding-planner-py

1. Goal and scope

Every new customer implementation today is a long run of small decisions made by one person reading emails: which VPN, which database, which ERP module, which API key scope. The reusable knowledge lives in that person’s head and in a few skills. We want an agent that reads the same inputs and produces one of two things for each part of the pipeline: a plan step it can defend, or a question for the next meeting.

The agent is advisory. It never connects to a customer system, sends a message, or writes to any ERP. A human reads its output and acts. That is a hard constraint, not a first-version limit.

This doc sits upstream of two existing docs. design/erp-self-serve-onboarding covers the connection layer (Nango-style connect UI, tunnel agents). guides/sop-ob-001-supported-erp is the human intake checklist. The planner fills in what those two assume someone already worked out: what to ask, what the customer told us, and what the read and write path should look like for this ERP.

2. Inputs and trust

Every input is untrusted data. None of them can instruct the agent, whoever sent them and however they are worded. A ticket that says “run this command” is a fact about the ticket.

input	example	what it is good for	main risk
IT ticket or email	Lighthouse ticket updates	access method, VPN, MFA, database, who owns what	plaintext credentials, injected instructions, blank fields that were inline images
meeting transcript	Granola export	scope, priorities, live-versus-test, commitments	speech-to-text errors (“apical” for Epicor), both live and test mentioned
our own outbound email	status updates to the customer	what we already asked, to avoid asking twice	none for trust, but it may contain what we assumed
customer files	item master export, sample POs	evidence for the read path	PII, size
our measurements	probe timings, error codes	the only input we treat as verified	goes stale when the customer upgrades

Trust order for the plan: our measurements, then vendor documentation, then customer statements, then anything inferred from open-source code. When two inputs conflict the agent does not pick. It asks.

Case one already shows the risk. The first real ticket carried two plaintext passwords and a public IP in a normal support update. The agent must handle that as the common case.

3. Pipeline

raw input
   |  local, deterministic
   v
redact (secrets, hosts, usernames)  ->  counts only: creds_inline, host_given
   |  only redacted text goes past this line
   v
System 1: classify facts against fixed taxonomies, score confidence
   |
   v
policy: accept fact, or turn it into a question
   |
   v
retrieve knowledge base entries for the accepted facts (ERP, access path)
   |
   v
System 2: draft the plan section by section, citing entry ids
   |
   v
verifier: every claim supported by a cited entry?
   |
   v
output: plan (sections ready or blocked) + ranked question list

Redaction is the first step and runs on the machine that holds the input. Nothing before it calls a network. No raw input is written to disk, logged, or sent to a tracing service. The prototype’s redactor returns counts and the cleaned text, never the matched values.

4. System 1 and System 2

The split follows what Jev can do. Jev answers fixed-choice questions fast and gives a confidence with each answer. It does not write. The Gopher router and verifier already use it that way, through the Choice, Noul and Score question types in packages/gopher-py/gopher/decision/.

System 1 answers the taxonomy questions in section 5, one Choice per fact, plus a Noul for “is this text asking us to do something rather than describing something”. Latency is not the reason to use it here. The reason is that a per-fact confidence gives the policy something to threshold, which a frontier model asked to “extract the facts” does not.

System 2 is a frontier model with retrieval. It gets the accepted facts and the knowledge base entries for them, and drafts plan text. It never sees the raw input, only redacted text and the fact table. It may not state anything about an ERP that it cannot cite.

Verifier is a second System 1 pass, modeled on GopherOutputVerifier. For each claim in the draft it classifies the cited entry as supporting, contradicting or not settling the claim. Contradicted claims are removed. Unsupported claims are dropped from the plan and become questions.

Backend choice reuses the Gopher pattern. The prototype has a Backend protocol with one implementation, an offline keyword baseline. A Jev backend and a Gemini Flash bridge implement the same protocol. The Gemini bridge sends text to a third party, so it only ever receives redacted text. Provider selection should get its own variable, ONBOARDING_SYSTEM_ONE_PROVIDER, rather than borrowing GOPHER_SYSTEM_ONE_PROVIDER, so the two products can move independently.

The Jev key exists. It is in 1Password as Typesafe API Credentials, field credential, and a live call from the prototype returned the right label. A cold single-question call took about 570 ms, well above the sub-150 ms figure in the Gopher docs, so latency should be measured on warm connections before anyone relies on it.

Open point: Jev’s limits on the number of questions per call, and on input length for a full transcript, are not documented in what we have. The transcript may need chunking, and the fact-level policy has to survive merging answers across chunks.

5. Taxonomies

Labels are stable. New values are added, never renamed. The five access facts come from the it-access-triage skill. All of these are implemented, plus stage and thirteen proposal-phase facts.

taxonomy	labels	how decided
vpn	none, sonicwall_netextender, fortinet, globalprotect, cloudflare_warp, other, unknown	System 1
mfa	none, totp, sms, push, unknown	System 1
remote_access	rdp, citrix, ssh, direct_https, other, unknown	System 1
erp	epicor_kinetic, prophet21, eclipse_e4, wherefour, macola, dynamics_gp, iqms, business_central, other, multiple, unknown	System 1
stage	discovery, proposal_out, verbal_yes, sow_sent, signed, paused, declined, access_pending, access_granted, read_path_running, unknown	System 1, threshold 0.90
hosting	on_prem, cloud, hybrid, unknown	System 1, per system
it_owner	internal_it, outside_vendor, none, unknown	System 1
email_platform	microsoft, google, other, unknown	System 1
inbox_structure	shared_inbox, per_rep, mixed, unknown	System 1
volume_band	under_1k, 1k_to_5k, 5k_to_20k, over_20k, unknown	System 1, order lines per month only
team_size	one_to_two, three_to_ten, over_ten, unknown	System 1
edi_share	none, minor, majority, unknown	System 1
pricing_source	in_erp, spreadsheet, other_system, unknown	System 1
erp_change	stable, upgrade_planned, migration_planned, unknown	System 1, per system
budget	allocated, timed_later, none_yet, unknown	System 1
approval	champion, exec_sign_off, committee, unknown	System 1
nda	signed, required, not_needed, unknown	System 1
samples	shared, promised, none, unknown	System 1
database	test, live, unknown	System 1, stricter threshold
creds_inline	true, false	deterministic, from the redactor
host_given	true, false	deterministic, from the redactor

multiple exists because one customer can run several ERPs and the answers differ per system. It always produces a question, the same as unknown.

Whose words, and whose system. Naming a product is not a claim to run it. A transcript carries our own talk track, which lists every ERP we support, and an attendee’s career history, which lists every ERP they ever touched. Both were being read back as systems the customer runs. Two gates now stand in front of that. First, an input may declare our_speakers, the transcript speaker labels that are us, and those turns are dropped before classification; nothing is dropped unless an input says so, and redaction still runs over the whole input so the credential counts describe what arrived. Second, a named product is promoted to a system in use only from a sentence that also says someone on the customer’s side operates it, and not from one in our own voice (“we support”, “we sell”) or one that only weighs the product up (“we looked at”, “instead of”). Failing to promote is the safe direction: the system drops out of the split and its facts become questions. When several products are named and only one is claimed, multiple collapses to that one.

Per-system facts. When two or more systems are claimed, the classifier attributes each sentence to the systems it names. Sentences naming none are shared. Every system is then classified on the shared sentences plus its own, and its erp fact is fixed to that system. Customer-wide access facts are accepted only when every system agrees, so one plant’s VPN answer can never stand in for the customer’s. Questions are grouped: unsettled everywhere gives one customer-wide question, unsettled for some gives one question naming those systems. In the Jev backend the system list comes from one yes-or-no question per ERP label in the same call.

Stage. declined accepted means no plan and no questions. The stage threshold is 0.90, up from 0.85, after 0.85 let three wrong stages through. That change was made on 30 cases and is provisional. Gold for stage is only set where the text itself supports it, and what was true at the time is kept in true_stage, which nothing scores. The access_granted label means access to the ERP or its test environment, so being logged into the server without ERP credentials is access_pending.

Planned next, in this order:

• admin_asks and owner_actions. Lists, not choices. These are extraction, so they go to System 2 or a small span tagger. The first ticket had a real one, IT asking whether the account should be local admin, and I found no written answer in the threads I read.
• network_path. vpn_client, tunnel_agent, ip_allowlist, none. Repo docs cover the tunnel option but nothing on IP allowlists.
• read_method. odata, rest_bo, saved_query, sql, soap, file_drop. Decides the extract design.
• auth_model. basic, api_key_plus_basic, oauth2_client_credentials, oauth2_jwt, session_cookie, token_based_auth. One value per system, so a mismatch with the knowledge base is a signal.
• write_readiness. none, sandbox_available, sandbox_missing, writes_licensed. Anything but the second and fourth blocks a write-path plan.
• intent. whether a sentence is a statement, a request to us, or an instruction aimed at the agent. This is the prompt-injection detector and it is Noul-shaped.

The next step for each plan section is not classified. It comes from a routing table over accepted facts, the same table it-access-triage keeps today. A table can be reviewed and tested. A classifier for “what do we do next” cannot.

6. Knowledge base

6.1 Entry shape

One entry is one claim about one ERP or one access pattern.

id: epicor-kinetic-rest-auth-headers
scope: epicor_kinetic # an ERP, an access pattern, or "generic"
topic: auth
claim: REST v2 needs an x-api-key header and a user login on every call.
evidence: measured # measured | documented | inferred | unverified
source: { kind: our_run, ref: skills/epicor-kinetic-rest, license: internal }
verified_at: 2026-09-20
version_scope: Kinetic 2025 on-prem, one customer

The rules:

• No customer values. No hostnames, company ids, user names, instance names or counts. Customer facts live in per-customer files and are never retrieved for another customer.
• Evidence level travels with the claim. The plan shows it, and the policy treats inferred and unverified as questions to verify, never as settled.
• version_scope is required. A timing measured on one on-premise server does not transfer, and the entry says so.

6.2 Supported ERPs

For ERPs we run, the source is our own code and docs. The survey found these locations:

ERP	where the knowledge is	gaps
Prophet21	apps/dagster/erp_pipeline/sources/prophet21/, webapp prophet21-service.ts, reference/erp-quirks	the P21 access requirements are .docx files that were not read for this survey
Eclipse E4	sources/eclipse_e4/, erp-sql-py, docs/poc/vms-epms-api WSDL, reference/erp-quirks	production auth doc is referenced but missing
WhereFour	wherefour-service.ts, sources/wherefour_fetch.py, docs/poc/wherefour-openapi-v1.0.20.json	not in the integration registry
Acumatica	sources/acumatica/, acumatica-rest-client.ts	quirks section empty
NetSuite	sources/netsuite/, netsuite-rest-client.ts	registry says token-based auth, client code does OAuth2 JWT. The docs contradict the code.
Business Central	sources/dynamics/, dynamics-rest-client.ts	docs thin
Dynamics GP	integration registry slug dynamics_gp, debt/projects/erp-integrations/dynamics-gp-integration-playbook.md	no connector or source. Playbook only.
SAP	design doc and mocks only	no implementation
Macola, IQMS, InSQL	nothing found	no code, no docs, and the license survey in 6.3 did not search for them
Epicor Kinetic	mock at packages/mockoon/mocks/epicor-kinetic.json, plus the workbench skills	no connector in this repo

The NetSuite row matters for the design. The knowledge base builder will find that contradiction on day one, and it needs a defined behavior: mark both entries conflicted, retrieve neither as settled, and open a review item.

6.3 Unsupported ERPs and open-source references

The point of this layer is to infer endpoints and auth for ERPs we do not run. Licenses below were read from the repositories’ license files on 2026-09-20. Section 6.3.1 lists what is still unconfirmed.

source	license	how to index	ERP coverage that matters	verdict
Nango providers.yaml, integration-templates	Elastic License 2.0	parse YAML for auth mode, base URL, token URL, scopes. TypeScript AST for template endpoints	Acumatica, Infor, NetSuite (three auth variants), Odoo, Sage, SAP B1, SAP OData, QuickBooks	extract facts only, store no code. Best single source for auth patterns.
Activepieces packages/pieces/community	MIT at the repo root. Only packages/ee/ and packages/server/api/src/app/ee are excluded. The pieces have no license file of their own, so they inherit the root.	TypeScript AST on PieceAuth and httpClient calls	NetSuite, Business Central, Odoo, Oracle Fusion, QuickBooks, Sage	index source, keep attribution
Airbyte connectors	Elastic License 2.0, including the connectors checked	parse manifest.yaml and spec.yaml (base URL, authenticator, stream paths)	NetSuite, QuickBooks, Sage HR, Oracle databases, SAP Fieldglass	extract facts only. Fieldglass is not B1 or S/4.
Meltano Hub	Apache-2.0	parse extractors/*.yml for settings and capabilities	NetSuite, Dynamics variants	index. Metadata only, no endpoints.
Singer taps	AGPL-3.0 or GPL-3.0 for the ones checked. hotgluexyz/tap-dynamics-bc declares Apache-2.0 in pyproject.toml only.	read for understanding	NetSuite, Dynamics, Oracle	reference only. Do not copy or store text. The Apache-2.0 tap may be indexed, with its declaration being the sole evidence.
dlt verified sources	Apache-2.0	Python AST	no ERP sources	useful only as the output format for extract configs
n8n nodes	Sustainable Use License, not OSI-open	none	Odoo, QuickBooks, Oracle	reference only. Whether internal retrieval counts as permitted use needs a legal read.
Composio	SDK is MIT. The terms of service bar data mining and derivative use of platform contents, and tool schemas are served from the API, not the repo.	none	unclear	facts only. Do not bulk-copy schemas from the API.
Community MCP servers	Epicor Kinetic server Apache-2.0. Odoo (erpipe-org) and Business Central servers MIT. CData wrappers MIT. dsvantien/netsuite-mcp-server declares MIT in package.json with no license file. BrokenRubik/netsuite-mcp-context-ql has no license at all. pantalytics/odoo-mcp-pro is Elastic License 2.0, with MPL-2.0 on files derived from another project.	Python or TypeScript AST for tool definitions and URLs	one Kinetic server, small. CData servers are JDBC wrappers and reveal no endpoints.	index the MIT and Apache ones. Reference only for pantalytics. Unusable for BrokenRubik, since no license means all rights reserved. Expect little.
APIs.guru OpenAPI directory	CC0	parse OpenAPI	no useful ERP specs found	low value
Postman public collections	the 2025 terms let users copy public content only through the service, and restrict scraping and copying, including by AI	none	varies	facts only. Do not bulk-fetch.

6.3.1 Still unconfirmed

• The current Postman terms. They render client-side and only the 2025 archived version was readable.
• Airbyte’s MIT directories. Every connector checked is Elastic License 2.0, and the boundary with any MIT code was not found.
• Nango: no other-licensed directory in four packages checked, but the full tree was not searched.
• Activepieces: the six ERP pieces inherit MIT from the root, and their source headers were not read.
• Composio has no explicit statement either way on caching tool schemas.
• The hotgluexyz GitLab mirror has no license file, so the Apache-2.0 claim rests on one line in pyproject.toml.

None of these changes a verdict, because every unconfirmed source is already handled as facts only or reference only. n8n’s fair-code license is the one that needs a legal read.

Three things follow from the table. First, nothing open-source covers Prophet21, Eclipse or Kinetic beyond one small MCP server, so for those the vendor’s own documentation and our measurements carry everything. Second, Elastic License 2.0 allows reading and preparing derivative works but not hosting the code as a service, so the safe pattern is to store extracted facts (a URL shape, an auth type) with a pointer to the source, not the source. Third, an entry built from any of these is inferred until a run against a real system confirms it.

6.4 Build and refresh

• Curated entries are markdown with frontmatter in the repo, in the same style as the workbench skills. This is where measured facts go.
• Derived entries come from small extractors, one per source type (YAML providers, TypeScript pieces, OpenAPI). They emit the same entry shape into a generated JSONL file that is committed, so a diff shows what changed.
• Promotion. After an engagement, someone strips customer values from a measured fact and adds it as a curated entry. The workbench skills already follow this rule, and the Epicor ones are the first entries.
• Refresh. Derived entries are rebuilt on a schedule. verified_at older than a set window downgrades documented to unverified in retrieval. A changed upstream file that alters a cited entry raises a review item on every plan that cited it.
• Retrieval. The knowledge base will be hundreds of entries, not millions. Filter by scope and topic first, then keyword or embedding ranking inside the filtered set. Start with the filter and plain ranking. Add embeddings if recall proves poor.
• Contamination. Case one’s labels come partly from skills written after case one. Those skills cannot be scored on case one without leaking the answer. See section 9.

7. Confidence policy

The policy mirrors decide_route(): explicit thresholds, deterministic, no model in the decision.

Per fact. A fact is accepted when the label is not unknown and confidence meets the threshold for that fact. Otherwise it becomes a question. Thresholds start at 0.85, and database at 0.90 because a wrong guess between test and live is the costliest error in the set. These numbers are placeholders until the eval in section 9 has enough cases to calibrate them.

Unknown is an answer. “Nothing in this text says” is a valid label, and it always produces a question. Two labels present in one text (“live” and “test”) lowers confidence by design, so the agent asks instead of choosing.

Per plan section. Each section lists the facts it needs. If any is unaccepted, the section is blocked, the section shows what it would say if unblocked, and the blocking facts appear in the question list. A section is ready only when all its facts are accepted and every claim in it passed the verifier.

Write path. Sections about writing to an ERP are always design-only. They carry the required feature flag, the sandbox requirement and the audit requirement from AGENTS.md and operations/feature-flag-policy-erp-writes, and they are never ready on their own. The write_readiness taxonomy gates even drafting them.

Question selection. Questions are ranked by how many plan sections they unblock, deduplicated against questions we already sent (from outbound email), capped at five per meeting, and each names its owner by role. Each carries its reason so the person asking can drop it if they already know.

Failure. Any error from a backend, a timeout, or a missing key yields questions, never a guess. The router’s ad_hoc_react fallback is the model: the safe result is the one that does less.

7.1 Incremental operation

An implementation is not one input in and one plan out. Inputs arrive over weeks, and the agent should get as far as the evidence allows at each point, then keep a record of what is left. Three parts make that work.

State per customer. Every input is triaged alone, then merged into that customer’s state in date order. State holds accepted facts, per-system facts, stage history, and the ids of the sources behind each value. It holds no input text. It lives in corpus/customers/<slug>/state.json, so it is gitignored and syncs to R2 with the rest of that customer’s files. Replaying all of a customer’s inputs rebuilds it from scratch, which keeps the state honest and makes it testable.

Conflicts. When two accepted answers to the same question disagree, neither is kept. The pair is recorded and becomes a question naming both sources and dates. A later statement on the same fact resolves it. A VPN that was retired and replaced is a legitimate change, and the agent asks so that a person confirms which is current.

Questions are gated by stage. Each fact has an unlock stage, the earliest point at which asking is reasonable. The stages come from how twelve earlier calls actually went, eleven with prospects: three read directly and nine read by a research pass over the raw Granola transcripts on 2026-09-20. Prospects answered discovery questions readily and were cautious about anything touching data or access.

fact	unlock stage	evidence from earlier calls
erp, stage	discovery	Needed to write any proposal. Answered readily in every first call.
hosting	discovery	Asked in four calls and answered at once: “it’s on prem”, “on Epicor’s cloud. Yes.”
volume_band	discovery	Asked in most first calls. Two prospects had to look the number up (“I could send that back”), so ask for a rough figure.
team_size, edi_share, pricing_source	discovery	Answered readily each time they were asked. Some prospects volunteered them.
it_owner, email_platform, inbox_structure	discovery	Asked on the ASF, Federal Hose and Aurora calls and answered. Other prospects volunteered them.
erp_change	discovery	Volunteered, and it decided timing: Federal Hose put implementation off for a migration.
budget	proposal_out	One prospect answered at once, but only after we floated a price. Another was vague. Ask once a number exists.
approval	proposal_out	Federal Hose answered on the first call when we asked who the case goes to. Three others named the approver only after the demo. Asking earns a real answer once a proposal exists.
nda, samples	proposal_out	Raised by us after the first demo. No prospect made an NDA a condition of talking, and requests for data or an inbox drew “what we’re willing to let you into”.
vpn, mfa, remote_access, database	signed	Held back in every call. “We won’t be able to do the full implementation unless we get under contract.”

Two limits on that evidence. The nine calls read by the research pass were not re-checked line by line, so treat its quotes as reported. And one prospect started a proof of concept on a sandbox before signing, so a pre-contract sandbox is possible when the customer offers one. The rule treats that as a customer’s choice and not a question we ask.

Everything open is split into ask_now, queued and deferred. A meeting carries at most five questions, chosen in priority order, and the rest are queued for the next one. Before signing, proposal facts outrank access facts. From signed on, access questions come first.

Deferred items are the ones below their unlock stage. They keep their unlock stage, so the ledger of what is left is always visible and nothing is lost by waiting. Below the unlock stage a question is deferred. When the stage is unknown, only erp and stage are asked. paused defers everything and declined drops everything.

The metric to watch is a question asked before its unlock stage. The hold-out check reports it as “asked too early”.

8. Output formats

The plan is JSON that renders to markdown. The question list is a separate object so it can be pasted into a reply.

{
  "customer": "<slug>",
  "generated_from": ["ticket-2026-09-16", "meeting-2026-09-18"],
  "facts": { "vpn": { "label": "sonicwall_netextender", "confidence": 0.9, "action": "accept" } },
  "risks": ["credentials_inline", "live_and_test_named"],
  "sections": [
    {
      "id": "access_path",
      "status": "ready",
      "steps": [{ "text": "...", "cites": ["customer-it-access-rdp"] }],
      "blocked_on": []
    },
    {
      "id": "read_path",
      "status": "blocked",
      "would_say": "Bulk reads through saved queries, paged with $top and $skip.",
      "blocked_on": ["erp", "database"]
    }
  ]
}

{
  "questions": [
    {
      "fact": "mfa",
      "ask": "Which second factor does the VPN use, and who holds it?",
      "why": "The ticket says 2FA but not the type.",
      "unblocks": ["access_path"],
      "owner_role": "customer IT provider"
    }
  ]
}

The question object gains unlock, and a plan carries two lists: ask_now and deferred. Each deferred item names the stage that unlocks it.

Rules for both. No secret, key, token or full hostname appears in any field. Credentials show up only as “present in ticket, needs rotation and a vault item”, with the field name. Risks are a closed list, not free text.

The plan sections follow the SOP: access path, credential handling, read path (method, paging, target bronze table), silver and gold implications, write path (design only), validation, risks. For the read path, the plan states the warehouse rule from CLAUDE.md: bronze tables are shared and partitioned by connection_id, writes are append or merge, never replace.

8.1 What the other examples change

Bison Bag was the first case. Four more customers in our pipeline, read from Granola notes and email on 2026-09-20, want quite different plans. The table shows why one template is not enough.

customer	ERP	access today	stage	what a useful output is
Bison Bag	Epicor Kinetic, on-prem	VPN, RDP, REST API key, test database	access granted, read path running	a read-path plan and the few questions left (mailbox archive, key scope)
All State Fasteners	Prophet21 for core branches, InSQL for a separate unit	no live access. Spreadsheet exports of order lines, a staging environment we built	SOW sent	a plan that starts from files and says what live access would add. Ask which ERP the InSQL unit needs.
Federal Hose	Macola, on-prem, several versions behind, an upgrade under way. IT is an outside vendor.	none discussed. Seen in person on a site visit	declined by email on 2026-09-17	no plan. The right output is a stage of declined and no questions. If it reopens, Macola is not in the knowledge base, so the plan says what we would have to learn.
Aurora Plastics	7 separate Great Plains instances, 2 IQMS, some Business Central, all on-prem	none. The demo is a simulation from sample POs	proposal out, budgeting for 2027	a routing-first plan. The read path is per site, and it says none is possible before access exists.

Five design changes follow.

• Stage is an input. Most of the plan for Federal Hose or Aurora cannot be written yet, and that is the correct answer. The agent says so and asks only what moves the stage forward, and section 7.1 defers access questions until a contract is signed. The stage taxonomy in section 5 drives this, and declined suppresses all questions.
• Facts belong to a system, not a customer. Aurora has ten or more ERP instances and ASF has two ERPs. The fact table is keyed by (customer, system), built as described in section 5. erp = multiple is the trigger.
• “No live access” is a first-class access path. ASF is being served from exports and a staging environment. The read-path section needs a file-first variant with its own risks: exports go stale, and nothing proves the export matches what the ERP holds.
• Sources contradict each other, and summaries are stale or wrong. Four real examples from this pass. An email summarizer said Aurora runs Eclipse E4 through VMS’s IT team. That mixes two customers, and the transcript of Aurora’s first call lists Great Plains, Business Central and IQMS. Two summaries put Federal Hose at “verbal yes” and “paused”, while the last message in the thread, from 2026-09-17, says they will not move forward. A Granola summary spelled a migration target “McCollough” for what the raw transcript renders as “mcola”, “micola” and “mola”, meaning Macola. The same engine wrote “apical” for Epicor in case one. The rule stands: when two inputs disagree the agent asks, and an LLM summary of a thread is never an input. Only the thread or transcript itself is.
• The knowledge base has coverage holes that the plan must show. Macola, IQMS and InSQL have nothing in the repo. Great Plains has a registry entry and a playbook but no connector. A plan for these says “not covered, evidence level unverified” in the section header.

The corpus now has sixteen real, verbatim, redacted cases for these three customers under corpus/customers/. Each records expected_stage, which nothing scores yet. Three real findings came out of them:

• Federal Hose’s call transcript never spells Macola correctly. A keyword baseline cannot read it, while live Jev returned macola at 0.95.
• Aurora’s and ASF’s own words describe several ERPs (“nine different ERP systems”, “another business unit that is on ERP called in SQL”). The keyword baseline accepted a single ERP on both, which are two wrong accepts on real data. Jev returned multiple for Aurora at 0.99. On the ASF call it leaned Prophet21 at 0.76, under the threshold, so the policy asked.
• Only the plural “ERPs” or an unusual name signals the second system, so multiple must be decided on the whole text and not a keyword hit.

8.2 What a proposal needs

The proposal-phase facts exist to fill a business case and price. Our prices so far are an implementation fee, a monthly fee and a per-line usage rate, so line volume matters most.

proposal input	facts it needs	if unknown
Usage price and monthly estimate	volume_band, edi_share	The estimate is a range, and the question is asked first.
Time saved	team_size, volume_band	Skip the savings claim.
Implementation effort and timeline	erp, hosting, it_owner, erp_change	Use the widest timeline and say what would narrow it.
Data and pricing approach	pricing_source, samples, nda	Plan from files only, and ask for samples after an NDA.
Closing path	approval, budget	Ask once the proposal is out.

The plan drafter will list these as blocked or ready in the same way as the access sections.

9. Evaluation

Corpus

packages/onboarding-planner-py/corpus/. One JSON file per input, with the redacted text, provenance, source pointer, and gold labels.

• customers/<slug>/ holds real inputs. These never go in git. The directory is gitignored and is a local cache of a private R2 bucket, pulled with onboarding-planner corpus pull and pushed with corpus push. Push refuses any file that still looks like it holds a secret. Redaction still happens before the file is written, because the bucket is a second line of protection and not the first.
• synthetic/ holds cases written to hit specific failures: prompt injection, non-Epicor ERPs, no-signal meetings, live-only access.
• provenance is real_redacted or synthetic. Reports show them separately, because synthetic cases are easier and inflate accuracy.

Corpus storage

Real cases live in a dedicated private R2 bucket under onboarding-corpus/customers/<slug>/. It should be its own bucket with its own token scoped to that bucket alone. The existing evl bucket is shared by prod and staging, its orphan-cleanup crons already deleted each other’s objects once, and its keys are wide. Customer conversations should not sit behind those.

Credentials come from 1Password through op run, as env vars, never files. onboarding.env.tpl in the package lists the references. The bucket item, Onboarding Corpus R2, does not exist yet.

Gold labels

Gold is what the text supports, not what turned out to be true later. In the first real ticket, “Epicor server” does not name a product, so gold for erp is unknown even though we later learned it was Kinetic. A classifier that says Kinetic there is guessing. What actually happened is recorded separately and used in the replay evaluation below.

Metrics

1. Wrong accepts. Facts the policy accepted with a label different from gold. Must be zero. This is the release gate.
2. Unnecessary asks. Facts asked about that the text did settle. This is the cost of caution, tracked but allowed above zero.
3. Redaction leaks. Any secret pattern surviving redaction. Zero, and tested by planting known fake secrets.
4. Question quality. For the replay set, whether each question the agent would have asked matches one a human eventually had to ask, and how many days earlier.
5. Groundedness. Share of plan claims whose cited entry the verifier supports. Contradicted claims must be zero in the output.

Replay

Run case one in order, one input at a time, and compare against the real timeline. The agent should have asked the database question after the Sep 16 sponsor email that said “the database of Epicor I want you to use”, which is what happened, and it should have flagged the local-admin question in the ticket, which I could not find answered in the threads I read. Replay is the closest thing to a measure of “self-service” we have.

Hold-out

The workbench skills were written after case one, so retrieving them and then scoring on case one measures memory. The plan-quality eval needs a second customer whose inputs were not used to write any entry. All State Fasteners is the first hold-out. It runs Prophet21, which we support, with data exports and a staging environment already built. Aurora is a poor hold-out for plan quality because there is no live access to plan around, but it is a good test of multi-ERP handling. The VMS Eclipse E4 work is the next candidate.

What exists for ASF. Seven dated, verbatim, redacted inputs from 2026-08-05 to 2026-09-18, and a gold ledger in holdout.json beside them: the stage, the systems, what is known, what should be asked now, what should be deferred, and the expected status of each plan section. onboarding-planner holdout all-state-fasteners replays the inputs in order and compares. The gold was written by the agent’s author from the same inputs, so it is not independent. Someone who knows the account should review it before it counts.

What it scores today. State only, as constraints. The stage reached must be sow_sent and both systems must be found. Known facts must be known: EDI share, line volume band, and email platform. The questions that must be asked (now or queued) are the ERP in scope, hosting, IT owner and approver. The questions that must not be asked are stage, EDI share and volume, which are already answered. VPN, MFA, remote access and database must be deferred. Jev passes. The keyword baseline has one documented gap: the call names Microsoft and Google in a single question and answer, so keywords cannot settle the email platform. The plan sections in the gold file are not scored yet, because the plan drafter does not exist. They record what a good plan says: a file-first read path is ready, a live read path and the write path are deferred to signed with the write path design-only, and the InSQL unit is blocked because it has no knowledge base coverage.

Prototype results

Current corpus: 51 cases. Twenty are real, verbatim and redacted (four Bison Bag, six Federal Hose, three Aurora, seven All State Fasteners). Thirty-one are synthetic, including fourteen that target one proposal-phase fact each.

measure	keyword baseline	Jev
wrong accepts	0	0
correct accepts	60	51
unnecessary asks	10	19
correct asks	244	244
system detection errors	0	0

Jev’s own system detection is not reliable. It missed InSQL on the ASF call in two of three runs on identical input. Named ERP products are a dictionary lookup, so system detection now unions the dictionary matches with Jev’s answers. With that, Jev passed the ASF hold-out on three consecutive runs. That union is then gated on an ownership claim, because the same breadth turned every product merely mentioned on a call into a system the customer runs. An unknown ERP label is skipped rather than raising, so a taxonomy that grows past the keyword dictionary no longer aborts a replay.

Stage and per-system findings from the run:

• Jev’s stage labels were mostly right but its confidence was low on short texts. “We have decided not to move forward” scored 0.48 and was asked. On the thin sponsor email that only proposed a call, it said discovery at exactly 0.85. On our own email about holding off implementation while waiting to sign a contract, it said paused at 0.89 against a gold of verbal_yes, which is a defensible reading. Raising the threshold to 0.90 stopped both from being accepted, and that is why the change is called provisional.
• One gold label of mine was wrong. I had marked “I am into the server via RDP” as access_granted. Under the label definition, ERP credentials still to come make it access_pending. I corrected it after seeing Jev disagree, so treat that case as adjusted.
• The design caught a real bug of its own. With two systems and different VPN answers, the customer-wide vpn came back as one system’s value. It is now accepted only when every system agrees, with tests for both directions.
• Per-system attribution works on the two-system case for both backends. Great Plains got a SonicWall VPN and Business Central got none.
• Jev missed the second system on the ASF call. InSQL did not trigger its other question, so no split happened. Its erp answer there scored 0.76 and was asked, so nothing wrong was accepted, but detection recall is a gap.
• The negation window ran across commas. “You don’t have a shared inbox, it’s all separate” wrongly negated “all separate”, and “We don’t have a budget, but we have a desire to budget” lost its second match. It now stops at a comma. Negated mentions are also dropped instead of forcing a conflict, so “We don’t have an internal IT. We use an outside vendor” resolves to the vendor.
• One gold label was changed after seeing Jev. “We are not in the cloud” was unknown. Jev said on_prem at 0.98, and that is right because the sentence rules out cloud and hybrid. Treat it as adjusted.
• Attribution is exercised. With it disabled the baseline gains two unnecessary asks and the unit tests fail.

The keyword baseline needed fixes twice. Outside review and new cases surfaced wrong accepts on negated mentions (“We do not use Fortinet”, “This is not the production database”, “RDP is disabled”) and “a sandbox is unlikely soon” read as test access. Then the real Aurora and ASF calls surfaced two more, on multi-ERP text and speech-to-text spellings. Live Jev got all of these right unaided, except that it hedged on the ASF call. Its extra asks come from correct labels scoring 0.72 to 0.83, under the 0.85 placeholder. Its one wrong label, mfa on “authenticator app”, scored 0.36 and was asked, not accepted. Confidence tracks correctness, so this is a calibration problem, and a 51-case corpus is still too small to set thresholds.

Findings from the real inputs that shaped the taxonomy:

• The ticket title said “test/pilot Epicor database” and the body never repeated it. The text given to the classifier must include the subject line.
• “2FA” appeared with no type. The agent must ask, and does.
• The transcript has a speech-to-text error and blank placeholders (“temporary password below, then just brackets”). The redactor must not treat the word “password” without a value as a secret, and a test covers it.
• Neither real input says which ERP product this is. A bare “Epicor” is ambiguous across three products, so the baseline requires “Kinetic” or “ERP 10/11”.

10. Security

• Redaction runs first and locally. Only redacted text goes to any model, tracing service or log.
• Nothing raw is persisted. Corpus files are redacted before they are written, and the redactor test plants secrets to prove none survive.
• The agent has no tools that reach the network, a mailbox, or an ERP. Its output is a document.
• Instructions inside inputs are data. The intent taxonomy exists to flag them, and the synthetic corpus includes an injection case that expects no acceptance of any fact that came from an instruction.
• Regex redaction will miss things. A pass by a second System 1 question (“does this text still contain a credential”) is planned as a backstop, and any positive blocks the model call.
• Real corpus files stay out of git. They live in a private R2 bucket with a bucket-scoped token, and R2 encrypts objects at rest.
• Secrets for the CLI (Jev key, bucket keys) come from 1Password at run time with op run. Nothing is read from a file or stored in the environment of a long-lived shell.
• Running the eval with the Jev backend sends the redacted corpus to TypeSafe. That is expected for the real product, and it means the corpus text must be redacted before it is stored, not only before it is sent.
• The redactor is best effort and has_leak is the guard that fails closed. They are written separately, and the guard blocks a model call or an R2 upload on anything that still looks like a credential. Both were rewritten after pre-landing review found that JSON password fields, prefixed variable names, Password is: value, values on the next line, bearer tokens, credentialed URLs of any scheme, PEM keys, hyphenated tokens and zero-width characters all slipped through, and that a placeholder typed into the input could vouch for the text after it. Input is normalized first, typed placeholders are neutralized, and input over 200,000 characters is refused. Every pattern is bounded so a hostile run of characters cannot make redaction quadratic.
• One carve-out in the long-token rule. An unlabeled token of 24 or more characters carrying both letters and digits is redacted as a secret, and a corpus source id (<customer>-<YYYY-MM-DD>-<kind>) looks exactly like that, so citing one blocked every brief. The exemption is cut to that literal shape, embedded ISO date required, because the generic slug shape is also what a memorable passphrase generator emits. Widen it only with a test per shape admitted.
• Known limits. An unquoted passphrase with spaces is redacted only up to its first word, and the guard blocks it only when the rest looks like a credential. Personal data other than credentials, such as names, phone numbers and addresses, is not redacted. A customer statement can steer the classifier’s labels, so a fact that unlocks a stage or suppresses a question comes from one untrusted source and is not corroborated. State cannot retire a system once it has been seen, so a wrong ERP name is removed by fixing or deleting its source case and replaying. These are open, and the last three are listed under decisions.

11. Proposed build order

For review. None of this is started beyond step 1.

1. Done. Redactor, taxonomies, policy, keyword backend, Jev backend, local CLI, eval harness, R2 corpus sync, 20 cases.
2. Done: stage, per-system facts, per-customer state, stage-gated questions, thirteen proposal-phase facts with evidence-based unlock stages, a five-question cap, and the ASF hold-out for state. Someone who knows the account should review the ASF gold. Second real customer in the corpus. Label a Prophet21 or Eclipse onboarding. Without it the plan-level numbers stay contaminated. With more cases, calibrate the per-fact thresholds against Jev.
3. Knowledge base v0. Curated entries from the workbench Epicor skills and our own supported-ERP docs. Filter-and-rank retrieval over files. No derived entries yet.
4. Plan drafter with verifier, one section at a time, each scored against the ASF gold. Start with the business case inputs in section 8.2 and the file-first read path, since those can be written before a contract is signed. Access-path sections come later, when access questions unlock.
5. Extractors for Nango providers.yaml and the Activepieces pieces, the two best sources for unsupported-ERP auth patterns, after the license read for Activepieces’ pieces directory.
6. Write-path section, behind a feature flag, design-only, after a reviewer signs off on how it states its limits.
7. Wiring beyond the CLI. The local CLI is the starting point and is shareable across the dev team through 1Password. Move to a Temporal workflow only if something needs to run without a person. Decide then whether output lands in the docs site or a Linear issue.

12. Decisions and open items

Settled:

• Questions about access wait for a signed contract, and the work is incremental. The agent gets as far as it can and keeps a ledger of what is left.
• Jev key exists and works. Backend is switchable, default auto uses Jev when TYPESAFE_API_KEY is set.
• Corpus files live in R2, not git.
• Start as a local CLI, secrets through 1Password.

Open:

• Whether signed is the right unlock point for every access question, or whether database scope and MFA should unlock later than VPN. The current rule treats them together.
• Whether the unlock stage for budget should move to discovery. It works best after a price is floated, but one prospect answered at the first call.
• Whether a customer-offered sandbox before signing should get its own path, since one prospect started a proof of concept that way.
• Jev’s recall for a second system named only in passing, and stage confidence on short texts. Both need more cases before thresholds move again.
• The corpus bucket and its scoped token need creating. The Cloudflare credential in 1Password cannot manage R2 (authentication error on a bucket list), so this needs someone with R2 admin. Steps are in packages/onboarding-planner-py/corpus/README.md.
• Who reviews a generated plan before it reaches a customer.
• Legal read on n8n’s license.
• Whether a high-impact fact (stage signed, VPN none, database live) may enter state from a single untrusted input, or should need a second source or a person’s confirmation.
• Whether state needs an explicit way to retire a system or a fact, instead of rebuilding from source cases.
• Resolve the NetSuite auth contradiction between the integration registry and the client code before it becomes the first knowledge base conflict.