Spike: real P21 order acknowledgement into Zendesk

Status: research complete, and the working route is now implemented (2026-08-13) — see What shipped. m_orderacknowledgements via /process/pdfreport is broken on both Genfit tenants — but document generation itself works on production, and a working, re-runnable route exists for orders that already exist. Start at The path that works; the option ranking below it predates that finding. Prod build is 26.1.5910.3. See Probe results — play and Probe results — production. Question asked: how do we get an order acknowledgement out of P21 so it can be posted back onto the Zendesk ticket the order came from? Primary source: mrwuss/p21-api-documentation (unofficial community docs — not Epicor-endorsed; every claim below cites the file it came from so it can be re-checked).

TL;DR

The ERP→Zendesk write-back pipeline already exists end to end and already accepts an arbitrary PDF. The gap is not plumbing — it is that the PDF is currently Ordermatic restating what it sent, not P21 confirming what it accepted.

P21 exposes a real Order Acknowledgement document via the report service m_orderacknowledgements on POST {ui_server}/api/v2/process/pdfreport. Fetch that, swap it into the existing pipe, keep the current synthesized PDF as a fallback.

Update 2026-08-11 — read this before option 1. Two findings that change the plan:

1. m_orderacknowledgements cannot render on EITHER Genfit tenant (play 2026-08-11, production 2026-08-12). Its criteria are now known and identical on both (probed live; two field names guessed below were wrong), but /process/pdfreport answers with a bodyless HTTP 500 — on play for every report service including the verified m_reprintpurchaseorders control. On prod a probe matrix locates the fault at document-generation time: the window opens, permissions are fine, an under-specified payload returns 200 [], and a complete payload 500s whether or not the target order exists. Upstream hit the same symptom set on a third tenant. Our payload passes upstream’s offline validator with 0 errors. This is an Epicor bug, not something we can work around.
2. The print-flag path DOES work here, and produces the real document. print_orderack = ON on a savable Order transaction returned a genuine 2-page OR164921 ORDER_ACK.pdf (101,864 bytes, %PDF-1.7, producer “Powered By Crystal”) titled ORDER ACKNOWLEDGEMENT, carrying the live order number and the real bill-to / ship-to. See Print-flag path.

So the rendering engine is alive on this tenant; the report service endpoint is what is broken. That is a sharper thing to take to Epicor, and it means option 2 below — previously rejected as unnecessary risk — is currently the only route that works here.

What already exists in this repo

Do not rebuild any of this. The loop runs today:

Step	Location
Fires after a successful ERP write (post-billing, failure-swallowed on purpose)	apps/ts-temporal-worker/src/workflows/erp-submission.ts:70 — acknowledgeSourceTicket()
Resolves the ticket, claims the work, builds + posts the PDF	apps/webapp/src/pages/api/ticketing/internal/ack.ts
Posts the Zendesk internal note (public: false) with the PDF upload token	packages/integrations/src/ticketing/zendesk/index.ts:393

Details that matter to anyone changing it:

• ack.ts walks orders → extracted_orders → pdf_documents to reach ticketExternalId / ticketingConnectionId, because that is what the submission workflow actually has in hand.
• Concurrency is handled by an atomic conditional claim on orders.ticketAcknowledgedAt, plus ACK_CLAIM_STALE_MS (5 min) takeover for abandoned claims, plus a duck-typed provider.hasPostedAcknowledgement() check to disambiguate “never posted” from “posted but the response was lost”. Read the comments in ack.ts before touching any of it — each branch exists because of a specific duplicate-note or stuck-order failure.
• The provider contract already takes a PDF: provider.postAcknowledgement(ref, { body, pdf: { bytes, filename, contentType }, tag }). Sourcing a different PDF requires no interface change.
• Gated per-org on zendesk_order_source_v1 (the generic ${provider}_order_source_v1 convention; Chatwoot is the one exception, gating on ordermatic_inbox_v1).

The actual gap

buildAcknowledgementPdf() (ack.ts:317) renders a one-page PDF from our own extracted_order_items rows. The comment at ack.ts:339 concedes the timestamp is render-time, not ERP-accept time. Nothing in that document is authoritative — it is our input echoed back, so it cannot confirm what P21 actually committed (which matters given P21’s non-atomic item-row behaviour, below).

P21 options, ranked

1. m_orderacknowledgements report service — recommended

POST {ui_server}/api/v2/process/pdfreport (docs/03-Transaction-API.md:1764; worked recipe at docs/recipes/generate-pick-ticket-pdf.md).

Constants for every report payload — these differ from the record-edit shape:

• Status and Type are numeric 0 (not "New")
• the DataElement carries Keys: []
• only Name, the DataElement name, and the criteria Edits change per service

Shape, using the verified PO-reprint service as the template:

{
  "Name": "m_reprintpurchaseorders",
  "UseCodeValues": false,
  "Transactions": [
    {
      "Status": 0,
      "DataElements": [
        {
          "Keys": [],
          "Type": 0,
          "Name": "TABPAGE_1.poreportcriteriadw",
          "Rows": [
            {
              "Edits": [
                { "Name": "company_id", "Value": "ACME" },
                { "Name": "beg_po_no", "Value": "500100" },
                { "Name": "end_po_no", "Value": "500100" },
                { "Name": "reprint_flag", "Value": "Y" }
              ]
            }
          ]
        }
      ]
    }
  ]
}

Success response is a JSON array even for one document:

[
  {
    "DocumentId": "PO500100 PURCHASE_ORDER",
    "FileName": "PO500100 PURCHASE_ORDER.pdf",
    "DocumentContentType": "application/pdf",
    "DocumentFormat": 5,
    "DocumentData": "JVBERi0xLjQK...",
    "ResponseStatus": { "StatusCode": "Success", "Message": "..." }
  }
]

Why this one: it is read-only, decoupled from the ERP write, and re-runnable against any order number — which is exactly what the existing retry-safe ack activity needs. Unlike m_picktickets (which creates the pick-ticket record as a side effect), an acknowledgement reprint should have no ERP-side effect.

Caveat — resolved 2026-08-11. The DataElement name and criteria fields were unknown when this was written, and two of the names guessed above are wrong. Use the verified shape in Probe results, not the PO-reprint template. Note also that the “should have no ERP-side effect” claim just above is still an expectation, not a finding — see Open questions.

2. print_orderack = ON on the createOrder transaction — rejected for production, but it is the only route that WORKS on the playground

The Order definition carries print_orderack / email_orderack / fax_orderack (type Char, ValidValues: ["ON","OFF"]) on the TP_FRONTCOUNTER.tp_frontcounter DataElement. Print flags on a savable transaction return PDFs at Results.Transactions[].Documents[].DocumentData (docs/03-Transaction-API.md:2102), and a new order is savable — so this would technically work in a single call.

Rejected anyway. createOrder already has a documented non-atomic partial-commit failure mode (docs/incidents/2026-06-01-genfit-p21-partial-submission.md) and a tight timeout budget: makeP21Request caps at 4 min, deliberately under the 6-min activity startToCloseTimeout. Adding a report render inside that transaction buys one saved round-trip at the cost of the most dangerous call in the system.

Separately, treat email_orderack as a non-goal: it makes P21 email the customer directly, behind the CSR’s back. The point of this work is to put the document in Zendesk where a human decides what to send. fax_orderack likewise.

Print-flag path — verified working (2026-08-11)

Run with scratch-style harness against the playground: one throwaway quote (approved = OFF), one line, the same five header edits our working createOrder sends, plus a third DataElement carrying the flag. POST /uiserver0/api/v2/transaction → 200, Status: "Passed", order 1013389, and one document:

Documents[0].FileName            = "OR164921 ORDER_ACK.pdf"
Documents[0].DocumentContentType = "application/pdf"
                                   101,864 bytes, %PDF-1.7, 2 pages

The flag lives on its own DataElement — not on TABPAGE_1.order:

{
  "Name": "TP_FRONTCOUNTER.tp_frontcounter",
  "Type": "Form",
  "Keys": [],
  "Rows": [{ "Edits": [{ "Name": "print_orderack", "Value": "ON" }] }]
}

Re-verified with a committed harness (2026-08-12)

The 2026-08-11 run used a throwaway script. There is now a real one: apps/dagster/scripts/probe_p21_print_orderack.py. It is dry-run by default, writes a quote (approved = OFF) unless --approve is passed, stamps po_no with a findable marker, discovers a valid customer/ship-to/item triple off the tenant’s newest order rather than guessing, and saves the PDF to --out-dir.

infisical run --env=staging --path=/runtime/webapp -- \
  uv run python scripts/probe_p21_print_orderack.py \
  --connection-id 566f4628-4771-4ea0-8ad2-5eeb2a69e66a --submit --out-dir /tmp

Second run, play tenant, quote 1013390 (po_no=ACK-PDF-TEST-2026-08-12): HTTP 200, Summary {"Failed": 0, "Succeeded": 1}, Status: Passed, one document — OR164922 ORDER_ACK.pdf, 101,858 bytes, %PDF-, application/pdf.

pdftotext confirms it is the genuine customer-facing document, not just well-formed bytes: title ORDER ACKNOWLEDGEMENT, Genfit letterhead, Order Number 1013390, order date, bill-to and ship-to for Humphrey Service Parts, Customer ID 10057, our PO marker, the line (H C2480-6, 1 EA @ 4.4100), SUB-TOTAL/TAX/AMOUNT DUE, and Genfit’s terms boilerplate. Producer is Crystal Reports.

One correction to the 2026-08-11 entry above: this document is 1 page, per its own “Page 1 of 1” and a /Count 1 page tree. The earlier “2-page” claim was almost certainly a miscount — the two files differ by 6 bytes, which a whole extra page would not.

Note what the document says. It reads “This acknowledgement constitutes acceptance of our terms and conditions” and carries a restocking-fee clause — on a record that is only a quote. That is fine for the current design, which posts it as an internal note for a CSR to judge. It would not be fine to send to a customer automatically off a quote.

What the P21 web client actually does (HAR capture, prod, 2026-08-12)

A DevTools capture of ticking Print Options → Order Acknowledgement → Print and pressing Save on the prod Order Entry window. None of these are the documented /api/ui/interactive/ surface — the web client drives two others, and this is the fastest way to learn field and tool names for any window:

Call	Purpose
PUT /uiserver0/ui/full/v2/data/data?dw=tp_frontcounter&fn=print_orderack&tn=TP_FRONTCOUNTER&wn=w_order_entry_sheet&wid={guid} body {"DatawindowName":"tp_frontcounter","FieldName":"print_orderack","Value":"Y","WindowId":"…"}	Set one field
GET /uiserver0/api/ui/emawindow/{wid}/ribbon/	Enumerate the window’s toolbar tools
PUT /uiserver0/api/ui/emawindow/{wid}/tools/Quick.Save body {"windowName":"w_order_entry_sheet"}	Invoke the Save tool

Note the UI sends Value: "Y", while the Transaction API declares ValidValues: ["ON","OFF"] and accepts ON. Different layers, different encodings.

Success: true does not mean saved. The Quick.Save above returned Success: true, State: "Normal" — and did not write anything, because Messages[] carried [Information] "A valid customer ID must first be entered." and [Error] "No line items have been entered." Any client on this surface must gate on Messages[].Type == "Error", exactly as the upstream Interactive API guidance says (“don’t trust save status alone”). Nothing was created on production by this capture.

The Order Entry ribbon has no order-acknowledgement reprint tool. Its 38 tools include m_reprintpicktickets (“Reprint Pick Tickets”) and m_inventorycardreport, but nothing for acknowledgements — which confirms print-on-save is the only route from this window. Usefully, ribbon tool names are report service names, so the UI’s report launcher and /process/pdfreport address the same namespace. That makes “run the Order Acknowledgement report from the UI” a true apples-to-apples test of the report engine.

Unresolved, and it is the whole question: because that save never committed, the capture says nothing about whether a successful save returns the PDF on this surface. Repeat it on the play tenant against an order that actually saves.

Lead worth chasing first — EnablePdf. The Quick.Save response carries Result.EnablePdf: false on prod. Its meaning is undocumented and this is speculation, but it is cheap to test: open the same w_order_entry_sheet window on play, where the print flag demonstrably does produce a PDF, and compare. If play reports true and prod false, that is a tenant-level switch for document generation — which would explain the report engine failing on prod while the same payload works elsewhere, and would turn the Epicor ticket into a specific configuration question.

Constraints that shape any use of this:

• Savable only. A bare reprint errors with “Save is not enabled”, so this can only ever produce the acknowledgement for an order the same call is creating. There is no way to re-fetch it later for an order that already exists.
• One shot. The bytes exist only in that response. Our ack activity retries independently, minutes later — so adopting this means persisting the PDF inside the submit path and having the ack read it from there, not re-asking P21. No new infrastructure is needed for that: packages/integrations/src/storage/r2-storage.ts (R2StorageClient) already exists and is what the source PDFs use.
• The other objections above still hold: createOrder is the partial-commit-prone call, and rendering inside it eats into the 4-minute makeP21Request budget.

Side effects — measured, not assumed. oe_hdr for 1013389 (flag ON) vs 1013388 (same customer, same day, flag absent) differs in 7 columns, and every one is an unavoidable per-order field: order_no, oe_hdr_uid, po_no, order_date, date_created, date_last_modified, rma_expiration_date. No ack-sent or print-date marker appeared. Caveat: this compares two different orders rather than one order before and after, so it proves the flag leaves no distinct oe_hdr signature — it does not prove the absence of effects elsewhere (print queues, doclink rows, per-customer ack counters).

4. The path that works on production (2026-08-12)

This supersedes the ranking above. A HAR capture of the prod web client doing the thing by hand proves three things the rest of this document had wrong or open:

1. Document generation works fine on Genfit production. A real 105,282-byte %PDF-1.7 acknowledgement came back, StatusCode: "Success", message Form request '1' for Form ID OR209119 has completed. — the same response envelope /process/pdfreport is documented to return (DocumentType, DocumentId, DocumentFormat: 5, FileName, DocumentContentType, DocumentData, ResponseStatus). So “report generation isn’t available in that environment” is wrong for prod. The engine is alive; only /uiserver0/api/v2/process/pdfreport is broken. That is a much sharper Epicor ticket.
2. The print flag works on an order that already exists. The captured run was against order 1012793, created 2025-03-28 — not a new order. So the “savable-only, one-shot” constraint is a limit of the Transaction API, not of P21. An acknowledgement is re-fetchable for any existing order.
3. The whole flow is replicable server-to-server, because the datastream arrives inline. No UNC share read is needed.

The call sequence

1. PUT  /uiserver0/ui/full/v2/data/data
        ?dw=tp_frontcounter&fn=print_orderack&r=-1&tn=TP_FRONTCOUNTER
        &wn=w_order_entry_sheet&wid={windowId}
   body {"DatawindowName":"tp_frontcounter","FieldName":"print_orderack",
         "Value":"Y","WindowId":"{windowId}"}

2. PUT  /uiserver0/api/ui/emawindow/{windowId}/tools/Quick.Save
   body {"windowName":"w_order_entry_sheet"}
   -> Events[] contains:
      { Name: "formdatastreamready", Publisher: "crystaloutput",
        EventData: { formType: "ORDER_ACK", jobType: "PRINT", formId: "OR209119",
                     datastreamFilename: "\\\\…\\shared\\printing\\OR209119.cfd",
                     xmlDatastream: "<?xml …>"   <-- the full ~17KB Crystal job, INLINE
                     printerName: "", publishPath: "", prompted: "N" } }
      { Name: "savesucceeded", EventData: { __order_no: "1012793" } }

3. POST /api/document/generate/form          <-- NOTE: not under /uiserver0
   body {"ClientId":null,"RequestId":1,"Batch":null,
         "DocumentId":"{formId}","DocumentFormat":"PDF","CompressDocument":false,
         "DatastreamXml":"{base64 of xmlDatastream}"}
   -> { FileName:"OR209119.pdf", DocumentContentType:"application/pdf",
        DocumentData:"JVBERi0…", ResponseStatus:{ StatusCode:"Success" } }

xmlDatastream is a complete Crystal job: REPORTNAME Acknowledgements.rpt on \\p21us…\generationfittingFiles\Shared\Reports\, USERREPORT Y with an Acknowledgements_user.rpt override, the logo path, and then the entire order as data — header, every line, release schedules, and totals. P21 renders from that payload, not from a fresh DB read, which is why the window has to load the order first; you cannot ask this endpoint for “order X’s acknowledgement” out of nowhere.

Reassuring on side effects: ARGPRINTER = __!uiserver!__ (render to the caller, not a physical printer), SENDPDFCOPYTODOCUMENTIMAGING = False, printerName and publishPath empty, prompted = N. Nothing spooled or archived.

Verified server-to-server (play, 2026-08-12)

apps/dagster/scripts/probe_p21_interactive_orderack.py runs the whole flow without a browser. Dry-run by default: it opens the window, loads the order, proves the order is really in the window, and stops. --submit sets the flag, saves, and renders.

infisical run --env=staging --path=/runtime/webapp -- \
  uv run python scripts/probe_p21_interactive_orderack.py \
  --connection-id 566f4628-4771-4ea0-8ad2-5eeb2a69e66a --order-no 1013390 --submit

Against existing order 1013390 on play:

savesucceeded : {"datawindowname":"order","__order_no":"1013390"}
formType      : ORDER_ACK    jobType: PRINT    formId: OR164923
datastream    : 9,736 chars  (build 26.1.5910.0)
render        : StatusCode "Success", Form request '1' for Form ID OR164923 has completed.
saved         : OR164923.pdf — 101,858 bytes, %PDF-, application/pdf

pdftotext confirms it is order 1013390’s own acknowledgement: right customer (Humphrey Service Parts, Customer ID 10057), right PO marker, right line. 101,858 bytes — the same size as OR164922, the Transaction API print-flag output for the same order, so both routes render the identical document. The interactive route simply also works on an order that already exists.

Interactive session open → window → load → flag → save → render, with the session always deleted afterwards (a ghost session can only be reaped by the token that made it).

Gotchas found while building it

• Two message conventions. The emawindow surface reports Messages[].Type as a string ("Error", "Information"); the interactive /v2/change surface reports it numerically (3). A client must treat both as errors, or it will sail past a real failure. /v2/change also returns no Success/State at all.
• Play emits [3] "No periods have been set up for this company." when the order loads. It looks fatal and is not — the save succeeded and the document rendered anyway.
• GET /v2/data?id= returns a list, not a dict. Note also the parameter inconsistency: /v2/data and /v2/window take ?id=, /v2/tools takes ?windowId=.
• Open by ServiceName: "Order". Opening w_order_entry_sheet by Name or Title is unreliable on 26.1.
• Read the build from the datastream, not from session-create. Play’s session-create returned 26.1.5910.0 once and 0.0.0.0 on later calls in the same hour; parserApplicationVersion on the datastream was correct every time.
• Confirm the order actually loaded before saving. On 26.1 a nonexistent key leaves the window empty instead of erroring, and a save then acts on nothing. The script refuses to save unless it can find the order number in the window data.

What it costs

• It re-saves the order. The datastream shows DATE_LAST_MODIFIED 08/12/2026 16:41:21 and LAST_MAINTAINED_BY TEST.ACCOUNT2 on an order originally taken by MICHELE.NICHOLS. No business data changed, but the audit trail now attributes a modification to our service account. For orders Ordermatic created that is unremarkable; doing it to arbitrary customer orders is audit-trail pollution and a real design constraint.
• More moving parts than one report call: session, window open, field set, save, document POST — each a failure point, and the save carries the Success: true-with-an-Error-message trap.
• Undocumented endpoints. None of these three appear in the upstream docs, so they carry no version-compatibility promise.

Why it is still probably the right answer

It is re-runnable, which the Transaction API print flag is not. That removes the one-shot capture, removes the need to persist bytes inside the submit path, removes the R2 dependency, and decouples the acknowledgement from createOrder — the call the 2026-06-01 incident says not to add work to. The ack activity can fetch on its own retry schedule, which is what it already wants to do.

Untested: whether the flag can be set and the document produced without a save (the Order Entry ribbon exposes no acknowledgement reprint tool, so probably not), and whether Quick.Save on an unmodified order is even accepted when nothing else changed — in the captured run the flag itself was the change.

3. OData read-back for the note body — do alongside option 1

fetchVerifiedOrderItems (apps/webapp/src/services/erp/implementations/prophet21-service.ts:2782) already queries p21_view_oe_line for confirmed qty/price and feeds executeOrderVerification. Extending to oe_hdr yields real ERP facts — order date, totals, promise/ship date, approved status — to replace today’s bare note body:

Order submitted to the ERP as ${erpOrderNumber}. Acknowledgment attached.

Note that string is also the idempotency marker hasPostedAcknowledgement() searches for (ack.ts:165). It is deliberately built only from the ERP order number so a retry finds the same comment a previously-ambiguous attempt may have left. If you enrich the body, keep a stable order-number-only substring as the marker, or the duplicate-note guard silently stops working.

Probe results (Genfit playground, 2026-08-11)

Probed with apps/dagster/scripts/probe_p21_report_service.py against https://generationfitting-play-api.epicordistribution.com, using the P.F. Markey - Zendesk POC connection (566f4628-4771-4ea0-8ad2-5eeb2a69e66a). Read-only by default; --render opts into the POST plus an oe_hdr before/after diff.

GET /definition/m_orderacknowledgements and GET /defaults/... both return 200. GET /api/ui/router/v1/?urlType=external returns …/uiserver0/, so the hardcoded /uiserver0 segment makeP21Request uses is correct for this instance — that trap is cleared here, not in general.

Corrected criteria shape

Guessed above	Actually
TABPAGE_1.poreportcriteriadw	TABPAGE_1.tp_1_dw_1 (datawindow d_order_ack_sheetreport)
beg_order_no / end_order_no	order_number1 / order_number2, and they are Char, not numeric

22 fields, only two required: company_id (Char) and print_qty (Decimal, default 1). A second DataElement, TIMESTAMP.timestamp (d_master_report_timestamp), carries 6 optional created/modified-date fields.

The three switches that matter:

Field	ValidValues	Default
print_option	Order Acknowledgement, Delivery Ticket	Order Acknowledgement
bucket_3	Order, Quote, Both	Order
bypass_customer_acknowledgements_setting	ON, OFF	OFF

bucket_3 settles the quote question mechanically — the report targets quotes explicitly, so isQuote maps to a criteria value instead of needing a policy call. bypass_customer_acknowledgements_setting probably wants ON for our use: at OFF, a customer flagged in P21 as not receiving acknowledgements likely yields no document at all, and the point here is to put the PDF in front of a CSR, not to email the customer.

The date default is a trap

Every default edit carries IgnoreIfEmpty: true, and the date criteria default to today only (beg_order_date = today 00:00, end_order_date = now). So omitting the dates does not mean “any date” — it means “today”. A Temporal retry after midnight, or an order placed yesterday, would render nothing and look like a malformed payload rather than a date filter. Always send an explicit wide range. Build the payload from the service’s own /defaults and override only what you mean to set.

company_id defaults to GENFIT here. It must come from the connection, never a constant — upstream verified that a wrong company_id also returns a bodyless 500.

Report-service rendering is blocked on this instance

POST /uiserver0/api/v2/process/pdfreport returns HTTP 500 with a zero-byte body for every combination tried against order 1013386:

• both UseCodeValues settings, 3 retries each (6 attempts, no attempt ever differed)
• the server’s own /defaults echoed back verbatim
• first DataElement only, dropping TIMESTAMP.timestamp
• numeric Status/Type: 0 and string "New"/"Form"
• order-number-only, and order number plus a wide date range

Evidence that this is the environment and not us:

• The verified control fails identically. m_reprintpurchaseorders — documented as running at production volume elsewhere — returns the same bodyless 500 here.
• The endpoint routes fine. A bogus service name returns 200 [], and so does an empty Transactions array. Auth, /definition and /defaults all work.
• Upstream reproduced it on the Play tenant at 26.1.5910.3: every report empty-500, criteria from each service’s own /defaults, both UseCodeValues, six Accept variants, while /definition and /defaults returned 200 — “treat a blanket failure like that as an environment property, not a payload bug” (docs/03-Transaction-API.md, An empty 5xx has several causes).
• Our payload is independently valid. Upstream’s offline scripts/validate_payload.py reports 0 errors, 0 warnings on the generated acknowledgement payload, and confirms numeric Status is the report-process shape.

Two negative results worth keeping: POST /api/v2/process/pdfreport without /uiserver0 returns 404 HTML, and POST /uiserver0/api/v2/transaction returns 200 with Status: "Failed" and no messages — the wrong-endpoint trap, confirmed live.

The failure is scoped to report rendering, not to P21 or the Transaction API. On the same tenant, in the same API: GET /definition/{service} and GET /defaults/{service} return 200, POST /transaction returns 200, and OData reads work. Only POST /process/pdfreport fails. The Interactive API is healthy too — an api/ui/interactive/sessions create/read/delete cycle returns 200/200 with no ghost session left behind. So the dead component is the report engine specifically.

The middleware build cannot be read from this tenant, so “same version as upstream” stays unverified — do not repeat it as fact. P21 exposes no version endpoint; the build rides the session-create response, and here it is zeroed: fullversion: "0.0.0.0", shortversion: "0.0" (those are the only two version-shaped fields in the payload). The match to upstream is a match of symptoms, not of a confirmed build number.

Superseded for production (2026-08-12). Prod’s build is now known — 26.1.5910.3 — read from parserApplicationVersion on the Crystal datastream (see The path that works). That is exactly the build upstream reproduced the blanket report empty-500 on, so for prod this is a confirmed build match, not a symptom match. Play’s build is still unread, so the paragraph above still stands for play. A third way to read the build: any formdatastreamready event carries it.

Probe results (Genfit production, 2026-08-12)

Same script, against the Genfit Production connection (a0b89aa8-d4ed-42e7-8e30-7db3f6e5d05b, in the production database — every prophet21 row in staging points at play). API base https://generationfitting-api.epicordistribution.com; default_company_id in extra_config is GENFIT, same as play.

Read-only half only. Rendering has NOT been attempted on production — see Open questions.

What this establishes:

• Auth works (Bearer, expires_in=86400), and GET /definition/m_orderacknowledgements and GET /defaults/... both return 200 — same as play.
• The criteria shape is identical to play, so it is a property of the service and not of one tenant: DataElement TABPAGE_1.tp_1_dw_1, datawindow d_order_ack_sheetreport, 22 fields, exactly two required (company_id Char, print_qty Decimal). Second DataElement TIMESTAMP.timestamp (d_master_report_timestamp), 6 optional fields. order_number1/order_number2 are Char; customer_id/ship_to_id are Decimal.
• The date trap reproduces on prod. /defaults returns beg_order_date = today 00:00 and end_order_date = now. Omitting the dates means “today”, not “any”.
• company_name defaults to 'Generational Fittings, LLC' — a display string on the report, distinct from company_id.
• OData p21_view_oe_hdr reads fine on prod too — 218 columns, so Data Services permissions are not a factor here either. The existence-check in client requirement 4 is available.

process_doclinks — the field to keep OFF

Prod (and play) expose process_doclinks (Transmit Document Links, ON/OFF, default OFF). The name suggests it transmits linked documents somewhere on render. Nothing here establishes what it actually does, and that is the point: leave it at OFF, and never let a payload builder set it ON “for completeness”. The related bypass_customer_acknowledgements_setting is the other switch with outward-facing potential — the spike’s earlier note that we “probably want ON” is about matching the report to our CSR-facing use, but on a production tenant ON overrides a customer’s own “do not send acknowledgements” preference, so it should stay OFF until someone establishes what P21 does with it.

Safe render targets on prod (read-only query)

Oldest orders that are both completed = 'Y' and cancel_flag = 'Y' — closed and cancelled in 2020, so nothing downstream is waiting on them and a stray print-date stamp would land on a dead record:

order_no	order_date	customer_id	po_no
1000002	2020-08-12	10144	631445
1000003	2020-08-12	10144	633264
1000004	2020-08-12	10144	633505

Rendering fails on production too — and the fault is now precisely located

POST /uiserver0/api/v2/process/pdfreport returns HTTP 500 with a zero-byte body on production, exactly as on play. Tried both UseCodeValues settings, 3 attempts each (6 per order), against two orders:

Order	State	Result
1000002	completed and cancelled (2020)	6 × empty 500
1000005	completed, not cancelled (2020)	6 × empty 500

The second order exists to kill a confound the first one introduced: a cancelled record is one of the documented causes of an empty 500, and the safest order to pick was also the most likely to be non-printable. It isn’t the cause. The harness also sends an explicit 1990–2049 range plus order_number1/order_number2, so the date trap is not the cause either.

A zero-side-effect probe matrix then located the fault (nothing real rendered):

Request	Prod result	What it proves
Bogus service name	400 "Window <<…>> is not available or user does not have permission to open it."	Service names are validated, and m_orderacknowledgements is not rejected — so the window exists and this user may open it. Permission is not the problem.
Empty Transactions	200 []	The endpoint routes and parses.
Complete criteria, nonexistent order 99999999	empty 500	Identical to a real order. Criteria are never evaluated — a missing record should yield No records to print for this range.
Criteria missing required company_id	200 []	An under-specified payload short-circuits cleanly. Only a payload complete enough to attempt a render reaches the 500.

Read together: the request is accepted, the window is permitted, the criteria shape is right, and the failure happens at document-generation time, before record selection. The report engine itself is broken on this tenant, deterministically.

Note a tenant difference worth remembering: on play a bogus service name returned 200 [], on prod it returns a 400 naming the window. Prod validates more strictly, which is what makes the permission conclusion above available here and not there.

This kills “it’s a play-environment quirk.” Two independent tenants — a sandbox and a production system — fail identically, with upstream reporting the same symptom set on a third. This needs an Epicor ticket, and the four rows above are the ticket: they rule out payload shape, permissions, record existence, and criteria evaluation, and leave only the renderer. It does not mean the feature is unavailable in P21 generally — see Is this P21-wide, or per-instance?.

Is this P21-wide, or per-instance? — per-instance, and other tenants have it working

This is settled by upstream’s own correction, not by inference. An earlier draft of their changelog reported /process/pdfreport as simply broken, on exactly our evidence (every attempt on a Play tenant returning an empty 500). They retracted it:

That reading was wrong: the endpoint runs at production volume — one integration emails supplier POs all day, logging 154 successes against 3 empty 500s and one dropped connection in a single afternoon, with every affected PO succeeding moments later. — docs/10-Changelog.md

So the capability exists and is used in anger on other tenants. Their enumerated causes of an empty 500 (03-Transaction-API.md, An empty 5xx has several causes) put ours in the third row: “the record isn’t printable, or report generation isn’t available in that environment” — deterministic, as opposed to the transient fault that dominates in production.

Ruled out as explanations:

• A 26.1 version regression. 26.1’s breaking changes are documented in detail (14-Breaking-Changes.md) and pdfreport is not among them. The 26.1 empty-500 defect is Interactive-endpoint-specific and Accept-header-driven; our probe and prophet21-service.ts both send Accept: application/json at every call site, and upstream tested six Accept variants against the same failure. Their own verification build (26.1.5910.3) is the same generation as the tenant where reports work.
• Payload shape. Upstream’s offline validator passes it; an under-specified payload returns 200 [] here rather than 500.
• Permissions. A bogus service name draws a 400 naming the window on prod, while m_orderacknowledgements does not — the window is available and this user may open it.
• A missing record. A nonexistent order number 500s identically to a real one.

What that leaves is per-tenant configuration or licensing of the report/document-generation component behind /process/pdfreport — and one sharp piece of evidence about which: on the play tenant, the print-flag path rendered a genuine Crystal-produced ORDER_ACK.pdf (“Powered By Crystal”) while /process/pdfreport empty-500’d on the same tenant, in the same API, minutes apart. The document engine itself is alive there; the pdfreport process path is not. That asymmetry is the most useful single fact for an Epicor ticket, because it excludes “reporting isn’t installed” and points at the report-service/output-destination configuration for that path.

Untested and worth knowing: whether the same asymmetry holds on production. The print-flag half of it requires creating a savable order, so it has not been run there.

Practical read: there is a real prospect of this working on PF Markey’s instance, or on Genfit’s after an Epicor configuration change. It is not a dead end — but nothing we control makes it work, so the POC cannot depend on it landing.

Client requirements this surfaced

Any producer must do all four of these, or it will misreport:

1. Retry a bodyless 5xx. It is usually a transient report-engine fault — upstream logged 154 successes against 3 empty 500s and a dropped connection in one afternoon, every one clearing on retry. Rendering is an idempotent read: 3 attempts, 0.5s × attempt. (This is also what proves our failure is deterministic rather than the fault.)
2. Classify the ErrorType/ErrorMessage envelope BEFORE the status code — it can arrive on a 200 as well as a 5xx. Branching on the status alone hides the one message that explains the failure, e.g. No records to print for this range.
3. A 200 does not mean a usable document. Each element carries its own ResponseStatus.StatusCode string, a 200 can carry a failed document, and a multi-document response can mix both. Require StatusCode == "Success" and non-empty DocumentData, per element.
4. Existence-check the order over OData first. A missing record and an engine fault produce the same unhelpful 5xx; checking first turns one of them into our own clear error. oe_hdr reads fine here (218 columns), so permissions are not a factor.

Also checked and not a problem for us: on 26.1, any request whose Accept header lacks application/json returns an empty 500 — including */*, the httpx and .NET default. prophet21-service.ts already sends Accept: application/json at every call site.

What shipped (2026-08-13)

The interactive route from The path that works, ported out of probe_p21_interactive_orderack.py into the ack pipeline. Not the report service — that stays blocked on Epicor.

Piece	Where
generateOrderAcknowledgementPdf(connection, erpOrderNumber) — the six-call interactive flow, session always deleted	apps/webapp/src/services/erp/implementations/prophet21-service.ts
Optional on the ERP contract, so other adapters simply don’t offer it	packages/integrations/src/services/ierp-service.ts
Connection lookup + deadline + fallback-on-anything, behind one call	apps/webapp/src/lib/ticketing/erp-acknowledgement.ts
Uses the ERP document when it arrives, buildAcknowledgementPdf() when it doesn’t	apps/webapp/src/pages/api/ticketing/internal/ack.ts

Decisions baked in:

• No flag of its own. It rides the ack route’s existing order-source gate (${provider}_order_source_v1, ordermatic_inbox_v1 for Chatwoot), which the route has already passed before it sources a PDF. Sam’s call, 2026-08-13, over the alternative of a separate erp_order_acknowledgement_pdf_v1. The consequence to remember: order-source and “may re-save orders in this ERP” are now one switch, so every organization with ticket ingestion on gets the last_maintained_by stamp, with no per-ERP-instance rollout. Split the gate if that ever has to be two decisions.
• Fallback is unconditional. No ERP connection on the order, an ERP without the capability, a thrown error, or a 30s deadline overrun all resolve to the locally rendered PDF. A note with our own document beats no note. The deadline exists because the calling activity aborts its HTTP request at 60s (apps/ts-temporal-worker/src/activities/ticket-acknowledgement.ts), and the fallback render plus the Zendesk post still have to fit in what is left.
• The load step’s messages are not fatal, because play’s benign [3] "No periods have been set up for this company." arrives there. The real gate is the window-data check: refuse to save unless the order number is actually in the window (the empty-window trap).
• Success: true is not trusted on the flag or save steps — both Messages[].Type conventions (string "Error", numeric 3) count as failures.
• The note body and filename are unchanged, so the hasPostedAcknowledgement idempotency marker still matches.

Still untested against a live tenant: the port itself. The Python probe verified the flow end to end on play against order 1013390; the TypeScript is covered by prophet21-service.order-acknowledgement.test.ts (mocked HTTP) but has not yet run against P21.

Recommended implementation

1. Probe before writing code. Done 2026-08-11 — see Probe results. Re-run it against PF Markey’s own instance before production; the criteria shape should carry over, but company_id and whether rendering works at all will not. apps/dagster/scripts/probe_p21_report_service.py --connection-id <id> does the read-only half; add --render <order_no> for the write test.
2. Add generateOrderAcknowledgementPdf(connection, erpOrderNumber) to prophet21-service.ts, routed through makeP21Request to /uiserver0/api/v2/process/pdfreport. It must satisfy all four client requirements — retry the bodyless 5xx, classify the error envelope before the status code, gate on each document’s own ResponseStatus, and existence-check the order first. _checkP21MessagesForErrors does not parse this endpoint’s envelope; it needs its own handling.
3. In ack.ts, try the P21 acknowledgement first and fall back to buildAcknowledgementPdf() on failure. The existing error path already does the right thing (502 → release the claim → Temporal retries), so a fallback keeps notes flowing on connections where the report service isn’t enabled.
4. Enrich ackBody from an oe_hdr read-back — preserving the idempotency marker.
5. Gate the P21 path behind its own flag, separate from zendesk_order_source_v1, so it can roll out per connection.

Traps

• Wrong-endpoint trap. POST /api/v2/transaction accepts an m_* report payload and returns Succeeded — but emits nothing. The docs call this “the single biggest gotcha.” Reports must go to /api/v2/process/pdfreport.
• UseCodeValues varies per report service. m_reprintpurchaseorders works with false; m_picktickets requires true with code values and returns HTTP 500 with false. Still unresolved for m_orderacknowledgements: both settings fail identically on the playground, so this cannot be settled until an instance renders at all.
• A bodyless 5xx is usually transient, and its causes are indistinguishable. Transient engine fault, criteria that match nothing (a wrong company_id), a non-printable record, and an environment without report generation all look the same. Retry first; treat a blanket failure across services as the environment.
• Omitting a date criterion means “today”, not “any”. Every default edit is IgnoreIfEmpty: true and the date defaults are same-day. Build from /defaults and override explicitly.
• Different error envelope. /process/pdfreport returns { ErrorType, ErrorMessage } — e.g. "No records to print for this range." — not the Summary/Messages shape /transaction uses. _checkP21MessagesForErrors in prophet21-service.ts will not parse it; needs its own handling.
• An unknown $select column returns 404, not 400 on OData. If a table you know exists starts 404ing, drop $select and retry before chasing Data Services permissions.
• OData base host vs. UI server. OData runs on the P21 base host; Transaction and Interactive APIs use the UI server URL. Our code papers over this: makeP21Request concatenates connection.apiUrl with a hardcoded /uiserver0 path segment, where the docs say the UI server should come from GET /api/ui/router/v1/?urlType=external. Works today, but is instance-specific — confirm against PF Markey’s setup rather than assuming. Note the existing OData paths in our code (/data/erp/views/v1/p21_view_oe_line, /uiserver0/api/v2/odata/Order) also differ from the docs’ /odataservice/odata/view/{viewname}, i.e. path layout varies by P21 version.
• Data Services permissions (user-credential auth only): the P21 user needs User Maintenance → Application Security → “Allow OData API Service” = Yes and Role Maintenance → Dataservice Permission for each table/view. Missing either gives "You are not authorized to access API" on a valid token. Consumer Key auth skips both.

Open questions

• Still open for the report service. Does m_orderacknowledgements have an ERP-side effect (e.g. stamping oe_hdr.print_date or an ack-sent flag)? m_picktickets does create a record. Unanswerable on the playground because no report render succeeds. It matters because the ack activity is retry-safe by design, so a side-effecting service would re-stamp on every Temporal retry. The harness diffs oe_hdr automatically; the first successful render on any instance must be on a throwaway order. (For the print-flag path this question is largely settled — see Print-flag path.)
• Which route do we actually build? The print-flag path works here today but only for the order being created, one shot, inside the riskiest call. The report path is read-only and re-runnable but needs an instance whose report endpoint works. A defensible answer is to build the report path as designed and treat the print flag as a playground-only demo mechanism — but that leaves the POC demoing something production would not use.
• Does /process/pdfreport render on Genfit production? Answered 2026-08-12: no. Same deterministic empty 500 as play, on two orders, with the fault located at document-generation time. See Rendering fails on production too.
• Who owns the Epicor ticket, and does the POC wait on it? Two tenants failing identically makes this Epicor’s bug, not a payload problem we can work around. The probe matrix is the evidence. Until it’s fixed, the report path cannot be verified end to end on any instance we have — so either the POC ships on the print-flag path (with the one-shot/persist-to-R2 consequences in Print-flag path), or it ships on the existing synthesized PDF and the real acknowledgement lands later.
• Is PF Markey’s own instance worth probing now anyway? The POC customer (P.F. Markey) and this ERP tenant (Genfit = General Fittings LLC) are different companies. A third data point would tell us whether this is Epicor-wide or specific to Genfit’s two tenants — which changes how the ticket is framed.
• Should the acknowledgement be fetched at submission time or lazily when a CSR opens the ticket? Submission time matches the existing flow; lazy would avoid a P21 call for orders nobody looks at.
• Should a quote (isQuote: true) get an acknowledgement at all, or only an approved order? createOrderXml defaults isQuote to true when undefined (prophet21-service.ts:1013), so this needs an explicit decision rather than a default.