Fulfil Error Alerting (TransactionError → Sentry → Slack)

How a failed Fulfil call becomes a persisted TransactionError row, a Sentry event, and a Slack message.

Deep-dive doc split out of .claude/rules/architecture.md (which is now an index). Append new findings about this area here, not to the index.

There is code-level Slack alerting (src/clients/slackClient.ts)

⚠️ Older revisions of the architecture index claimed "There is no code-level Slack integration anywhere in the codebase" and that Fulfil Slack alerts come only from Sentry alert rules. That is stale. Since INFRA-520 the repo posts to Slack directly via chat.postMessage:

• src/clients/slackClient.ts — an axios client (makeRetryClient) with baseURL: env.SLACK_FULFIL_ALERT_WEBHOOK_URL (in practice https://slack.com/api/chat.postMessage), Authorization: Bearer ${env.SLACK_BOT_TOKEN}, and channel: env.SLACK_CHANNEL_ID injected by _postMessage.
• isEnabled() requires all three of SLACK_FULFIL_ALERT_WEBHOOK_URL, SLACK_CHANNEL_ID, SLACK_BOT_TOKEN; otherwise every notifier is a silent no-op returning { enable: false }. This is what keeps local/CI runs quiet — not a NODE_ENV check.
• _chunkLines() packs detail lines into ≤2900-char section blocks because Slack rejects the whole message (invalid_blocks) if any single section text exceeds 3000 chars.

Sentry alert rules may also be configured in the Sentry project, but the primary Fulfil Slack path is this client.

Slack triggers: TransactionErrorService.create (Fulfil) and postWorkOrderPdfFailureAlert (WorkOrder PDF)

src/services/transactionError/transactionError.service.ts#create is the single funnel for Fulfil alerts:

1. Writes a TransactionError row (prisma/schema.prisma model TransactionError, table transaction_errors).
2. Derives errorCode (params.errorCode → error.statusCode → error.status → 500), errorMessage/errorStack (stack truncated to ~10KB / ≥5 frames by truncateStack), and sentryUrl from sentryEventId.
3. Fire-and-forget postFulfilFailureAlert(...) — but only when transactionType === FULFIL_DONE || transactionType === FULFIL_ACKNOWLEDGE. Any Slack failure is caught and logged, never propagated.

Consequence: a new TransactionType does not get Slack alerting for free — the if in create is an explicit allowlist and must be extended. (This contradicts the older index claim that new acknowledge failure modes "inherit Slack alerting for free"; that only holds for new call sites reusing an already-allowlisted transaction type.)

Call sites today, both in src/utils/sentryUtils.ts:

• captureFulfilShipmentAcknowledge → TRANSACTION_TYPES.FULFIL_ACKNOWLEDGE, entityRecName: poRecName, entityId: poRecord.poNumber, fires on responseType 'partial' | 'error' ('success' only logs).
• captureFulfilShipmentDone → TRANSACTION_TYPES.FULFIL_DONE, entityRecName: shipmentRecName, entityId: shipmentRecord.shipmentNumber.

Both pass requestId: 'mes_call_fulfil' and createdBy: email (returned by captureAndLog → captureMessage, which reads the AsyncLocalStorage adminUser.email).

A second Slack trigger exists since INFRA-655 — postWorkOrderPdfFailureAlert (slackClient.ts:278) is not funneled through TransactionErrorService.create. It is called directly from the WorkOrder PDF generation flow (updateJobFailedAndLog in workOrderPdfGeneration/utils.ts:32, and _pollUntilAllDone in runJobs.ts:281) when a run ends with no zip produced. It reuses the same _postMessage/_makeAlertBlocks/isEnabled() helpers and posts to the same SLACK_CHANNEL_ID. See work-order-pdf-export.md for the exact alert condition (failedStep !== WaitingPrereqs && !generatedAt) and the two disjoint fire sites.

TransactionType is a Prisma enum — adding a value needs a migration

TRANSACTION_TYPES (src/constants/transactionTypes.ts) is a plain as const object, but TransactionErrorCreateParams.transactionType and the Prisma column are typed as the generated Prisma enum TransactionType (prisma/schema.prisma:41): PO_CREATE, SHIPMENT_WORKORDER_CREATE, FULFIL_ACKNOWLEDGE, FULFIL_DONE. Adding a value therefore touches both — the TS constant and a hand-authored migration. Precedent: prisma/migrations/20260622120000_add_fulfil_done_transaction_type/migration.sql, whose whole body is:

-- AlterEnum
ALTER TYPE "public"."TransactionType" ADD VALUE 'FULFIL_DONE';

Keep the new value in its own migration file — Postgres forbids using an enum value added in the same transaction.

Downstream surfaces that also key off the enum and need review when a value is added:

• src/clients/slackClient.ts#postFulfilFailureAlert — the header text is a binary isDone ? 'Done' : 'Acknowledge', so any third type silently renders as "… Fulfil Acknowledge Failed". Add a branch, don't rely on the fallback.
• src/services/transactionError/transactionError.service.ts — the Slack allowlist if (above).
• scripts/backfill-transaction-error-factory.ts#resolveFactoryId — maps type → which table resolves factoryId (PO-ish types → PurchaseOrder.poRecName; FULFIL_DONE → CustomerShipment.shipmentRecName; unknown → null).
• src/routers/admin/resources/transactionError/ — the AdminJS list/filter reads the enum from Prisma, so the new value appears in the filter dropdown automatically; listActionHandler.ts maps transactionType to an exact match.

⚠️ entityRecName is @db.VarChar(25) and required. Whatever identifier a new capture path supplies must fit in 25 chars — a URL or endpoint path will not, and overflow surfaces as a Prisma write error inside a fire-and-forget path.

Acknowledge chunk outcomes: responseType: 'error' does not mean "these shipments failed"

_makeShipmentsAcknowledgeByChunk (src/services/shipment/shipment.service.ts) derives the chunk's responseType purely from the success count:

const responseType = chunkSuccess.length === chunk.length ? 'success' : chunkSuccess.length > 0 ? 'partial' : 'error';

So 'error' (zero successes) is reached by two different paths, and they populate different arrays:

Path	failedFulfilShipments	unknownStateShipments	Per-shipment error text
Fulfil responded, every shipment carried errors[]	the whole chunk	[]	yes — FulfilShipmentAcknowledgeShipmentRes.errors
The HTTP call threw (timeout, bad header, 5xx)	[] — empty	the whole chunk	no — only the batch-level err

⚠️ The trap: on an HTTP throw, chunkFailed is never populated (the .forEach over fulfilRes.data.shipments never runs), so failedFulfilShipments is empty even though responseType === 'error'. Any consumer that wants "which shipments were affected" must read both arrays — failedFulfilShipments alone silently yields nothing for the entire HTTP-failure class, which is the most common production alert.

The distinction is semantic, not cosmetic: unknownStateShipments is Scenario B — MES cannot know whether Fulfil acknowledged them, so their labelStatus deliberately stays Pending and they need re-checking rather than blind retry. failedFulfilShipments are confirmed-rejected by Fulfil.

Note also that an "already acknowledged" error is treated as success, not failure (item.errors.some((str) => str?.indexOf('already acknowledge') < 0)), so it never reaches either list.

Slack chat.postMessage returns HTTP 200 on logical failure

Slack signals errors like invalid_blocks / channel_not_found in the body ({ ok: false, error: '...' }) with a 200 status, so axios does not throw and makeRetryClient never retries. _postMessage currently returns the raw { enable, res, err } without inspecting res.data.ok — any caller that depends on the post having actually landed (e.g. reading res.data.ts to thread replies under it) must check res.data.ok itself.

_postMessage accepts thread_ts (was dead until INFRA-647)

_postMessage(msgObj, thread_ts?) has always forwarded thread_ts into the chat.postMessage body, but no caller passed it before INFRA-647 — threading was half-plumbed. The parent message's ts (from res.data.ts) is the thread root; replies pass it as thread_ts.

Sentry tag vocabulary (src/constants/sentryTags.ts)

Captures are tagged with module (SENTRY_MODULE), processName (SENTRY_PROCESS_NAME), and eventType (one of the per-domain SENTRY_*_EVENT_TYPE maps). Fulfil's event types live in SENTRY_FULFIL_EVENT_TYPE: acknowledge, done, factoryResolutionFailed, rtsShipmentPackFailed, retrieveTrackingNumberFailed. A new Fulfil failure mode should add a processName and an eventType here rather than reusing done/acknowledge, since Sentry alert rules filter on these tags.

captureAndLog(message, { tags, extra }, level, expandExtra) is the shared private helper: it calls Sentry.captureMessage (attaching the session user's email) and logPinoSentry[level], and returns { sentryEventId, email } for the TransactionErrorService.create call that usually follows.