Skip to main content

Fulfil Error Alerting (TransactionError → Sentry → Slack)

How a failed Fulfil call becomes a persisted TransactionError row, a Sentry event, and a Slack message.

Deep-dive doc split out of .claude/rules/architecture.md (which is now an index). Append new findings about this area here, not to the index.

There is code-level Slack alerting (src/clients/slackClient.ts)​

⚠️ Older revisions of the architecture index claimed "There is no code-level Slack integration anywhere in the codebase" and that Fulfil Slack alerts come only from Sentry alert rules. That is stale. Since INFRA-520 the repo posts to Slack directly via chat.postMessage:

  • src/clients/slackClient.ts — an axios client (makeRetryClient) with baseURL: env.SLACK_FULFIL_ALERT_WEBHOOK_URL (in practice https://slack.com/api/chat.postMessage), Authorization: Bearer ${env.SLACK_BOT_TOKEN}, and channel: env.SLACK_CHANNEL_ID injected by _postMessage.
  • isEnabled() requires all three of SLACK_FULFIL_ALERT_WEBHOOK_URL, SLACK_CHANNEL_ID, SLACK_BOT_TOKEN; otherwise every notifier is a silent no-op returning { enable: false }. This is what keeps local/CI runs quiet — not a NODE_ENV check.
  • _chunkLines() packs detail lines into ≤2900-char section blocks because Slack rejects the whole message (invalid_blocks) if any single section text exceeds 3000 chars.

Sentry alert rules may also be configured in the Sentry project, but the primary Fulfil Slack path is this client.

The only Slack trigger is TransactionErrorService.create​

src/services/transactionError/transactionError.service.ts#create is the single funnel:

  1. Writes a TransactionError row (prisma/schema.prisma model TransactionError, table transaction_errors).
  2. Derives errorCode (params.errorCode → error.statusCode → error.status → 500), errorMessage/errorStack (stack truncated to ~10KB / ≥5 frames by truncateStack), and sentryUrl from sentryEventId.
  3. Fire-and-forget postFulfilFailureAlert(...) — but only when transactionType === FULFIL_DONE || transactionType === FULFIL_ACKNOWLEDGE. Any Slack failure is caught and logged, never propagated.

Consequence: a new TransactionType does not get Slack alerting for free — the if in create is an explicit allowlist and must be extended. (This contradicts the older index claim that new acknowledge failure modes "inherit Slack alerting for free"; that only holds for new call sites reusing an already-allowlisted transaction type.)

Call sites today, both in src/utils/sentryUtils.ts:

  • captureFulfilShipmentAcknowledge → TRANSACTION_TYPES.FULFIL_ACKNOWLEDGE, entityRecName: poRecName, entityId: poRecord.poNumber, fires on responseType 'partial' | 'error' ('success' only logs).
  • captureFulfilShipmentDone → TRANSACTION_TYPES.FULFIL_DONE, entityRecName: shipmentRecName, entityId: shipmentRecord.shipmentNumber.

Both pass requestId: 'mes_call_fulfil' and createdBy: email (returned by captureAndLog → captureMessage, which reads the AsyncLocalStorage adminUser.email).

TransactionType is a Prisma enum — adding a value needs a migration​

TRANSACTION_TYPES (src/constants/transactionTypes.ts) is a plain as const object, but TransactionErrorCreateParams.transactionType and the Prisma column are typed as the generated Prisma enum TransactionType (prisma/schema.prisma:41): PO_CREATE, SHIPMENT_WORKORDER_CREATE, FULFIL_ACKNOWLEDGE, FULFIL_DONE. Adding a value therefore touches both — the TS constant and a hand-authored migration. Precedent: prisma/migrations/20260622120000_add_fulfil_done_transaction_type/migration.sql, whose whole body is:

-- AlterEnum
ALTER TYPE "public"."TransactionType" ADD VALUE 'FULFIL_DONE';

Keep the new value in its own migration file — Postgres forbids using an enum value added in the same transaction.

Downstream surfaces that also key off the enum and need review when a value is added:

  • src/clients/slackClient.ts#postFulfilFailureAlert — the header text is a binary isDone ? 'Done' : 'Acknowledge', so any third type silently renders as "… Fulfil Acknowledge Failed". Add a branch, don't rely on the fallback.
  • src/services/transactionError/transactionError.service.ts — the Slack allowlist if (above).
  • scripts/backfill-transaction-error-factory.ts#resolveFactoryId — maps type → which table resolves factoryId (PO-ish types → PurchaseOrder.poRecName; FULFIL_DONE → CustomerShipment.shipmentRecName; unknown → null).
  • src/routers/admin/resources/transactionError/ — the AdminJS list/filter reads the enum from Prisma, so the new value appears in the filter dropdown automatically; listActionHandler.ts maps transactionType to an exact match.

⚠️ entityRecName is @db.VarChar(25) and required. Whatever identifier a new capture path supplies must fit in 25 chars — a URL or endpoint path will not, and overflow surfaces as a Prisma write error inside a fire-and-forget path.

Acknowledge chunk outcomes: responseType: 'error' does not mean "these shipments failed"​

_makeShipmentsAcknowledgeByChunk (src/services/shipment/shipment.service.ts) derives the chunk's responseType purely from the success count:

const responseType = chunkSuccess.length === chunk.length ? 'success' : chunkSuccess.length > 0 ? 'partial' : 'error';

So 'error' (zero successes) is reached by two different paths, and they populate different arrays:

PathfailedFulfilShipmentsunknownStateShipmentsPer-shipment error text
Fulfil responded, every shipment carried errors[]the whole chunk[]yes — FulfilShipmentAcknowledgeShipmentRes.errors
The HTTP call threw (timeout, bad header, 5xx)[] — emptythe whole chunkno — only the batch-level err

⚠️ The trap: on an HTTP throw, chunkFailed is never populated (the .forEach over fulfilRes.data.shipments never runs), so failedFulfilShipments is empty even though responseType === 'error'. Any consumer that wants "which shipments were affected" must read both arrays — failedFulfilShipments alone silently yields nothing for the entire HTTP-failure class, which is the most common production alert.

The distinction is semantic, not cosmetic: unknownStateShipments is Scenario B — MES cannot know whether Fulfil acknowledged them, so their labelStatus deliberately stays Pending and they need re-checking rather than blind retry. failedFulfilShipments are confirmed-rejected by Fulfil.

Note also that an "already acknowledged" error is treated as success, not failure (item.errors.some((str) => str?.indexOf('already acknowledge') < 0)), so it never reaches either list.

Slack chat.postMessage returns HTTP 200 on logical failure​

Slack signals errors like invalid_blocks / channel_not_found in the body ({ ok: false, error: '...' }) with a 200 status, so axios does not throw and makeRetryClient never retries. _postMessage currently returns the raw { enable, res, err } without inspecting res.data.ok — any caller that depends on the post having actually landed (e.g. reading res.data.ts to thread replies under it) must check res.data.ok itself.

_postMessage accepts thread_ts (was dead until INFRA-647)​

_postMessage(msgObj, thread_ts?) has always forwarded thread_ts into the chat.postMessage body, but no caller passed it before INFRA-647 — threading was half-plumbed. The parent message's ts (from res.data.ts) is the thread root; replies pass it as thread_ts.

Sentry tag vocabulary (src/constants/sentryTags.ts)​

Captures are tagged with module (SENTRY_MODULE), processName (SENTRY_PROCESS_NAME), and eventType (one of the per-domain SENTRY_*_EVENT_TYPE maps). Fulfil's event types live in SENTRY_FULFIL_EVENT_TYPE: acknowledge, done, factoryResolutionFailed, rtsShipmentPackFailed, retrieveTrackingNumberFailed. A new Fulfil failure mode should add a processName and an eventType here rather than reusing done/acknowledge, since Sentry alert rules filter on these tags.

captureAndLog(message, { tags, extra }, level, expandExtra) is the shared private helper: it calls Sentry.captureMessage (attaching the session user's email) and logPinoSentry[level], and returns { sentryEventId, email } for the TransactionErrorService.create call that usually follows.