Work Order PDF batch export — design

Status: design agreed, validated by spike. Hosting revised by INFRA-621 — the renderer runs as a Cloud Run job, not a DigitalOcean worker. Implementation tracked in INFRA-617 (generation) and INFRA-618 (delivery + automation). Spike findings and measurements: infra-616-spike-findings.md Hosting decision, cost comparison and Terraform: infra-621-cloud-run-hosting-decision.md

Aolong and Milly both reported that downloading a PO's work order templates is unusably slow. This document describes why, and the design that replaces it: PDFs generated on the server, started automatically when the PO is created, delivered by email and from the existing Download All page.

---

1 · The problem

The whole change in five steps, before any of the detail. The rendering work itself is identical — same template, same 500-page files; what changes is who runs it and when:

Steps 1–4 of the new flow need nobody. The download stays a click in MES because that click is what moves the work orders Open → Printed, and Done is only reachable from Printed (§3). What MES waits on — two blocking gates plus a settle floor — is detailed in §3 · When generation fires; image coverage is checked during render prep and never blocks.

Rendering happens entirely in the operator's browser. WorkOrderDownloadAll opens one tab per 500 work orders (BATCH_PROCESS_THRESHOLD = 500); each tab mounts 500 React templates, polls every 50 ms until they settle, waits on every image, then calls window.print() — which stops on a print dialog a human has to answer. Progress is tracked in localStorage and polled by the parent tab every 2 s.

Consequences: nothing can be left unattended, progress is lost if the browser restarts, there is no server-side record of what was exported, and the operator's machine does all the work.

The two factories are slow for opposite reasons, which is why one fix has to serve both:

	Milly	Aolong
Cadence	Weekly (Thursday)	Daily
Work orders per PO	up to 9,000 (PO23182 = 7,135)	706–855 measured
Distinct styles per PO	—	50–56 measured
Template content	Header + barcode; measurements on Scarf	Header + measurements + construction notes + 2 images
Tabs today	15–18	2
Real bottleneck	Sheer count — a dialog per tab to babysit	Page weight — ~1,500 image elements decoded at print time

Milly's page content is thin because _resolveEnablementFlags turns both measurements and construction notes off for Milly + non-Scarf. Aolong falls through to the defaults and gets everything, including the 2-image materials table.

---

2 · The design

Shipment creation does one cheap thing: it writes a job row. A Cloud Run job renders the same work order template server-side through headless Chromium, uploads 500-page PDFs plus a zip to GCS, and MES emails the factory a link.

MES stays the brain throughout — it checks the gates, snapshots what each render covers, records progress and writes work order status. The renderer is deliberately dumb: it reads a payload object, writes a PDF back, and holds no database connection, no config and no credentials of its own.

Note what the renderer is not connected to: there is no edge between it and Postgres. Every arrow it touches is the bucket.

Running it outside MES means Chromium can never contend with the web service that serves AdminJS and n8n's PO creation — that container already has an OOM history (INFRA-599) — and its 821 MB Chromium layer never enters the MES deploy pipeline or DOCR, whose garbage collection is already awkward (INFRA-503).

Cloud Run rather than an always-on worker because the work is tiny and bursty: ~7 minutes of rendering a week. A dedicated 4 GB instance would idle over 99.9% of the time at ~$50/month, while the same work sits inside Cloud Run's Jobs free tier at $0 (~1.5% of the monthly allowance). The renderer needing no database is what makes it possible — no VPC connector, Cloud NAT, static IP or trusted-sources change.

One pipeline, two ways to collect the output

This is not a server-side renderer plus a separate email feature. Generation happens once, on its own schedule, and the factory picks the files up whichever way suits them:

• Push — email. A deep link that opens the PO's export page and starts the zip. One per PO: Aolong daily, Milly on Thursdays.
• Pull — the Download All page. Every 500-page PDF listed individually, plus progress, staleness and Regenerate.

The UI path is quick because the files already exist, not because the renderer got faster. Clicking a file is a plain download: no print dialog, no render wait, no working through tabs in order. That is why the automatic trigger and the server-side renderer must ship together — build only the renderer and trigger it on click, and an operator still waits out 9,000 pages, just with a progress bar instead of eighteen dialogs.

Same page, different list

There is no new screen. Work Order → Download All keeps its place in the nav, its toolbar button, its PO picker and its one-row-per-500 layout. What changes inside it:

• Every row is enabled at once. The Pending/Locked sequencing existed only to stop an operator opening fifteen rendering tabs simultaneously.
• A zip row above the per-file rows — the same object the email's link fetches.
• The progress bar changes meaning — from "batches the operator has downloaded" (localStorage) to generation progress while a job renders. Once complete it becomes a generated-at line, a staleness indicator and a Regenerate button.
• The emailed link lands here with the PO pre-selected, reusing the existing pre-selection behaviour (INFRA-195).

One page disappears, though nobody at the factory has ever seen it: BatchDownloadWorkorder, the invisible tab that opened itself, mounted 500 templates, polled for layout and called window.print(). Single-work-order printing (listWorkorder?mode=single) is untouched.

---

3 · When generation fires

Creation time is the wrong moment to render — the data the template needs is still landing. So the job parks in waiting_prereqs, and MES promotes it only when the prerequisites hold.

Two of those prerequisites are state predicates that block. A third is a clock. A fourth thing — image coverage — reads like a gate but never blocks, so it is not one; it belongs to render prep. Keeping those three kinds apart is what makes the promotion logic small.

The job is finished when the email goes out. There is no state after Notified — how old the files are is a property of the artifacts, which outlive the job, and the export page reports it (§4).

Note that Regenerate does not re-enter this machine. It enqueues a new job with its own id and object prefix (§4), entering at Queued since the settle floor is skipped. Only Partial and Blocked route back into the same job — retrying the missing files, or proceeding once the gates finally pass.

The promotion condition

status = 'waiting_prereqs'
AND eligibleAt <= now()              -- settle floor
AND every work order has a barcode   -- gate 1
AND every creation chunk succeeded   -- gate 2
→ status = 'queued'

Gate 1 · Barcodes exist. updateBarcodesWhenInit runs fire-and-forget after the create response returns, and the template reads the stored workOrderBarcode column. Rendering before it finishes produces work orders with no scannable barcode — useless on the floor. Settles in well under a minute even for 9,000 work orders.

Gate 2 · Every creation chunk succeeded. Chunked shipment creation can partially fail or fail-fast-skip (INFRA-599/601/602), and a fully failed batch is reverted for retry — so a 200 response does not mean the work order set is complete.

Both are genuine correctness gates: the data is not there yet, and a file rendered now would be wrong.

The settle floor is a clock, not a predicate

Nothing becomes true here, so there is nothing to react to — which is why it is not numbered as a gate. It exists so a correction made in the first few minutes does not produce an email pointing at files that are already superseded. The file itself is covered — the export page flags files that have fallen behind the data (§4) — so what the floor protects is what the factory was told.

Because it is per-factory, the delay is not evaluated in the sweep. eligibleAt is computed once at enqueue — where the PO, and therefore the factory, is already in scope — and persisted:

const factory = await FactoryService.findUnique({ where: { factoryId: poRecord.factoryId } });
const delayMinutes = await FactoryConfigurationService
  .getPdfExportSettleDelayMinutes(factory?.factoryCode ?? null);   // default 10

eligibleAt: new Date(Date.now() + delayMinutes * 60_000)

The sweep is then factory-agnostic and indexable — @@index([status, eligibleAt]) — with no config read per tick. Three things not to get wrong:

• 0 must be a legal value. Unlike getLeadTimeDays, which treats non-positive as invalid and falls back to 14, the guard here is n >= 0. Setting the delay to 0 for a factory is the intended way to disable it once corrections prove rare; a > 0 guard would make that impossible.
• Compare through Prisma, not raw now(). These columns are @db.Timestamp(6) — without time zone. A JS new Date() through Prisma keeps both sides on one conversion path; Postgres now() mixes session-timezone semantics with Prisma-written values.
• Persisting means a config change does not move jobs already waiting. Acceptable at ten minutes, but it is a real behaviour difference from computing on the fly.

A retried creation chunk landing after eligibleAt has passed leaves its newly-added work orders with no settle window. Bumping eligibleAt when a chunk is stored on a job still in waiting_prereqs closes that; small enough to note rather than build.

Image coverage is render prep, not a gate

Its old definition ended "generate anyway", so it never blocked — it was never a gate. It also rested on a premise that is no longer true: images used to arrive only via the standalone POST /product/v1/syncImages/:sku pipeline with no tie to style creation. The Airtable style webhook now syncs images itself (webhooks.service.ts → syncStyleImagesBestEffort), on create and update, so an update doubles as a retry. Images therefore land when the style is created, days or weeks before it appears on a PO — nothing is in flight at PO time.

(For the record, "50–56 styles arrive daily" was a misreading of the production numbers. That is distinct styles referenced per PO, not new styles per day. POs reuse existing styles.)

What remains is narrow: syncStyleImagesBestEffort is void-ed with no internal retry, so a style stays imageless if its sync failed and no later Airtable edit re-fired it and nobody read the captureSyncProductImageFailed alert. So during render prep, non-blocking:

1. One indexed query over the PO's distinct styles → which lack images
2. For each missing, fire syncStyleImagesBestEffort — repairs exactly that case
3. Record the still-missing styles on the job and name them in the email
4. Proceed regardless

The coverage query matters: syncImages always calls Shopify with no short-circuit when images already exist, so blanket re-syncing a PO's 50-odd styles would be 50-odd GraphQL calls a day for nothing. Milly skips 1–3 trivially — no images in its template.

What evaluates all this

Worth stating plainly because the answer is "nothing yet": this repo has no recurring-work mechanism. No cron dependency, and the only server-side setTimeout calls are the shutdown force-exit and the retry client's backoff. monitorPendingShipmentsAfterCreate is not a precedent — it runs immediately and awaited inside runPostCreateFlow.

A sweep is required regardless of the settle floor, because gates 1 and 2 also become true asynchronously with nothing to react to. Preferred shape: n8n polls a apiKeyAuth-protected MES endpoint every few minutes — n8n already drives PO creation, so it is a proven scheduler here, and a single caller means no duplicate-promotion problem. Cloud Scheduler hitting the same endpoint is a fine alternative; it is already enabled in the project.

Note this does not contradict §5's "no Cloud Scheduler" decision, which is about invoking the renderer — a scheduled renderer would have to read the database to discover work, which is the coupling this design removes. A scheduler that pings MES is unrelated; MES already owns the database.

Promote with a conditional update rather than read-then-write, following the pattern INFRA-599 used for Created → Open, so a concurrent sweep or an n8n retry throws P2025 instead of double-promoting:

update({ where: { jobId, status: 'waiting_prereqs' }, data: { status: 'queued' } })

Work is addressed by task index, not claimed

There is no SELECT ... FOR UPDATE SKIP LOCKED claim loop, and no worker-liveness question — Cloud Run assigns work by index instead. MES writes one chunk-NNN/payload.json per file plus an index.json mapping task index → exportChunkId, then calls jobs:run once with taskCount = N. Each task reads CLOUD_RUN_TASK_INDEX and resolves its own slice.

The indirection is not decoration. RunJob overrides apply per execution, not per task — containerOverrides[] is keyed by container name, so there is no way to hand task 3 a different argument than task 7. And task indices are always contiguous 0..N-1, so retrying 3 files out of 15 cannot reuse the original indices; MES writes a fresh index.json covering only the gaps. Without it, partial retry would re-render everything.

Recovery follows from that: every finished task leaves a result.json beside its PDF, so MES can see exactly which files exist and start a fresh render for the rest. A crash costs the unfinished files, never the whole PO. Note the renderer never writes to the database — MES polls executions.get plus those result objects and updates the ExportChunk rows itself.

Two consequences worth carrying into INFRA-617. Files render concurrently, so nothing should assume ordering — Google documents that with parallelism > 1, task 2 may start before task 1. And CLOUD_RUN_TASK_ATTEMPT exists because Cloud Run retries tasks, so each render must be idempotent to a deterministic object key rather than appending.

Work order status: Open → Printed

Unchanged from today: it happens on the Download click, whole PO, via the existing markAllPrintedByPoNumber. updateManyStatus already permits only Open → Printed and stamps printedAt, so repeat clicks and regenerations are idempotent for free.

This is the last place to get inventive, because Done is only reachable from Printed — a work order that never gets marked fails the factory's completion scan with "status transition not allowed". Two rejected alternatives:

• Marking at enqueue or on render would flip every work order in every PO to Printed minutes after creation, since nobody asked for it. That breaks the status filter and the aging report, and erases the "factory hasn't picked this up yet" signal.
• Marking on observed download cannot be relied on, because a raw signed URL is fetched directly from GCS and MES never sees it.

Which is why the email deep-links to the export page, not to the GCS object. A factory downloading straight from a signed URL clicks nothing in MES, so those work orders would sit at Open forever.

The only change to marking anywhere: generation passes markPrinted = false (makeBatchInfosByPoId defaults it to true).

---

4 · What gets stored

work-order-pdf/<factoryCode>/<poRecName>/<jobId>/
    index.json               task index -> exportChunkId, written per execution
    BG_PO23224.zip           what the email's link fetches
    chunk-000/
        payload.json         the WorkOrderData this task renders
        BG_PO23224_001.pdf   500 pages
        result.json          pageCount · sizeBytes · sha256
    chunk-001/
        payload.json
        BG_PO23224_002.pdf   355 pages
        result.json

The payload.json / index.json / result.json objects are the entire interface between MES and the renderer — the reason the renderer needs no database, and the reason a retry can target individual files.

File naming and the 500-page unit match what the print shop already batches by. Each 500-work-order file is rendered as a single Chromium document — image and font dedupe is per-document, so splitting into smaller render units inflates output ~26% and would force a merge step.

Measured sizing

Measured in the INFRA-616 spike at production row densities, not estimated:

	Milly	Aolong
Per work order	8.4 KB	39.8 KB
Files per PO	15	2
Per 500-page file	4.1 MB	19.4 MB
Zip per PO	~16 MB	~7 MB
Stored per PO (zip + loose PDFs)	~76 MB	~41 MB
Render time per PO, sequential	~1.5 min	~40 s
Jobs	1 / week	1 / day

⚠️ The old table said ~6 min for Milly. That was wrong — it applied Aolong's 24.0 s/500 rate to Milly's file count. Milly's shape is header + barcode only (8.4 KB per work order against Aolong's 39.8 KB) and measures 6.1 s per 500, so its 15 files are ~92 s. The figures above are worst case, one file after another; with parallelism = 4 each PO finishes in under 30 s. Derive per-PO timings from the per-factory rates, never from one shared number.

Storage settles near 1.5 GB at a 30-day lifecycle — a few cents a month. A single 4 GiB Cloud Run job covers both factories, renders one PO's files in parallel, and scales to zero in between; compute falls inside the free tier, so the renderer's own running cost is nil.

The 4 GiB figure comes from the measured 2,391 MB peak at 500 work orders. It is paired with 2 vCPU not for speed but for headroom — Cloud Run allows up to 4 GiB at 1 vCPU and up to 8 GiB at 2, so the page-doubling case below is answered by a one-line job_memory = "8Gi" rather than a re-sizing exercise. On Cloud Run that costs double a number that rounds to zero; on an always-on instance it would be a plan upgrade.

⚠️ Page counts can double. An Aolong work order renders 1107px against a 1123px A4 page — 16px of headroom. A style at the observed measurement maximum already spills to two pages, and adding ~4 rows to the global Material library pushes every Aolong work order to two pages, because every library row renders on every work order. Budget up to 2× on storage and memory, and never derive behaviour from page counts.

Expiry

	Email link	Files in GCS	What the factory does
Day 0–7	live	live	Click the link in the email
Day 7–30	expired	live	Export page re-signs a fresh link
Day 30+	expired	deleted	Export page offers Regenerate

Links expire at 7 days (the V4 signed-URL maximum); objects at 30. The job rows have no lifecycle of their own, so expiresAt is stamped on the job from the same config value as the bucket rule, and the page renders "expired — Regenerate" rather than signing URLs to deleted objects. Lifecycle deletion is asynchronous and can lag a day, so the download path also handles a missing object mid-click.

Regenerate

Enqueues the same job the automatic trigger would — gates re-checked, new files, a fresh 30-day clock. Four rules:

• No email by default. Whoever clicked is already on the page. An opt-in checkbox covers not wanting to wait out a large PO's render.
• A new job, not a reused one — its own jobId and object prefix, so paths stay immutable and every export of a PO is on the record. Old prefixes expire on their own schedule.
• One active job per PO. A second click shows progress rather than starting a second render.
• Only Open work orders get marked printed, so audit columns aren't churned while anything added since the first export is still caught.

Regeneration always works — the database is the source of truth, not the PDF — but it is not a time machine. Measurements may have been revised and work orders added or cancelled since, so a regenerated file can differ from what was originally printed.

Staleness is a label on the files, not a job state

Files can fall behind the data, and the export page says so. This is deliberately not a lifecycle state — the job's work ends when the email goes out, and how current the files are is a property of the artifacts, which outlive it.

Worth stating why it exists at all, since the happy path looks finished at Notified. Today's flow cannot go stale: the browser renders at click time, so the factory always prints current data. Pre-generating introduces the possibility of printing a measurement that was corrected two days ago — a new failure mode, not an inherited one, in a workflow where the output drives cutting fabric.

The check is a timestamp comparison, not a content hash — max(updatedAt) > generatedAt across the contributing tables, evaluated when the page loads. That is coarser than hashing: touching an unrelated column yields a false "may be out of date". That direction is free, because it only offers an unnecessary Regenerate. A false negative is the one that cuts fabric to the wrong measurement.

⚠️ The input most likely to be missed is the shared Measurement and Material libraries. Every library row renders on every Aolong work order, so a single row added centrally invalidates every generated file for every open PO — without touching one WorkOrder record. A staleness check that walks only WorkOrder and its relations will not see it. Same fact as the page-count-doubling warning above.

The page already needs a Regenerate control for expiry, so staleness reuses it: the marginal cost is a query, not a feature. And because a timestamp comparison replaces hashing, the job model needs no fingerprint column — worth telling INFRA-617 while that migration is still being written.

---

5 · Decisions

Decision	Rationale
GCS for storage	Chosen over DO Spaces. Needs a new service account — the Gmail integration uses a user-delegated OAuth refresh token, so there is nothing to reuse
Signed URLs are reachable from both factories	Fulfil already delivers shipping labels to these same users via V4 signed URLs on storage.googleapis.com, so the mechanism is proven in production with them
7-day link expiry	The V4 maximum. A 10-day ask was walked back to avoid needing a MES token route; if longer is ever wanted, a token route that 302s to a short signed URL is a drop-in addition
30-day retention	Comfortably clear of the 7-day link life, so a live link can never point at a deleted object
500-page files	Matches what the print shop already batches by, and the existing BG_<poRecName>_NNN naming
Email links to the export page	Preserves click-driven Printed, which Done depends on
One email per PO	Aolong daily, Milly weekly. Recipients per factory via FactoryConfiguration, body in Chinese
Render unit = delivery unit = 500	Dedupe is per-document; smaller units inflate output ~26% and force a merge. Costs 4 GiB per render
Separate renderer image	Chromium adds 821 MB; the web service has no use for it, and on Cloud Run it never enters the MES deploy pipeline or DOCR (INFRA-503)
Cloud Run job, not a DO worker (INFRA-621)	~7 min of work a week. An always-on 4 GB instance is ~$50/month at >99.9% idle; Cloud Run scales to zero and falls inside the free tier. Decisive enabler: the renderer needs no database, so the networking objection never arises
Renderer holds no DB access	Verified under env -i in the spike — correct PDFs with nothing configured. Keeps MES the single writer of status and removes any cross-cloud private networking
Work addressed by task index	RunJob overrides are per execution, not per task, so per-file work must derive from CLOUD_RUN_TASK_INDEX via an index.json. Also what makes partial retry possible
Zip built by a second --mode zip run	With files rendered in parallel no single task holds them all. MES zipping on DO would push ~20 MB × N through the OOM-prone web service
asia-east2 (Hong Kong) for bucket, job and Artifact Registry	Closest region to the factories, who download directly. Costs the same as a US region — asia-east2 is Cloud Run Tier 2, but at ~1.5% of the free tier that cannot reach the bill — so the choice is purely download speed. AR must share the job's region: same-location pulls are free and the ~1 GB image is pulled cold every execution
jobs:run on demand, no Cloud Scheduler	A scheduled job would have to discover work, i.e. read the database, reintroducing the coupling this design removes. MES already knows the moment a job becomes eligible
Pilot Aolong first	A daily PO surfaces problems the next day; Milly gives one attempt per week and a bad Thursday costs seven days

Open

• Does anyone internal get copied on the factory email, or is it factory-only?
• Is there an archive requirement beyond 30 days — does anyone ever need the exact sheet that was on the floor months later (QA dispute, customer claim)? If so that is an archive concern, not a retention setting, because regeneration cannot guarantee fidelity.
• Do factory users stay signed in to MES? The emailed deep link needs their session, or it needs to carry a token. Either works; it decides whether the link is one click or two.
• Is asia-east2 acceptable as a data location, and does any org policy pin cloud resources to US regions? Owned by whoever owns the GCP org, not by this design — and cheaply reversible either way, since the objects are a 30-day regenerable cache. docs/infra-621-cloud-run-hosting-decision.md § 8 has a runnable test for timing a factory download against both regions before committing.
• Does an org-level GitHub Workload Identity Federation pool already exist to reuse for pushing the renderer image, rather than creating a second one?

(The former open question here — how to build the worker image — was settled by INFRA-621: the renderer is a Cloud Run job, its image is built and pushed by its own keyless GitHub Actions workflow, and the MES release workflow is untouched.)