INFRA-621 — Hosting the Work Order PDF renderer: Cloud Run vs a DigitalOcean worker
Decision spike for INFRA-621. Companion to
infra-616-spike-findings.md (the measurements this reasons
over) and work-order-pdf-export-design.md.
Verdict: run the renderer as a Cloud Run job. Retire the DigitalOcean worker component from INFRA-617.
The Terraform this implies is sketched in ../terraform/ — terraform validate passes against hashicorp/google ~> 6.0, but nothing has been applied.
1 · Why
Four things decide it, in descending order of weight.
The renderer needs no database, and that was the only real blocker. Verified in
INFRA-616, not assumed: scripts/spike-render-wo-pdf.mts --fixture produces correct PDFs
under env -i with nothing set at all. MES stays the brain — gates, chunk bookkeeping,
loading WorkOrderData, status writes — and the job is a pure JSON→PDF function. So
"Cloud Run reaching DigitalOcean Postgres over an allowlistable IP" never arises, and with
it goes the VPC connector, Cloud NAT, static IP and trusted-sources work.
The duty cycle makes an always-on instance absurd. Measured end-to-end the workload is ~7 minutes of compute per week. An always-on 4 GB instance is idle 99.93% of the time.
The 821 MB Chromium layer never touches the DigitalOcean pipeline. The renderer image is
~1.05 GB against the web service's ~225 MB. On DigitalOcean that rides along on the deploy
pipeline and into a registry whose garbage collection is already awkward — manual GC returns
412 while automated GC is enabled (INFRA-503). On Cloud Run it lives in Artifact Registry,
entirely outside the MES release path, and deploy-staging.yml is untouched.
The pattern already runs here. inventory-integration-shipmonk-sync is a Cloud Run job
in production in the same projects, and its terraform/main.tf is a complete working
blueprint. This is "copy the neighbour", not a new platform.
What it costs us
Honest counterweights, none of them decisive:
- A second cloud in the request path, so a MES failure mode now includes "GCP is unhappy".
- A GCP service account key on the DigitalOcean side. Unavoidable — see §7.
- A new Terraform stack and two state buckets to bootstrap and own (§9).
- The chunk queue changes shape. The
FOR UPDATE SKIP LOCKEDclaim loop INFRA-617 specifies is no longer the mechanism, and the zip step moves. This is the biggest concrete consequence — §12.
2 · Cost
Rates fetched from the live pricing pages on 2026-07-30. The Cloud Run figures are from the Jobs table specifically (the Services instance-based and request-based tables carry different numbers — $0.000024/vCPU-s — and are the easy thing to misread).
| Rate (Tier 1) | |
|---|---|
| Cloud Run Jobs CPU | $0.000018 / vCPU-second |
| Cloud Run Jobs memory | $0.000002 / GiB-second |
| Cloud Run Jobs free tier | 240,000 vCPU-s + 450,000 GiB-s per month |
| DigitalOcean App Platform, 2 vCPU / 4 GiB scalable | $50.00 / month |
| Artifact Registry storage | first 0.5 GiB-month free, then $0.000136986 / GiB-hour (≈ $0.10 / GiB-month) |
| Artifact Registry → Cloud Run pull, same location | $0.00 |
us-central1 and asia-east1 are Tier 1; asia-east2 (the chosen region, §8) is Tier 2.
It makes no difference here — this workload sits at ~1.5% of the free tier either way.
The workload
Derived from the INFRA-616 container measurements at unit=500, lang=zh:
| Per 500 WOs | Peak RSS | Real PO | Chunks | Task-seconds | |
|---|---|---|---|---|---|
| Aolong (measurements + notes + 2 images) | 24.0s | 2,391 MB | ~855 WOs, daily | 2 | ~48s/day |
| Milly (header + barcode) | 6.1s | 1,753 MB | 7,135 WOs, Thursdays | 15 | ~92s/week |
≈ 428 task-seconds/week → ~1,853/month. At 2 vCPU / 4 GiB that is 3,706 vCPU-seconds and 7,412 GiB-seconds a month — about 1.5% of the free tier.
| Monthly | Annual | |
|---|---|---|
| Cloud Run job | $0.00 (inside free tier; $0.08 at list price if the free tier were consumed elsewhere) | ~$1 |
| DigitalOcean 4 GB worker | $50.00 | $600 |
Artifact Registry adds roughly $0.15/month for the retained renderer images. GCS storage (~1.5 GB steady state) and factory egress are identical under both options, so they cancel out of the comparison — as does image-registry storage, which DOCR would also carry.
Even multiplied by 100× the traffic, this stays inside the free tier. The comparison isn't close enough to need refining.
⚠️ Correction: the "~6-minute Milly job" is wrong
INFRA-621's background and INFRA-616's findings §4 both say Milly's PO takes about 6 minutes. That applied Aolong's 24s/500 rate to Milly's chunk count (15 × 24s = 360s). Milly's own measured rate is 6.1s/500 — its template is header + barcode only, 8.4 KB per work order against Aolong's 39.8 KB. Milly's 15 chunks are ~92 seconds sequentially.
So total weekly compute is ~7 minutes, not the ~12–13 the ticket assumed. The error points the same way as the decision.
3 · Payload handoff — via GCS, keyed by task index
MES writes the payload to GCS and passes only a prefix. The invocation body carries a
path, not megabytes: the rendered HTML for 500 Aolong work orders measured 10.4 MB, and the
payload JSON behind it is the same order of magnitude. That is not going in a RunJob
override.
The mechanism matters, because of a Cloud Run constraint worth stating plainly:
RunJoboverrides apply to the whole execution, not per task. Containerargsandenvare set once for all tasks. There is no way to hand task 3 a different argument than task 7.
So per-chunk work must be derived from the one thing that differs per task:
CLOUD_RUN_TASK_INDEX, which Cloud Run injects.
gs://<bucket>/work-order-pdf/<factoryCode>/<poRecName>/<jobId>/
index.json written by MES: { "0": <chunkId>, "1": <chunkId>, ... }
chunk-000/payload.json written by MES: the WorkOrderData[] for that chunk
chunk-000/BG_<po>_001.pdf written by the renderer
chunk-000/result.json written by the renderer: pageCount, sizeBytes, sha256
...
Task N reads index.json, resolves its own chunk, renders, writes back. One jobs:run per
job with taskCount = chunkCount.
index.json is what makes retries clean. Task indices are always 0..taskCount-1
contiguous, so a re-run covering only the 3 failed chunks out of 15 cannot reuse the
original indices. MES writes a fresh index.json for the retry execution mapping 0,1,2
onto the chunk ids that actually need work. Without that indirection, partial retry is
either impossible or re-renders all 15.
4 · Memory — 4 GiB, with a verified path to 8
Measured peak was 2,391 MB rendering 500 Aolong work orders as a single Chromium document. Chromium floors at ~1 GB regardless of unit size, so no unit size makes a small instance viable; and dedupe is per-document, so splitting the unit inflates output ~26%. 500-in-one-document stays the right call — this decision does not revisit it.
4 GiB at 2 vCPU, EXECUTION_ENVIRONMENT_GEN2.
The reason 2 vCPU rather than 1: Cloud Run allows up to 4 GiB at 1 vCPU and up to 8 GiB at
2 vCPU (verified against the memory-limits doc). INFRA-616 found Aolong work orders render
1107px against a 1123px page — a 1.4% margin — and about four extra rows in the global
Material library doubles every Aolong page count, and with it peak memory. 2,391 MB doubled
overruns 4 GiB. Sitting at 2 vCPU makes the response a one-line job_memory = "8Gi".
And this is where scale-to-zero pays off a second time. On Cloud Run, provisioning double the memory costs double a number that rounds to zero. On a $50/month always-on instance it is a plan upgrade. The cheap-insurance option only exists on the Cloud Run side.
Gen2 caveat: container filesystem writes count against the memory limit. The renderer must stream the PDF and zip into the upload rather than staging on disk — which is already an INFRA-617 acceptance criterion, but for a different reason. It is now load-bearing.
5 · Parallelism — task_count = chunks, parallelism = 4
Projected, not measured. Truthfully: measuring real Cloud Run parallelism needs the job deployed, which is INFRA-617's first task, not this spike's. What follows is arithmetic over the measured per-chunk times, and should be re-measured on the first staging run.
| Chunks | Sequential | parallelism = 4 | |
|---|---|---|---|
| Aolong (855 WOs) | 2 | ~41s | ~24s (both at once) |
| Milly (7,135 WOs) | 15 | ~92s | ~25s (4 waves × 6.1s) |
Each task gets its own 4 GiB, so unlike one DigitalOcean worker grinding chunks sequentially
in a single 4 GB box, concurrency here costs nothing in memory contention. parallelism = 4
caps concurrent draw at 8 vCPU / 16 GiB so one PO cannot saturate the project's regional
quota — which matters because these projects are shared with inventory-integration.
The number to actually measure first is cold start, not parallelism. The job scales to zero, so every execution pulls a ~1.05 GB image cold. If that costs 20s, it dominates Milly's entire 25s render and is the only latency worth optimising — and it is the reason Artifact Registry must sit in the same region as the job (§8). Nothing here is latency-critical (generation is triggered automatically after PO creation, and the factory reads a link later), so this is a tuning question, not a risk.
6 · Invocation — MES calls jobs:run, no Cloud Scheduler
POST https://run.googleapis.com/v2/projects/{p}/locations/{l}/jobs/{j}:run, authenticated
as mes_sa, with an overrides block setting taskCount and the job prefix.
Rejected: a Cloud Scheduler tick. A scheduled job would have to discover work, which means reading the database — reintroducing exactly the coupling that makes this decision easy. It also adds up to a full tick of latency for nothing, and MES already knows the precise moment a job becomes eligible (barcodes written, the four gates satisfied).
main.tf therefore omits the sibling stack's google_cloud_scheduler_job deliberately.
Status comes back by polling, not a callback: MES reads executions.get for task
success/failure counts and the per-chunk result.json objects for page counts and sizes.
No inbound webhook, so no public MES endpoint to authenticate and no signature scheme. The
renderer stays a pure function with exactly one output channel — the bucket.
7 · Auth on the DigitalOcean side — a service account key
Workload Identity Federation is not available, and that decides it. WIF works by exchanging a token the workload's own platform issues; DigitalOcean App Platform does not issue an OIDC workload identity token, so there is nothing to federate with. It is not a preference between two options — there is one option.
Mitigations, since the key is unavoidable:
- Both storage grants are bucket-scoped (
google_storage_bucket_iam_member), not project-scoped, and useroles/storage.objectUserrather thanobjectAdmin— neither identity ever manages object ACLs. A leaked key reaches one bucket and cannot rewrite permissions inside it. - Both Cloud Run grants are bound to the single job, not the project.
- No project-level IAM binding exists in the stack at all — see the escalation note below.
Why the bucket role is objectUser and must keep delete
objectUser versus objectAdmin drops object-level ACL/IAM management
(storage.objects.get/setIamPolicy). It does not drop delete — Google's own description
is "create, read, update and delete objects and multipart uploads" — and that matters,
because delete must stay.
Overwriting an object needs more than storage.objects.create: objectCreator holds create
and is documented as unable to overwrite, and the legacy writer roles bundle "create,
overwrite, and delete". Cloud Run retries tasks (CLOUD_RUN_TASK_ATTEMPT) and every render
is idempotent by rewriting the same deterministic key — so a genuinely delete-less role
would fail every retried task. Anyone "tightening" this to objectCreator breaks retries.
objectUser is bucket-level grantable and carries storage.objects.list (the export page
and zip mode both enumerate) plus storage.multipartUploads.* (the zip streams into the
upload), so nothing needed is lost.
The Terraform runner's roles live outside this stack, on purpose
An earlier revision granted the runner its own project roles from inside the stack. That was
wrong twice over. It was self-referential — resourcemanager.projectIamAdmin is the exact
permission needed to manage the block granting it — and iam.serviceAccountAdmin plus
projectIamAdmin is a privilege escalation path: create a service account, then grant it
any role on the project, up to owner.
Granting those roles once during bootstrap instead means no project-level binding remains
in the stack, so the runner never needs projectIamAdmin and the escalation path is
removed rather than mitigated. It also drops the requirement that a human with project-admin
rights run the first apply per environment.
Residual and accepted knowingly: iam.serviceAccountAdmin still carries
iam.serviceAccounts.setIamPolicy, so the runner could grant itself impersonation on a
pre-existing privileged service account. Closing that means dropping to
iam.serviceAccountCreator, which cannot set IAM policy — and therefore means dropping the
mes_sa_token_creator binding, which is what keeps the keyless path a config change rather
than a Terraform change. That trade is deliberate; revisit it if the org tightens.
⚠️ roles/run.invoker is the wrong role here
Worth knowing before writing the invoke code, because the failure is a 403 on every single call rather than something that degrades gracefully:
roles/run.invoker grants run.jobs.run but not run.jobs.runWithOverrides. Those
are separate permissions, and every invocation we make passes an overrides block —
taskCount is the chunk count and the job prefix rides in args, both of which vary per
PO. So the obvious-looking role fails 100% of the time in this design.
The role that carries it is roles/run.jobsExecutorWithOverrides (run.jobs.run +
run.jobs.runWithOverrides + run.executions.cancel — the last being wanted anyway, to
kill a stuck job). run.admin and run.developer also carry it and are far too broad.
One thing to confirm on the first apply: Google documents only
run.{admin,developer,invoker,viewer} as grantable at the Cloud Run job level, and
jobsExecutorWithOverrides is not on that list. main.tf binds it at job level anyway,
since that is the correct intent and the docs' annotations are not necessarily exhaustive.
If the API rejects it, move it to a project-level binding — still far narrower than
run.admin, but it would then cover every job in the project, which is shared with
inventory-integration. Flagged in a comment at the resource.
Polling is separate and does scope cleanly: roles/run.viewer carries exactly
run.executions.get/list + run.tasks.get/list, and is documented as job-level
grantable, so it is bound to the one job.
- Base64 into
GCS_SERVICE_ACCOUNT_KEY_BASE64(DigitalOcean env vars are single-line), and rotate on a schedule. mes_saalso holdsserviceAccountTokenCreatoron itself, sosignBlobworks. With a key present, V4 signing happens locally and the binding is unused — it is there so that going keyless later is a config change, not a Terraform change.
Inside GCP there is no key. Cloud Run attaches job_sa to the job and the client library
picks it up as application default credentials — the same thing the sibling job already does
(service_account = google_service_account.job_sa.email, GCS_CLIENT_EMAIL rather than a
key). Any suggestion in INFRA-617 that the renderer needs a key JSON is wrong.
CI is keyless too, because GitHub Actions does issue OIDC tokens — §10.
8 · Region — everything in asia-east2 (Hong Kong)
Cost does not discriminate between any of the candidates, so this is decided purely on factory download performance. Establishing that first, because it removes the argument most likely to be had about it:
-
Compute is free in every candidate region, with room to spare.
asia-east2is a Tier 2 region — verified — so it is dearer per vCPU-second than Tier 1asia-east1/us-central1. That cannot reach our bill. The pricing page states the free tier is "applied as a spending based discount using Tier 1 pricing", so the monthly allowance is worth240,000 × $0.000018+450,000 × $0.000002= $5.22 of credit. This workload is 3,706 vCPU-seconds + 7,412 GiB-seconds a month = $0.082 at Tier 1 list. Tier 2 would have to be 63× dearer than Tier 1 before a cent was billed. (The exact Tier 2 rate is not quoted here because the pricing page renders it via JavaScript and it could not be verified from a static fetch — but the 63× headroom makes the precise number irrelevant.)Worth knowing when someone proposes "just use US": the US is not uniformly Tier 1. us-central1/us-east1/us-east4/us-east5/us-south1/us-west1 are Tier 1, but us-west2 (LA), us-west3 and us-west4 are Tier 2 — the same bracket as Hong Kong. And conversely, asia-east1 (Taiwan) is the only Tier 1 region near the factories; Hong Kong, Singapore and Seoul are all Tier 2. If tier mattered, Taiwan would be the pick.
-
Storage cannot matter. 1.5 GB steady state. Even at the priciest regional Standard rate the whole bill is under $0.05/month, so any inter-region difference is smaller than that.
-
Egress is the largest line and it is still cents. ~1.5 GB/month of factory downloads, so ~$0.20/month, varying by a few cents across regions.
The entire GCP bill for this feature is under $1/month in any region under discussion.
So it comes down to which leg a human waits on
Four legs cross a network boundary. Only one of them is felt by a person:
| Leg | Volume | Frequency | Latency-sensitive? |
|---|---|---|---|
| MES (DigitalOcean) → GCS: upload payload | ~10–25 MB per job | 1×/day + 1×/week | No — background |
MES → Cloud Run API: jobs:run, then polling | a few KB | per job | No |
| Cloud Run job ↔ GCS: read payload, write PDF + zip | ~20 MB per chunk | per chunk | Must be same region (free, fast) |
| Factory (mainland China) → GCS: download the zip | 7–16 MB | once per PO | Yes — this is the one |
So the bucket location decides, and the job plus Artifact Registry follow it.
asia-east2 (Hong Kong) over asia-east1 (Taiwan) for mainland China peering.
| Location | Why | |
|---|---|---|
| GCS bucket | asia-east2 | The only human-facing leg. Hong Kong generally peers better with mainland China than Taiwan. |
| Cloud Run job | asia-east2 | Follows the bucket so the render↔GCS leg stays free and in-region. Tier 2 pricing, immaterial at this volume. |
| Artifact Registry | asia-east2 | Must match the job. AR→consumer pulls are free only in the same location, and the ~1.05 GB image is pulled cold on every execution because the job scales to zero. |
Two framings that were wrong, recorded so they don't come back
- "Asia is the cost-saving choice." It isn't, in either direction — cost is flat across all of these. An earlier version of this doc argued asia-east1 partly because it was Tier 1 like us-central1; that was true but load-bearing on nothing.
- "Put GCS near the DigitalOcean app." Intuitive, but it optimises the wrong leg. MES only uploads a payload and polls status — it never moves the PDFs. The factory downloads them directly from GCS via a signed URL, so MES↔GCS proximity buys nothing a person notices. (Also unverified: the DigitalOcean app's region is not recorded anywhere in this repo — no app spec is committed — so any argument resting on it would be guesswork regardless.)
Reachability was never the question: Fulfil already delivers shipping labels to these same
factories from storage.googleapis.com today. This is a throughput optimisation.
And it is cheaply reversible. The objects are a 30-day regenerable cache with the database as source of truth, so changing region later is a variable change plus a re-apply — not a data migration. If a download from a factory workstation shows Taiwan or even us-central1 performing just as well, switching back costs almost nothing.
How to settle "would US actually be slower?" — measure it
Worth doing rather than arguing, because the honest answer is nobody here can predict it. The files are small (~7 MB Aolong zip, ~16 MB Milly zip), so this is seconds either way, not minutes.
Note the naive "China→Iowa is 200ms RTT vs 40ms to Hong Kong, so 5× worse" reasoning overstates the gap. GCS requests enter Google's edge network at the POP nearest the client (Google Front End terminates TLS there) and then reach the bucket over Google's private backbone. So the congested, throttle-prone leg — factory to nearest POP in HK/Taiwan/Japan — is identical for both regions. Only the backbone leg differs, and that is Google's own network.
What can't be reasoned about is the variance: trans-Pacific links suffer peak-hour congestion and throttling, and "fewer bad days" is the real argument for keeping bytes in Asia — not a few seconds of mean transfer time.
Also worth knowing: a CDN would not rescue a distant bucket. Each object is downloaded roughly once, so nearly every request is a cache miss.
The test, run from an actual factory workstation:
# setup, from any machine with gcloud
head -c 16000000 /dev/urandom > /tmp/test16.bin # random = incompressible, like a zip
for R in us-central1 asia-east2; do
gcloud storage buckets create "gs://bg-latency-test-$R" \
--location="$R" --project=infrastructure-staging-1
gcloud storage cp /tmp/test16.bin "gs://bg-latency-test-$R/test16.bin"
gcloud storage sign-url "gs://bg-latency-test-$R/test16.bin" --duration=2h
done
# on the factory workstation, alternating, 3x each
curl -o /dev/null -s -w "us-central1 %{time_total}s %{speed_download} B/s\n" "<signed-url-us>"
curl -o /dev/null -s -w "asia-east2 %{time_total}s %{speed_download} B/s\n" "<signed-url-hk>"
Two details that decide whether the result means anything: use random bytes (a compressible file lets transfer-encoding distort it, and the real payload is a zip), and run it at the hour the factory actually downloads — Thursday morning for Milly — since congestion is time-of-day dependent. Delete both buckets afterwards; the whole exercise costs fractions of a cent.
Two things still to confirm with whoever owns the GCP org
- Any org policy pinning resources to US regions, which would override all of the above. Not verifiable from this repo.
- Whether Hong Kong is acceptable to the org as a data location. A judgement call
rather than a technical one, but worth asking before the first
applyrather than after.
9 · State buckets and bootstrap
manufacturing-admin-tf-staging and manufacturing-admin-tf-production, versioning on,
created out-of-band before the first init, injected via
terraform init -backend-config="bucket=...". One state bucket per service per environment,
matching the sibling repo's convention.
Exact commands are in ../terraform/README.md § Bootstrap, along
with the per-env mes-tf-runner service account. Two notes that bite:
- The runner SA's project roles are granted during bootstrap, not by this stack (§ 7), so no
project-admin rights are needed to run
apply— only to do the one-time bootstrap itself. - Use
terraform init -reconfigurewhen switching environments. Without it, Terraform offers to migrate staging state into the production bucket.
10 · Sequencing — Terraform first
Terraform lands before anything is created by hand.
Console-first-then-import is an accepted move here (the sibling ships import-staging.sh and
a "Re-importing staging from scratch" section), and INFRA-621's brief explicitly permits
creating the bucket by hand to unblock INFRA-617. But that permission was written when the
resource set was assumed to be about four resources — bucket, lifecycle rule, service
account, binding. Under the Cloud Run decision main.tf is 10 resource blocks: bucket,
Artifact Registry repo, two service accounts, five IAM bindings, and the job. Hand-creating that and then importing it is
strictly more work than one apply, and it leaves import debt behind.
Bootstrap is ~1 hour of one-time work. Do it first.
The escape hatch stays documented: if INFRA-617 is genuinely blocked, create only the bucket
by hand and terraform import it — README § "Console-first, then import" has the command and
the "plan until empty" check.
11 · CI — build and push to Artifact Registry, keyless
The renderer gets its own Dockerfile.renderer, because the 821 MB Chromium layer must
not reach the web service image. The existing deploy-staging.yml and the production release
workflows are untouched.
A separate workflow, triggered on changes to the renderer paths. Keyless via GitHub's OIDC — no key anywhere in this path:
permissions:
contents: read
id-token: write # required for OIDC; the whole point
steps:
- uses: actions/checkout@v4
- uses: google-github-actions/auth@v2
with:
workload_identity_provider: projects/<n>/locations/global/workloadIdentityPools/github/providers/github
service_account: mes-ci-pusher@<project>.iam.gserviceaccount.com
- uses: google-github-actions/setup-gcloud@v2
- run: gcloud auth configure-docker asia-east2-docker.pkg.dev --quiet
- run: |
IMAGE=asia-east2-docker.pkg.dev/<project>/manufacturing-admin/wo-pdf-renderer:${{ github.sha }}
docker build -f Dockerfile.renderer -t "$IMAGE" .
docker push "$IMAGE"
gcloud run jobs update mes-work-order-pdf-render \
--region=asia-east2 --image="$IMAGE"
Two things this depends on, both already in place:
main.tfsetslifecycle.ignore_changeson the job's image,clientandclient_version, so CI moving the tag does not show up as Terraform drift.artifact_registry_immutable_tagsisfalsein staging (re-push while iterating) andtruein production (a tag always means the same bytes).
A WIF pool + provider for GitHub, and the mes-ci-pusher SA, are not in this stack. They
are org-level identity plumbing shared with other repos, and the sibling repo very likely
already has a pool to reuse. Confirm before creating a second one — that is a bootstrap
question for whoever owns the GCP org, listed in §13.
12 · What this changes in INFRA-617
The ticket is mostly unaffected: both Prisma models, the workOrderIds snapshot, the partial
unique index, the renderer itself, the SSR entry point, the CJK and page-count tests, the
markPrinted = false rule and the Open → Printed semantics all stand exactly as written.
What changes:
| INFRA-617 says | Under Cloud Run |
|---|---|
| A DigitalOcean worker component with its own image | Gone. A Cloud Run job, image in Artifact Registry, outside the MES deploy pipeline. |
Workers claim chunks via SELECT … FOR UPDATE SKIP LOCKED | Gone. Cloud Run assigns work by CLOUD_RUN_TASK_INDEX; MES writes index.json mapping index → chunk. No claim loop, no lock, no worker-liveness question. |
| A crashed worker resumes at the first unstored chunk | Becomes: MES re-runs with a fresh index.json covering only the chunks with no result.json. Same guarantee, different mechanism. |
| Chunk rows updated by the worker that rendered them | The renderer has no database. Each task writes result.json beside its PDF; MES polls executions.get plus the result objects and updates the chunk rows itself. |
| Step 0: decide the worker build strategy; confirm a 4 GB DO tier | Replaced by: bootstrap Terraform (§9), then CI pushes to Artifact Registry (§11). No DO tier needed. |
| Provision GCP by hand in the console | Terraform, this repo, terraform/ (§10). |
| Bucket "in an Asia region" | asia-east2 (Hong Kong), alongside job and Artifact Registry (§8). |
GCS_SERVICE_ACCOUNT_KEY_BASE64 on the web service | Still needed — but only for MES on DigitalOcean. The renderer gets no key (§7). |
| Zip streamed into the upload by the worker | Needs a decision. See below. |
The zip step needs somewhere new to live
INFRA-617 assumes a single sequential worker, so whichever pass handles the last chunk can stream the zip. With chunks rendering in parallel tasks, no task holds all the PDFs.
Recommended: a second, single-task invocation of the same job in --mode zip, run by MES once
every chunk has a result.json. It streams the chunk PDFs out of GCS, through archiver, and
back into the bucket — MES never touches the bytes, and the renderer stays a pure function.
Costs one extra cold start per job, on a workload that rounds to zero.
The alternative — MES builds the zip on DigitalOcean — puts ~20 MB × N through the web service that already has an OOM history (INFRA-599). Not recommended.
13 · Open items for someone with GCP org access
Everything below is a lead-time dependency I could not resolve from the repo:
- Org policy on resource regions — does one pin resources to the US? Decides §8.
- An existing GitHub WIF pool to reuse rather than creating a second (§11).
- Who runs the first
apply— needs project-admin on both projects (§9). - Whether Hong Kong is acceptable as a data location for the org, and ideally a timed
download from a real factory workstation to confirm
asia-east2earns its place (§8).
None of these block writing INFRA-617's code; all of them block its first deploy.