Skip to main content

Release Process

This document describes the complete release process for manufacturing-admin, from creating a release to deploying to production.

Overview​

The manufacturing-admin release process follows a "build once, deploy many" pattern with three GitHub Actions workflows:

  1. Tag & Build - Automatically triggered when creating a pre-release
  2. Deploy to Production (Specific Tag) - Manual deployment with a specific tag
  3. Deploy to Production (Latest) - Manual deployment with the latest tag

Why Pre-Releases?​

We create GitHub pre-releases (not full releases) when tagging to:

  • Keep the release in draft state until deployed to production
  • Prevent confusion about which version is actually live
  • Allow the "Deploy Latest" workflow to mark it as the official release after successful deployment

Prerequisites​

  • Access to the GitHub repository
  • Permission to push tags to the repository
  • Access to manually trigger GitHub Actions workflows

Complete Release Flow​

Step 1: Create a Pre-Release on GitHub​

Version tags must follow semantic versioning: v\{MAJOR\}.\{MINOR\}.\{PATCH\}

  1. Go to GitHub → Releases → Draft a new release
  2. Click "Choose a tag"
  3. Type your new version tag (e.g., v1.2.3) and select "Create new tag on publish"
  4. Select the target branch (usually main)
  5. Enter release title (e.g., v1.2.3)
  6. Check "Set as a pre-release" ✅
  7. Click "Publish release"

What happens automatically:

  • ✅ CI/CD checks run (Biome, TypeScript, Tests)
  • ✅ Docker image is built
  • ✅ Image is tagged with the version (e.g., web-portal:v1.2.3)
  • ✅ Image is pushed to production registry: registry.digitalocean.com/manufacturing-admin-production
  • ✅ AI-generated release notes are created
  • ✅ GitHub release is updated with formatted notes
  • ✅ Slack notification posted to #infra-releases with release summary

Important:

  • This creates a pre-release (not marked as latest)
  • This does NOT deploy to any environment
  • The Docker image is ready in the registry for deployment

Step 2: Deploy to Production​

You have two options for deploying to production:

  1. Go to GitHub Actions → Release tag to Production
  2. Click "Run workflow"
  3. Enter the tag you want to deploy (e.g., v1.2.3)
  4. Click "Run workflow"

What happens:

  • ✅ Checks if this is a rollback (deploying older version)
  • ✅ Runs CI/CD checks on the tag
  • ✅ Runs database migrations on production database
  • ✅ Seeds reference tables
  • ✅ Updates app spec to use the specified image tag
  • ✅ Deploys to production: https://mes.birdygrey.com
  • ✅ Posts deployment notification to #infra-releases
    • Shows rollback warning if deploying older version
    • Includes link to production site

Option B: Deploy Latest Tag​

  1. Go to GitHub Actions → Release latest tag to Production
  2. Click "Run workflow"
  3. Click "Run workflow" (no input needed - auto-detects latest tag)

What happens:

  • ✅ Automatically detects the latest tag
  • ✅ Runs CI/CD checks on the tag
  • ✅ Runs database migrations on production database
  • ✅ Seeds reference tables
  • ✅ Updates app spec to use the latest image tag
  • ✅ Deploys to production: https://mes.birdygrey.com
  • ✅ Marks GitHub release as "published" and "latest" (removes pre-release status)
  • ✅ Posts deployment notification to #infra-releases

Note: This workflow will convert your pre-release into a full release and mark it as the latest.

Rollback Process​

To rollback to a previous version:

  1. Use Option A (Deploy a Specific Tag)
  2. Enter the tag of the version you want to rollback to (e.g., v1.2.2)
  3. The workflow will detect this is a rollback and notify in Slack

Note: The rollback detection compares commit dates. If the tag you're deploying is older than the current latest release, it will be flagged as a rollback.

Release Notes​

Release notes are automatically generated using AI (Claude) and include:

  • Categorized changes: Bug Fixes, Features, Dependencies, etc.
  • Linear ticket links: Automatically converted (e.g., INFRA-123 → Linear link)
  • GitHub PR links: With proper formatting
  • Contributor mentions: GitHub usernames → Slack mentions
  • Dependency updates: Grouped together at the bottom

Release notes are posted to:

  • Slack #infra-releases channel (Slack markdown format)
  • GitHub release page (GitHub markdown format)

Workflow Files​

  • .github/workflows/build-tag-production.yml - Tag & Build workflow (triggered by pushing a v*.*.* tag)
  • .github/workflows/release-tag-production.yml - Deploy specific tag
  • .github/workflows/release-latest-production.yml - Deploy latest tag
  • .github/workflows/deploy-staging.yml - Build & deploy staging (on push to main)

All four install doctl and must keep an explicit pinned version: — see doctl Version Pinning & the Production VPC Guard.

Environment Configuration​

Staging Environment​

Staging Container Image Management​

Staging uses a tag rotation strategy to maintain rollback capability while automatically cleaning up old images:

Tag Rotation Flow:

  1. :candidate - New image is built and pushed with this tag
  2. :staging - The :candidate is promoted to :staging (active deployment), then :candidate is deleted
  3. :previous - The old :staging is re-tagged as :previous (rollback target)
  4. Garbage collection - Untagged images are automatically cleaned up

Rollback: The :previous tag enables rollback to the last deployed version. However, the next push to main will automatically deploy new code again, so rollback is temporary unless commits are reverted in the main branch.

Registry: registry.digitalocean.com/manufacturing-admin-staging

Production Environment​

  • Trigger: Manual workflow dispatch
  • URL: https://mes.birdygrey.com
  • Database: Separate production database with firewall protection
  • Registry: registry.digitalocean.com/manufacturing-admin-production

doctl Version Pinning & the Production VPC Guard (INFRA-625)​

Every digitalocean/action-doctl@v2 step carries an explicit version: — 7 steps across 4 workflows (release-latest-production.yml ×2, release-tag-production.yml ×2, deploy-staging.yml ×2, build-tag-production.yml ×1), all pinned to 1.166.0. Omitting version: defaults to latest, which the action resolves with an unauthenticated GitHub API call from the runner's shared IP; when that call is rate-limited the action emits a core.warning (never an error) and installs its hardcoded fallback, doctl 1.98.1. Passing an explicit version skips the releases API call entirely, so pinning removes the exposure rather than narrowing it.

The literal is repeated at each call site rather than hidden behind a workflow-level env var so that grep -A6 action-doctl .github/workflows/* shows the effective version everywhere.

Bumping the version touches 8 places, not 7 — all 7 workflow steps plus PINNED_DOCTL_VERSION in .github/workflows/deploy-workflows.spec.ts. Bump only the workflows and CI fails with a bare expected [ '1.180.0' ] to deeply equal [ '1.166.0' ], which doesn't name the constant you missed. Verify with:

grep -rn "1\.166\.0" .github/workflows/    # expect 8 hits: 7 steps + the spec constant
npx vitest run .github/workflows/deploy-workflows.spec.ts

Whichever version you move to must preserve vpc on a spec round-trip. 1.135.0 and 1.166.0 are both verified; anything at or below 1.98.1 is known-broken.

Why this matters — the 2026-08-04 production outage (~2 hours)​

doctl 1.98.1 (April 2023) predates App Platform VPC support and has no vpc field in its AppSpec struct, so doctl apps spec get silently omits it. Both production release workflows read-modify-write the entire app spec:

doctl apps spec get → jq patches .services[] | select(.name == "web-portal").image.tag → doctl apps update --spec

So the VPC-less spec was written straight back, disconnecting the app from its VPC (374820ff-b6bf-4d76-9b1d-64253091fce8, manufacturing-admin-sgp1, 10.104.0.0/20). Because src/server.ts runs await pgPool.query('SELECT 1') at module load, the process died on connect ETIMEDOUT 10.104.0.8:25060 before binding :8080 — readiness probes failed and DigitalOcean auto-rolled back, 7 attempts in a row.

Only attempt 1 actually installed 1.98.1; attempts 2-7 got 1.166.0 but faithfully round-tripped the already-broken stored spec. One transient rate limit, then self-perpetuating. Resolved by re-ticking Connect app to VPC network in the DigitalOcean console.

The guard​

Both production workflows run this immediately after doctl apps spec get, before the jq patch:

jq -e '.vpc.id' app-spec.json > /dev/null || {
  echo "❌ app spec is missing vpc.id — refusing to deploy"
  ...
  exit 1
}

Placement is load-bearing: the field is lost on the read, so a check after doctl apps update would only fire once the damage is already written. It is presence-only rather than an equality assert on the id, so a legitimate VPC migration doesn't require a workflow edit. Staging is not VPC-connected and its deploy job just calls apps create-deployment --force-rebuild with no spec round-trip, so staging gets the pin but no guard.

Known-fragile, deliberately out of scope: the whole-spec read-modify-write drops any field the installed doctl doesn't model — the pin and guard only close this for vpc. A targeted image-tag-only update would eliminate the class of bug; tracked as a separate follow-up.

Automated coverage​

.github/workflows/deploy-workflows.spec.ts enforces both invariants and runs in CI as part of npm run test:ci. It needs no network, no doctl, and no DigitalOcean token:

  1. Every action-doctl step across all 4 workflows carries an explicit version:, and they all agree.
  2. Any new workflow that adds an action-doctl step fails the spec until it's added to DOCTL_STEP_COUNTS — so a new unpinned step can't slip in unexamined.
  3. In both production workflows, the guard's line index falls between apps spec get and apps update. This is the assertion a human reviewer is least likely to catch by eye.
  4. The guard block, extracted verbatim from the committed workflow and executed under bash -e, exits 0 with vpc.id, exits 1 with no vpc key and with vpc: {}, and names the expected VPC id in its failure output.

All four assertions were mutation-tested: unpinning a step, drifting one file's pin, moving the guard below apps update, and deleting the guard each fail the spec.

Two notes if you extend it:

  • It parses the YAML as raw text on purpose. js-yaml and yaml are present in node_modules but are transitive-only (yaml arrives via vite) and neither is declared in package.json — fine for an ad-hoc one-off, but importing one from a committed spec would couple it to a dependency any bump could remove.
  • The file sits outside tsconfig.json's include (src/**, prisma/**), so npx tsc --noEmit does not typecheck it — the same as the existing scripts/*.spec.ts. Vitest transpiles it regardless, and Biome does lint it (files.includes is ** minus a few paths, and .github isn't excluded).

Best Practices​

  1. Always test in staging first before creating a production tag
  2. Use semantic versioning for tags (v{MAJOR}.{MINOR}.{PATCH})
  3. Always create as pre-release when tagging (not full release)
  4. Review AI-generated release notes in Slack before deploying
  5. Deploy during maintenance windows for major releases
  6. Use "Deploy Latest" workflow to mark releases as official after successful deployment
  7. Verify production after deployment using the "View Production" link in Slack

Troubleshooting​

Tag workflow failed​

  • Check GitHub Actions logs for the specific failure
  • Common issues:
    • Tests failing
    • Docker build errors
    • Registry authentication issues

Production deployment failed​

  • Check if database migrations succeeded
  • Verify app spec was updated correctly
  • Check DigitalOcean App Platform logs

App boots then dies with connect ETIMEDOUT 10.x.x.x:25060​

The app has been disconnected from its VPC, so it can't reach the database's private IP. Because the DB is checked at module load, the process exits before binding its port, readiness probes fail, and DigitalOcean auto-rolls back — which makes it look like a bad release rather than an infrastructure change.

  1. Check doctl apps spec get <app-id> --format json | jq '.vpc' — if it's null, the spec lost the VPC block.
  2. Re-tick Connect app to VPC network in the DigitalOcean console to restore it.
  3. Check the failed job's log for the installed doctl version — an old doctl silently drops vpc on read. See doctl Version Pinning & the Production VPC Guard. Deploys since INFRA-625 fail loudly before writing a VPC-less spec.

Deploy fails with "app spec is missing vpc.id — refusing to deploy"​

Working as intended — the guard caught a spec read that lost the VPC block, before it could be written back. Check the installed doctl version printed earlier in the same job (>>> doctl version vX installed); it should be the pinned 1.166.0. Do not bypass the guard: verify the live spec still has .vpc.id, and fix the doctl version rather than the check.

Rollback needed urgently​

  • Use the rollback process above
  • The previous Docker image is still in the registry and can be deployed immediately

Questions?​

For questions about the release process, reach out in #infra-releases on Slack.