ci-cd-best-practices

Hollywood workflow generation

Do not handwrite new GitHub Actions workflow YAML. Author or edit the Hollywood TypeScript source, then regenerate .github/workflows/*.yml.

Default workflow flow:

1. Put workflow source under ci/ and export a Hollywood workflow(...) object.
2. Generate with pnpm exec hollywood generate "ci/**/*.ts" --output ..
3. Commit both the source and generated YAML. Generated YAML must begin with # @generated by Hollywood. Do not edit by hand.
4. If Hollywood cannot express a required GitHub Actions field, extend Hollywood first instead of hand-editing YAML.

Existing handwritten workflows are legacy precedent, not permission to add more handwritten YAML. When touching a legacy workflow for substantial behavioral changes, prefer moving that workflow behind Hollywood in the same PR or split the generator migration into a preparatory PR.

Runner placement

Choose the cheapest runner that matches the job's real bottleneck.

ubuntu-24.04

Use for I/O-bound or orchestration-heavy jobs:

• git / GitHub API orchestration
• AWS/Terraform apply jobs that do not compile code
• artifact upload/download
• release-please
• branch promotion and audit tagging
• Copybara sync jobs
• IAM preflight / policy simulation
• ECS force deploy
• link checking
• small lint/validation jobs that do not build or test significant code
• self-hosted runner provision / cleanup control jobs

blacksmith-4vcpu-ubuntu-2404

Use for compute-bound jobs:

• Docker builds
• Rust / Go / Python / TypeScript builds
• unit, integration, and end-to-end tests
• typechecking over real codebases
• static analysis that scans large codebases (for example Semgrep)
• Kind / KWOK / Chaos / stress workloads
• anything that spends real time compiling, bundling, or executing test suites

self-hosted AWS runners

Use only when GitHub-hosted runners cannot do the job:

• KVM / nested virtualization
• ARM64-native builds that must run on our own infrastructure
• privileged or hardware-specific workloads
• deploy jobs whose payload must run inside the target architecture/runtime

If the job is only provisioning or cleaning up the self-hosted runner, run that control job on ubuntu-24.04. Only the payload belongs on self-hosted.

Docker artifact policy

PR CI

Every deployable image must be built in CI with push: false.

Rules:

• CD must never be the first place a Dockerfile runs
• do not push unreviewed application images to ECR from pull requests
• verify the exact Dockerfile, build context, and target platform the CD workflow uses
• make CI gates and CD path filters cover every file copied into that Docker build context
• if the Dockerfile depends on private ECR base images, log in only for pull access; still keep push: false

merge / branch CD

After merge, CD may build and publish the application image for the merged commit.

Current repo rule:

• PRs verify builds
• merged branches publish images

If you later refactor a service toward true build-once/deploy-later, keep the same invariant: no unreviewed artifact is written to ECR.

Workflow structure

Prefer these shapes:

1. verify-build in CI
2. publish in CD
3. deploy/apply in CD

Split build from deploy when runner classes differ. A pure Terraform/ECS apply job should not stay on Blacksmith just because the build job in the same workflow needs it.

Use reusable workflows for generic behavior. Use service-specific wrapper workflows when a service needs custom setup such as:

• private base image bootstrap
• extra build contexts
• special build arguments
• architecture-specific setup

Self-hosted runner guidance

Prefer server-side runner registration when the platform supports it.

For GitHub Actions runners:

• prefer JIT runner configuration over registration-token plus polling loops
• avoid scanning the runner list in a retry loop when the API can return the runner identity directly
• keep provisioning and cleanup logic separate from the payload job

Validation

After editing workflows:

1. Regenerate workflows with pnpm exec hollywood generate "ci/**/*.ts" --output .
2. Run actionlint
3. Check that cheap jobs use ubuntu-24.04
4. Check that builds/tests remain on Blacksmith or self-hosted
5. Confirm PR image verification uses push: false
6. Confirm CD still publishes only from merged branches or explicit deploy triggers