🔗 The Pain That is Github Actions

For the past two weeks, I’ve been spending most of my time rewriting our CI scripts in GitHub Actions. This is the third time we’ve had to redo our CI setup—first GitHub Actions, then Earthly (which we moved away from because it was discontinued), and now, reluctantly, back to GitHub Actions.

Our CI is complex: merge queues, multiple runners (self-hosted, blacksmith.sh, GitHub-hosted), Rust builds, Docker images, and heavy integration tests. Every PR we merge burns through an hour of CI time, running across multiple parallel runners.

There are a few things we'd like to have (which we deem as "good software practice") but it's nothing unheard of:

1. Everything that goes into `main` must pass all tests.
2. Trivial mistakes (formatting, unused deps, lint issues) should be fixed automatically, not cause failures.
3. The artifacts we test with in CI should be the exact ones we release.
4. CI should complete quickly (to keep developers happy).

GitHub Actions technically allows all of this—but setting it up is a frustrating mess, full of hidden gotchas, inconsistent behavior, and a debugging experience that makes me question my choices.