Industry practices ยท Practice 9 of 10

Make the build an owned, measured product

Teams that fix this class of problem usually put a platform or developer-productivity team in charge of the build as a product, with published service levels. The commonly tracked measures are the four DORA metrics (deployment frequency, lead time for changes, change failure rate and time to restore) together with build-specific ones: time to green on main, percentage of main builds that are green, flake rate per pipeline and median pipeline duration. A rotating build cop, recommended in Continuous Delivery, enforces the rule that a red main build is fixed or reverted before anything else is merged. In this system the rule can only start once flaky tests are under control, and until then the build cop works as a triager.

Why the revert-or-fix rule waits for flake control

The rule assumes a red build points to a change that can be reverted or fixed. At the overview's 3% flake rate per UI pipeline, 95% of full runs fail from flakes alone, so a red build usually points to no change at all. A revert or a fix is a new commit, and the new commit starts another wave of 100 pipelines that fails on some unrelated flake with the same 95% probability. Under the rule, main would stay closed to merges while the build cop chased one red run after another, each two hours long.

While the flake rate stays near today's level, the build cop triages each red module pipeline and leaves main open. Rerunning the failed job on the same revision separates the two cases. A test that passes on the rerun is a flake, which goes to the quarantine with an owner (practice 3). A failure that repeats on the same revision is treated as a defect, and the owning team reverts or fixes it. UI pipelines that wait on a red UI pipeline have no run to triage, so the build cop starts with the upstream pipeline, and its rerun lets the ones below it run. The triage records double as the flake history that practice 3 needs, and they also measure progress: the share of red runs whose failure repeats on a rerun shows how often a red build now means a real defect.

With release assembly (practice 1), a red module pipeline holds back only its own module, so the rule can be adopted one module at a time. A module whose pipeline flakes in 0.5% of runs after quarantine and fixes (practice 4) goes red from flakes once in 200 runs, and its team can apply revert-or-fix to that pipeline while the full gate is still noisy. The rule reaches the whole system when the triage records show that most red runs reproduce.

In GoCD

Features to use

  • The GoCD analytics plugin charts pipeline duration, failure rate and time to recover per pipeline.
  • The pipeline history API provides the run data for computing deployment frequency, lead time and the share of green builds on main.
  • Pipeline templates and config repositories keep the 100 module pipelines consistent, and changes to them go through review in version control.
  • Pipeline groups and their permissions record which team owns which pipelines.
  • A failed job can be rerun on the same revisions without a new commit, which gives the build cop a flake-or-defect verdict without starting another wave.

Gaps

  • GoCD has no DORA dashboard and no flake-rate measure, so both are computed outside it.
  • There is no build-cop support, such as rotation or paging, beyond the notification plugins, and GoCD keeps no record of which red runs the build cop classified as flakes.

Pros and cons for this system

Pros

  • A named team is accountable for build health and has the time to improve it.
  • Published measures show whether the other practices are working.
  • Once flakes are under control, a build cop shortens the time main stays red after a break, and before then the same rotation produces the triage records that practice 3 depends on.

Cons

  • The owning team needs dedicated people, taken from other work.
  • Measures can be gamed, for example by quarantining tests to raise the share of green builds, so the quarantine size has to be tracked alongside them. Commenting tests out raises the same measures without showing up in any count, which is one more reason to replace comments with tagged quarantine (practice 3).
  • The build cop rotation takes an engineer off feature work for the length of each shift.
  • Starting the revert-or-fix rule before flakes are controlled keeps main red, because each revert or fix starts a wave that is likely to fail on an unrelated flake.

Sources

Addresses tragedy of the commons, normalisation of deviance, CI theatre.

Sources DORA research programme (Google Cloud), dora.dev and its "Continuous delivery" capability; Jez Humble and David Farley, Continuous Delivery (2010), for the build cop and stop-the-line rule.