Why a release candidate reaches green about once a week
An overview of the current build system, the arithmetic behind the release bottleneck, and an index of the documents in this study.
Current setup
- Frontend lives in one mono repo containing many UI modules and a shared platform folder used by all of them.
- Backend is a set of microservices, each in its own repository.
- Together the UI modules and microservices come to roughly 100 buildable units.
- GoCD runs a first wave of about 100 parallel pipelines, one per unit, with compilation and fast unit tests. The pipelines are independent of each other, so a commit to one service's repository runs that service's pipeline alone.
- A fan-in waits for that wave. After it, a fan-out starts a second wave of about 100 pipelines running the system tests, one suite per frontend module. Each suite drives the deployed system through that module's UI, and this study calls these suites the UI suites and their pipelines the UI pipelines.
- The fan-out can only be triggered manually. A product change reaches the system tests when someone starts the fan-out, and each fan-out carries every commit that has passed the first wave since the previous one.
- The system tests are modular. Each frontend module's suite sits in its own part of the system test code, and its UI pipeline takes that part as a material along with its materials from the first wave. A commit that fixes one suite triggers that suite's UI pipeline and the pipelines chained below it, without a first-wave run or the manual fan-out, and the other UI pipelines keep the results they already have. These runs test the new test code against the product revisions from the last fan-out. Many UI pipelines also take another UI pipeline in the second wave as a material. Such a pipeline starts only after the UI pipeline upstream of it has passed on the same revisions, so the second wave runs as chains of suites and not as one flat layer.
- A release candidate exists only when both waves are green for the same set of revisions.
- The first wave, service builds included, rarely holds up a candidate. The time and the red results come from the second wave of frontend UI tests.
- Each module team looks after the flaky tests in its own UI suite. When a team sets out to fix its system tests, it comments a flaky test out, fixes it, or makes a call on whether the underlying issue is worth fixing at all.
- A release is always the full system. No microservice or frontend module is released on its own, and this release model is a fixed constraint for this study.
Observed problems
- A fully successful run takes about two hours, and the wait before someone starts the fan-out comes on top of that.
- The UI tests are comprehensive and flaky.
- A flaky failure in a UI pipeline that others depend on leaves its downstream UI pipelines without a result for that revision, so one flake removes several results from the run.
- Flaky tests are handled when a module team decides to work on its suite, and a test that has been commented out leaves no record in the pipeline of what it covered or why it was taken out.
- Producing a releasable build is hard, and some weeks yield only one.
The arithmetic of a 100-wide green gate
When a release requires every one of N independent pipelines to pass, the probability of a fully green run is (1 − p)N, where p is the chance that a single pipeline fails for reasons unrelated to the change under test. With N = 100, small per-pipeline flake rates compound into a very low chance of a green candidate.
| Flake rate per UI pipeline run | Chance all 100 pass | Expected green runs per week (25 runs) |
|---|---|---|
| 0.5% | 60.6% | 15.1 |
| 1% | 36.6% | 9.2 |
| 2% | 13.3% | 3.3 |
| 3% | 4.8% | 1.2 |
| 5% | 0.6% | 0.15 |
The weekly column assumes about five two-hour runs per working day, which needs someone to start the fan-out as soon as each run finishes. With a manual fan-out, the number of runs in a week is the number of times someone starts it, so the green runs per week fall below the column whenever starts are missed, even at the same flake rate. A flake rate near 3% per pipeline is enough on its own to produce the one-green-build-a-week pattern, before counting any genuine defects. A pipeline that runs 30 UI tests, each flaking once in a thousand runs, already sits at about 3%, so the per-test numbers that look acceptable to a module team become unacceptable at the level of the whole gate. A module team deciding whether a flaky test is worth fixing sees the first number, a test that fails once in a thousand runs, while the cost of that test is paid in the second.
The chains inside the second wave leave the formula unchanged, because every UI pipeline still has to pass, and they add to what each flake costs in time. A downstream UI pipeline starts only when the pipeline above it has passed, so the two-hour run is set by the longest chain of suites. When a pipeline near the top of a chain flakes, the pipelines below it wait for its rerun to pass and then run their own suites, which adds the duration of the rest of the chain to the time before the candidate can turn green.
The modular suites limit what a repair costs. When a team fixes a flaky or broken test, the commit reruns that team's UI pipeline and the pipelines chained below it, and the rest of the second wave keeps its results for the same product revisions. A repair therefore takes as long as the chain it sits in, and only the longest chain takes the full two hours. The formula still applies to every fan-out, because each fan-out runs all the UI pipelines against new product revisions, and each repair run is exposed to flakes in the pipelines it reruns.
Genuine failures make the picture worse because of batch size. Every commit made between two manual fan-outs goes into the same candidate, which is at least two hours of commits from many teams and more when the fan-out waits, and a single real defect in any of them turns the entire candidate red. The next candidate carries that defect forward until someone fixes it, along with a fresh set of commits that each carry their own risk.
p (determinism of tests and environments) and reducing N (how many pipelines must pass together for one candidate). The full-system release does not force N to be 200: the release can still be assembled from module versions that each passed on an earlier run. Most of the industry practices in this study pull one of those two levers, and the rest shorten the cycle so that more attempts fit into a week.
Documents in this study
Animated walkthroughs of the pipeline, from the happy path to a genuine defect whose fix reaches the second wave with new commits and fails again, and a promotion gate that lets the fix through alone.
What still keeps a candidate red once the UI suites stop flaking, from genuine defects in large batches to tests that were commented out and never restored.
The standard industry names for what the current pipeline exhibits, where each name comes from, and how each one shows up here.
What other organisations running large mono repos and many services do about slow, flaky, all-or-nothing pipelines, with a mapping back to the anti-patterns.