Industry practices · Practice 3 of 10

Treat flaky tests as defects with an owner and a deadline

Large engineering organisations converge on a similar loop for flakiness, and Google's published approach ("Flaky Tests at Google and How We Mitigate Them") is the best documented. Each test result is recorded with its revision, and a test that both passes and fails on the same revision is marked flaky automatically. Flaky tests are moved into quarantine, where they still run and report but no longer block the gate, and a ticket is raised against the owning team. A test that stays in quarantine past an agreed period is deleted or rewritten. Fowler's 2011 article "Eradicating Non-Determinism in Tests" recommends the same quarantine step and adds that the quarantine list must be kept short, or it becomes a place where tests go to be forgotten.

Microsoft Research's study of the lifecycle of flaky tests across six large proprietary projects measured how often flaky tests recur after being fixed, how long fixes take and how often a claimed fix actually removes the flakiness. Its findings are the right input for setting the quarantine deadline and for deciding when a returning test needs a second look before it rejoins the gate.

Result recorded with its revision Pass and fail on one revision? yes Quarantine runs, does not block Ticket to owner with a deadline Fixed by deadline? no Blocking gate yes no Delete or rewrite
The quarantine loop. A flaky test keeps running and reporting, stops blocking the gate, and has an owner and a deadline to return or be removed.

Automated retries are used carefully in this model. A test that passes only on retry is recorded as flaky and counted against its owner, so retries keep the gate moving without hiding the problem. Spotify, Uber (Testopedia), Atlassian (Flakinator) and Slack have each published descriptions of internal tooling that follows this pattern, and commercial CI products such as Buildkite Test Engine, CircleCI and Datadog CI Visibility offer flaky-test detection built on the same idea of tracking pass and fail on unchanged code.

Teams also publish the flake rate per pipeline as a service-level objective. Given the arithmetic in the overview, a 100-wide gate needs each pipeline below roughly 0.5% to get most runs green, and a target of that kind makes the cost of one team's flakiness visible to everyone who depends on the gate.

Quarantine in place of commenting tests out

Module teams here already take flaky tests out of the gate, by commenting them out, and they also fix them or decide that an issue is not worth fixing. Quarantine keeps all three outcomes and changes what each one leaves behind. A commented-out test stops running, so the pipeline records no further results for it, the flake history that would show whether its cause has gone never builds up, and nothing outside the team's own memory says which behaviour has lost its check. A quarantined test keeps running on every pipeline run without blocking it, its entry names the owner and the deadline, and the list of entries is the record of what the gate no longer checks.

The test runner can carry the quarantine without any new tooling at first. Replacing the comment with a skip or tag that holds a ticket reference, such as a Playwright @quarantine tag that the gate run excludes and a separate non-blocking run includes, keeps the test executing and makes the number of quarantined tests per suite something a script can count.

The call on whether an issue is worth fixing stays with the module team, and two things change about it. The team makes it from the test's measured contribution to the gate: the overview's arithmetic shows that a test flaking once in a thousand runs is cheap in one pipeline and expensive across 100, and the flake history gives each test's actual rate. A decision not to fix ends in deletion with a note of the behaviour that is no longer checked, and where that behaviour still matters, the check moves to a component or contract test (practice 5). The deadline also changes when the work happens: today a flaky test is handled when its team sets out to fix the suite, and with a deadline it is handled within an agreed period of being detected.

Flakes in UI pipelines that others wait on

Some UI pipelines depend on another UI pipeline through a pipeline material and start only after it passes on the same revisions. A flake in such an upstream pipeline leaves every pipeline below it without a result for that revision and delays them by its rerun, so its cost to the gate is higher than its own flake rate suggests. Ranking quarantine and fix work by flake rate multiplied by the number of UI pipelines that wait on the suite puts these pipelines first.

In GoCD

Features to use

  • Test artifacts are stored per job and per pipeline instance, together with the material revisions of that instance, and the API exposes both, which is the raw data flake detection needs.
  • A stage, or selected jobs within it, can be rerun on the same revisions. A rerun that passes after a failure is the pass-and-fail-on-one-revision signal.
  • Pipeline groups map pipelines to teams, which gives each flaky test an owning team.

Gaps

  • GoCD has no flaky-test detection, no quarantine and no test history across runs. Results have to be exported to an external store or to a product such as Buildkite Test Engine or Datadog CI Visibility, both of which accept JUnit XML from any CI server.
  • Reruns in GoCD repeat whole jobs and are not recorded as flake evidence. Per-test retries belong in the test runner, and Playwright, for example, reports a test that passes on retry with the status flaky.
  • The quarantine list has to be applied by the test runner, for example through tags or a skip list read at start-up.

Pros and cons for this system

Pros

  • Quarantine directly lowers the effective per-pipeline flake rate, which is the lever with the largest effect in the overview's arithmetic.
  • Quarantined tests keep running, so their results still build the history needed to fix them, which a commented-out test does not.
  • The quarantine list records which behaviour the gate no longer checks, while commented-out tests leave no such record.
  • Owners and deadlines make the cost of each team's flakiness visible to the teams that depend on the gate.

Cons

  • A quarantined test no longer blocks, so a genuine regression in the area it covers can pass the gate until the test returns.
  • Detection needs every result stored with its revision. GoCD keeps test reports as artifacts and computes no flake rates, so this needs an external store or product.
  • Without enforced deadlines the quarantine list grows until it holds a large share of the suite.

Sources

Addresses non-deterministic tests, normalisation of deviance, tragedy of the commons.

Sources Google, John Micco, "Flaky Tests at Google and How We Mitigate Them" (Google Testing Blog, 2016); Martin Fowler, "Eradicating Non-Determinism in Tests" (2011); Microsoft Research, Wing Lam, Kıvanç Muşlu, Hitesh Sajnani and Suresh Thummalapenta, "A Study on the Lifecycle of Flaky Tests" (ICSE 2020); Spotify, Jason Palmer, "Test Flakiness: Methods for Identifying and Dealing with Flaky Tests" (2019); Uber, "Flaky Tests Overhaul"; Atlassian, "Taming Test Flakiness"; Slack Engineering, "Handling Flaky Tests at Scale: Auto Detection and Suppression".