Industry practices ยท Practice 7 of 10

Keep main green with a merge queue

Graydon Hoare's "not rocket science rule" (from the Rust project's bors bot) states that the main branch should only ever contain code that has passed the full test suite. A merge queue enforces this by testing each change, or a small group of changes, against the current tip of main before merging it. When a group fails, the queue bisects it to find the culprit and merges the rest. GitHub merge queue, GitLab merge trains, Mergify, Aviator and Graphite implement this, and Uber's SubmitQueue paper ("Keeping Master Green at Scale", EuroSys 2019) describes how it scales to thousands of changes a day by speculatively building likely outcomes in parallel.

Every commit in this system goes directly to main, so a change never waits anywhere a queue could test it. Adopting a queue means each change arrives as a short-lived branch or pull request, which the queue lands on main once its check passes. Trunk Based Development allows short-lived branches of this kind for larger teams, and the branch lives for hours, so the code still integrates on main daily. The required check is the build and unit pipelines of the modules a change touches, and the UI suites stay after the fan-in as in practice 2.

The build and unit pipelines are rarely what turns a candidate red here, so a queue whose check is that tier keeps main green only for a tier that is already mostly green, and UI regressions still land on main. The queue starts to address the actual problem once the affected UI suites of a change (practice 6) are fast and deterministic enough to join the required check, which in turn depends on stubbing services at the network boundary (practice 4) and on low flake rates. Until then, this practice comes after practices 3, 4 and 6.

Merge queue PR A PR B PR C PR D Test A+B+C+D on tip of main fails Bisect the batch A+B pass C fails D passes Merge A, B, D main stays green C back to author
A merge queue tests waiting changes together against the tip of main and bisects a failing batch, so a defect is attributed to one change before it reaches main.

With a merge queue, a genuine defect is caught against the change that introduced it and never reaches the shared candidate, which removes most of the batch-diagnosis cost. For failures found in the UI tier after the fan-in, Google's "Flake Aware Culprit Finding" paper (ICST 2023) describes how to locate the responsible commit when some of the results being bisected are themselves flaky, which is the situation a two-hour batch of many teams' commits creates today. Combined with trunk-based development and short-lived branches, it keeps batches small without relying on everyone running the full suite locally.

In GoCD

Features to use

  • Pull-request pipelines from the SCM plugins, with a build status notifier plugin, let GoCD act as the required check that a hosted merge queue waits for.
  • The scheduling API can start a pipeline on a given revision, which lets a webhook from the queue trigger GoCD for each candidate batch.

Gaps

  • GoCD has no merge queue. GitHub's merge queue tests each batch on a temporary branch, while a GoCD git material tracks one fixed branch, so the batch has to be built through a branch-aware plugin or a webhook that calls the scheduling API.
  • Each repository needs its own queue, and a change that spans the mono repo and a service repository is not tested as one unit.
  • Bisecting a failed batch is done by the queue, and GoCD only runs the builds it is asked for.

Pros and cons for this system

Pros

  • Each defect in the blocking tier is attributed to one change before it reaches main.
  • Main stays green for every check in the queue, so once the affected UI suites are part of it, every revision on main is a frontend candidate.
  • Small, tested batches replace the large unattributed batch that each two-hour candidate carries today.

Cons

  • Every team gives up direct commits to main for short-lived branches, which adds a review or queue step to each change.
  • Queue throughput is bounded by how long the required check takes, so the check has to be fast.
  • A flaky required check makes the queue bisect healthy changes and stall, so flakiness has to be under control first (practices 3 and 4).
  • GoCD has no merge queue of its own, and an external queue has to call GoCD as its required check.
  • With build and unit tests as the only required check, the queue misses the UI failures that hold up candidates today.

Sources

Addresses large batch size, big-bang integration, normalisation of deviance.

Sources Graydon Hoare, "The Not Rocket Science Rule of Software Engineering" (2014); GitHub documentation, "Managing a merge queue"; GitLab documentation, "Merge trains"; Uber, Sundaram Ananthanarayanan et al., "Keeping Master Green at Scale" (EuroSys 2019) and the later Uber blog post "Slashing CI Costs at Uber"; Google Research, "Flake Aware Culprit Finding" (ICST 2023).