How a batch of commits moves through the pipeline

Animated walkthroughs of the current pipeline described in the overview, followed by one change to it that resolves the second scenario. Each scenario steps through the same diagram, with a caption for every step and a time track along the bottom.

Happy path

Nothing flakes and no commit carries a defect. The scenario follows six commits from the moment the first one lands until both waves are green on the same revisions and a release candidate exists, including the time the green fan-in spends waiting for someone to start the fan-out.

Genuine defect, then a fix that travels with new commits

A commit to svc-billing breaks the order journey, and ui-orders goes red in the second wave. By the time the fix has passed the first wave, other teams have added commits to the batch, and one of them breaks ui-onboarding. Because ui-orders takes ui-onboarding as a material, the second run ends red without running the suite that would test the fix. The after the flakes page works through the arithmetic behind this pattern.

Resolution: a promotion gate that admits only the fix

The second scenario goes red twice because the fan-out takes every revision that has passed the first wave. The fix for the first defect therefore cannot reach the second wave without the commits that landed during the diagnosis, and any one of them can break a suite above ui-orders in its chain. This resolution changes what the fan-out takes as input and leaves wave 1, wave 2 and the manual start as they are.

Where the gate sits

The gate sits between the end of wave 1 and the fan-in, shown in the diagram as the Promote column. Today a revision becomes eligible for the fan-out the moment its wave-1 pipeline goes green. With the gate, each wave-1 pipeline ends in an extra stage called promote, and a revision becomes eligible only when that stage has run. The fan-in reads the last promoted revision of every unit, so a commit whose promote stage has not run is still built and tested in wave 1 but stays invisible to the fan-out, and the fan-in keeps using the revision of that unit it promoted earlier. The system test code is unaffected, since each UI pipeline still takes its suite straight from source control.

The promotion gate box under the column holds one value, off or on, and a script uses it to decide which promote stages to approve. The promote stages are switched on by default and stay in every wave-1 pipeline permanently, while the gate is off by default. With the gate off, the script approves every promote stage as soon as the build before it passes, and the pipeline behaves as it does today apart from a few seconds per promotion. When a candidate goes red, the pipeline engineer switches the gate on, and the script then approves only commits marked as fixes for that run. Every other commit waits at its own promote stage, shown in the diagram as held, without being reverted. The next fan-out then carries the revisions run 1 tested with only the fixed unit moved forward, so its result answers whether the fix works and nothing else.

Switching the gate on and off on demand

The gate can be switched on demand, because the switch is the one value the script reads and changing it touches no pipeline definition. The pipeline engineer commits the new value to the gate file, the script picks it up on its next pass, and from then on it holds or approves promote stages accordingly. Switching the gate off approves every held promote stage, so the held commits enter the next fan-out together. Keeping the switch in a reviewed file gives a record of who switched the gate, when and against which red run.

GoCD offers a second way to switch, which avoids the script in normal times. A stage's approval type can be set to automatic, which GoCD calls success, or to manual, and the Groovy DSL config can set it from the gate file. With the gate off, every promote stage is automatic and GoCD promotes green builds without help, and switching the gate on regenerates the config with manual approval, so new builds wait until someone or the script approves them. Builds already promoted stay promoted, which is the intended behaviour, and whether a build that is running when the config changes picks up the new approval type is worth checking on the GoCD version in use. The script approach switches faster and keeps the GoCD config unchanged, so it is the better default, and the config approach suits teams that want to see the gate state in the pipeline definitions themselves.

Removing the promote stages when the gate is off is a third option and the weakest one. The UI pipelines would need their dependency materials changed back to the build stage each time, and GoCD treats a changed material as a new one, so each switch would trigger fresh runs.

The GoCD pieces

• A promote stage at the end of each wave-1 pipeline, with manual approval and a trivial job.
• Dependency materials in the UI pipelines and the fan-in that name the promote stage instead of the build stage. GoCD triggers a downstream pipeline from a named upstream stage, so this is a config change in the existing pipeline definitions.
• A script that approves promote stages through the stage run API, POST /go/api/stages/{pipeline}/{counter}/{stage}/run, triggered by a notification plugin or by polling.
• The gate state in a file under review, and a convention such as a fix-for: line in the commit message that marks a fix and names the red run.

What the gate changes and what it leaves

The candidate in this scenario arrives one run after the fix, where the second scenario had none after two runs. The svc-users defect still exists, and it turns the following run red once the gate releases the held commits. Its cost moves to a run of its own, and the candidate from run 2 is already available for release while that run is red.

The held commits grow the next batch, because everything that waited at the gate goes into one fan-out once the gate is switched off. A long diagnosis therefore produces a large release of held commits, with the batch-size risk described on the after the flakes page. Switching the gate off as soon as the fix produces a candidate limits this, and the script can also release held commits in smaller groups across several fan-outs.

The gate selects revisions per unit. A commit to svc-billing that lands between the defect and the fix comes along with the fix, because promoting the fix promotes the whole repository at that revision. Excluding such commits needs a short-lived fix branch cut from the revision run 1 tested, with its own pipeline generated by the Groovy DSL config plugin and promoted in place of the main line. Commits to the frontend mono repo outside the shared platform folder do not have this problem, because each ui-* pipeline builds only its own module folder.

The same selection can be made with a manifest repository that lists the revision of each unit and serves as the only material of the second wave, with the script committing to the manifest instead of approving stages. The manifest keeps a history of what each candidate contained, and the cost is that UI pipelines have to fetch artifacts by version from an artifact store, since GoCD's fetch artifact task works only through dependency materials.