Industry practices · Practice 1 of 10

Assemble the full-system release from versions already verified

Organisations that ship a whole system as one release, such as browser vendors, mobile app teams and enterprises using a SAFe "agile release train", usually separate verifying a module from releasing the system. Each module and service has its own pipeline that builds it, tests it and publishes a versioned artifact when it is green. The release is then a manifest (sometimes called a bill of materials) that lists one verified version of every module. Chrome and Firefox follow this on a fixed cadence: at the branch cut, whatever is green goes on the train, and anything that is not ready waits for the next one.

The working rule that makes this effective is "last known good". When a module's newest build is red, the release candidate takes that module's previous green version, so a red module holds back only its own new changes and the other 99 modules still move forward. Under the current setup, one red module among 100 means no candidate at all. With last-known-good assembly, the same red module means a candidate that contains everyone else's work and the previous version of that module.

Uber's continuous deployment system applies the same rule to each deployment stage. In "Continuous deployment for large monorepos" (2024), Rasmus Vestergaard and Kasper Munck describe how each stage takes the newest commit that passed the previous stage and meets its gating conditions, so a stage moves straight to the newest qualifying commit instead of working through every commit in order. Uber deploys each service on its own, and here the rule selects versions for a single manifest, with the selection logic unchanged.

Where the red pipelines are in this system

In this system the red results come from the UI pipelines of frontend modules, and the service pipelines seldom stop one from forming. Last known good per module therefore matters most for the frontend, which is where it is hardest to apply. If the UI modules are built from one mono repo revision into one bundle, the manifest cannot take ui-search from one revision and ui-orders from another, and the rule turns into a choice of revision: the candidate takes the newest mono repo revision on which every UI module has a valid UI pass. A pass carries forward from an earlier revision when the dependency graph shows that the module's inputs have not changed since (practice 6). A red UI module then holds back every frontend commit made after the last revision where it had a valid pass, from all teams, which is a weaker result than pinning back one module and still far better than needing all 100 suites green in a single run. Pinning UI modules one at a time, as the diagram above shows for ui-search, needs the frontend packaging changes described under the defect section below, while services keep the per-module rule as it stands.

UI pipelines chained to other UI pipelines tie their passes together. A downstream suite that passed after a particular run of its upstream suite has a pass that is valid only alongside that run, so the assembler has to treat the chain as one unit when it reads passes, and a red upstream suite leaves the downstream module with no new pass to select. Removing a dependency that only orders the runs, or making the downstream suite create the state it takes from the upstream one, lets each module's pass count on its own.

System-level checks still run, because the full system is what ships, and in this system they are the UI suites of the second wave, which already drive the deployed system through each module's UI. They run against the assembled manifest, and they can focus on the modules whose versions changed since the last released manifest, together with the modules that consume a changed service, since the unchanged versions have already passed system tests together. A module's UI pass recorded against deployed services can stand for the manifest when those services ran the versions the manifest lists. Both of these depend on building each artifact once and promoting the same binary through every later stage, a rule from Continuous Delivery, so that a version that passed its module tests is byte for byte the version that goes into the system test and the release.

Feature flags complete the picture for a full-system release. A team whose feature is unfinished can still commit to main and ship its module on the train with the feature switched off, which keeps that module's version current in the manifest and avoids a large catch-up change later.

When system tests find a defect in one module

System tests on an assembled manifest can still find a genuine defect. The release-assembly model handles it by pinning the faulty module back to its last known good version and assembling again, while the owning team fixes forward. Suppose system tests on manifest M42 fail and the failure is traced to svc-users v21.

1. The version is marked as rejected in the artifact store or the manifest history, so the assembly pipeline no longer selects svc-users v21.
2. Manifest M43 is assembled with the same versions as M42 except svc-users v20, the last version that passed system tests. v20 passed alongside the earlier versions of its neighbours, so M43 is a combination that has not run before. The contract check (can-i-deploy, practice 5) runs first and confirms whether v20 is compatible with the current version of every other module.
3. System tests run on M43. Because M43 differs from M42 in one module, a green run makes it a release candidate. The release goes out without the change in v21, and only that change misses the train.
4. The owning team fixes the defect in v22 and adds a test that reproduces it at the lowest layer able to catch it, usually a unit, contract or module-level UI test. A defect that only the system tests can detect will surface late and inside a large batch the next time something similar breaks.
5. Once v22 passes its module pipeline, the next assembly selects it. Until then the assembler keeps svc-users at v20, even though v21 is newer.

A release manager or the rotating build cop (practice 9) chooses between pin-back and fix-forward by the rule in the section on pinning risk below, and the module team owns the fix. Keeping the two roles apart stops a defect in one module from holding up the other 99 while its fix is discussed.

When the defect cannot be narrowed to one module

The difference between the failing manifest and the last green one lists the suspects. Reverting one changed module at a time to its last known good version and re-running the system tests bisects the failure. Bisection is cheap when manifests are assembled often and consecutive manifests differ in a few modules each, and it is expensive in the current setup, where a two-hour candidate carries many teams' changes at once.

When pinning back drags other modules with it

Pinning one module back assumes the other modules still work with its previous version. A coordinated change breaks this assumption: if ui-orders v33 calls an endpoint that exists only in svc-users v21, rolling back svc-users forces ui-orders back too, and the contract check in step 2 reports the conflict. Expand-contract changes keep single-module pin-back possible by adding the new endpoint alongside the old one, migrating consumers, and removing the old endpoint only after every consumer has moved. Shipping new behaviour behind a feature flag that is off by default has the same effect, since the release can carry the code without depending on it.

The frontend mono repo needs a separate check. If the UI modules are built from one mono repo revision into a single bundle, ui-search cannot be pinned back on its own, and the remedy is a revert commit on main followed by a rebuild, which also picks up everything committed since the candidate. A release branch cut across repositories (practice 10) keeps that rebuild to the cut revisions plus the revert. Versioning the platform packages and each UI module separately, as published packages or as micro-frontends loaded through Module Federation, lets the manifest pin frontend modules one at a time in the same way as services. The way the frontend is packaged today decides which of these applies.

When pinning back is riskier than fixing forward

The other modules in M43 kept moving while svc-users went back to v20, so M43 can fail in ways that neither M41 nor M42 did. Another module may already rely on behaviour introduced in v21, and can-i-deploy sees only the recorded request and response formats, so a change in meaning passes it. A schema migration shipped with v21 has already run against the system-test database, and v20 may fail against the migrated schema unless the environment is reset or the migration was backward compatible. Records written by v21, or messages in a format it introduced, can break v20 in the same way. Each failure on M43 leads to another pin-back or a second module rolled back, and every step produces another combination nobody has tested.

Fixing forward costs the time to write the fix plus a full system-test run, and on main it also brings in every commit made since v21. When the defect is understood and the fix is small, the fixed candidate is still the more predictable of the two, since it differs from M42 in one known change. Both paths end in a system-test run, and the choice is about which candidate is more likely to pass it:

• Pin back when the change since v20 carries no schema migration, changes no contract that another module consumes, and has no other module in the manifest depending on it.
• Fix forward when any of these holds, or when the fix is already known and small. The fix goes onto a release branch (practice 10) so the candidate picks up only the fix and none of the other commits on main.

Recording, for each module version, whether it ships a migration or changes a consumed contract lets the build cop apply the rule from the manifest diff without asking the owning team.

In GoCD

Pros and cons for this system

Sources

Addresses distributed monolith in the build, big-bang integration at the fan-in, large batch size.

Sources Jez Humble and David Farley, Continuous Delivery (2010), for build once and promote; Uber, Rasmus Vestergaard and Kasper Munck, "Continuous deployment for large monorepos" (2024), for per-stage promotion of the newest passing version; Scaled Agile, "Agile Release Train"; Google Chromium project, "Chrome Release Cycle"; Mozilla, Firefox release calendar; Pete Hodgson on martinfowler.com, "Feature Toggles (aka Feature Flags)"; Module Federation documentation; Sam Newman, Building Microservices, 2nd edition (O'Reilly, 2021).