Is the Build Still Green After the Merge?

One-line summary

It is not rare for a branch to be green and then turn red after merging. This is because the branch's check looked at my change plus the trunk at that time, not at the trunk after merging. Narrowing this gap is what short-lived branches and server-side rules do.

Why this is needed

Trunk-based development is defined as "a source-control model in which developers collaborate on a single branch called the trunk and resist the pressure to create other long-lived development branches". Short-lived branches are used for code review and build checks but are premised on disappearing soon. Release branches too are cut from the trunk when needed and deleted some time after the release. The same property is required of the version name attached to a release. The semantic versioning convention states firmly that once a version is released, its content must not be modified, and if something needs fixing, it must come out as a new version. Leaving alone what has already gone out is what leaves a target to roll back to.

Why avoid long-lived branches? The longer a branch gets, the wider its distance from the trunk grows, and the widened distance is billed all at once at merge time. And what is scary on that bill is not the text conflicts. Text conflicts are reported by the tool. What is scary is semantic conflicts. If someone else changed on the trunk the behavior of a function I was using on my branch, the two changes are on different lines, so they merge cleanly and only the result is wrong. Version control tools cannot detect this.

From the feedback viewpoint the answer is the same. The more integration is postponed, the later you learn of a problem, and the later you learn, the more candidate causes there are. Kent Beck's sentence, "no code stays unintegrated for more than a couple of hours", states the target line most briefly.

Branch protection must be a server-side rule

If you leave a rule as a promise between people, it is not kept. It breaks on a busy day, during an incident response, in a newcomer's first week. So put the rule not on the side that pushes but on the side that receives.

Git itself has this distinction. Hooks are by default in $GIT_DIR/hooks, and do not come along with a clone. git init can copy a template, but the pre-commit hook I made does not appear by itself in a colleague's copy. So a local hook is a convenience device, not an enforcement means. Enforcement is done by server-side hooks.

• pre-receive runs once for the whole receiving operation. If it ends with a non-zero value, no reference is updated. The push is rejected as a whole.
• update runs once for each reference to be updated. If it is non-zero, only that reference is not updated.
• post-receive runs after the updates are done. It is for notifications and follow-up work.

A hosting service's branch protection is a feature in the same place. GitHub's protection rules include requiring a pull request before merging, required status checks, requiring conversation resolution, requiring signed commits, requiring linear history, requiring a merge queue, requiring deployments to succeed, locking the branch, restricting who can push, allowing force pushes, and allowing deletions.

There is one default you must know here. Protection rules by default do not apply to administrators. The documentation states that the constraints do not apply to people with repository administrator permission or bypass permission, and guides that you must turn on "do not allow bypassing the above settings" for them to apply to administrators too. This is where it happens that you turn the rule on and feel safe, yet the most dangerous change passes right by that rule.

The problem of green turning red after a merge

Even if you turn on required status checks, the gap remains. Branch A and branch B were each green relative to the trunk, but after A is merged first, the check result of B is no longer about the current trunk. If you merge it as it is, the trunk breaks.

There are two directions for a solution.

• Incorporate the latest trunk just before merging and check again. The requirement "keep the branch up to date" is this. It is simple, but if merges are frequent, another merge happens while you are rechecking, so you keep falling behind.
• Use a merge queue. It lines up the things to be merged, creates the merged state in advance and checks it, and puts only what passes into the trunk. It bundles several items and checks them at once, and if it fails, removes only the culprit and tries again, so even if the line is long, the number of checks does not explode. GitLab's merge trains are also a feature that solves the same problem.

Feature flags instead of long branches

"This feature is not finished yet so it cannot be merged" is the common reason for a long-lived branch. The answer is to put the unfinished thing into the trunk but not turn it on. Feature flags and branch by abstraction are the ways. The unfinished code is in the trunk but not visible to users, so you can integrate every day while deciding the release time separately.

In exchange, flags have a cost. As flags increase, the combinations that actually run increase, and paths that tests do not cover appear. Deciding, when you create a flag, at the same time the point at which you will delete it is the only management method that works.

What it looks like in the field

• Half of the "I pushed directly because it was urgent" incidents come from an administrator account. Because the bypass permission was not turned off.
• The name of a required status check was changed but the name in the protection rule stayed the same, so weeks pass in a state that requires no check at all. There is no way to confirm other than trying it directly to see whether the rule actually blocks.
• If you require linear history, reverting and binary search get easier. Narrowing down the culprit commit in a history tangled with merge commits is noticeably more troublesome.
• If you start measuring branch lifetime, it usually turns out that branches that last several days create most of the problems.

References

• Trunk-based development: https://trunkbaseddevelopment.com/
• Branch protection: https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches
• Merge queue: https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/configuring-pull-request-merges/managing-a-merge-queue
• GitLab merge trains: https://docs.gitlab.com/ci/pipelines/merge_trains/
• githooks: https://git-scm.com/docs/githooks
• Continuous integration: https://martinfowler.com/articles/continuousIntegration.html
• Branch by abstraction: https://martinfowler.com/bliki/BranchByAbstraction.html
• Semantic versioning: https://semver.org/

What you will do in the next lab

You create a bare repository locally and set up the receiving-side rules yourself. You use pre-receive and update hooks to block direct pushes to the protected branch, and confirm by exit code the difference between rejecting only a specific reference and rejecting the entire push. Then you make a local hook and see with your own eyes that the hook does not come along to the cloned copy. You imitate the required check with a hook that verifies a check result attached to the commit, and reproduce a semantic conflict with an example that breaks when two branches that each passed are merged. At the end you write, as a script, a queue imitation that incorporates the trunk and rechecks before merging, and count how the number of checks changes when the line gets long.