Reproducible Builds Come First

One-line summary

CI is not a robot that presses the build button for you; it is a reproducibility mechanism that forces the same input to produce the same output no matter when or where it runs.

Why this is needed

Accidents from the era of building by hand always had the same shape. It works on my laptop but not on the server, and the image built yesterday differs from the one built today, and nobody can explain what changed. The cause is usually "something that was not pinned". Dependency versions were not pinned, actions and base images were referenced only by tag, or the artifact name does not record what it was built from.

A tag is a label that a person can move. In the tj-actions/changed-files supply-chain attack, the attacker re-pointed the action's tags to a malicious commit. They did not break into the repository; they only moved the labels, and countless pipelines that referenced those tags ran the malicious code as is. So the rule is short. A tag can be changed maliciously, but a commit SHA cannot be changed. Pin actions and images not by tag but by commit SHA or digest.

If you use latest in production, you lose two things at once. It becomes impossible to trace which version was deployed, and when an incident occurs there is no target to roll back to. So the tag used for deployment must be immutable. It must be a value, like a commit SHA or a semantic version, that never points to something else once it is set.

How it works

A reproducible build is made on four axes.

1. Pin the input. Always commit the lock file, and in CI use commands such as npm ci, --frozen-lockfile and -lockfile=readonly that install exactly what is written without updating the lock file. The ordinary install family quietly updates the lock file and produces a different dependency tree on every run.
2. Identify the output. Put an immutable value that changes with each commit into artifact names and image tags. latest is just an alias layered on top of that, not an identifier.
3. Cache. Put the lock file hash in the cache key so that when dependencies change, the key changes by itself and the cache is invalidated automatically. The standard form is ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }} with a restore-keys prefix chain attached. An effective cache key strategy cuts build time by 50–70%. One team's measurement showed the average CI build falling from 28 minutes to 8 minutes, a 71% reduction. However, the limit is 10 GB per cache store, and when it is exceeded the oldest entries are evicted, so if you split the scope finely to the commit level, the caches push each other out and the hit rate actually drops. The cache is also a trust boundary. If a fork PR injects a malicious dependency into the cache, cache poisoning becomes possible, where later builds use that cache as it is.
4. Failure handling. The default of fail-fast in a matrix build is true, so when one fails, the rest are cancelled. Change it to true if fast feedback matters, and to false if checking full compatibility matters. If you do not set concurrency control, consecutive pushes cause several deployment jobs to run at the same time, leading to accidents in which rollback becomes complicated.

What it looks like in the field

The sentence that comes up most often in incident retrospectives is "what exactly was deployed at that time?". If the image tag is only latest, nobody can answer this question. Conversely, if the commit hash is embedded in the tag, the answer comes out within 5 minutes from the registry and the Git log alone.

Teams that manage by numbers set their targets like this. Lead time of 1 hour or less, deployment frequency of 10 or more per day, change failure rate of 5% or less, MTTR of 30 minutes or less, pipeline run of 15 minutes or less, coverage of 80% or more. Among these, once the pipeline run time starts to exceed 15 minutes, people stop waiting for the CI result and go off to do other work, and the feedback loop collapses entirely.

A build that gives the same result for the same input

When you first attach CI, "it worked on my computer" just turns into "it worked in CI". Eliminating that gap is the real goal of build automation.

If you do not pin versions, yesterday's success does not guarantee today. This is the difference between npm install and npm ci. The former fetches the latest within the range of package.json, so it updates the lock file. The latter fails outright if it differs from the lock file. In CI, the side that fails is the right one. Python's pip install -r also gets the same property only when you use a requirements file with hashes together with --require-hashes.

The tag of the base image is also a version. FROM python:3.12 may point to something different tomorrow. If you nail it down with a digest, it is reproducible, but you stop receiving security updates. So pinning by digest and automating the updates must go together. If you only pin and do not update, the vulnerability list grows long a few months later.

A cache is a benefit only when it is accurate. Put the hash of the lock file in the key.

key: deps-${{ runner.os }}-${{ hashFiles('**/package-lock.json') }}
restore-keys: deps-${{ runner.os }}-

If the key is too loose, you get builds that pass with stale dependencies. This is worse than having no cache: because something that should fail succeeds.

Build the artifact only once. If you build once when deploying to the development environment and build again when deploying to production, what you tested and what you deployed become different things. Build once and push it to the registry, and later stages promote the same digest. A tag is merely a label for humans to read, and what guarantees it is the same thing is the digest.

Keep evidence of a failed build. If you keep only the log, you end up running the same build again to reproduce it. If you upload test reports, coverage and generated configuration files as artifacts, you can see the state at the very moment of failure as it was. The things that disappear when you run it again are usually the cause.

What you will do in the next lab

This Pod has neither Jenkins nor a GitHub Actions runner. So you build the three stages build → test → package yourself with a shell script. What matters is not the vendor but the mechanism. You make just one artifact named by the source hash, reuse it when you run again with the same input, and at the end attach the hash tag and latest to the same image, to see with your own eyes what is an immutable identifier and what is a moving alias.