TT Lab
Get started
Learn Learning paths Courses

CI/CD Pipelines

Two builds from the same source, two different digests

Continue in TT Lab

Goal

You find the nondeterminism of build artifacts at the byte level, make two builds identical down to the bytes with SOURCE_DATE_EPOCH and tar options, and then set up a verification script and a gate that protect it.

Why it matters

Using a hash as a name and that hash being reproducible are different problems. If you build the same commit twice and the digests of the artifacts differ, nobody can prove that the deployed bytes came from that source — when an incident occurs, the place to go back to disappears, the cache misses every time, and a signature vouches only for 'these bytes' and cannot vouch for 'this source'. Nondeterminism almost always comes from a few common places: file modification times, the time and name engraved in the compression header, the directory read order, the uid and gid of the build user, and the build time written into the artifact. If you pin those places one by one, the build becomes a function of its input, and from then on a single digest becomes a name that points to the entire source.

Steps

  1. Create a source tree at /root/repro/src — src/app.py, src/lib/util.py, src/conf/app.conf, src/README.md, and src/VERSION (the content is the single line 1.4.0). Then create /root/repro/naive-build.sh <출력파일> [소스디렉터리] (output file, optional source directory). The default source directory is /root/repro/src, and it copies the source into a temporary staging directory and bundles it with tar -czf <출력파일> -C <스테이징> .. Give it execute permission and run it twice with at least 1 second in between to create /root/repro/out/naive-1.tar.gz and /root/repro/out/naive-2.tar.gz, then, in /root/repro, save the output of sha256sum out/naive-1.tar.gz out/naive-2.tar.gz as it is to /root/repro/naive.sha256. The two values must differ.
  2. Leave evidence of why the two artifacts differ. First, in /root/repro/tar-diff.txt, save as it is the output of diff <(tar --full-time -tvf /root/repro/out/naive-1.tar.gz) <(tar --full-time -tvf /root/repro/out/naive-2.tar.gz) (it must not be empty). Second, check the gzip header — into /root/repro/out/inner-1.tar save the result of gzip -dc /root/repro/out/naive-1.tar.gz, and then compress the same file in two ways to create /root/repro/out/hdr-keep.gz (the default compression that keeps the original name and time) and /root/repro/out/hdr-none.gz (compression that leaves out the name and time). Third, write two lines in /root/repro/gzip-header.txt in the form <파일이름> <FLG> <MTIME> (file name, FLG, MTIME). The file names are in the order hdr-keep.gz, hdr-none.gz, FLG is the 4th byte (byte 3 if counted from 0) as two hexadecimal digits, and MTIME is the 4 bytes starting from the 5th byte as a little-endian unsigned decimal.
  3. Create /root/repro/build.sh <출력파일> [소스디렉터리] (output file, optional source directory). The default source directory is /root/repro/src. Copy to staging like naive-build.sh, but use SOURCE_DATE_EPOCH (the default is 1735689600 if not given from outside), bundle the tar sorted by name, pin the modification time of every member to that epoch and the owner and group to the number 0, and compress so that the original name and time do not remain in the gzip header. Give it execute permission, run it twice with at least 1 second in between to create /root/repro/out/det-1.tar.gz and /root/repro/out/det-2.tar.gz, then, in /root/repro, save the output of sha256sum out/det-1.tar.gz out/det-2.tar.gz to /root/repro/det.sha256. The two values must be the same.
  4. Create /root/repro/verify-repro.sh <빌드스크립트> (build script). It runs the script it receives twice in a temporary directory (with at least 1 second in between) and compares the sha256 of the artifacts. If they are the same, it prints one line SAME <다이제스트> and ends with 0; if different, it prints DIFF <다이제스트1> <다이제스트2> on the first line, then shows the difference in tar -tvf of the two artifacts, and ends with 1; and if the build itself fails, it prints a line whose first line starts with ERROR and ends with 2. It does not leave temporary artifacts in the working directory. After giving it execute permission, save the result of running it on /root/repro/build.sh to /root/repro/verify-good.txt and the result of running it on /root/repro/naive-build.sh to /root/repro/verify-naive.txt.
  5. Create /root/repro/bad-build.sh <출력파일> [소스디렉터리] (output file, optional source directory). Bundle deterministically exactly like build.sh, but just before bundling, put one more file BUILDINFO into the staging directory. The content is the three lines built_at=<나노초까지 찍은 UTC 시각> (UTC time printed down to nanoseconds), built_on=<호스트 이름> (host name) and nonce=<난수> (random number). Give it execute permission and save the output of /root/repro/verify-repro.sh /root/repro/bad-build.sh to /root/repro/verify-bad.txt. The first line must start with DIFF. Do not fix this script; leave it as it is — you will use it as a counterexample in the next step and the last step.
  6. Fix /root/repro/build.sh to put BUILDINFO into the staging just before bundling, but fill it with values that come only from the input. It is three lines and the order is as given — version=<소스디렉터리의 VERSION 내용> (the content of VERSION in the source directory), source_tree_sha256=<소스 트리 해시> (the source tree hash), source_date_epoch=<쓰고 있는 에포크> (the epoch in use). The source tree hash is the first 64 digits of the value from running LC_ALL=C find . -type f -print0 | sort -z | xargs -0 sha256sum | sha256sum in the source directory. After fixing it, confirm that /root/repro/verify-repro.sh /root/repro/build.sh still gives SAME, and save the BUILDINFO taken out of the newly built artifact as it is to /root/repro/buildinfo.txt (you can build once as /root/repro/out/prov.tar.gz and then use tar -xOf /root/repro/out/prov.tar.gz ./BUILDINFO).
  7. With skopeo, read /opt/images/alpine_3.20.tar (an oci-archive). Save the raw manifest as it is to /root/repro/manifest.json, and write the sha256 of that file in /root/repro/manifest.sha256 as one line sha256:<64자리 16진수> (64 hexadecimal digits). Then copy the same archive twice into the OCI layout /root/repro/oci/alpine to create the tags v1 and v2 (the digests of the two tags must be the same). Finally, write three lines in /root/repro/layer-recompress.txt in the form <이름> <64자리 16진수> (name, 64 hexadecimal digits) — the first line original is the sha256 of the layer blob file the manifest points to, the second line uncompressed is the sha256 of the bytes obtained by unpacking that blob with gzip -dc, and the third line recompressed is the sha256 of the bytes obtained by compressing the unpacked bytes again with gzip -n -9. original and recompressed must differ.
  8. Create /root/repro/repro-gate.sh <빌드스크립트> <보고서파일> (build script, report file). It runs the build twice (with at least 1 second in between); if they are the same, it prints REPRODUCIBLE <다이제스트> on the screen and on the first line of the report and ends with 0; if different, it prints DIFFERENT <다이제스트1> <다이제스트2> on the screen and on the first line of the report and ends with 3; and if the build fails, it prints a line starting with ERROR and ends with 1. When they differ, the report lists after the first line the difference in tar -tvf of the two artifacts, the content difference from unpacking and comparing them (diff -ru), and the position of the first differing byte. If the parent directory of the report file does not exist, it must create it. After giving it execute permission, run it with /root/repro/build.sh and leave the report at /root/repro/report/good.txt, and run it with /root/repro/bad-build.sh and leave the report at /root/repro/report/bad.txt.

Notes

The source is unchanged but the artifact's digest changed

Create a source tree at /root/repro/src — src/app.py, src/lib/util.py, src/conf/app.conf, src/README.md, and src/VERSION (the content is the single line 1.4.0). Then create /root/repro/naive-build.sh <출력파일> [소스디렉터리] (output file, optional source directory). The default source directory is /root/repro/src, and it copies the source into a temporary staging directory and bundles it with tar -czf <출력파일> -C <스테이징> .. Give it execute permission and run it twice with at least 1 second in between to create /root/repro/out/naive-1.tar.gz and /root/repro/out/naive-2.tar.gz, then, in /root/repro, save the output of sha256sum out/naive-1.tar.gz out/naive-2.tar.gz as it is to /root/repro/naive.sha256. The two values must differ.

Create the temporary directory with mktemp -d and delete it with trap ... EXIT. If you use cp -r for copying, the modification times of the copy become 'now' — where that value gets recorded is the starting point of this lab. The reason for the 1-second gap is that the resolution of modification times is 1 second.

Digging into the differing bytes turns up the modification times and the gzip header

Leave evidence of why the two artifacts differ. First, in /root/repro/tar-diff.txt, save as it is the output of diff <(tar --full-time -tvf /root/repro/out/naive-1.tar.gz) <(tar --full-time -tvf /root/repro/out/naive-2.tar.gz) (it must not be empty). Second, check the gzip header — into /root/repro/out/inner-1.tar save the result of gzip -dc /root/repro/out/naive-1.tar.gz, and then compress the same file in two ways to create /root/repro/out/hdr-keep.gz (the default compression that keeps the original name and time) and /root/repro/out/hdr-none.gz (compression that leaves out the name and time). Third, write two lines in /root/repro/gzip-header.txt in the form <파일이름> <FLG> <MTIME> (file name, FLG, MTIME). The file names are in the order hdr-keep.gz, hdr-none.gz, FLG is the 4th byte (byte 3 if counted from 0) as two hexadecimal digits, and MTIME is the 4 bytes starting from the 5th byte as a little-endian unsigned decimal.

The default output of tar -tvf shows only up to the minute, so a 1-second difference is buried — that is why --full-time is needed. od -An -tx1 -j3 -N1 <파일> turns the FLG one byte, and od -An -tu4 -j4 -N4 <파일> the MTIME 4 bytes, into numbers people can read. Delete the spaces with tr -d ' '. When gzip keeps the original name, it sets one bit in FLG. Recall the header diagram of RFC 1952.

Once the order, time and owner were pinned, it became the same down to the bytes

Create /root/repro/build.sh <출력파일> [소스디렉터리] (output file, optional source directory). The default source directory is /root/repro/src. Copy to staging like naive-build.sh, but use SOURCE_DATE_EPOCH (the default is 1735689600 if not given from outside), bundle the tar sorted by name, pin the modification time of every member to that epoch and the owner and group to the number 0, and compress so that the original name and time do not remain in the gzip header. Give it execute permission, run it twice with at least 1 second in between to create /root/repro/out/det-1.tar.gz and /root/repro/out/det-2.tar.gz, then, in /root/repro, save the output of sha256sum out/det-1.tar.gz out/det-2.tar.gz to /root/repro/det.sha256. The two values must be the same.

Look up GNU tar's --sort, --mtime, --owner, --group and --numeric-owner. tar's -z calls gzip over stdin, but to give the option that leaves the name and time out of the header yourself, you must pipe the tar output into gzip. Sorting can differ with the locale, so it is safer to attach LC_ALL=C.

Hand the question of whether it reproduces to a script

Create /root/repro/verify-repro.sh <빌드스크립트> (build script). It runs the script it receives twice in a temporary directory (with at least 1 second in between) and compares the sha256 of the artifacts. If they are the same, it prints one line SAME <다이제스트> and ends with 0; if different, it prints DIFF <다이제스트1> <다이제스트2> on the first line, then shows the difference in tar -tvf of the two artifacts, and ends with 1; and if the build itself fails, it prints a line whose first line starts with ERROR and ends with 2. It does not leave temporary artifacts in the working directory. After giving it execute permission, save the result of running it on /root/repro/build.sh to /root/repro/verify-good.txt and the result of running it on /root/repro/naive-build.sh to /root/repro/verify-naive.txt.

The contract of the build script was set in an earlier step — the first argument is the output file path. If you turn on set -e, the script dies first the moment diff finds a difference. The listing may be the same while the bytes differ, so decide in advance what to show in that case too.

When the build time was written into the artifact, the verification script caught it

Create /root/repro/bad-build.sh <출력파일> [소스디렉터리] (output file, optional source directory). Bundle deterministically exactly like build.sh, but just before bundling, put one more file BUILDINFO into the staging directory. The content is the three lines built_at=<나노초까지 찍은 UTC 시각> (UTC time printed down to nanoseconds), built_on=<호스트 이름> (host name) and nonce=<난수> (random number). Give it execute permission and save the output of /root/repro/verify-repro.sh /root/repro/bad-build.sh to /root/repro/verify-bad.txt. The first line must start with DIFF. Do not fix this script; leave it as it is — you will use it as a counterexample in the next step and the last step.

date -u +%FT%T.%NZ prints down to nanoseconds. If you print only in seconds, the two builds may end in the same second and pass by chance. Do not forget that the artifact must still be a normal tar.gz that opens — a broken build and a non-reproducible build are different problems.

Make the provenance information only from the input and regain reproducibility

Fix /root/repro/build.sh to put BUILDINFO into the staging just before bundling, but fill it with values that come only from the input. It is three lines and the order is as given — version=<소스디렉터리의 VERSION 내용> (the content of VERSION in the source directory), source_tree_sha256=<소스 트리 해시> (the source tree hash), source_date_epoch=<쓰고 있는 에포크> (the epoch in use). The source tree hash is the first 64 digits of the value from running LC_ALL=C find . -type f -print0 | sort -z | xargs -0 sha256sum | sha256sum in the source directory. After fixing it, confirm that /root/repro/verify-repro.sh /root/repro/build.sh still gives SAME, and save the BUILDINFO taken out of the newly built artifact as it is to /root/repro/buildinfo.txt (you can build once as /root/repro/out/prov.tar.gz and then use tar -xOf /root/repro/out/prov.tar.gz ./BUILDINFO).

The hash must be calculated not from the staging but from the source directory — if BUILDINFO itself gets mixed into the hash, it becomes impossible to tell what the value is about. Values that come along with the input, such as a commit hash, may be put in, but the build time, host name and random numbers may not. The grader also builds from a copy with one character of the source changed and checks whether the hash changes along with it.

A digest vouches for the bytes, not for the build

With skopeo, read /opt/images/alpine_3.20.tar (an oci-archive). Save the raw manifest as it is to /root/repro/manifest.json, and write the sha256 of that file in /root/repro/manifest.sha256 as one line sha256:<64자리 16진수> (64 hexadecimal digits). Then copy the same archive twice into the OCI layout /root/repro/oci/alpine to create the tags v1 and v2 (the digests of the two tags must be the same). Finally, write three lines in /root/repro/layer-recompress.txt in the form <이름> <64자리 16진수> (name, 64 hexadecimal digits) — the first line original is the sha256 of the layer blob file the manifest points to, the second line uncompressed is the sha256 of the bytes obtained by unpacking that blob with gzip -dc, and the third line recompressed is the sha256 of the bytes obtained by compressing the unpacked bytes again with gzip -n -9. original and recompressed must differ.

skopeo inspect --raw oci-archive:<파일> outputs the raw manifest as it is — the digest is exactly the sha256 of those bytes. Copying is skopeo copy --insecure-policy oci-archive:<파일> oci:<디렉터리>:<태그>, and the parent directory must exist first. Inside the layout, the blob file name is that blob's own digest. The architecture differs per machine, so do not memorize the values; calculate them on the spot.

Stand up a non-reproducible build in the pipeline

Create /root/repro/repro-gate.sh <빌드스크립트> <보고서파일> (build script, report file). It runs the build twice (with at least 1 second in between); if they are the same, it prints REPRODUCIBLE <다이제스트> on the screen and on the first line of the report and ends with 0; if different, it prints DIFFERENT <다이제스트1> <다이제스트2> on the screen and on the first line of the report and ends with 3; and if the build fails, it prints a line starting with ERROR and ends with 1. When they differ, the report lists after the first line the difference in tar -tvf of the two artifacts, the content difference from unpacking and comparing them (diff -ru), and the position of the first differing byte. If the parent directory of the report file does not exist, it must create it. After giving it execute permission, run it with /root/repro/build.sh and leave the report at /root/repro/report/good.txt, and run it with /root/repro/bad-build.sh and leave the report at /root/repro/report/bad.txt.

Do not use the verify-repro.sh of the earlier step as it is; make this script itself handle the three outcomes and the report — a gate is useful only if it leaves what differs when it fails. To see the content difference, you must unpack the two artifacts into temporary directories each. The grader tests this gate with three build scripts it builds itself: a normal one, a nondeterministic one and a broken one.