TT Lab
Get started
Learn Learning paths Courses

CI/CD Pipelines

The tag never changed, yet production was running a different image

Continue in TT Lab

Goal

You move an artifact built once between environments by hand, without rebuilding it. You promote by pointing at the digest, set up a checker that catches a tag having moved and a gate that blocks anything not on record, carry out a rollback as a re-promotion of an old digest, and leave in one file what is where.

Why it matters

If you rebuild for each environment, each environment gets something different. The base image tag moves, dependency patches go up, and the tool versions of the build machines differ. So a test that passed in the development environment tells you nothing about the production artifact — because what you tested and what you deployed are different things. If you build once and only move it, this variable disappears, but that principle leaks out through the name that points to what you are moving. A tag is a label a person attached, so it can be made to point to something else at any time, and at that moment what you verified yesterday and what is deployed today split apart. A digest is an address computed from the content, so it cannot be moved. So the rule in the field comes down to one: tags for reading, digests for pointing. Rollback stands on the same rule. As long as the old digest remains in the registry, rollback is not rebuilding but pointing at the old address again, and you can prove in bytes that what you rolled back to is the same thing as before.

Steps

  1. Create a practice git repository at /root/promote/app — put in the files main.py, orders.conf and README.md, stack two or more commits, and then attach an unannotated tag v1.2.0 to the last commit. Next create /root/promote/release-name.sh <저장소경로> (the placeholder stands for the repository path). It is a script that extracts a one-line release name from the repository and prints it to standard output. The rule is <가장 가까운 태그>-<그 태그 이후 커밋 수>-g<커밋 7자리> (nearest tag, number of commits since that tag, 7 characters of the commit), and if there are uncommitted changes, append -dirty at the end. For a repository with no tags at all, write v0.0.0 in the tag slot and the total commit count in the commit count slot. If it is given a path that is not a git repository, print nothing to standard output and end with a non-zero code. Give it execute permission and save the result of running it on /root/promote/app as one line in /root/promote/release.txt.
  2. You imitate per-environment registries with OCI image layout directories. Under /root/promote/reg, put dev, staging and prod. Suppose that what the build machine produced as the artifact of this release is /opt/images/busybox_1.36.tar (an oci-archive). Put it just once into the development layout /root/promote/reg/dev with the release name attached as the tag (the release name is the value in /root/promote/release.txt). Then create /root/promote/resolve.sh <레이아웃> <태그> (layout, tag) — it prints the digest that tag currently points to as a single line sha256:<64자리 16진수> (64 hexadecimal digits) and ends with 0. If there is no such tag, it prints nothing to standard output and ends with 3, and if given a path that is not a layout, ends with 2. After giving it execute permission, save that digest as one line in /root/promote/dev-digest.txt. Finally, create /root/promote/releases.tsv and write in the first line <다이제스트><탭><판이름><탭><커밋40자리> (digest, tab, release name, tab, 40-character commit). The commit is the full hash of the commit that made that release.
  3. You create a situation in which someone pushed a different artifact into the development environment under the same release name tag. Copy /opt/images/nginx_1.27-alpine.tar to the same tag (the release name) in /root/promote/reg/dev. Then write exactly three lines in /root/promote/tag-moved.txt — the first line before <다이제스트> (digest) is the value the tag pointed to before the push, the second line after <다이제스트> is the value it points to now, and the third line, orphan yes or orphan no, is whether the old digest's entry still remains in index.json (even if it lost its label, yes if the entry and blob remain). Do not memorize and write the values; check them on the spot and write them.
  4. Create /root/promote/promote.sh <원본레이아웃> <대상레이아웃> <다이제스트> <대상태그> (source layout, target layout, digest, target tag). It finds that digest in the source and puts it into the target layout with that tag. It does not rebuild — after moving, the manifest bytes of the target must not differ by a single byte from the source. If the source has no entry or blob for that digest, it prints NOTFOUND <다이제스트> on the first line and ends with 3; on success it prints PROMOTED <다이제스트> and ends with 0; and for any other fault (the source is not a layout, it is not in digest form, the copy failed) it prints a line starting with ERROR and ends with 2. If the target layout does not exist yet, it must create it, and it must also be able to move a digest that has no label (one that lost its tag). After giving it execute permission, move the digest in /root/promote/dev-digest.txt from /root/promote/reg/dev to /root/promote/reg/staging with the release name tag attached, and save the digest read back from staging after moving as one line in /root/promote/staging-digest.txt. It must equal the value written for the development environment.
  5. Create /root/promote/tag-drift.sh <레이아웃> <태그> <기대다이제스트> (layout, tag, expected digest). It compares the value that tag points to now with the expected value. If equal, it prints OK <다이제스트> and ends with 0; if different, prints MOVED <기대> <실제> (expected, actual) and ends with 3; if there is no such tag at all, prints MISSING <태그> and ends with 4; and if the path is not a layout, prints a line starting with ERROR and ends with 2. After giving it execute permission, using /root/promote/dev-digest.txt as the expected value, save the result of running it on the development environment to /root/promote/drift-dev.txt and the result on staging to /root/promote/drift-staging.txt. The development environment must give MOVED and staging OK.
  6. Create /root/promote/approved.txt and write on a single line the digest that finished confirmation in staging (the very value written for the development environment). Comment lines starting with # and blank lines are allowed. Then create /root/promote/promote-gate.sh <승인목록> <원본레이아웃> <대상레이아웃> <다이제스트> <대상태그> (approval list, source layout, target layout, digest, target tag). It promotes only when that digest is written as an entire line in the approval list, and if it is not, it prints REFUSED <다이제스트> and ends with 5 — at this time the target layout must not change by a single byte. It passes the promotion on to the promote.sh of an earlier step and relays its output and exit code as they are (PROMOTED 0 · NOTFOUND 3 · ERROR 2). If the approval list file does not exist, it prints a line starting with ERROR and ends with 2. After giving it execute permission, promote the approved digest from /root/promote/reg/staging to /root/promote/reg/prod with the tag current attached. Finally, try the same promotion with the digest that the development tag points to now (the unapproved one) and save its output in /root/promote/gate-refused.txt. The first line must start with REFUSED.
  7. You put out one more release and roll it back. Stack one more commit on /root/promote/app and attach the tag v1.3.0, then extract the new release name with release-name.sh. Taking /opt/images/alpine_3.20.tar as that release's artifact, put it into /root/promote/reg/dev under the new release name tag, append a second line to /root/promote/releases.tsv in the same format, and append that digest as a line to /root/promote/approved.txt too. Then, with the gate, promote the new release from /root/promote/reg/dev to the /root/promote/reg/prod tag current, and right after that roll back by re-promoting the old release's digest under the same tag. Finally write three lines in /root/promote/rollback.txt — before <되돌리기 직전 current 가 가리키던 다이제스트> (the digest that the current tag pointed to just before the rollback), after <되돌린 뒤의 다이제스트> (the digest after the rollback), and identical yes or identical no (the result of checking with cmp whether the bytes of the rolled-back manifest are the same as those of the old release in the development environment).
  8. Create /root/promote/ledger.sh <레지스트리루트> <릴리스표> <출력파일> (registry root, release table, output file). Treat each directory under the registry root as one environment (only those with an index.json inside), collect only the entries that have labels and write a tab-separated table to the output file. The first line is the header env<탭>tag<탭>digest<탭>release<탭>commit (tab-separated), and the following lines are <환경><탭><태그><탭><다이제스트><탭><판이름><탭><커밋> (environment, tag, digest, release name, commit). Look up the release name and commit by digest in the release table, and if absent write - in both cells. Sort the body lines with LC_ALL=C sort in the order environment, tag, digest. On standard output, print one line LEDGER <본문 줄 수> (the number of body lines) and end with 0. If the root does not exist or the release table does not exist, print a line starting with ERROR and end with 2. If the parent directory of the output file does not exist, create it. After giving it execute permission, run it with /root/promote/reg and /root/promote/releases.tsv and leave /root/promote/ledger.tsv.

Notes

When people started naming releases themselves, nobody knew which commit it was

Create a practice git repository at /root/promote/app — put in the files main.py, orders.conf and README.md, stack two or more commits, and then attach an unannotated tag v1.2.0 to the last commit. Next create /root/promote/release-name.sh <저장소경로> (the placeholder stands for the repository path). It is a script that extracts a one-line release name from the repository and prints it to standard output. The rule is <가장 가까운 태그>-<그 태그 이후 커밋 수>-g<커밋 7자리> (nearest tag, number of commits since that tag, 7 characters of the commit), and if there are uncommitted changes, append -dirty at the end. For a repository with no tags at all, write v0.0.0 in the tag slot and the total commit count in the commit count slot. If it is given a path that is not a git repository, print nothing to standard output and end with a non-zero code. Give it execute permission and save the result of running it on /root/promote/app as one line in /root/promote/release.txt.

If you give git describe the options --tags --long --abbrev=7, you get a string exactly as the rule says, except when there are no tags. Also look at --dirty. If there are no tags, describe itself fails, so you must set up a separate branch for that case. If you turn on set -e, the script dies first at that failure. The commit count is git rev-list --count, and the short hash is git rev-parse --short=7.

It built the artifact only once and wrote down its digest

You imitate per-environment registries with OCI image layout directories. Under /root/promote/reg, put dev, staging and prod. Suppose that what the build machine produced as the artifact of this release is /opt/images/busybox_1.36.tar (an oci-archive). Put it just once into the development layout /root/promote/reg/dev with the release name attached as the tag (the release name is the value in /root/promote/release.txt). Then create /root/promote/resolve.sh <레이아웃> <태그> (layout, tag) — it prints the digest that tag currently points to as a single line sha256:<64자리 16진수> (64 hexadecimal digits) and ends with 0. If there is no such tag, it prints nothing to standard output and ends with 3, and if given a path that is not a layout, ends with 2. After giving it execute permission, save that digest as one line in /root/promote/dev-digest.txt. Finally, create /root/promote/releases.tsv and write in the first line <다이제스트><탭><판이름><탭><커밋40자리> (digest, tab, release name, tab, 40-character commit). The commit is the full hash of the commit that made that release.

skopeo copy --insecure-policy oci-archive:<파일> oci:<디렉터리>:<태그> creates the layout — the parent directory must exist first. The layout's index.json is a list of manifest descriptors, and the tag goes into the annotations, under org.opencontainers.image.ref.name. With jq, you can pull out the .digest of the entry whose annotation equals the tag. Be careful not to let through the null that jq outputs when there is no value. Put in tabs with printf '%s\t%s\t%s\n'.

When a different image was pushed into the same tag, only the label moved

You create a situation in which someone pushed a different artifact into the development environment under the same release name tag. Copy /opt/images/nginx_1.27-alpine.tar to the same tag (the release name) in /root/promote/reg/dev. Then write exactly three lines in /root/promote/tag-moved.txt — the first line before <다이제스트> (digest) is the value the tag pointed to before the push, the second line after <다이제스트> is the value it points to now, and the third line, orphan yes or orphan no, is whether the old digest's entry still remains in index.json (even if it lost its label, yes if the entry and blob remain). Do not memorize and write the values; check them on the spot and write them.

Look at jq . reg/dev/index.json once before and once after the push. Here it becomes clear that a tag is merely a label attached to an entry and the substance is blobs/sha256/<다이제스트>. Compare the annotations with your eyes to see how the old entry changed. The before value was already written in a file in an earlier step.

It moved to staging by pointing at the digest, not the tag

Create /root/promote/promote.sh <원본레이아웃> <대상레이아웃> <다이제스트> <대상태그> (source layout, target layout, digest, target tag). It finds that digest in the source and puts it into the target layout with that tag. It does not rebuild — after moving, the manifest bytes of the target must not differ by a single byte from the source. If the source has no entry or blob for that digest, it prints NOTFOUND <다이제스트> on the first line and ends with 3; on success it prints PROMOTED <다이제스트> and ends with 0; and for any other fault (the source is not a layout, it is not in digest form, the copy failed) it prints a line starting with ERROR and ends with 2. If the target layout does not exist yet, it must create it, and it must also be able to move a digest that has no label (one that lost its tag). After giving it execute permission, move the digest in /root/promote/dev-digest.txt from /root/promote/reg/dev to /root/promote/reg/staging with the release name tag attached, and save the digest read back from staging after moving as one line in /root/promote/staging-digest.txt. It must equal the value written for the development environment.

Measured: this skopeo (1.13.3)'s oci: transport does not accept digest references — it rejects both oci:<디렉터리>@sha256:... and oci:<디렉터리>:sha256:.... But the layout's index.json is just a list of descriptors, and attaching a label to one descriptor can be done with jq. If you make an index.json containing only that one digest in a temporary directory and make blobs point to the source, skopeo reads that temporary layout as the source. That pointing is resolved inside the temporary directory, so the source path must be an absolute path. Clean up the temporary directory with mktemp -d and trap ... EXIT.

Once the checker was attached, it became clear that the development tag had moved

Create /root/promote/tag-drift.sh <레이아웃> <태그> <기대다이제스트> (layout, tag, expected digest). It compares the value that tag points to now with the expected value. If equal, it prints OK <다이제스트> and ends with 0; if different, prints MOVED <기대> <실제> (expected, actual) and ends with 3; if there is no such tag at all, prints MISSING <태그> and ends with 4; and if the path is not a layout, prints a line starting with ERROR and ends with 2. After giving it execute permission, using /root/promote/dev-digest.txt as the expected value, save the result of running it on the development environment to /root/promote/drift-dev.txt and the result on staging to /root/promote/drift-staging.txt. The development environment must give MOVED and staging OK.

Reuse the resolve.sh you made in an earlier step — if you implement the same thing twice, the two places fall out of sync. For a script to call the script next to it, use $(dirname "$0"). The case where the tag does not exist and the case where the tag points to a different value have different causes and different responses, so do not mix up the exit codes. The grader tests all three outcomes with a layout it builds itself.

A digest not on record stopped at the production threshold

Create /root/promote/approved.txt and write on a single line the digest that finished confirmation in staging (the very value written for the development environment). Comment lines starting with # and blank lines are allowed. Then create /root/promote/promote-gate.sh <승인목록> <원본레이아웃> <대상레이아웃> <다이제스트> <대상태그> (approval list, source layout, target layout, digest, target tag). It promotes only when that digest is written as an entire line in the approval list, and if it is not, it prints REFUSED <다이제스트> and ends with 5 — at this time the target layout must not change by a single byte. It passes the promotion on to the promote.sh of an earlier step and relays its output and exit code as they are (PROMOTED 0 · NOTFOUND 3 · ERROR 2). If the approval list file does not exist, it prints a line starting with ERROR and ends with 2. After giving it execute permission, promote the approved digest from /root/promote/reg/staging to /root/promote/reg/prod with the tag current attached. Finally, try the same promotion with the digest that the development tag points to now (the unapproved one) and save its output in /root/promote/gate-refused.txt. The first line must start with REFUSED.

When searching the list with grep, give -x (match the whole line) and -F (do not read it as a regular expression) together. If either one is missing, a digest written in a comment or a part of a longer string is read as an approval — the grader tests exactly those two things. When refusing, it must not begin the copy at all. To relay the exit code as it is, capture the $? of the script you called and exit with it at the end.

Rollback was not rebuilding but re-promoting the old digest

You put out one more release and roll it back. Stack one more commit on /root/promote/app and attach the tag v1.3.0, then extract the new release name with release-name.sh. Taking /opt/images/alpine_3.20.tar as that release's artifact, put it into /root/promote/reg/dev under the new release name tag, append a second line to /root/promote/releases.tsv in the same format, and append that digest as a line to /root/promote/approved.txt too. Then, with the gate, promote the new release from /root/promote/reg/dev to the /root/promote/reg/prod tag current, and right after that roll back by re-promoting the old release's digest under the same tag. Finally write three lines in /root/promote/rollback.txt — before <되돌리기 직전 current 가 가리키던 다이제스트> (the digest that the current tag pointed to just before the rollback), after <되돌린 뒤의 다이제스트> (the digest after the rollback), and identical yes or identical no (the result of checking with cmp whether the bytes of the rolled-back manifest are the same as those of the old release in the development environment).

Rollback needs no new mechanism — calling the gate of the earlier step once more with the old digest is all there is to it. For this to hold, you must not have deleted the old blob, and its value must be written in releases.tsv. The two files to compare with cmp are blobs/sha256/<다이제스트의 16진수 부분> (the hexadecimal part of the digest) of each layout. cmp gives a non-zero value when they differ, so if you turn on set -e, it stops there.

It answered in one file what is running where

Create /root/promote/ledger.sh <레지스트리루트> <릴리스표> <출력파일> (registry root, release table, output file). Treat each directory under the registry root as one environment (only those with an index.json inside), collect only the entries that have labels and write a tab-separated table to the output file. The first line is the header env<탭>tag<탭>digest<탭>release<탭>commit (tab-separated), and the following lines are <환경><탭><태그><탭><다이제스트><탭><판이름><탭><커밋> (environment, tag, digest, release name, commit). Look up the release name and commit by digest in the release table, and if absent write - in both cells. Sort the body lines with LC_ALL=C sort in the order environment, tag, digest. On standard output, print one line LEDGER <본문 줄 수> (the number of body lines) and end with 0. If the root does not exist or the release table does not exist, print a line starting with ERROR and end with 2. If the parent directory of the output file does not exist, create it. After giving it execute permission, run it with /root/promote/reg and /root/promote/releases.tsv and leave /root/promote/ledger.tsv.

Do not write the environment list into the script — only if you sweep the root will it keep working even when environments are added. jq's @tsv joins with tabs. An entry without a label has no annotations at all or lacks that key. For joining two tables by digest, the short way is to give awk -F'\t' two files and remember the first file with NR == FNR. You must sort before attaching the header so that the header does not get wedged in the middle.