Verification and Rollback

Goal

You verify the integrity of the imported items, detect tampering, pin versions, and actually perform a rollback.

Why it matters

In an air-gapped network, you do not make changes that have no rollback path. If you deleted something and want it back, you must go through import review again, and that takes days. So half of the work procedure is "recording what you changed and deciding in advance how to undo it".

And verification must go through the layers in order. If you try only the final check and it fails, you do not know which layer is the problem. If you go up in the order kernel → user space → container injection → actual execution, the point of failure is the cause.

Steps

Create a SHA-256 manifest for every .deb in /root/gpu/pool as /root/gpu/manifest.sha256, run the verification, and save the result to /root/gpu/verify-ok.txt. Every line must be OK.
Create the /root/gpu/tamper/ directory, copy one .deb into it, tamper with the copy, confirm that verification of that copy fails, and save the output to /root/gpu/verify-fail.txt. The original pool must stay as it is.
Check the file integrity of the installed nvidia-container-toolkit and save the result to /root/gpu/dpkg-verify.txt.
Make /root/gpu/verify-plan.txt with the following 4 lines. It is the verification order in an environment with an actual GPU. L1=lsmod / L2=nvidia-smi / L3=nvidia-ctk / L4=podman-run
Exclude nvidia-driver-550 and nvidia-container-toolkit from upgrades and save the pin list to /root/gpu/holds.txt. Both packages must show.
Save the part of apt's installation transaction record that relates to this installation to /root/gpu/history.txt. It must contain the string nvidia.
Completely remove (roll back) nvidia-container-toolkit, including its configuration files, and save the state after removal to /root/gpu/after-rollback.txt. That package must not be in the installed state, and nvidia-driver-550 must still be in the installed state.
Make /root/gpu/final-report.txt with the following 6 lines. BUNDLE_VERIFY=OK / TAMPER_DETECTED=yes / HELD=2 / DRIVER_STILL=installed / TOOLKIT_NOW=removed / SNAPSHOT=<오늘 날짜 YYYY-MM-DD> (the placeholder is today's date)

Notes

Make the manifest in the form cd /root/gpu/pool && sha256sum *.deb > ../manifest.sha256, and verify in the same directory with sha256sum -c ../manifest.sha256.
For tampering, appending a byte is enough, as in printf 'x' >> <사본> (the placeholder is the copy).
File integrity is checked with dpkg -V <패키지> (the placeholder is the package). If the output is empty, it is normal.
To release a pin, use apt-mark unhold <패키지> (the placeholder is the package). You may need it before step 7.
Common mistake 1: in step 2, tampering with the original inside the pool, so that all later steps break.
Common mistake 2: in step 7, using only remove leaves the configuration files and puts the package in the rc state. A complete removal is needed.

Create the integrity manifest

Create a SHA-256 manifest for every .deb in /root/gpu/pool as /root/gpu/manifest.sha256, run the verification, and save the result to /root/gpu/verify-ok.txt. Every line must be OK.

Create checksums for every .deb in the pool. For convenience in verification, make it with relative paths.

Reproduce tamper detection

Create the /root/gpu/tamper/ directory, copy one .deb into it, tamper with the copy, confirm that verification of that copy fails, and save the output to /root/gpu/verify-fail.txt. The original pool must stay as it is.

Tamper with a copy, not the original. Save the failing output of the verification command as it is.

Check the integrity of installed files

Check the file integrity of the installed nvidia-container-toolkit and save the result to /root/gpu/dpkg-verify.txt.

dpkg has an option that compares the hashes from install time with the current files. If the output is empty, it is normal.

Verification plan document

Make /root/gpu/verify-plan.txt with the following 4 lines. It is the verification order in an environment with an actual GPU. L1=lsmod / L2=nvidia-smi / L3=nvidia-ctk / L4=podman-run

List the layers in order. Write what is checked at each layer as key=value.

Pin versions

Exclude nvidia-driver-550 and nvidia-container-toolkit from upgrades and save the pin list to /root/gpu/holds.txt. Both packages must show.

Pin both the driver and the toolkit. Query the pin list to check.

Check the installation transactions

Save the part of apt's installation transaction record that relates to this installation to /root/gpu/history.txt. It must contain the string nvidia.

apt's history log keeps the time and command line of each transaction.

Perform the rollback

Completely remove (roll back) nvidia-container-toolkit, including its configuration files, and save the state after removal to /root/gpu/after-rollback.txt. That package must not be in the installed state, and nvidia-driver-550 must still be in the installed state.

Roll back only the toolkit. Use the option that deletes the configuration files too, and if a pin is in place, release it first.

Import, installation, verification, and rollback report

Make /root/gpu/final-report.txt with the following 6 lines. BUNDLE_VERIFY=OK / TAMPER_DETECTED=yes / HELD=2 / DRIVER_STILL=installed / TOOLKIT_NOW=removed / SNAPSHOT=<오늘 날짜 YYYY-MM-DD> (the placeholder is today's date)

Gather the results of the whole process. The values must be what you actually checked.