Automating Backup Verification

Goal

Verify the integrity of a backup in three stages, and write a verification script with a distinct exit code per stage.

Why it matters

"Backups are running" and "the backup is valid" are different statements. Verification has three stages — does it exist, is it readable, does it actually recover. The first two can be automated, and that alone catches most silent failures. And giving a different exit code per stage lets monitoring see immediately where it broke — "compressed stream corrupted" is a far more useful alarm than "backup verification failed."

Steps

1. Create the /root/bv directory, and create a sha256 manifest for the files under /root/bk whose names end in .tar.gz as /root/bv/manifest.sha256.
2. Run verification using the manifest and save the result to /root/bv/check.txt. Every line must be OK.
3. Check the compressed stream integrity of /root/bk/full.tar.gz, and write the exit code to /root/bv/gzip.txt in the form rc=0.
4. Check the archive structure of the same file, and write the exit code to /root/bv/tarlist.txt in the form rc=0.
5. Create /root/bv/corrupt.tar.gz. Copy /root/bk/full.tar.gz and then corrupt a byte in the middle. Then confirm that the compression check fails, and save the output to /root/bv/corrupt-detected.txt.
6. In the /root/bv/last-success file, record the current time in ISO format (date -Is), and write the result of determining whether that file is within 24 hours to /root/bv/fresh.txt in the form FRESH=yes.
7. Write /root/bv/verify.sh. It must take the archive path as the first argument and behave as follows.
   • If the file is missing or has size 0, exit code 1
   • If the compressed stream is corrupted, exit code 2
   • If the archive structure is corrupted, exit code 3
   • If everything passes, exit code 0, with a message that includes OK The grader runs it against both a healthy archive and the corrupted archive from step 5.
8. Create /root/bv/report.txt with the following 4 lines. MANIFEST_OK=yes / CORRUPT_DETECTED=yes / SCRIPT_OK_RC=0 / SCRIPT_BAD_RC=<손상 아카이브에 대해 스크립트가 낸 종료 코드> (The last placeholder stands for the exit code the script returned for the corrupted archive.)

Notes

• Creating the corruption: cp <원본> <사본> and then dd if=/dev/urandom of=<사본> bs=1 seek=200 count=32 conv=notrunc (the placeholders are the original and the copy).
• The compression check is gzip -t <파일>, and the structure check is tar -tzf <파일> >/dev/null (the placeholder is the file).
• Check a file's age with find <파일> -mtime -1, or by comparing stat -c %Y with the current time (the placeholder is the file).
• Common mistake 1: in step 5, if you append bytes to the end of the file, gzip may only warn and pass. Corrupt the middle.
• Common mistake 2: if you use only set -e in the script for step 7, then when a check fails it ends with the command's own code instead of the exit code you want. Use an explicit exit N.

Checksum manifest

Create the /root/bv directory, and create a sha256 manifest for the files under /root/bk whose names end in .tar.gz as /root/bv/manifest.sha256.

If you build it with relative paths from the directory being verified, checking later is easier.

Verify with the manifest

Run verification using the manifest and save the result to /root/bv/check.txt. Every line must be OK.

sha256sum has an option that reads a manifest and checks against it. Every line must be OK.

Check the compressed stream

Check the compressed stream integrity of /root/bk/full.tar.gz, and write the exit code to /root/bv/gzip.txt in the form rc=0.

gzip has an option that checks only the integrity of a compressed file. On success it is silent.

Check the archive structure

Check the archive structure of the same file, and write the exit code to /root/bv/tarlist.txt in the form rc=0.

Print the listing but discard it from the screen, and look only at the exit code.

Reproduce and detect corruption

Create /root/bv/corrupt.tar.gz. Copy /root/bk/full.tar.gz and then corrupt a byte in the middle. Then confirm that the compression check fails, and save the output to /root/bv/corrupt-detected.txt.

Make a copy and then append or flip bytes in it. You must not touch the original.

Record the last success time

In the /root/bv/last-success file, record the current time in ISO format (date -Is), and write the result of determining whether that file is within 24 hours to /root/bv/fresh.txt in the form FRESH=yes.

The approach is to leave the success time in a file and check that file's age. ISO format is easy to work with.

Per-stage verification script

Write /root/bv/verify.sh. It must take the archive path as the first argument and behave as follows.

• If the file is missing or has size 0, exit code 1
• If the compressed stream is corrupted, exit code 2
• If the archive structure is corrupted, exit code 3
• If everything passes, exit code 0, with a message that includes OK The grader runs it against both a healthy archive and the corrupted archive from step 5.

Take the archive path as an argument, and give a different exit code for each failing stage. The grader runs it against both the healthy and the corrupted archive.

Verification report

Create /root/bv/report.txt with the following 4 lines. MANIFEST_OK=yes / CORRUPT_DETECTED=yes / SCRIPT_OK_RC=0 / SCRIPT_BAD_RC=<손상 아카이브에 대해 스크립트가 낸 종료 코드>

The values must be actual execution results. The exit code is the value the script really returned.