Automating Backup Verification
Goal
Verify the integrity of a backup in three stages, and write a verification script with a distinct exit code per stage.
Why it matters
"Backups are running" and "the backup is valid" are different statements. Verification has three stages — does it exist, is it readable, does it actually recover. The first two can be automated, and that alone catches most silent failures. And giving a different exit code per stage lets monitoring see immediately where it broke — "compressed stream corrupted" is a far more useful alarm than "backup verification failed."
The final incident shows why these layers must be separate. In an incremental chain, the three links with wrong hashes all look the same, FAILED, but one of them even passes the compression check; had you trusted only the compression check, you would have used that link and restored a wrong point in time. An incremental stacks on the previous link, so if one link collapses, everything after it becomes unusable, which is why "how far can I trust it?" becomes the first question of recovery.
Steps
- Create the
/root/bvdirectory, and create a sha256 manifest for the files under/root/bkwhose names end in.tar.gzas/root/bv/manifest.sha256. - Run verification using the manifest and save the result to
/root/bv/check.txt. Every line must be OK. - Check the compressed stream integrity of
/root/bk/full.tar.gz, and write the exit code to/root/bv/gzip.txtin the formrc=0. - Check the archive structure of the same file, and write the exit code to
/root/bv/tarlist.txtin the formrc=0. - Create
/root/bv/corrupt.tar.gz. Copy/root/bk/full.tar.gzand then corrupt a byte in the middle. Then confirm that the compression check fails, and save the output to/root/bv/corrupt-detected.txt. - In the
/root/bv/last-successfile, record the current time in ISO format (date -Is), and write the result of determining whether that file is within 24 hours to/root/bv/fresh.txtin the formFRESH=yes. - Write
/root/bv/verify.sh. It must take the archive path as the first argument and behave as follows.- If the file is missing or has size 0, exit code 1
- If the compressed stream is corrupted, exit code 2
- If the archive structure is corrupted, exit code 3
- If everything passes, exit code 0, with a message that includes
OKThe grader runs it directly against a healthy archive, the corrupted archive from step 5, a missing file, an empty file, a file whose compression is fine but is not tar, and a tar file with a broken entry in the middle (including names it has never seen before).
- Incident: raise
incident bk-chain-broken, determine the five links of the incremental chain (/srv/backup/app) one by one, then restore the latest trustworthy point in time exactly to/root/incident/bk-chain-broken/restore/. The report is/root/incident/bk-chain-broken/report.txt(candidate codes are in the step description).
Notes
- Always introduce the damage into a copy. You must overwrite bytes in the middle while keeping the size the same for the CRC to mismatch; check the option that keeps the overwriting tool from truncating the file.
- The compression check and the structure check are different options of different tools. Both are silent when they pass, so record them by exit code.
- A file's age is the difference between its modification time and now. Look at the file's timestamp, not a date string.
- Common mistake 1: in step 5, if you append bytes to the end of the file, gzip may only warn and pass. Corrupt the middle.
- Common mistake 2: if you use only
set -ein the script for step 7, then when a check fails it ends with the command's own code instead of the exit code you want. Use an explicitexit N. - Incident tools:
incident listshows the list, andincident show <이름>(name) shows the ticket (a colleague's report) again. Raising the same incident again returns it to its original broken state. - Write the report to
/root/incident/<이름>/report.txt(name is the incident name): one line<이름>: <원인 코드>(name: cause code) per target, plusevidence: <관찰한 출력 한 줄>(one line of output you observed). The evidence line is checked against the output actually seen in the incident. - Grading does not look only at the report. The actual state of what you restored, such as hashes, ownership, and modes, must be right, and workarounds that only erase the symptom fail.
Checksum manifest
Create the /root/bv directory, and create a sha256 manifest for the files under /root/bk whose names end in .tar.gz as /root/bv/manifest.sha256.
If you build it with relative paths from the directory being verified, checking later is easier.
Verify with the manifest
Run verification using the manifest and save the result to /root/bv/check.txt. Every line must be OK.
sha256sum has an option that reads a manifest and checks against it. Every line must be OK.
Check the compressed stream
Check the compressed stream integrity of /root/bk/full.tar.gz, and write the exit code to /root/bv/gzip.txt in the form rc=0.
The compression tool has an option that only checks integrity without extracting and writing. It says nothing on success, so to keep a record you must write the exit code down separately.
Check the archive structure
Check the archive structure of the same file, and write the exit code to /root/bv/tarlist.txt in the form rc=0.
Print the listing but discard it from the screen, and look only at the exit code.
Reproduce and detect corruption
Create /root/bv/corrupt.tar.gz. Copy /root/bk/full.tar.gz and then corrupt a byte in the middle. Then confirm that the compression check fails, and save the output to /root/bv/corrupt-detected.txt.
Make a copy and then append or flip bytes in it. You must not touch the original.
Record the last success time
In the /root/bv/last-success file, record the current time in ISO format (date -Is), and write the result of determining whether that file is within 24 hours to /root/bv/fresh.txt in the form FRESH=yes.
The approach is to leave the success time in a file and check that file's age. ISO format is easy to work with.
Per-stage verification script
Write /root/bv/verify.sh. It must take the archive path as the first argument and behave as follows.
- If the file is missing or has size 0, exit code 1
- If the compressed stream is corrupted, exit code 2
- If the archive structure is corrupted, exit code 3
- If everything passes, exit code 0, with a message that includes
OKThe grader runs it directly against a healthy archive, the corrupted archive from step 5, a missing file, an empty file, a file whose compression is fine but is not tar, and a tar file with a broken entry in the middle (including names the grader made that you have never seen).
Take the archive path as an argument, and give a different exit code for each failing stage. Check from the outer layer (existence) to the inner layer (structure) so that each code has only one meaning.
Incident: restoring from an incremental chain stopped partway
Running incident bk-chain-broken raises the incident. Symptom: restoring the app server from the incremental chain /srv/backup/app (the full full-1101, the incrementals inc-1102 through inc-1105, and the MANIFEST.sha256 and backup.log produced at backup time) stopped partway. Determine the state of each link and restore the latest trustworthy point in time exactly to /root/incident/bk-chain-broken/restore/: keep the in-archive path app/ as it is, and no file that did not exist at that point (including files deleted by that point) may be mixed in. The chain files are the only copies, so do not delete, modify, or move them. In /root/incident/bk-chain-broken/report.txt, write five lines, one per link from full-1101: <코드> to inc-1105: <코드> (code), plus evidence: <관찰한 출력 한 줄> (one line of output you observed). Each link's code is one of the following: bitrot (same size, but part of the content changed), ok (intact), overwritten (readable, but replaced by a different archive), truncated (cut off midway and short).
Direction: the layers of verification answer different questions: the hash list, "are these the bytes from that time?"; the compression check, "does it decompress?"; the structure check, "can the listing be read to the end?". Trust only one layer and you miss one. And incrementals stack on the previous link. Tools: sha256sum -c, gzip -t, tar -t (the verify.sh from step 7 works too), the size records in backup.log, tar's incremental restore mode. Concretely: for each link, write the three check results and the current and recorded sizes side by side on one line. When restoring, extract in a way that lets the directory listings the links recorded reflect even the files deleted in between.