TT Lab
Blog Get started
Learn Learning paths Courses

Backup and Restore

Automating Backup Verification

Continue in TT Lab

Goal

Verify the integrity of a backup in three stages, and write a verification script with a distinct exit code per stage.

Why it matters

"Backups are running" and "the backup is valid" are different statements. Verification has three stages — does it exist, is it readable, does it actually recover. The first two can be automated, and that alone catches most silent failures. And giving a different exit code per stage lets monitoring see immediately where it broke — "compressed stream corrupted" is a far more useful alarm than "backup verification failed."

The final incident shows why these layers must be separate. In an incremental chain, the three links with wrong hashes all look the same, FAILED, but one of them even passes the compression check; had you trusted only the compression check, you would have used that link and restored a wrong point in time. An incremental stacks on the previous link, so if one link collapses, everything after it becomes unusable, which is why "how far can I trust it?" becomes the first question of recovery.

Steps

  1. Create the /root/bv directory, and create a sha256 manifest for the files under /root/bk whose names end in .tar.gz as /root/bv/manifest.sha256.
  2. Run verification using the manifest and save the result to /root/bv/check.txt. Every line must be OK.
  3. Check the compressed stream integrity of /root/bk/full.tar.gz, and write the exit code to /root/bv/gzip.txt in the form rc=0.
  4. Check the archive structure of the same file, and write the exit code to /root/bv/tarlist.txt in the form rc=0.
  5. Create /root/bv/corrupt.tar.gz. Copy /root/bk/full.tar.gz and then corrupt a byte in the middle. Then confirm that the compression check fails, and save the output to /root/bv/corrupt-detected.txt.
  6. In the /root/bv/last-success file, record the current time in ISO format (date -Is), and write the result of determining whether that file is within 24 hours to /root/bv/fresh.txt in the form FRESH=yes.
  7. Write /root/bv/verify.sh. It must take the archive path as the first argument and behave as follows.
    • If the file is missing or has size 0, exit code 1
    • If the compressed stream is corrupted, exit code 2
    • If the archive structure is corrupted, exit code 3
    • If everything passes, exit code 0, with a message that includes OK The grader runs it directly against a healthy archive, the corrupted archive from step 5, a missing file, an empty file, a file whose compression is fine but is not tar, and a tar file with a broken entry in the middle (including names it has never seen before).
  8. Incident: raise incident bk-chain-broken, determine the five links of the incremental chain (/srv/backup/app) one by one, then restore the latest trustworthy point in time exactly to /root/incident/bk-chain-broken/restore/. The report is /root/incident/bk-chain-broken/report.txt (candidate codes are in the step description).

Notes

Checksum manifest

Create the /root/bv directory, and create a sha256 manifest for the files under /root/bk whose names end in .tar.gz as /root/bv/manifest.sha256.

If you build it with relative paths from the directory being verified, checking later is easier.

Verify with the manifest

Run verification using the manifest and save the result to /root/bv/check.txt. Every line must be OK.

sha256sum has an option that reads a manifest and checks against it. Every line must be OK.

Check the compressed stream

Check the compressed stream integrity of /root/bk/full.tar.gz, and write the exit code to /root/bv/gzip.txt in the form rc=0.

The compression tool has an option that only checks integrity without extracting and writing. It says nothing on success, so to keep a record you must write the exit code down separately.

Check the archive structure

Check the archive structure of the same file, and write the exit code to /root/bv/tarlist.txt in the form rc=0.

Print the listing but discard it from the screen, and look only at the exit code.

Reproduce and detect corruption

Create /root/bv/corrupt.tar.gz. Copy /root/bk/full.tar.gz and then corrupt a byte in the middle. Then confirm that the compression check fails, and save the output to /root/bv/corrupt-detected.txt.

Make a copy and then append or flip bytes in it. You must not touch the original.

Record the last success time

In the /root/bv/last-success file, record the current time in ISO format (date -Is), and write the result of determining whether that file is within 24 hours to /root/bv/fresh.txt in the form FRESH=yes.

The approach is to leave the success time in a file and check that file's age. ISO format is easy to work with.

Per-stage verification script

Write /root/bv/verify.sh. It must take the archive path as the first argument and behave as follows.

Take the archive path as an argument, and give a different exit code for each failing stage. Check from the outer layer (existence) to the inner layer (structure) so that each code has only one meaning.

Incident: restoring from an incremental chain stopped partway

Running incident bk-chain-broken raises the incident. Symptom: restoring the app server from the incremental chain /srv/backup/app (the full full-1101, the incrementals inc-1102 through inc-1105, and the MANIFEST.sha256 and backup.log produced at backup time) stopped partway. Determine the state of each link and restore the latest trustworthy point in time exactly to /root/incident/bk-chain-broken/restore/: keep the in-archive path app/ as it is, and no file that did not exist at that point (including files deleted by that point) may be mixed in. The chain files are the only copies, so do not delete, modify, or move them. In /root/incident/bk-chain-broken/report.txt, write five lines, one per link from full-1101: <코드> to inc-1105: <코드> (code), plus evidence: <관찰한 출력 한 줄> (one line of output you observed). Each link's code is one of the following: bitrot (same size, but part of the content changed), ok (intact), overwritten (readable, but replaced by a different archive), truncated (cut off midway and short).

Direction: the layers of verification answer different questions: the hash list, "are these the bytes from that time?"; the compression check, "does it decompress?"; the structure check, "can the listing be read to the end?". Trust only one layer and you miss one. And incrementals stack on the previous link. Tools: sha256sum -c, gzip -t, tar -t (the verify.sh from step 7 works too), the size records in backup.log, tar's incremental restore mode. Concretely: for each link, write the three check results and the current and recorded sizes side by side on one line. When restoring, extract in a way that lets the directory listings the links recorded reflect even the files deleted in between.