Synchronising With rsync

Goal

You confirm rsync's key options by hand. In particular you build three things yourself: the trailing slash rule, the danger and guard of --delete, and a --link-dest generational backup.

Why it matters

rsync is the tool used most and gotten wrong most. The fact that -a does not include ACLs, extended attributes, or hard links, the fact that one slash at the end of the source path changes the directory depth, and the fact that --delete wipes out the target when the source is empty — all three have led to real incidents. --link-dest, on the other hand, is rsync's most beautiful feature. Each generation looks like a full snapshot while the disk is occupied only by the changes.

Steps

Create the following under /root/sync/src: app/main.py, app/util.py, conf/app.yaml, logs/app.log, and logs/error.log. Put at least one line of any content in each file.
Run a dry-run of synchronizing /root/sync/src/ to /root/sync/dst1/, with the itemized list of changes shown, and save it to /root/sync/dryrun.txt.
Actually perform the same synchronization over SSH. The target is root@127.0.0.1:/root/sync/dst1/, port 2222, and the key is /root/.ssh/id_labhub. After it completes, /root/sync/dst1/app/main.py must exist.
Demonstrate the trailing slash difference. If you send /root/sync/src (no slash) to /root/sync/dst2/, /root/sync/dst2/src/app/main.py must appear.
Write the script /root/sync/backup.sh. It must start with set -euo pipefail, include a check that aborts with exit code 1 if the source is empty, and synchronize /root/sync/src/ to /root/sync/dst3/ using --delete. The source path is taken as the first argument, and if there is no argument, it uses /root/sync/src/ — the grader passes an empty directory as the first argument to check that the guard really runs. Run the script to create /root/sync/dst3.
Synchronize /root/sync/src/ to /root/sync/dst4/ excluding what is under logs/. /root/sync/dst4/logs must not exist.
Make a generational backup. First do a full synchronization to /root/sync/gen/daily.1/, and then create /root/sync/gen/daily.0/ with --link-dest. For files that did not change, the two generations must share the same inode.
Make /root/sync/final.txt with the following 3 lines. PENDING=<3번과 같은 동기화를 다시 dry-run 했을 때 전송 예정 파일 수> / DST4_HAS_LOGS=no / SHARED_INODE=yes (PENDING= is followed by the number of files that would be transferred when you dry-run the same synchronization as step 3 again) PENDING must be 0.

Notes

rsync -ain <원본> <대상> is the dry-run + itemize combination (the placeholders are the source and the destination).
You specify SSH with -e 'ssh -p 2222 -i /root/.ssh/id_labhub'.
To check inode sharing, compare the files of the two generations with stat -c '%i %h' <파일> (the placeholder is the file).
Common mistake 1: if you leave off the slash at the end of the source in step 3, dst1/src/ is created and the check fails.
Common mistake 2: in step 7, if you give --link-dest a relative path, it is interpreted relative to the destination directory and the links are not made. Use an absolute path.

Creating the source tree

Create the following under /root/sync/src: app/main.py, app/util.py, conf/app.yaml, logs/app.log, and logs/error.log. Put at least one line of any content in each file.

If you mix subdirectories and log files, the exclude rule in a later step has meaning.

Simulate first

Run a dry-run of synchronizing /root/sync/src/ to /root/sync/dst1/, with the itemized list of changes shown, and save it to /root/sync/dryrun.txt.

Use together the option that only shows what would change without making actual changes and the option that shows the changes one per line.

Actual synchronization over SSH

Actually perform the same synchronization over SSH. The target is root@127.0.0.1:/root/sync/dst1/, port 2222, and the key is /root/.ssh/id_labhub. After it completes, /root/sync/dst1/app/main.py must exist.

rsync has an option that specifies the whole remote shell command. Put the port and key inside it.

Demonstrating the trailing slash difference

Demonstrate the trailing slash difference. If you send /root/sync/src (no slash) to /root/sync/dst2/, /root/sync/dst2/src/app/main.py must appear.

Send the same source to two targets, changing only the presence of the slash, and compare the resulting structures.

--delete with a guard

Write the script /root/sync/backup.sh. It must start with set -euo pipefail, include a check that aborts with exit code 1 if the source is empty, and synchronize /root/sync/src/ to /root/sync/dst3/ using --delete. The source path is taken as the first argument, and if there is no argument, it uses /root/sync/src/ — the grader passes an empty directory as the first argument to check that the guard really runs. Run the script to create /root/sync/dst3.

Put a check at the front of the script that aborts immediately if the source is empty. And set -euo pipefail too.

Applying an exclude rule

Synchronize /root/sync/src/ to /root/sync/dst4/ excluding what is under logs/. /root/sync/dst4/logs must not exist.

You can specify exclude patterns several times or read them from a file. The log files must not be in the target.

Hard-link generational backup

Make a generational backup. First do a full synchronization to /root/sync/gen/daily.1/, and then create /root/sync/gen/daily.0/ with --link-dest. For files that did not change, the two generations must share the same inode.

If you give --link-dest a relative path, it is interpreted relative to the destination. An absolute path is safe. Check with the link count of stat.

Proof that synchronization is complete

Make /root/sync/final.txt with the following 3 lines. PENDING=<3번과 같은 동기화를 다시 dry-run 했을 때 전송 예정 파일 수> / DST4_HAS_LOGS=no / SHARED_INODE=yes (PENDING= is followed by the number of files that would be transferred when you dry-run the same synchronization as step 3 again) PENDING must be 0.

If it was synchronized correctly, running the same command again as a dry-run should leave nothing to transfer.