scp, rsync and sftp — When to Use Which

In one line

All three tools run over an SSH channel. The difference is whether they send everything at once (scp), send only the differences (rsync), or exchange interactively (sftp).

Why this exists

There is a reason there are three tools for moving one file. The qualities you need differ when you send one 100MB file for the first time, when you synchronize a 10GB directory every day, and when you look at the file list on a server and pick what to download.

How it works

scp — simple copy

scp -P 2222 -i ~/.ssh/id_ed25519 ./app.tar.gz root@127.0.0.1:/tmp/
scp -r ./dist root@host:/srv/app/
scp -p ./config.yaml root@host:/etc/app/       # mtime/권한 보존
scp root@host:/var/log/app.log ./              # 내려받기

Two points to note.

The port option is uppercase -P. The lowercase -p means "preserve timestamps", so the meaning is completely different. In ssh, lowercase -p is the port, so it is easy to mix up.
Recent OpenSSH's scp uses the SFTP protocol internally. So some path expansion that used to work (the behavior where the remote shell handled globs) has changed. If you need the old behavior, you can force the legacy protocol with -O.

scp cannot resume. If it is cut at 90%, you start over from the beginning. So it is unsuitable for large transfers.

rsync — sending only the differences

rsync -aHAX --numeric-ids --delete --dry-run /srv/data/ backup@host:/backup/data/
rsync -avz -e 'ssh -p 2222 -i ~/.ssh/id_labhub' ./dist/ root@127.0.0.1:/srv/app/

You need to know the exact meaning of the options.

-a (archive) is the same as -rlptgoD. Recursive, symlinks, permissions, times, group, owner, and special files.
Some things are not included in -a. ACLs (-A), extended attributes (-X), and hard links (-H) must be specified separately. If you do not know this, the permissions after a restore come out subtly different.
--numeric-ids uses numeric UID/GID instead of names. It prevents owners being mapped wrongly when you restore to a different system.
It is good to get into the habit of always using -n (--dry-run) and -i (--itemize-changes) together.

And the trailing slash rule. This is the spot people get wrong most often in rsync.

rsync -a /srv/data  /backup/    # /backup/data/... 가 만들어진다
rsync -a /srv/data/ /backup/    # /backup/... 에 내용이 바로 들어간다

A slash at the end of the source path means "copy the contents of this directory", and without it means "copy this directory itself".

--delete is destructive. If it runs while the source is empty or the mount has come off, it deletes all the data on the target. It is an incident that really happens often in practice. Be sure to put a guard in scripts.

set -euo pipefail
[ "$(ls -A "$SRC" | wc -l)" -gt 0 ] || { echo "source empty, abort" >&2; exit 1; }
rsync -aHAX --numeric-ids --delete "$SRC/" "$DST"

You make generational backups cheaply with hard links.

rsync -aHAX --delete --link-dest=/backup/daily.1 /srv/data/ /backup/daily.0/

Instead of copying unchanged files, --link-dest makes hard links to the files in the specified directory. Each generation looks like a full snapshot, but the disk is occupied only by the changes.

sftp — interactive/batch

sftp -P 2222 -i ~/.ssh/id_labhub root@127.0.0.1
sftp -b /root/batch.txt -P 2222 root@127.0.0.1

If you write cd, put, get, ls, and mkdir in a batch file, it runs non-interactively. It is still widely used in automation for exchanging files with partner companies. Being able to create a transfer-only account with no shell with the combination ChrootDirectory + ForceCommand internal-sftp is also a strength of sftp.

What it looks like in the field

The incident where a generational backup is copied with a tool that does not know about hard links, and the size triples. The space saved with --link-dest is lost in a single copy. -H is not included in -a, so you must specify it separately.

A backup that eats all the bandwidth and disrupts the service. Lower the priority with the combination --bwlimit=50M and ionice -c 3 nice -n 19. If the backup does not disturb production, you actually gain room to shorten the interval.

What you will do in the next lab

You do two labs in a row. First you send files back and forth with scp and sftp and confirm integrity with checksums. Then you build, with rsync, the trailing slash difference, a --delete guard, and a --link-dest generational backup.