Multipart Uploads and the Arithmetic of ETag

In one line

A multipart ETag is not the md5 of the content. It is the md5 of the concatenated binary md5 values of each part, with -파트수 (a hyphen and the number of parts) appended.

Why it was needed

If you upload a 5GB file with a single PUT and the connection drops at the 90% mark, you have to start over from the beginning. Also, a single TCP connection cannot use all the bandwidth.

Multipart upload solves both. You start an upload and receive an UploadId, split the file into parts and upload them in parallel, and at the end signal completion with the list of parts. If it breaks midway, you only have to upload that part again.

How it works

The rules have numbers. There can be at most 10,000 parts, and each part except the last must be at least 5MiB. The maximum part size is 5GiB. Multiplying these gives the maximum size of a single object — 5GiB x 10,000 = about 5TB.

Choosing the part size is a trade-off. If it is small, the number of requests grows (a cost in services that charge per request), and if it is large, the retransmission cost grows. In practice, sizes between 8–64MiB are widely used. It must be larger than the file size divided by 10,000.

The ETag calculation is the interesting point. For a single part, it is the md5 of the content as it is. For multipart, it is calculated like this.

1. 각 파트의 md5 를 이진(16바이트)으로 구한다
2. 그것들을 순서대로 이어 붙인다 (파트 5개면 80바이트)
3. 그 80바이트의 md5 를 구한다
4. 뒤에 "-5" 를 붙인다

So if an ETag has a hyphen, the object was uploaded as multipart, and it never equals the md5 of the local file. Most inquiries of the form "I verified integrity with the ETag and it doesn't match" are this. To verify, you have to split with the same part size and do the same calculation.

Calculating the part size

You do not choose the part size by feel. Two constraints give the upper and lower bounds.

하한 = 파일 크기 ÷ 10,000      (파트 수 제한)
하한 = 최소 5 MiB              (마지막 파트 제외)
상한 = 5 GiB

100GB 파일이면 → 100GB ÷ 10,000 = 10MB 이상이어야 한다
                → 16MiB 로 잡으면 6,400 파트. 여유가 있다

Next, look at the retransmission cost. If the network is unstable and an average of 2% of parts fail, the larger the parts, the more bytes you throw away on each failure. Conversely, if parts are small, the number of requests grows and so do overhead and request charges.

File size	Recommended part	Number of parts	Notes
Up to 100MB	Multipart not needed	1	A single PUT is simple
100MB–1GB	8–16 MiB	12–64	Parallelism of 4–8 is enough
1GB–50GB	32–64 MiB	30–800	Parallelism 8–16
50GB or more	64–128 MiB	400–800	Check the part count limit first

Parallelism is decided by bandwidth and memory. Part size × parallelism is loaded in memory, so 64MiB × 16 = 1GB. If you exceed the container memory limit, it dies from OOM.

The right way to verify integrity with checksums

Attempts to verify integrity with the ETag break down in multipart. Instead, use the additional checksums that S3 provides. If you specify the algorithm at upload time, S3 verifies per part and also keeps a checksum for the whole object.

s3.put_object(Bucket=b, Key=k, Body=data, ChecksumAlgorithm="SHA256")

r = s3.head_object(Bucket=b, Key=k, ChecksumMode="ENABLED")
print(r.get("ChecksumSHA256"))

In multipart this value is also a composite (the part checksums are concatenated and hashed again), but S3 verifies every time it receives a part, so corruption in transit is caught on the spot. It is far safer than recalculating the ETag by hand.

CRC32C is fast to compute and favorable for large data, and SHA256 is slow but cryptographically strong. If the goal is detecting transmission errors, CRC32C is enough.

Finding and cleaning up aborted uploads

# 이 버킷에 떠도는 미완료 업로드
aws s3api list-multipart-uploads --bucket my-bucket

# 하나 지우기 — 올라간 파트가 모두 사라진다
aws s3api abort-multipart-upload   --bucket my-bucket --key big.tar --upload-id <UploadId>

If you look for them manually, you will forget. Automating with a lifecycle rule is the right answer (AbortIncompleteMultipartUpload, 7 days). This one rule is a setting you should include by default when creating a bucket.

What it looks like in the field

Aborted multipart uploads quietly eat money. The parts of an upload that was not completed take up storage space but do not appear as objects in the listing. If uploads cut off when a client died or during a deployment pile up for months, hundreds of GB with no explanation appear on the bill. It is a common cost-leak item on AWS.

There are two responses. The application should explicitly abort on failure, and you should set AbortIncompleteMultipartUpload to around 7 days in the bucket lifecycle rules. It is safe to set the latter by default.

What we do in the next lab

You do a multipart upload yourself, splitting a 24MiB file into 5MiB parts. You receive the UploadId, upload the parts, complete it, then calculate the multipart ETag by hand and match it against the server's value. At the end, you also try starting an upload and only aborting it.