Multipart Upload and Computing the ETag

Goal

Perform a multipart upload yourself with the low-level API, calculate the multipart ETag by hand and match it against the server's value, and then clean up aborted uploads.

Why it matters

Multipart is often handled by the SDK, so it is easy to pass over without knowing the internals. But there are two points you must know. First, the ETag. The ETag of an object uploaded as multipart is not the md5 of the content; it is the md5 of the concatenated binary md5 values of each part, with the number of parts appended at the end. If you don't know this, you spend hours on "I verified with the ETag and it doesn't match." Second, aborted uploads. The parts of an uncompleted multipart upload take up storage space but are not visible in the object list. If uploads cut off when a client died or during a deployment pile up for months, hundreds of GB with no explanation appear on the bill. It is a regular item in AWS cost leaks, and step 7 of this lab is the response to it.

Steps

Create /root/mp/big.bin at exactly 25165824 bytes (24MiB). The content must be reproducible (for example, repeating 0x00–0xFF).
In /root/mp/parts.txt, write three lines: part_size=5242880, parts=5 and last_part=4194304. Four 5MiB parts are followed by a last 4MiB part.
With /root/mp/mp.py, start a multipart upload for lab-media/big/big.bin and save the UploadId in /root/mp/upload_id.txt. It must be at least 32 characters.
Upload the 5 parts and write the ETag of each response to /root/mp/part_etags.txt in the form <파트번호> <etag> (part number, then etag), 5 lines.
Complete the upload. For aws --profile local s3api head-object --bucket lab-media --key big/big.bin, the ContentLength must be 25165824 and the ETag must end with -5.
With /root/mp/etag_calc.py, calculate the multipart ETag by hand and in /root/mp/etag_calc.txt, write two lines, calculated=<값> and server=<값> (each with the actual value in place of the placeholder). The two values must be the same.
For lab-media/big/orphan.bin, only start an upload and then abort it. In /root/mp/abort.out, write uploads_before=1 uploads_after=0.
Download the object to /root/mp/downloaded.bin, compare its sha256 with the original, and in /root/mp/verify.txt, write sha_match=true. In the same file, also write two lines, max_parts=10000 and min_part_bytes=5242880.

Notes

Part rules: at most 10,000 parts, each part except the last at least 5MiB, and a part at most 5GiB
ETag calculation: convert each part's md5 to binary with bytes.fromhex(), concatenate them, take the md5 of the whole, and append -<파트수> (the number of parts)
To look up incomplete uploads, use list_multipart_uploads or aws s3api list-multipart-uploads --bucket lab-media.
In /opt/fixtures/s3/creds.env, the credentials are S3_ENDPOINT, S3_ACCESS_KEY and S3_SECRET_KEY. The aws profile local is the one made in step 1 of the previous lab (s3-basics), used as it is.
Common mistake 1: concatenating the part md5 values as hexadecimal strings — they must be binary.
Common mistake 2: not aborting a failed upload, so parts quietly pile up.

Create a large file

Create /root/mp/big.bin at exactly 25165824 bytes (24MiB). The content must be reproducible (for example, repeating 0x00–0xFF).

The content must be the same every time so that you can compare hashes later. Fill it with a fixed pattern.

Work out the split into parts

In /root/mp/parts.txt, write three lines: part_size=5242880, parts=5 and last_part=4194304. Four 5MiB parts are followed by a last 4MiB part.

Each part except the last has a fixed minimum size. The remainder is the last part.

Start a multipart upload

With /root/mp/mp.py, start a multipart upload for lab-media/big/big.bin and save the UploadId in /root/mp/upload_id.txt. It must be at least 32 characters.

When you start, you receive an identifier. You need this identifier to upload parts.

Upload the parts and collect the ETags

Upload the 5 parts and write the ETag of each response to /root/mp/part_etags.txt in the form <파트번호> <etag> (part number, then etag), 5 lines.

Part numbers start at 1. You have to collect the ETag of each response in order to be able to complete.

Complete the upload

Complete the upload. For aws --profile local s3api head-object --bucket lab-media --key big/big.bin, the ContentLength must be 25165824 and the ETag must end with -5.

You pass the list of part numbers and ETags together. After completion, the object size must equal the original.

Reproduce the multipart ETag by hand

With /root/mp/etag_calc.py, calculate the multipart ETag by hand and in /root/mp/etag_calc.txt, write two lines, calculated=<값> and server=<값> (each with the actual value in place of the placeholder). The two values must be the same.

Concatenate the binary values of each part's md5, take the md5 again, and append the number of parts. It is the binary value, not the hexadecimal string.

Start and only abort

For lab-media/big/orphan.bin, only start an upload and then abort it. In /root/mp/abort.out, write uploads_before=1 uploads_after=0.

If you don't abort, the parts keep taking up storage space. After aborting, the list of incomplete uploads must be empty.

Download, verify integrity and write down the rules

Download the object to /root/mp/downloaded.bin, compare its sha256 with the original, and in /root/mp/verify.txt, write sha_match=true. In the same file, also write two lines, max_parts=10000 and min_part_bytes=5242880.

Compare with the original by sha256. And write the three part rules as numbers.