Generating and Verifying a Presigned URL

Goal

By creating, using and expiring presigned URLs, get hands-on with the method of delegating only access permission without passing the bytes through the app server.

Why it matters

An implementation in which the app server relays user file downloads works fine at small scale. The problem comes when 100 videos of 500MB are requested at the same time. The app server relays 50GB, workers get tied up, bandwidth is trapped by the app server's specs, and if buffering is done wrong, you get OOM. A presigned URL turns this structure around — the app decides permissions and only makes a signed URL a few bytes long, and the actual transfer happens directly between the client and the storage. If you take the URL apart in step 7, you can see why this is safe. The signature is made with the secret key, but the secret key itself is nowhere in the URL. So even if the URL leaks, only that object, that action and that expiry time are exposed.

Steps

To s3://lab-media/private/secret.txt, upload /opt/fixtures/s3/readme.txt.
Run curl -o /dev/null -w '%{http_code}' http://127.0.0.1:9000/lab-media/private/secret.txt without a signature and in /root/ps/anon.txt, write status=403.
With /root/ps/presign.py, create a presigned GET URL with a 300-second expiry and save it on one line in /root/ps/get_url.txt. The URL must have X-Amz-Signature and X-Amz-Expires.
Download with that URL to create /root/ps/downloaded.txt. Its content must be the same as the original.
Create a URL with a 1-second expiry, wait 3 seconds and then use it. In /root/ps/expired.txt, write status=403.
With a presigned PUT URL with a 300-second expiry, upload /opt/fixtures/s3/sales.csv to s3://lab-media/upload/direct.csv. Confirm its existence with aws --profile local s3api head-object.
Write a Markdown table in /root/ps/sig.md. The row titles are the six X-Amz-Algorithm, X-Amz-Credential, X-Amz-Date, X-Amz-Expires, X-Amz-SignedHeaders and X-Amz-Signature, and in each row write the meaning of that parameter. The file must not contain the secret key value.

Notes

The credentials are in /opt/fixtures/s3/creds.env (the S3_* values), and the aws profile local is created the same way as in step 1 of s3-basics.
aws s3 presign also makes a URL, but the CLI v1 defaults to the old signature (SigV2, Signature=), so the X-Amz-* items do not appear. This lab makes it with boto3, which lets you pin the signature version.
When using the URL in the shell, be sure to wrap it in quotes — & is interpreted as background execution.
The shorter the expiry, the safer. A few minutes for downloads and a few tens of minutes for uploads is typical.
Common mistake 1: not putting size and Content-Type limits on a presigned PUT — the client can then upload anything.
Common mistake 2: leaving a presigned URL as it is in logs — anyone can use it until it expires.

Prepare a private object

To s3://lab-media/private/secret.txt, upload /opt/fixtures/s3/readme.txt.

Use the bucket from the previous lab as it is, or create a new one. A default bucket has anonymous access blocked.

Check that anonymous access is denied

Run curl -o /dev/null -w '%{http_code}' http://127.0.0.1:9000/lab-media/private/secret.txt without a signature and in /root/ps/anon.txt, write status=403.

Try to access it by the plain URL, without a signature. Record what the status code is.

Create a presigned GET URL

With /root/ps/presign.py, create a presigned GET URL with a 300-second expiry and save it on one line in /root/ps/get_url.txt. The URL must have X-Amz-Signature and X-Amz-Expires.

Specify the expiry time with it. It is normal for the URL to have signature-related query parameters attached.

Actually download with the presigned URL

Download with that URL to create /root/ps/downloaded.txt. Its content must be the same as the original.

If you don't wrap it in quotes, the shell interprets the ampersand as background execution.

Check that an expired URL is rejected

Create a URL with a 1-second expiry, wait 3 seconds and then use it. In /root/ps/expired.txt, write status=403.

Create it with a very short expiry, wait a moment and try it. Record the rejection status code.

Upload directly with a presigned PUT

With a presigned PUT URL with a 300-second expiry, upload /opt/fixtures/s3/sales.csv to s3://lab-media/upload/direct.csv. Confirm its existence with aws --profile local s3api head-object.

The client uploads directly to the storage. The key is not going through the app server.

Analyze the signature elements

Write a Markdown table in /root/ps/sig.md. The row titles are the six X-Amz-Algorithm, X-Amz-Credential, X-Amz-Date, X-Amz-Expires, X-Amz-SignedHeaders and X-Amz-Signature, and in each row write the meaning of that parameter. The file must not contain the secret key value.

Break down the URL's query parameters and write what constraint each one is. Confirm that the secret key is not in the URL.