One redirect line blocked renewal

Goal

Pass an HTTP-01 challenge inside a Pod, reproduce the incident in which an HTTP→HTTPS redirect swallows the challenge and blocks renewal, and then fix it by making only the challenge path an exception. You write probe.sh for standing checks and a cert-manager Certificate manifest.

Why it matters

With ACME automatic renewal, once the first issuance succeeds there are no symptoms for the next two months. One redirect line added in the meantime blocks the challenge at renewal time, and by then nobody thinks of that change. This is an incident we actually ran into in the homelab, and it is obvious at a glance once you reproduce it. There are two pieces of material — /opt/app/edge.py is an edge (HTTP 8080, HTTPS 8443) that rereads its configuration file (/root/acme/edge.json) on every request, and /opt/app/acme_va.py is a validation server that follows redirects and compares the body with the key authorization.

Steps

1. The token is zk3r9QmZ7Vf1Yb2Ld8Ns4Hj6Xc5Pa0Wt and the account key thumbprint is Q7pVw2xK9mN4rT1sB6yL0cH3fJ8dG5eA. Put the key authorization string in the file /root/acme/challenges/zk3r9QmZ7Vf1Yb2Ld8Ns4Hj6Xc5Pa0Wt.
2. Create /root/acme/edge.json — http_port 8080, https_port 8443, redirect_to_https false, exempt_paths containing /.well-known/acme-challenge/, and challenge_dir /root/acme/challenges. Start python3 /opt/app/edge.py --config /root/acme/edge.json and save the body of curl -s http://127.0.0.1:8080/.well-known/acme-challenge/<토큰> (replace the placeholder with the token) to /root/acme/02-fetch.txt.
3. Save the output of python3 /opt/app/acme_va.py --url http://127.0.0.1:8080/.well-known/acme-challenge/<토큰> --expect <키인가> (replace the placeholders with the token and the key authorization) to /root/acme/03-valid.txt. The first line must be VALID.
4. Reproduce the incident: in edge.json, change redirect_to_https to true and exempt_paths to an empty list. Save the output of curl -sI http://127.0.0.1:8080/.well-known/acme-challenge/x to /root/acme/04-curl.txt and the validation server's output to /root/acme/04-broken.txt (it must be INVALID).
5. Fix it: with the redirect still on, put /.well-known/acme-challenge/ back into exempt_paths. Save the validation server's output to /root/acme/05-fixed.txt (VALID) and curl -sI http://127.0.0.1:8080/ to /root/acme/05-app.txt (still 301).
6. Create /root/acme/probe.sh <http://host:port>. If the status code of <base>/.well-known/acme-challenge/probe-check is 3xx, it must print REDIRECT and exit with 1; otherwise (200, 404, and so on) it must print OK and exit with 0. Here base means the URL you pass in as the argument. Save the output for http://127.0.0.1:8080 to /root/acme/06-probe.txt.
7. Write a cert-manager Certificate in /root/acme/certificate.yaml — apiVersion: cert-manager.io/v1, spec.secretName, www.lab.internal in spec.dnsNames, spec.duration and spec.renewBefore (in hours, h), and spec.issuerRef.kind: ClusterIssuer.
8. Summarize the incident in /root/acme/08-report.md. It must include HTTP-01, the 301 you saw as evidence, the path you made an exception, .well-known/acme-challenge, and why renewBefore makes the incident surface late.

Notes

• edge.json is reread on every request, so you do not need to restart the edge after editing.
• The validation server prints output even on failure. If you save it with > 파일 (redirecting to a file), it is kept regardless of the exit code.
• curl -sI sends a HEAD request and shows only the status line and headers. Look at where the Location: header points.
• Common mistakes: if you do not empty exempt_paths in step 4, the incident is not reproduced. If you write d as in renewBefore: 15d in step 7, cert-manager will not accept it.

Put the key authorization file in place

Put the key authorization string (token.thumbprint) in /root/acme/challenges/zk3r9QmZ7Vf1Yb2Ld8Ns4Hj6Xc5Pa0Wt.

In RFC 8555, the key authorization is a string made by joining the token, a period, and the account key thumbprint. The file name is the token itself, and the content is that one line. The thumbprint is given in the instructions.

Start the edge and fetch the challenge

Create /root/acme/edge.json (redirect off, challenge path exempted), start the edge, and save the body of the challenge URL to /root/acme/02-fetch.txt.

The JSON key names are exactly as in the instructions. Run the edge in the background with nohup and send its log to /root/acme/edge.log. The body you get with curl -s must equal the key authorization from step 1.

Get confirmation from the validation server

Run acme_va.py and save the output to /root/acme/03-valid.txt. The first line must be VALID.

--url is the challenge URL and --expect is the key authorization string. The first line of the output is the verdict, and below it the paths it followed are listed one per line.

The redirect swallows the challenge

Change edge.json to redirect on and no exemptions, then save the curl -sI output to /root/acme/04-curl.txt and the validation server output to /root/acme/04-broken.txt.

Editing the JSON with a python3 one-liner leaves fewer mistakes. In the first line of curl -sI you see 301 and in the Location header you see https, and the validation server follows 301 → 404 and ends with INVALID.

Exempt only the challenge path

With the redirect still on, put the challenge path in exempt_paths, and save the validation output to /root/acme/05-fixed.txt and curl -sI http://127.0.0.1:8080/ to /root/acme/05-app.txt.

What you change is one line in the exemption list. The app path (/) must still go to HTTPS with a 301, and only the challenge path must give 200 directly over HTTP.

One line to see whether renewal is blocked

Create /root/acme/probe.sh <http://host:port> (REDIRECT and exit 1 if 3xx, otherwise OK and exit 0) and save the output for http://127.0.0.1:8080 to /root/acme/06-probe.txt.

Get only the status code with curl -s -o /dev/null -w '%{http_code}' and branch on it with case. A 404 is OK — a missing file does not block renewal, only a redirect does. The grader starts one 301 server and one 404 server and runs your script against them.

Declare it with cert-manager

In /root/acme/certificate.yaml, write a Certificate (cert-manager.io/v1, secretName, www.lab.internal in dnsNames, duration and renewBefore in h units, issuerRef.kind ClusterIssuer).

Follow the structure of the example in the documentation, but with the values from this lab. duration must be greater than renewBefore, and both are Go duration strings (h, m, s). An issuer used from other namespaces as well is a ClusterIssuer.

Incident report

In /root/acme/08-report.md, write how HTTP-01 got blocked, the evidence (301), the path you made an exception, and why renewBefore makes the incident surface late.

The curl result from step 4 is the evidence, and the exemption list from step 5 is the fix. Right after issuance there are no symptoms, and the challenge runs again only at the renewBefore point — that is why it "surfaces late."