Envoy jwt_authn and the two identities

Goal

Verify RS256 tokens from a local JWKS with Envoy's jwt_authn filter to cut wrong requests off in front of upstream, pass only the verified claims as a header, and then confirm with real requests that the workload identity proven by mTLS and the user identity proven by a JWT are different things.

Why it matters

Once you bring in a mesh, you can move token verification into the setting of a single proxy that stands in front of every service instead of repeating it in every service. In exchange, upstream comes to trust the header the proxy passes as the identity, so if an imitated header leaks in on a path that skips verification, it becomes identity forgery right away. Also, that mTLS is on between services does not mean user authentication is done. All a certificate tells you is which workload connected, and only a token tells you on whose behalf the request was made. This lab lets you see the two identities side by side in one request.

Steps

1. Put an RSA 2048 private key at /root/st/mesh/private.pem and a JWK Set holding only its public components at /root/st/mesh/jwks.json (kid mesh-k1, RS256), and mint RS256 tokens with /root/st/mesh/mint.py (iss https://issuer.mesh.lab, aud mesh-api).
2. In /root/st/mesh/envoy.yaml, put admin 19901, listener 18000, cluster app (18081), and the router after jwt_authn (provider mesh), and check with envoy --mode validate.
3. Start the /root/st/mesh/upstream.py echo server at 18081 and Envoy, request /api/orders with alice's token, and write code=200 and path=/api/orders into /root/st/mesh/allow.txt.
4. Send a missing token, a forged one, an expired one, and a different audience, write 401, 401, 401, and 403 into /root/st/mesh/deny.txt, and make sure the rejected requests are not left in the upstream log.
5. Add forward_payload_header: x-jwt-payload and write the claims upstream received into /root/st/mesh/claims.txt.
6. Open /public as a verification exception, remove x-jwt-payload on that path, and write 200, 401, and absent into /root/st/mesh/public.txt.
7. Make a CA, server, and client certificates under /root/st/mesh/pki/, add an mTLS chain to 18000, test the two identities with /root/st/mesh/e2e.sh, and leave seven lines in /root/st/mesh/e2e.out.

Notes

• Envoy does not reread its static configuration. If you changed envoy.yaml, take it down with pkill -x envoy, bring it up again, and check that curl 127.0.0.1:19901/ready is LIVE.
• What upstream received is on the last line of /root/st/mesh/upstream.log. If a rejected request shows up there, the filter is not in front of upstream.
• Common mistake 1: emptying a rule's requires or using allow_missing_or_failed, so that the filter exists but a request with no token just passes.
• Common mistake 2: opening a public path without removing x-jwt-payload — anyone can then attach the header that upstream trusts.

Make the signing key and the JWKS

Make an RSA 2048 private key at /root/st/mesh/private.pem, and put a JWK Set holding only its public components at /root/st/mesh/jwks.json (one key: kty RSA, kid mesh-k1, alg RS256, use sig, n, e). /root/st/mesh/mint.py prints one line of an RS256 token with python3 /root/st/mesh/mint.py <sub> — the header kid is mesh-k1, and the claims are iss https://issuer.mesh.lab, aud mesh-api, sub, iat, and exp (5 minutes later by default). Make it accept the options --aud, --ttl (seconds, already expired if negative), and --key (a different private key).

A JWT is three pieces, base64url(헤더).base64url(페이로드).base64url(서명) (header, payload, signature), and the thing being signed is the string of the first two pieces joined with a dot. RS256 is cryptography's key.sign(데이터, padding.PKCS1v15(), hashes.SHA256()) (the first argument is the data). base64url strips the trailing =. For the JWK's n and e, you turn the public key numbers (public_numbers()) into big-endian bytes and encode them the same way. A JWKS is a file published to anyone, so the private components (d, p, q) must not be in it.

Validate the jwt_authn configuration

Make /root/st/mesh/envoy.yaml — admin 127.0.0.1:19901, listener 127.0.0.1:18000, and cluster app at 127.0.0.1:18081. The HTTP filters are in the order envoy.filters.http.jwt_authn and then envoy.filters.http.router. The provider name is mesh, issuer is https://issuer.mesh.lab, audiences is [mesh-api], local_jwks.filename is /root/st/mesh/jwks.json, and the rule is that prefix / requires provider_name: mesh. envoy --mode validate -c /root/st/mesh/envoy.yaml must pass.

The filter chain runs in order. If the verification filter is after router, the request has already gone upstream and it means nothing. If a rules entry's requires is empty, that path is not verified — it does not check when there is a token, it does not look at all. allow_missing_or_failed lets even a wrong token pass, so you do not use it at this step. --mode validate does not open ports and only reads the configuration, so you use it to catch field name mistakes before bringing it up.

A valid token reaches upstream

Bring up /root/st/mesh/upstream.py at 127.0.0.1:18081 — for any GET path it returns 200 and the JSON {"path": 요청 경로, "headers": {소문자 헤더 이름: 값}} (the placeholders are the request path and the lowercase header names and values), and appends the same JSON one line at a time to /root/st/mesh/upstream.log. Bring up Envoy with /root/st/mesh/envoy.yaml, request http://127.0.0.1:18000/api/orders with a mint.py alice token, and write the two lines code=200 and path=/api/orders into /root/st/mesh/allow.txt.

Upstream does no authentication. It believes the Envoy in front verified and just shows what it received as is — so what upstream received becomes the evidence of the lab. Detach the server and Envoy from the shell and bring them up in the background (setsid --fork nohup ...), and wait for Envoy's admin-port /ready to become LIVE. You send the token with the Authorization: Bearer <토큰> header (the placeholder is the token).

Cut a missing or forged token off in front of upstream

Send four kinds to the same /api/orders and write the codes you received into /root/st/mesh/deny.txt as four lines: missing= (no token), forged= (a token signed with a different RSA key with the kid left as is), expired= (--ttl -600), and wrong_aud= (--aud billing-api). The first three must be 401 and the last 403, and none of the rejected requests may be left in /root/st/mesh/upstream.log.

An attacker can write the kid and the claims however they like. The only thing they cannot have is the issuer's private key, and so signature verification is the whole of the defense. For a forged token, make one more key with openssl genpkey and mint with mint.py --key. Look at where 401 and 403 split — if the signature or expiry is wrong, it is 'we do not know who it is', and if the signature is right but the audience differs, it is 'we know who it is but this is not a token meant to come here'. The grader attaches a marker header to every request it sends and also checks whether that marker was printed in the upstream log.

Pass the verified claims downstream

Add forward_payload_header: x-jwt-payload to the provider mesh (leave forward at its default of false) and bring Envoy up again. Request with a mint.py alice token, unpack the header upstream received, and write four lines into /root/st/mesh/claims.txt: sub=alice, aud=mesh-api, iss=https://issuer.mesh.lab, and authorization_forwarded=no.

So that upstream does not have to verify the token again, Envoy carries the successfully verified payload as a header. The value is JSON encoded in base64url without padding. If forward is false, the original token is removed from the request after verification — it is the default that keeps what upstream received from being reused elsewhere. Also try sending once what happens if a client imitates and sends a header of the same name. The grader checks that too. Envoy does not reread its static configuration, so if you changed it, take it down and bring it up again.

Open a public path and remove the imitated identity

In the jwt_authn rules, put prefix /public without requires before the / rule, and put prefix /public separately in the route as well to remove x-jwt-payload with request_headers_to_remove. After bringing Envoy up again, write three lines into /root/st/mesh/public.txt: public= (the code of /public/status without a token), api= (the code of /api/orders without a token), and spoofed_payload= (present if upstream received that header when you attached a fake x-jwt-payload and sent to /public/status, otherwise absent). The expected values are 200, 401, and absent.

In the rules, the first match wins. If / comes first, every path is caught there and no public path is created. On a verified path, Envoy overwrites x-jwt-payload with the verified value, but on an unverified path no one touches that header. If upstream trusts that header as the identity, the public path becomes an identity forgery route. The route's request_headers_to_remove blocks that hole.

Look at the workload identity and the user identity separately

Make, under /root/st/mesh/pki/, a CA (ca.pem), a server certificate (server.pem and server.key, SAN IP:127.0.0.1 and URI:spiffe://mesh.lab/ns/shop/sa/orders), and a client certificate (client.pem and client.key, SAN URI:spiffe://mesh.lab/ns/shop/sa/frontend). Add to the 18000 listener a filter chain with tls_inspector and transport_protocol: tls — require_client_certificate: true, trusted_ca is ca.pem, and that chain's HCM has forward_client_cert_details: SANITIZE_SET and uri: true of set_current_client_cert_details. After bringing it up again, test with /root/st/mesh/e2e.sh and leave seven lines in /root/st/mesh/e2e.out: valid=200, missing=401, forged=401, workload=spiffe://mesh.lab/ns/shop/sa/frontend (the URI of the XFCC upstream received when you sent alice's token over mTLS), user=alice (the sub of the x-jwt-payload of the same request), mtls_without_jwt=401 (only a certificate and no token), and without_client_cert=rejected (only a token and no certificate, the handshake fails).

mTLS proves which workload opened the connection, and a JWT proves on whose behalf the request was made. That is why it must be a 401 when there is only a certificate and no token — that it is the frontend Pod does not mean it is Alice. To receive TLS and plain text on one port, the listener filter tls_inspector looks at the first bytes of the connection and lets it pick the chain. The XFCC default behavior is SANITIZE, so an XFCC imitated over a plain-text connection is removed. SANITIZE_SET discards what it received on mTLS and refills it with the certificate information it verified itself. If the server certificate has no IP:127.0.0.1 SAN, curl does not trust the server. You put the SAN in with openssl x509 -req ... -extfile.