Mint SPIFFE certificates and build STRICT and PERMISSIVE chains

Goal

Write a PeerAuthentication, create a mesh CA and SPIFFE certificates with openssl, set up by hand the Envoy filter chain each mode becomes and confirm how plain text and mTLS are split, and then derive the AuthorizationPolicy principal from the certificate's SAN.

Why it matters

Calls that get cut on the day you switch to STRICT, a policy that is quietly ignored when you try to leave just one port as plain text, and an authorization policy with a wrongly written principal that blocks everyone — istioctl validate passes all three. If you know which chain a mode becomes in Envoy and where identity comes from in the certificate, you can pin down the cause from one line of statistics and one certificate.

Steps

Write a PeerAuthentication to /root/ist2-mtls/pa.yaml — apiVersion: security.istio.io/v1, name reviews-strict, namespace default, selector.matchLabels.app: reviews, mtls.mode: STRICT. Put the output and exit code of istioctl validate -f pa.yaml in /root/ist2-mtls/01-validate.txt (with rc= on the last line).
In /root/ist2-mtls, create one self-signed CA (ca.crt and ca.key) with openssl, and create three certificates signed by that CA — reviews.crt/reviews.key (the server), orders.crt/orders.key (the client to allow) and intruder.crt/intruder.key (a client signed by the same CA but not to be allowed). The SAN of each certificate is one URI, spiffe://cluster.local/ns/default/sa/<이름> (the placeholder is the name). Then write the SANs actually read from the certificates to /root/ist2-mtls/02-ids.txt as three lines, reviews.crt=…, orders.crt=… and intruder.crt=….
Write an Envoy configuration to /root/ist2-mtls/strict.yaml — admin port 9986, a listener virtualInbound listening at 127.0.0.1:10086 with envoy.filters.listener.tls_inspector as a listener filter. There is one filter chain, with filter_chain_match.transport_protocol: tls, and give DownstreamTlsContext require_client_certificate: true, the server certificate /root/ist2-mtls/reviews.crt and /root/ist2-mtls/reviews.key, and the trusted CA /root/ist2-mtls/ca.crt (do not put in a SAN matcher yet). That chain sends every path to the cluster inbound|8109|| (127.0.0.1:8109). Start an upstream on 8109 as ok and start Envoy, then write four lines to /root/ist2-mtls/03-strict.txt — plain= (the code of plain http://localhost:10086/strict), mtls= (the code of https://localhost:10086/strict with the orders certificate), mtls_body= (that response body) and no_filter_chain_match= (the value of the statistic listener.127.0.0.1_10086.no_filter_chain_match).
Copy /root/ist2-mtls/strict.yaml to /root/ist2-mtls/strict-san.yaml, then add one match_typed_subject_alt_names to validation_context in that copy only — san_type: URI, and matcher.exact is exactly the SAN of orders.crt you read in step 2. Start Envoy again with that configuration and write three lines to /root/ist2-mtls/04-san.txt — orders= (the code of https://localhost:10086/san with the orders certificate), intruder= (the code of the same request with the intruder certificate) and fail_verify_san= (the value of the statistic listener.127.0.0.1_10086.ssl.fail_verify_san).
Based on /root/ist2-mtls/strict.yaml, make /root/ist2-mtls/permissive.yaml — leave the TLS chain of step 3 as it is, and add one plain-text chain with neither filter_chain_match nor transport_socket (to the same cluster inbound|8109||). Start Envoy again with that configuration and write three lines to /root/ist2-mtls/05-permissive.txt — plain= (the code of plain http://localhost:10086/permissive), plain_body= (its body) and mtls= (the code of https://localhost:10086/permissive with the orders certificate).
Write two documents to /root/ist2-mtls/pa-port.yaml — (1) a Service reviews (namespace default, selector app: reviews, port: 80 → targetPort: 10096), and (2) a PeerAuthentication reviews-port (the same selector, mtls.mode: STRICT, and only one port set to DISABLE through portLevelMtls). See the hint for which number to write for that port. Then write an Envoy configuration to /root/ist2-mtls/portlevel.yaml — make the same listener as step 3 (10086) also listen on 127.0.0.1:10096 through additional_addresses, and put one plain-text chain with filter_chain_match.destination_port: 10096 and the one TLS chain from step 3. Start Envoy again and write four lines to /root/ist2-mtls/06-portlevel.txt — validate_rc= (the exit code of istioctl validate -f pa-port.yaml), plain_10096= and plain_10086= (the code of plain /port on each port) and mtls_10086= (the code of https://localhost:10086/port with the orders certificate).
Read the SAN of /root/ist2-mtls/orders.crt with openssl and write two lines to /root/ist2-mtls/07-principal.txt — san= (the URI as read) and principal= (the string to put in principals of the AuthorizationPolicy). Then write an AuthorizationPolicy to /root/ist2-mtls/authz.yaml — apiVersion: security.istio.io/v1, name reviews-from-orders, namespace default, selector.matchLabels.app: reviews, action: ALLOW, and in from[0].source.principals of one rule only that principal. istioctl validate -f authz.yaml must pass.
In /root/ist2-mtls/08-report.md, write four lines — default_mode= (the mode when there is no PeerAuthentication at all), strict_plain= (the code the plain-text request received in step 3), permissive_chains= (the number of filter chains of the step 5 listener) and orders_principal= (the principal from step 7) — and below them write explanations starting with - in at least four lines.

Notes

This Pod has neither a real istiod nor a real sidecar. So you cannot see the actual generated output with istioctl proxy-config; instead you know the translation rules and build the equivalent Envoy configuration by hand to confirm the behavior. The same rules show up as they are in the proxy-config output of a production cluster.
The certificates are signed by the CA you made instead of a real istiod. The shape (the SPIFFE URI in the SAN) is the same as what istiod signs.
When a plain-text connection is cut, curl prints 000 as the code. Get the code with curl -s -o /dev/null -w '%{http_code}'.
In steps 3, 4, 5 and 6, the configuration file names differ from one another. Do not edit an earlier step's file; create a new one under a new name.
When you start Envoy, detach it completely from the shell with setsid --fork nohup envoy -c <파일> --log-level warn > <로그> 2>&1 </dev/null (the placeholders are the file and the log). Before you start it again, clean up with pkill -x envoy (pkill -f 'envoy -c' also kills the shell itself that contains that string).
A server for imitating an upstream is already in the image: python3 /opt/lab/envoy/upstream.py <포트> ok|fail|slow (the placeholder is the port). The response body is <모드>:<포트> <경로> (the placeholders are the mode, the port and the path).
After you edit the configuration, first filter it with envoy --mode validate -c <파일> before you start it. The cluster name contains |, so in YAML you must always wrap it in quotes.

Write a STRICT PeerAuthentication and filter it offline

Write a PeerAuthentication to /root/ist2-mtls/pa.yaml — apiVersion: security.istio.io/v1, name reviews-strict, namespace default, selector.matchLabels.app: reviews, mtls.mode: STRICT. Put the output and exit code of istioctl validate -f pa.yaml in /root/ist2-mtls/01-validate.txt (with rc= on the last line).

PeerAuthentication is configuration of the receiving workload. You are not putting it on the side that calls reviews; you are telling reviews itself "do not accept plain text", so the selector must be a label of the receiving side. If you leave out the selector, the whole namespace is the target, and if you put it in the root namespace (istio-system), the whole mesh is the target. istioctl validate looks only at the schema without a cluster — a typo in the mode value is caught here.

Create a mesh CA and three SPIFFE certificates

In /root/ist2-mtls, create one self-signed CA (ca.crt and ca.key) with openssl, and create three certificates signed by that CA — reviews.crt/reviews.key (the server), orders.crt/orders.key (the client to allow) and intruder.crt/intruder.key (a client signed by the same CA but not to be allowed). The SAN of each certificate is one URI, spiffe://cluster.local/ns/default/sa/<이름> (the placeholder is the name). Then write the SANs actually read from the certificates to /root/ist2-mtls/02-ids.txt as three lines, reviews.crt=…, orders.crt=… and intruder.crt=….

Istio's identity is not the name (CN) but the URI in the SAN — spiffe://<신뢰 도메인>/ns/<네임스페이스>/sa/<서비스 계정> (the placeholders are the trust domain, the namespace and the service account). After istiod checks the Pod's service account token, it signs a certificate of exactly this shape. Even if you put the SAN in the CSR with -addext "subjectAltName=URI:…", if you do not give openssl x509 -req the -copy_extensions copyall when signing, the extension is dropped. After signing, be sure to check with openssl x509 -noout -ext subjectAltName, and also look at the chain with openssl verify -CAfile ca.crt.

STRICT becomes a listener with only one TLS chain

Write an Envoy configuration to /root/ist2-mtls/strict.yaml — admin port 9986, a listener virtualInbound listening at 127.0.0.1:10086 with envoy.filters.listener.tls_inspector as a listener filter. There is one filter chain, with filter_chain_match.transport_protocol: tls, and give DownstreamTlsContext require_client_certificate: true, the server certificate /root/ist2-mtls/reviews.crt and /root/ist2-mtls/reviews.key, and the trusted CA /root/ist2-mtls/ca.crt (do not put in a SAN matcher yet). That chain sends every path to the cluster inbound|8109|| (127.0.0.1:8109). Start an upstream on 8109 as ok and start Envoy, then write four lines to /root/ist2-mtls/03-strict.txt — plain= (the code of plain http://localhost:10086/strict), mtls= (the code of https://localhost:10086/strict with the orders certificate), mtls_body= (that response body) and no_filter_chain_match= (the value of the statistic listener.127.0.0.1_10086.no_filter_chain_match).

When istiod receives STRICT, it removes the plain-text chain from the incoming listener. What remains is a chain that accepts only connections tls_inspector judged to be "TLS", and that chain requires a client certificate and verifies it against the mesh CA. A plain-text connection has no matching chain, so Envoy just closes it — there is not even an HTTP response, so curl's code is 000, and the trace of that is the no_filter_chain_match statistic. The server certificate's SAN is only a URI, so curl's hostname check does not pass; turn off only server verification with -k and present the client certificate with --cert and --key. (Istio's client checks the SPIFFE ID in the server SAN instead of the hostname.)

Even if the same CA signed it, a different SAN is rejected

Copy /root/ist2-mtls/strict.yaml to /root/ist2-mtls/strict-san.yaml, then add one match_typed_subject_alt_names to validation_context in that copy only — san_type: URI, and matcher.exact is exactly the SAN of orders.crt you read in step 2. Start Envoy again with that configuration and write three lines to /root/ist2-mtls/04-san.txt — orders= (the code of https://localhost:10086/san with the orders certificate), intruder= (the code of the same request with the intruder certificate) and fail_verify_san= (the value of the statistic listener.127.0.0.1_10086.ssl.fail_verify_san).

The chain of step 3 looks only at "did the mesh CA sign it". That is why the intruder, which came from the same CA, passed too. If you put a SAN matcher, it looks not only at the signature but also at "whose certificate is it". However, where Istio uses this matcher is mainly on the client side — the calling side checking "is the other side really reviews' identity" (secure naming), and "who may call" on the receiving side is the job of the AuthorizationPolicy in step 7. Here we apply the same principle on the receiving side to see how Envoy checks the SAN. The rejection happens in the TLS handshake, so the code is 000.

PERMISSIVE is two chains

Based on /root/ist2-mtls/strict.yaml, make /root/ist2-mtls/permissive.yaml — leave the TLS chain of step 3 as it is, and add one plain-text chain with neither filter_chain_match nor transport_socket (to the same cluster inbound|8109||). Start Envoy again with that configuration and write three lines to /root/ist2-mtls/05-permissive.txt — plain= (the code of plain http://localhost:10086/permissive), plain_body= (its body) and mtls= (the code of https://localhost:10086/permissive with the orders certificate).

PERMISSIVE means "accept both". In Envoy it is expressed by adding one more chain. tls_inspector looks at the first bytes and sends it to the transport_protocol: tls chain if it is TLS, and otherwise to the chain with no match condition. The chains Istio actually makes also carry ALPN conditions (istio-peer-exchange, istio), but the principle is the same. If you attach a transport_socket to the plain-text chain, that chain also waits for TLS and plain text is blocked again.

portLevelMtls is one chain that matches a workload port

Write two documents to /root/ist2-mtls/pa-port.yaml — (1) a Service reviews (namespace default, selector app: reviews, port: 80 → targetPort: 10096), and (2) a PeerAuthentication reviews-port (the same selector, mtls.mode: STRICT, and only one port set to DISABLE through portLevelMtls). See the hint for which number to write for that port. Then write an Envoy configuration to /root/ist2-mtls/portlevel.yaml — make the same listener as step 3 (10086) also listen on 127.0.0.1:10096 through additional_addresses, and put one plain-text chain with filter_chain_match.destination_port: 10096 and the one TLS chain from step 3. Start Envoy again and write four lines to /root/ist2-mtls/06-portlevel.txt — validate_rc= (the exit code of istioctl validate -f pa-port.yaml), plain_10096= and plain_10086= (the code of plain /port on each port) and mtls_10086= (the code of https://localhost:10086/port with the orders certificate).

When iptables diverts a connection coming to the Pod to 15006, it preserves the original destination port. That port is not the Service's port but the targetPort the container actually listens on, and istiod turns the number in portLevelMtls as it is into a chain matched by destination_port. That is why the documentation also says "workload port". Also, portLevelMtls is accepted only in a policy that has a selector — if you remove it, istioctl validate rejects it. In this Pod, instead of iptables, one listener listens on two ports to make the same shape.

Drop spiffe:// from the SAN and it becomes the principal

Read the SAN of /root/ist2-mtls/orders.crt with openssl and write two lines to /root/ist2-mtls/07-principal.txt — san= (the URI as read) and principal= (the string to put in principals of the AuthorizationPolicy). Then write an AuthorizationPolicy to /root/ist2-mtls/authz.yaml — apiVersion: security.istio.io/v1, name reviews-from-orders, namespace default, selector.matchLabels.app: reviews, action: ALLOW, and in from[0].source.principals of one rule only that principal. istioctl validate -f authz.yaml must pass.

When mTLS finishes, the receiving Envoy remembers the other side's certificate URI SAN as the connection's identity. The source.principals of an AuthorizationPolicy is checked against that identity, and it is written in the form <신뢰 도메인>/ns/<네임스페이스>/sa/<서비스 계정> (the placeholders are the trust domain, the namespace and the service account) with the scheme (spiffe://) dropped from the notation. In the shell you can strip it with ${san#spiffe://}. Note — even if you write it with the scheme attached, istioctl validate passes. That policy matches nobody, and if it is an ALLOW policy, it blocks everyone. And a principal exists only for connections that came in over mTLS, so on a PERMISSIVE port that allows plain text, no request is caught by this rule.

Summarize on one sheet what each mode creates in Envoy

In /root/ist2-mtls/08-report.md, write four lines — default_mode= (the mode when there is no PeerAuthentication at all), strict_plain= (the code the plain-text request received in step 3), permissive_chains= (the number of filter chains of the step 5 listener) and orders_principal= (the principal from step 7) — and below them write explanations starting with - in at least four lines.

Copy the values from the files of the earlier steps. In the explanation, it is good to pair "mode → what is created in Envoy → the symptom you see in production". As for why the default mode is that value, recall the middle of putting sidecars into a mesh one after another.