Run three TLS modes and check who presents the certificate

Goal

Translate into rules what the TLS mode of a Gateway becomes on the gateway Envoy listener, start SIMPLE, MUTUAL, PASSTHROUGH and httpsRedirect as equivalent listeners, and prove with the certificate the client received where TLS ends.

Why it matters

Gateway certificate outages usually come from misunderstanding "where does TLS end". You fixed the gateway certificate but the client is looking at the backend certificate, or you mistake a connection that could not choose a chain for lack of SNI for a certificate problem. If you know what is created in Envoy for each mode, you can tell them apart with one listener configuration and one line of statistics.

Steps

Write a Gateway to /root/ist2-gw/gw.yaml — name shop-gw, namespace default, selector istio: ingressgateway, two servers: (1) port 443, name https, protocol HTTPS, hosts shop.example.com, tls.mode: SIMPLE, tls.credentialName: shop-cert; (2) port 80, name http, protocol HTTP, hosts shop.example.com, tls.httpsRedirect: true. Put the output and exit code of istioctl validate -f gw.yaml in /root/ist2-gw/01-validate.txt (with rc= on the last line). Then make a copy with only the credentialName line removed as /root/ist2-gw/gw-nocred.yaml, check it with the same command, and put the result in /root/ist2-gw/01-nocred.txt.
Write five lines to /root/ist2-gw/02-modes.txt — for each of SIMPLE, MUTUAL, OPTIONAL_MUTUAL, PASSTHROUGH and ISTIO_MUTUAL, one line of the form <모드> terminates=<gateway|upstream> client_cert=<none|required|optional|mesh> filter=<hcm|tcp_proxy> needs_vs=<yes|no> (where the placeholder is the mode; separated by spaces). terminates is where TLS is decrypted, client_cert is how the gateway treats the client certificate (mesh means it requires a mesh certificate issued by istiod), filter is the last network filter of the chain, and needs_vs is whether a VirtualService must exist for traffic to flow. Treat the server's protocol as HTTPS (TLS for PASSTHROUGH).
Create certificates with openssl in /root/ist2-gw/certs — a CA (ca.crt and ca.key, subject /O=lab/CN=lab-ca) and a gateway certificate signed by that CA (gateway.crt and gateway.key, subject /O=gateway/CN=shop.example.com, SAN DNS:shop.example.com). Then write an Envoy configuration to /root/ist2-gw/gw-simple.yaml — admin port 9988, on the chain of the listener 127.0.0.1:10088 a DownstreamTlsContext (the gateway certificate) + HCM, the route configuration name is https.443.https.shop-gw.default, the name Istio attaches to this HTTPS server, domains shop.example.com, and the cluster outbound|8117||shop.default.svc.cluster.local → 127.0.0.1:8117 (a plain-text upstream, upstream.py 8117 ok). After you start it, request with shop.example.com and write four lines to /root/ist2-gw/03-simple.txt — code= (the HTTP code), body= (the response body), seen_o= (only the O value of the subject of the certificate the client received) and seen_sha256= (the SHA-256 fingerprint of that certificate, what comes after the = in the output of openssl x509 -fingerprint -sha256).
With the same CA, create a client certificate /root/ist2-gw/certs/client.crt and client.key (subject /O=client/CN=client, SAN DNS:client). /root/ist2-gw/gw-mutual.yaml is the same as gw-simple.yaml with require_client_certificate: true and validation_context.trusted_ca (/root/ist2-gw/certs/ca.crt) added to the DownstreamTlsContext. After you start it, write four lines to /root/ist2-gw/04-mutual.txt — without_cert_code= and without_cert_rc= (the HTTP code and the curl exit code of a request without a client certificate), with_cert_code= (the HTTP code of a request with --cert and --key) and fail_verify_no_cert= (the value of the statistic listener.127.0.0.1_10088.ssl.fail_verify_no_cert).
With the same CA, create a backend certificate /root/ist2-gw/certs/backend.crt and backend.key (subject /O=backend/CN=shop.example.com, SAN DNS:shop.example.com), and start a TLS upstream with openssl s_server -accept 8111 -cert /root/ist2-gw/certs/backend.crt -key /root/ist2-gw/certs/backend.key -www -quiet. Write a PASSTHROUGH gateway to /root/ist2-gw/gw-pass.yaml — admin port 9988, the tls_inspector listener filter on the listener 127.0.0.1:10088, and for the chain filter_chain_match.server_names: ["shop.example.com"] and one tcp_proxy (no transport_socket), cluster outbound|8111||shop.default.svc.cluster.local → 127.0.0.1:8111. After you start it, request with shop.example.com and write three lines to /root/ist2-gw/05-passthrough.txt — code=, seen_o= and seen_sha256= (the same way as in step 3).
Connect twice to the gateway started with gw-pass.yaml — (1) with the SNI as other.example.com (--resolve other.example.com:10088:127.0.0.1), (2) without SNI, by IP (https://127.0.0.1:10088/). Give both --cacert /root/ist2-gw/certs/ca.crt. Write five lines to /root/ist2-gw/06-sni.txt — sni=other.example.com, sni_rc= (the curl exit code of (1)), no_sni_rc= (the exit code of (2)), stat= (the full name of the listener statistic that counts connections that could not choose a chain) and count= (the value of that statistic after the two connections).
Write a plain-text listener corresponding to the port 80 server of step 1 to /root/ist2-gw/gw-redirect.yaml — admin port 9988, listener 127.0.0.1:10098 (no transport_socket), the HCM's route configuration name is http.80, the name Istio attaches to a plain-text port 80 server, shop.example.com in the virtual host's domains, require_tls: ALL on that virtual host, and the route outbound|8117||shop.default.svc.cluster.local → 127.0.0.1:8117. After you start it, write the result of curl -H 'Host: shop.example.com' localhost:10098/cart to /root/ist2-gw/07-redirect.txt as two lines — code= (the HTTP code) and location= (the Location header value as it is).
In /root/ist2-gw/08-report.md, write five lines — simple_seen_o= (the O you saw in step 3), passthrough_seen_o= (step 5), mutual_without_cert_code= (step 4), sni_mismatch_rc= (the exit code of (1) in step 6) and redirect_code= (step 7). Below them, write explanations starting with - in at least four lines.

Notes

This Pod has neither a real istiod nor a real sidecar. So you cannot see the actual generated output with istioctl proxy-config; instead you know the translation rules and build the equivalent Envoy configuration by hand to confirm the behavior. The same rules show up as they are in the proxy-config output of a production cluster.
All certificates are signed by the one CA in /root/ist2-gw/certs. Create the CA only once, in step 3 — if you create it again, the certificates you signed earlier cannot be verified by the new CA. To carry the SAN over, you need -copy_extensions copyall when signing.
The listeners in steps 3, 4 and 5 use the same port 10088. Start only one at a time, and keep a separate configuration file for each step.
Also start the TLS upstream (openssl s_server) detached from the shell with setsid --fork nohup … </dev/null, and when you start it again, clean up with pkill -x openssl.
When you start Envoy, detach it completely from the shell with setsid --fork nohup envoy -c <파일> --log-level warn > <로그> 2>&1 </dev/null (the placeholders are the file and the log). Before you start it again, clean up with pkill -x envoy (pkill -f 'envoy -c' also kills the shell itself that contains that string).
A server for imitating an upstream is already in the image: python3 /opt/lab/envoy/upstream.py <포트> ok|fail|slow (the placeholder is the port). The response body is <모드>:<포트> <경로> (the placeholders are the mode, the port and the path).
After you edit the configuration, first filter it with envoy --mode validate -c <파일> before you start it. The cluster name contains |, so in YAML you must always wrap it in quotes.

Write the two Gateway servers and see what istioctl demands of SIMPLE

Write a Gateway to /root/ist2-gw/gw.yaml — name shop-gw, namespace default, selector istio: ingressgateway, two servers: (1) port 443, name https, protocol HTTPS, hosts shop.example.com, tls.mode: SIMPLE, tls.credentialName: shop-cert; (2) port 80, name http, protocol HTTP, hosts shop.example.com, tls.httpsRedirect: true. Put the output and exit code of istioctl validate -f gw.yaml in /root/ist2-gw/01-validate.txt (with rc= on the last line). Then make a copy with only the credentialName line removed as /root/ist2-gw/gw-nocred.yaml, check it with the same command, and put the result in /root/ist2-gw/01-nocred.txt.

A Gateway is the resource that opens only ports and certificates on the Envoy of the gateway Pod. Routes appear only when a VirtualService is attached. SIMPLE means "the gateway ends TLS", so a server certificate must exist, and Istio gets it from the Secret that credentialName points to. That is why, if you remove that line, istioctl rejects it even without a cluster — put the rejection reason in as it is. The exit code is the $? right after the command. You can remove just one line with sed '/credentialName/d'.

Write the five TLS modes out in Envoy's shape

Write five lines to /root/ist2-gw/02-modes.txt — for each of SIMPLE, MUTUAL, OPTIONAL_MUTUAL, PASSTHROUGH and ISTIO_MUTUAL, one line of the form <모드> terminates=<gateway|upstream> client_cert=<none|required|optional|mesh> filter=<hcm|tcp_proxy> needs_vs=<yes|no> (where the placeholder is the mode; separated by spaces). terminates is where TLS is decrypted, client_cert is how the gateway treats the client certificate (mesh means it requires a mesh certificate issued by istiod), filter is the last network filter of the chain, and needs_vs is whether a VirtualService must exist for traffic to flow. Treat the server's protocol as HTTPS (TLS for PASSTHROUGH).

There is one fork — whether a DownstreamTlsContext is created on the gateway Envoy's chain. If it is, the gateway decrypts, and since it decrypted, it can read HTTP and route with the HCM. If it is not, the gateway can see only ciphertext, so it has no choice but to choose a chain by SNI and hand the bytes over. Think of the client certificate as a combination of Envoy's two values, require_client_certificate and validation_context. The AUTO_PASSTHROUGH in the sample line is an exception where the destination is written in the SNI itself, so no VirtualService is needed.

SIMPLE — the gateway ends TLS and presents its own certificate

Create certificates with openssl in /root/ist2-gw/certs — a CA (ca.crt and ca.key, subject /O=lab/CN=lab-ca) and a gateway certificate signed by that CA (gateway.crt and gateway.key, subject /O=gateway/CN=shop.example.com, SAN DNS:shop.example.com). Then write an Envoy configuration to /root/ist2-gw/gw-simple.yaml — admin port 9988, on the chain of the listener 127.0.0.1:10088 a DownstreamTlsContext (the gateway certificate) + HCM, the route configuration name is https.443.https.shop-gw.default, the name Istio attaches to this HTTPS server, domains shop.example.com, and the cluster outbound|8117||shop.default.svc.cluster.local → 127.0.0.1:8117 (a plain-text upstream, upstream.py 8117 ok). After you start it, request with shop.example.com and write four lines to /root/ist2-gw/03-simple.txt — code= (the HTTP code), body= (the response body), seen_o= (only the O value of the subject of the certificate the client received) and seen_sha256= (the SHA-256 fingerprint of that certificate, what comes after the = in the output of openssl x509 -fingerprint -sha256).

One SIMPLE server is, in Envoy, one transport_socket on one chain. Istio puts in the certificate not as a file but through SDS (kubernetes://shop-cert), but even if you put it in as a file, what Envoy sees is the same. To connect by name, use curl --resolve 호스트:포트:127.0.0.1 --cacert (the placeholders are the host and the port), and for the certificate the client actually received, pass the output of openssl s_client -connect … -servername … to openssl x509. To carry the SAN over into the certificate, you need -copy_extensions copyall when signing. The route name rule is https.<포트>.<포트이름>.<Gateway이름>.<네임스페이스> (the placeholders are the port, the port name, the Gateway name and the namespace).

MUTUAL — a client certificate requirement is added to the same chain

With the same CA, create a client certificate /root/ist2-gw/certs/client.crt and client.key (subject /O=client/CN=client, SAN DNS:client). /root/ist2-gw/gw-mutual.yaml is the same as gw-simple.yaml with require_client_certificate: true and validation_context.trusted_ca (/root/ist2-gw/certs/ca.crt) added to the DownstreamTlsContext. After you start it, write four lines to /root/ist2-gw/04-mutual.txt — without_cert_code= and without_cert_rc= (the HTTP code and the curl exit code of a request without a client certificate), with_cert_code= (the HTTP code of a request with --cert and --key) and fail_verify_no_cert= (the value of the statistic listener.127.0.0.1_10088.ssl.fail_verify_no_cert).

MUTUAL is two values stacked on top of SIMPLE — "you must present a certificate" (require_client_certificate) and "verify it against what" (validation_context). In Istio, the CA for verification comes from ca.crt of the credentialName Secret (or <이름>-cacert, where the placeholder is the name). If it connects without a certificate, it cannot reach HTTP, so the code is the value that means there was no response, and the exit code differs by TLS version — write it as it is. The rejection is left in Envoy's statistics (grep in /stats).

PASSTHROUGH — the gateway reads only the SNI and the backend certificate is what you see

With the same CA, create a backend certificate /root/ist2-gw/certs/backend.crt and backend.key (subject /O=backend/CN=shop.example.com, SAN DNS:shop.example.com), and start a TLS upstream with openssl s_server -accept 8111 -cert /root/ist2-gw/certs/backend.crt -key /root/ist2-gw/certs/backend.key -www -quiet. Write a PASSTHROUGH gateway to /root/ist2-gw/gw-pass.yaml — admin port 9988, the tls_inspector listener filter on the listener 127.0.0.1:10088, and for the chain filter_chain_match.server_names: ["shop.example.com"] and one tcp_proxy (no transport_socket), cluster outbound|8111||shop.default.svc.cluster.local → 127.0.0.1:8111. After you start it, request with shop.example.com and write three lines to /root/ist2-gw/05-passthrough.txt — code=, seen_o= and seen_sha256= (the same way as in step 3).

In PASSTHROUGH the gateway holds neither a certificate nor a key. Even so, it has to choose a destination, so it peeks with tls_inspector at only the SNI carried in plain text in the first message of TLS (the ClientHello), and matches that name against server_names to choose the chain. It did not decrypt, so it cannot use the HCM, and tcp_proxy hands the bytes over as they are. So the other party in the handshake is the backend, and the certificate the client receives is the backend's too — compare the fingerprint with the result of step 3. s_server -www answers an HTTP request with 200 and a status page.

If the SNI does not match or is absent, there is no chain to choose

Connect twice to the gateway started with gw-pass.yaml — (1) with the SNI as other.example.com (--resolve other.example.com:10088:127.0.0.1), (2) without SNI, by IP (https://127.0.0.1:10088/). Give both --cacert /root/ist2-gw/certs/ca.crt. Write five lines to /root/ist2-gw/06-sni.txt — sni=other.example.com, sni_rc= (the curl exit code of (1)), no_sni_rc= (the exit code of (2)), stat= (the full name of the listener statistic that counts connections that could not choose a chain) and count= (the value of that statistic after the two connections).

The PASSTHROUGH listener has only one chain with server_names attached and no default chain. If the name is different, or curl connects by IP and sends no SNI at all, Envoy has no chain to choose and closes the connection right away — to curl it looks like a handshake failure. The listener statistic names start with listener.<주소>_<포트>. (the placeholders are the address and the port). Look for it in /stats with filter_chain.

httpsRedirect is one line of require_tls on the virtual host

Write a plain-text listener corresponding to the port 80 server of step 1 to /root/ist2-gw/gw-redirect.yaml — admin port 9988, listener 127.0.0.1:10098 (no transport_socket), the HCM's route configuration name is http.80, the name Istio attaches to a plain-text port 80 server, shop.example.com in the virtual host's domains, require_tls: ALL on that virtual host, and the route outbound|8117||shop.default.svc.cluster.local → 127.0.0.1:8117. After you start it, write the result of curl -H 'Host: shop.example.com' localhost:10098/cart to /root/ist2-gw/07-redirect.txt as two lines — code= (the HTTP code) and location= (the Location header value as it is).

Istio puts require_tls: ALL into the virtual host of a server with httpsRedirect: true. Then Envoy sends a plain-text request back to https before it even looks at the route — it does not go as far as the upstream. The route name of a plain HTTP server is http.<포트> (the placeholder is the port), without the gateway name, so the hosts of several Gateways that use the same port are merged into one route configuration. Look at the headers with curl -s -D - -o /dev/null. The host in Location comes from the request's Host header.

Summarize who ends TLS for each mode

In /root/ist2-gw/08-report.md, write five lines — simple_seen_o= (the O you saw in step 3), passthrough_seen_o= (step 5), mutual_without_cert_code= (step 4), sni_mismatch_rc= (the exit code of (1) in step 6) and redirect_code= (step 7). Below them, write explanations starting with - in at least four lines.

Copy the values from the files of the earlier steps. If in the explanation lines you pair "what is created on the gateway Envoy chain in this mode, and so what the client sees", you will have sorted out where to look first when you meet a certificate problem in production.