Istio Deep Dive — Why It Flows That Way
Plaintext is blocked because only one filter chain is left
In one line
PeerAuthentication changes the incoming listener of the receiving side's sidecar. STRICT becomes one filter chain that accepts only connections judged to be TLS (a client certificate is required and is verified against the mesh CA), and PERMISSIVE is that plus one more plain-text chain with no match condition. The SPIFFE ID in the certificate's SAN is the identity, and the string with the leading spiffe:// removed becomes the principal of an AuthorizationPolicy.
Why this was needed
A Kubernetes network is plain text by default, and a Pod IP is not an identity. When a Pod starts again, its IP changes, and another workload inherits the same IP. So if you write "only orders may call reviews" in terms of IPs, it is soon wrong. Istio tied identity to the service account. After istiod checks the Pod's service account token, it signs a certificate that has a URI such as spiffe://cluster.local/ns/default/sa/orders in its SAN, and sidecars talk to one another over mTLS, verifying each other with this certificate.
The problem is that a mesh is not built all at once. Sidecars go in one after another, namespace by namespace and deployment by deployment. If the receiving side rejects plain text in the middle of that, calls from Pods that do not yet have a sidecar are all cut. So the default is PERMISSIVE — it accepts both mTLS and plain text. If the other side has a sidecar, the calling sidecar sends over mTLS by itself (auto mTLS), so once injection is finished you can tighten to STRICT. PeerAuthentication is the resource that writes down that tightening.
How it works
istiod translates PeerAuthentication into the receiving sidecar's virtualInbound (15006) listener. The key is the listener filter tls_inspector. It looks at the first bytes of the connection, and if it is a TLS ClientHello it marks transport_protocol as tls, and Envoy chooses the filter chain by that mark.
| PeerAuthentication | Envoy's inbound listener |
|---|---|
| STRICT | One transport_protocol: tls chain. require_client_certificate: true, and trusted_ca is the mesh root |
| PERMISSIVE (default) | The chain above + a plain-text chain with no match condition |
| DISABLE | Only the plain-text chain |
portLevelMtls |
A separate chain is created that matches that port with destination_port |
Under STRICT, a plain-text connection has no matching chain, so Envoy closes it without even an HTTP response. To the client it looks like connection reset, and on Envoy only the no_filter_chain_match statistic is left. The number in portLevelMtls is not the Service's port but the workload port the container listens on. Because iptables diverts to 15006 while preserving the original destination port, that is the number the chain sees. portLevelMtls is accepted only in a policy that has a selector.
Identity verification has two layers. Signature verification checks only "did the mesh CA sign it". A certificate from the same CA passes whoever it belongs to. "Who is it" is answered by the SAN. Istio applies the SAN check mainly on the calling side — this is secure naming, where the client checks the SPIFFE ID of the server certificate against the account the service is expected to have. On the receiving side, "who may call" is the AuthorizationPolicy's job. When mTLS finishes, Envoy remembers the other side's certificate URI SAN as the connection's principal, and source.principals: ["cluster.local/ns/default/sa/orders"] is checked against it. It is the same value with only the scheme dropped from the notation.
What it looks like in the field
After switching to STRICT, only some calls get connection reset by peer. They are from Pods with no sidecar on the calling side (cron jobs, batch jobs with the sidecar left out, monitoring outside the mesh). In the receiving side's log there is no request at all — because it was cut at the stage of choosing the chain. If no_filter_chain_match is rising, this is the case.
I want to leave only the health check or metrics port as plain text. Set DISABLE on just that port with portLevelMtls. If you write the Service port number there, it matches no chain and is quietly ignored. Write the targetPort.
I added an AuthorizationPolicy and everything is 403. Either you put spiffe:// on the principals, or the trust domain differs (an installation that is not cluster.local), or the call came in as plain text and the principal is empty. istioctl validate passes in all three cases.
Official documentation: PeerAuthentication · Security concepts · AuthorizationPolicy · Envoy TLS
What you will do in the next lab
You write a STRICT PeerAuthentication and validate it, and then create a mesh CA and three SPIFFE certificates with openssl. With those certificates you set up by hand, as Envoy chains, STRICT, a SAN matcher, PERMISSIVE and a port-level exception, and confirm what happens to plain text and to mTLS in each, and you derive the AuthorizationPolicy principal from the SAN.