Istio Deep Dive — Why It Flows That Way
Why Every Connection Passes the Proxy Without the App Knowing
In one line
Sidecar injection adds two containers to a Pod. istio-init, which runs once and finishes, plants numbers (15001, 15006, 1337) into iptables, and istio-proxy, which lives long, waits at those numbers. These numbers are why every connection passes through the proxy even though the app changed nothing.
Why this was needed
The promise of a service mesh is "attach mTLS, retries and observability to every call without changing code". For that, a connection the app sent to http://reviews:9080 has to be received by the proxy without the app's knowledge. The approach of planting a library has to be built separately for each language, and the app has to use that library. So Istio intercepts on the kernel side — it puts iptables rules into the Pod's network namespace and diverts outgoing connections to 15001 and incoming connections to 15006.
Two problems arise right away. First, if the proxy sends a request it received back out, that connection is also caught by the rules and returns to 15001. It is an endless loop. Second, changing the rules needs the NET_ADMIN permission, and if you give that permission to the always-running proxy, then when the proxy is compromised, it can change the Pod's whole network. The shape of the injection output is the answer to these two problems.
How it works
If you open the manifest produced by istioctl kube-inject, the numbers are scattered. Gathered together, they look like this.
| Number | Where it is written | Meaning |
|---|---|---|
| 15001 | istio-init -p |
Where connections the app sends outward are diverted to (virtualOutbound) |
| 15006 | istio-init -z |
Where connections coming in from outside are diverted to (virtualInbound) |
| 1337 | istio-init -u, istio-proxy runAsUser |
Packets sent by this user are not intercepted |
| 15021 | readinessProbe | The proxy's own readiness state |
| 15020 | the prometheus.io/port annotation |
Where the app's and the proxy's metrics are exported merged together |
| 15090 | the container port http-envoy-prom |
Envoy's own metrics |
The loop problem is solved with -u 1337. The proxy container is started with UID 1337, and packets sent by that UID are taken out of the rules. If the numbers in the two places do not match, the proxy receives its own requests again and burns CPU. The permission problem is solved by splitting the containers. istio-init runs as root holding NET_ADMIN and NET_RAW, plants only the rules and then finishes, while istio-proxy is not root, drops all capabilities, and has a read-only file system. -d 15090,15021,15020 are the ports to exclude from inbound interception. This is because the kubelet's health checks and Prometheus scraping must not go through Envoy's routing.
From the Envoy side, 15006 is the virtualInbound listener, and it looks at the original destination port and passes to the cluster inbound|<포트>|| (the placeholder is the port). 15001 is virtualOutbound, and if the destination is one the mesh knows, it goes to that service's listener, and if not, it goes to PassthroughCluster (just let it flow through) or BlackHoleCluster (no endpoints, so 503). Which one it is is decided by outboundTrafficPolicy in the mesh configuration.
What it looks like in the field
After injection, requests sent before the app is up fail. If the app starts before the proxy is ready and connects outward, the rules already exist but nobody at 15001 is there to receive. That is why holdApplicationUntilProxyStarts exists. If you look at the order of containers in the injection output, you can see how that setting is reflected.
"Only external APIs get a 503." If BlackHoleCluster is printed in the log, the mesh is REGISTRY_ONLY and there is no ServiceEntry telling it about that host. The response body no healthy upstream was made not by the app but by the sidecar.
The app starts as 1337. If you happen to start the app container with UID 1337, the app's connections are also excluded from interception and go out in plain text with no mTLS. No error appears, so it only comes to light in a security check.
Official documentation: Application requirements · Debugging Envoy and Istiod
What you will do in the next lab
You produce a real injection output offline and pull out, one by one with yq, the places where the numbers are written. Next you set up virtualInbound and virtualOutbound by hand and see for yourself a request going all the way to the app and a request falling to BlackHoleCluster.