Read the Injected Numbers and Build Both Entrances

Goal

Produce a sidecar injection output offline, find the places where the interception numbers are written, and set up by hand to confirm which Envoy listeners and clusters those numbers become.

Why it matters

When a sidecar acts strangely, the first thing you look at is the injection output. If one number is off, the proxy receives its own requests again (UID), health checks go through routing (-d), or every external call becomes a 503 (BlackHoleCluster). If you can read the numbers, you can tell these three apart from the symptoms alone.

Steps

Create /root/ist2-inject, and inside it use istioctl kube-inject to put a sidecar into /opt/lab/fixtures/istio/inject-target.yaml and save it as /root/ist2-inject/inject.yaml (pass all three injection configuration files /opt/istio/inject-config.yaml, mesh-config.yaml and values-config.yaml). Then write three lines to /root/ist2-inject/01-containers.txt — containers= (the container names after injection, in order, separated by commas), init= (the init container name) and proxy_image= (the image of the istio-proxy container).
Read the args of the istio-init container in /root/ist2-inject/inject.yaml and write five lines to /root/ist2-inject/02-redirect.txt — outbound_port= (the value of -p), inbound_port= (-z), proxy_uid= (-u), mode= (-m) and excluded_inbound_ports= (-d, exactly as joined by commas).
Find three ports in /root/ist2-inject/inject.yaml and write them to /root/ist2-inject/03-ports.txt — readiness_port= (readinessProbe.httpGet.port of istio-proxy), merged_metrics_port= (the Pod template annotation prometheus.io/port) and envoy_prom_port= (the one named http-envoy-prom among the ports of istio-proxy). On the last line, all_excluded=, write yes if the three ports are all in the -d list of istio-init, otherwise no.
Read the securityContext in /root/ist2-inject/inject.yaml and write five lines to /root/ist2-inject/04-uid.txt — proxy_uid= (runAsUser of istio-proxy), proxy_run_as_non_root= (runAsNonRoot of the same container), proxy_caps_drop= (capabilities.drop, separated by commas), init_run_as_user= (runAsUser of istio-init) and init_caps_add= (capabilities.add of istio-init, separated by commas, in manifest order).
Write an Envoy configuration to /root/ist2-inject/mesh.yaml — admin port 9981, a listener named virtualInbound listening at 127.0.0.1:15006 with traffic_direction: INBOUND, sending every path to the cluster inbound|8101|| (playing the role of the app, 127.0.0.1:8101). Start an upstream on 8101 as ok and start Envoy, then write the result of curl localhost:15006/orders to /root/ist2-inject/05-inbound.txt as two lines, status= (the HTTP code) and body= (one line of the response body).
Add a second listener to /root/ist2-inject/mesh.yaml (leave the earlier virtualInbound as it is) — named virtualOutbound, 127.0.0.1:10081, traffic_direction: OUTBOUND, sending every path to the cluster BlackHoleCluster. BlackHoleCluster is a cluster with type: STATIC and no endpoints at all. After you start it again, write the result of curl localhost:10081/ to /root/ist2-inject/06-outbound.txt as three lines, status=, body= and cluster= (the name of the cluster this listener's route points to).
From localhost:9981/config_dump?resource=static_listeners of the running Envoy, pull out one line per listener, with 이름 방향 포트 (the placeholders are the name, the direction and the port) separated by spaces, and save it to /root/ist2-inject/07-listeners.txt (two lines, the order does not matter).
In /root/ist2-inject/08-report.md, write four lines, outbound_capture=, inbound_capture=, proxy_uid= and unknown_destination= (respectively the port outgoing traffic is diverted into, the port for incoming traffic, the user ID excluded from interception, and the name of the cluster the request with an unknown destination went to in step 6), and below them write explanations starting with - in at least four lines.

Notes

This Pod has neither a real istiod nor a real sidecar. So you cannot see the actual generated output with istioctl proxy-config; instead you know the translation rules and build the equivalent Envoy configuration by hand to confirm the behavior. The same rules show up as they are in the proxy-config output of a production cluster.
The injection output has an empty document (--- and null) attached at the end. When you read it with yq, always filter with select(.kind=="Deployment") | …. kube-inject has no -o json, so if you need JSON, use yq -o=json.
mikefarah yq does not know jq's "\(.a)" interpolation — use .a + " " + .b or pass it to jq.
When you start Envoy, detach it completely from the shell with setsid --fork nohup envoy -c <파일> --log-level warn > <로그> 2>&1 </dev/null (the placeholders are the file and the log). Before you start it again, clean up with pkill -x envoy (pkill -f 'envoy -c' also kills the shell itself that contains that string).
A server for imitating an upstream is already in the image: python3 /opt/lab/envoy/upstream.py <포트> ok|fail|slow (the placeholder is the port). The response body is <모드>:<포트> <경로> (the placeholders are the mode, the port and the path).
After you edit the configuration, first filter it with envoy --mode validate -c <파일> before you start it. The cluster name contains |, so in YAML you must always wrap it in quotes.

Produce the injection output and count what grew

Create /root/ist2-inject, and inside it use istioctl kube-inject to put a sidecar into /opt/lab/fixtures/istio/inject-target.yaml and save it as /root/ist2-inject/inject.yaml (pass all three injection configuration files /opt/istio/inject-config.yaml, mesh-config.yaml and values-config.yaml). Then write three lines to /root/ist2-inject/01-containers.txt — containers= (the container names after injection, in order, separated by commas), init= (the init container name) and proxy_image= (the image of the istio-proxy container).

There is no istiod in this Pod, so kube-inject cannot fetch the injection configuration from the cluster as it usually does. You must pass all three configuration files for it not to go outside — if you give only two, it goes out to look for the remaining one. An empty document is attached at the end of the output, so when you read it with yq, filter with select(.kind=="Deployment"). Pull the values out with yq instead of copying them by eye, and you will not get them wrong.

Read the interception design from the arguments of istio-init

Read the args of the istio-init container in /root/ist2-inject/inject.yaml and write five lines to /root/ist2-inject/02-redirect.txt — outbound_port= (the value of -p), inbound_port= (-z), proxy_uid= (-u), mode= (-m) and excluded_inbound_ports= (-d, exactly as joined by commas).

In istio-iptables, each argument is one chunk of rules. -p is the port that connections the app sends outward are diverted into, -z is the port that connections coming in from outside are diverted into, and -u is the user ID not to intercept. args is a list of strings, so the element right after a flag is its value — you can pull out "the line after a flag" with awk. There are also flags like -x whose value is an empty string, so do not count by position.

Find out whom the ports excluded from interception are for

Find three ports in /root/ist2-inject/inject.yaml and write them to /root/ist2-inject/03-ports.txt — readiness_port= (readinessProbe.httpGet.port of istio-proxy), merged_metrics_port= (the Pod template annotation prometheus.io/port) and envoy_prom_port= (the one named http-envoy-prom among the ports of istio-proxy). On the last line, all_excluded=, write yes if the three ports are all in the -d list of istio-init, otherwise no.

I said incoming connections are all diverted to 15006, but if the kubelet's health checks and Prometheus scraping were diverted too, they would go through Envoy's routing rules and end up somewhere else. So the ports the proxy must receive directly are taken out of interception with -d. The annotation name has a dot and a slash, so read it with brackets, as in .metadata.annotations["prometheus.io/port"].

See where UID 1337 and the permissions are divided

Read the securityContext in /root/ist2-inject/inject.yaml and write five lines to /root/ist2-inject/04-uid.txt — proxy_uid= (runAsUser of istio-proxy), proxy_run_as_non_root= (runAsNonRoot of the same container), proxy_caps_drop= (capabilities.drop, separated by commas), init_run_as_user= (runAsUser of istio-init) and init_caps_add= (capabilities.add of istio-init, separated by commas, in manifest order).

Changing the iptables rules needs NET_ADMIN, and that permission is given only to the init container, which runs once and finishes. The long-lived proxy starts neither as root nor with any capabilities. And the proxy's UID must be the same as the -u value in step 2 — if not, requests the proxy sent get diverted back into the proxy and go around endlessly. Join lists with join(",") in yq.

Set up virtualInbound at 15006 and send to the app

Write an Envoy configuration to /root/ist2-inject/mesh.yaml — admin port 9981, a listener named virtualInbound listening at 127.0.0.1:15006 with traffic_direction: INBOUND, sending every path to the cluster inbound|8101|| (playing the role of the app, 127.0.0.1:8101). Start an upstream on 8101 as ok and start Envoy, then write the result of curl localhost:15006/orders to /root/ist2-inject/05-inbound.txt as two lines, status= (the HTTP code) and body= (one line of the response body).

This listener is exactly the sidecar's incoming entrance. iptables diverts all connections coming to the Pod to 15006, and here it looks at the original destination port and passes to the cluster inbound|<포트>|| (the placeholder is the port) — the middle slot of the name is empty because there is no subset on the incoming side. The cluster name contains |, so wrap it in quotes in YAML. traffic_direction does not change behavior, but the direction is left in the statistics and in config_dump.

A request with an unknown destination goes to BlackHoleCluster

Add a second listener to /root/ist2-inject/mesh.yaml (leave the earlier virtualInbound as it is) — named virtualOutbound, 127.0.0.1:10081, traffic_direction: OUTBOUND, sending every path to the cluster BlackHoleCluster. BlackHoleCluster is a cluster with type: STATIC and no endpoints at all. After you start it again, write the result of curl localhost:10081/ to /root/ist2-inject/06-outbound.txt as three lines, status=, body= and cluster= (the name of the cluster this listener's route points to).

When blocking requests to destinations the mesh does not know about (outboundTrafficPolicy: REGISTRY_ONLY), Istio sends those requests to a cluster with no endpoints. In the default (ALLOW_ANY), they go instead to PassthroughCluster, which just lets them flow through to the original destination. A cluster with 0 endpoints is made by not writing load_assignment at all. Copy the body Envoy returns as it is — if you know this wording, you will recognize it right away in production logs.

Read back the name, direction and port of the two entrances from config_dump

From localhost:9981/config_dump?resource=static_listeners of the running Envoy, pull out one line per listener, with 이름 방향 포트 (the placeholders are the name, the direction and the port) separated by spaces, and save it to /root/ist2-inject/07-listeners.txt (two lines, the order does not matter).

What you wrote in the file and what Envoy actually read in can differ. What istioctl proxy-config listeners shows in production is in the end this dump. If you narrow it with resource=static_listeners, each listener is in .configs[].listener one by one. Join the three values on one line with jq -r string interpolation.

Summarize it as a table of interception numbers

In /root/ist2-inject/08-report.md, write four lines, outbound_capture=, inbound_capture=, proxy_uid= and unknown_destination= (respectively the port outgoing traffic is diverted into, the port for incoming traffic, the user ID excluded from interception, and the name of the cluster the request with an unknown destination went to in step 6), and below them write explanations starting with - in at least four lines.

You can copy the numbers from the files of the earlier steps — do not write them by guessing; look again at 02-redirect.txt and 06-outbound.txt. If in the explanation lines you write "why it was designed that way" rather than "what I did", this table will be useful the next time you run into an outage.