Istio Deep Dive — Why It Flows That Way
Reading Hundreds of proxy-config Lines as a Table
In one line
The sidecar's cluster name is made by rule in four slots, 방향|포트|서브셋|호스트 (direction, port, subset, host). One subset of a DestinationRule becomes one cluster, and statistics and error messages all come out wearing this name. If you can read the name, hundreds of lines of proxy-config output look like a table.
Why this was needed
The first time you open istioctl proxy-config clusters <파드> (the placeholder is the Pod), lines like these come out endlessly.
outbound|9080||reviews.default.svc.cluster.local
outbound|9080|v1|reviews.default.svc.cluster.local
outbound|9080|v2|reviews.default.svc.cluster.local
inbound|9080||
If you cannot read this name, you cannot do two things. First, when you look at "why is a request going to v2 a 503", you cannot confirm whether the v2 subset cluster exists at all or how many endpoints it has. Second, you cannot tell what service and what a metric such as cluster.outbound|9080|v2|…upstream_rq_503 on a dashboard is about. Istio has to create thousands of clusters without human hands, so it fixed the names as a rule, and that rule is the way to read them.
How it works
| Slot | Value | When it is empty |
|---|---|---|
| Direction | outbound (when this Pod calls others) / inbound (when others call this Pod) |
Never |
| Port | The service port (9080) |
Never |
| Subset | subsets[].name of the DestinationRule |
The default cluster where no subset was picked |
| Host | The service's FQDN | The incoming side (because it sends to its own app) |
For every service and port in the mesh, istiod always creates a cluster with no subset, and if a DestinationRule has subsets, it creates one more for each subset. The endpoints of a subset cluster are a subset of the endpoints of the same service, picked by matching labels. So a Pod without the label is in none of the subset clusters.
A VirtualService's destination: {host: reviews, subset: v1} is translated as a route pointing to the cluster outbound|9080|v1|reviews…. This is where a common accident comes from. If you point to a subset that is not in the DestinationRule, istiod sends down a route that points to a cluster that does not exist. Envoy does not check in advance the clusters of dynamically received routes (the default of validate_clusters is false), so the configuration is accepted, and a 503 comes when a request arrives, with NC (no cluster) printed in the access log. A static configuration written in a file has the default true, so it rejects the same configuration at startup — where the same mistake shows up differs.
Statistics accumulate as cluster.<이름>.<지표> (the placeholders are the cluster name and the metric). Names containing | are troublesome to carry over into Prometheus labels, so if you give a pattern to outboundClusterStatName in the mesh configuration, istiod attaches an alt_stat_name to each cluster. Routing is done with the original name and only the statistics name changes.
What it looks like in the field
I set up a canary and only the requests going to v2 all get a 503. This is the case where the VirtualService was deployed first and the DestinationRule later. For a few seconds to a few minutes in between, the v2 subset cluster does not exist. First look at whether the v2 line is in proxy-config clusters. That is why the order is always DestinationRule first and VirtualService after.
The subset cluster has 0 endpoints. If the name is there but proxy-config endpoints is empty, the subset's labels and the Pod labels have drifted apart (version: v2 versus version: 2.0). The response flag at that time is not NC but UH (no healthy upstream), so from the two letters of the flag alone you can tell "the cluster does not exist" from "the cluster exists but is empty".
The dashboard suddenly went empty. If someone changes outboundClusterStatName in the mesh configuration, the statistics names change wholesale and every query built on the old names goes empty. Routing is fine, so it is discovered only much later.
Official documentation: Debugging Envoy and Istiod · DestinationRule
What you will do in the next lab
You write a DestinationRule and derive four cluster names by the rule, then set up Envoy with exactly those names and split requests by subset. You change the statistics name with alt_stat_name, point at a nonexistent subset to create a 503 and no_cluster, and then count with /clusters that a subset is a subset of the same endpoints.