Turn Subsets into Clusters and Trace Them by Name

Goal

Derive cluster names by rule from a DestinationRule, configure Envoy with exactly those names, and confirm for yourself per-subset routing, statistics names and the 503 for a nonexistent subset.

Why it matters

Half of Istio outage investigation is reading proxy-config clusters and statistics. Cluster names are made by rule, so if you know the rule, you can recognize the service, port and subset from the name alone, and you can tell "the subset does not exist", "the endpoints are empty" and "the statistics name changed" apart in seconds.

Steps

Write a DestinationRule to /root/ist2-name/dr.yaml — name reviews, namespace default, host reviews.default.svc.cluster.local, subsets v1 (label version: v1) and v2 (label version: v2). Put the output and exit code of istioctl validate -f /root/ist2-name/dr.yaml in /root/ist2-name/01-validate.txt (the last line is rc=0).
The service reviews uses port 9080. From /root/ist2-name/dr.yaml, write the four cluster names the sidecars will have to /root/ist2-name/02-names.txt, one per line — three used when other Pods go out to reviews (one with no subset and one for each subset), and one used when the reviews Pod itself passes incoming requests to the app.
Write an Envoy configuration to /root/ist2-name/name.yaml — admin port 9983, listener 127.0.0.1:10083 (HTTP), route_config name 9080, sending every path to outbound|9080|v1|reviews.default.svc.cluster.local. There are three outgoing clusters — outbound|9080||reviews.default.svc.cluster.local (two endpoints, 127.0.0.1:8103 and 127.0.0.1:8104), outbound|9080|v1|reviews.default.svc.cluster.local (8103 only) and outbound|9080|v2|reviews.default.svc.cluster.local (8104 only). Start two upstreams on 8103 and 8104 as ok and start Envoy, then write the result of curl localhost:10083/ to /root/ist2-name/03-v1.txt as two lines, status= and body=.
Copy /root/ist2-name/name.yaml to /root/ist2-name/split.yaml and change the route to a weighted route — outbound|9080|v1|reviews.default.svc.cluster.local 75 and outbound|9080|v2|reviews.default.svc.cluster.local 25. Start Envoy again with split.yaml and --concurrency 1, send curl localhost:10083/ exactly 40 times, count from the response bodies which subset each went to, and write three lines to /root/ist2-name/04-split.txt: v1=, v2= and total=.
Add alt_stat_name: outbound_9080_v2_reviews to the outbound|9080|v2|reviews.default.svc.cluster.local cluster in /root/ist2-name/split.yaml (leave everything else as it is), start it again and send 20 or more requests. Copy two lines as they are from /stats on the admin port and save them to /root/ist2-name/05-stats.txt — the upstream_rq_200 line of the v1 cluster, and the upstream_rq_200 line of the v2 cluster (which now comes out under the changed name).
Copy /root/ist2-name/split.yaml to /root/ist2-name/missing.yaml and change two things — add validate_clusters: false to route_config, and add before the existing routes a route that sends the path prefix /v3 to outbound|9080|v3|reviews.default.svc.cluster.local (you do not create this cluster). Also make /root/ist2-name/missing-strict.yaml, a copy of missing.yaml with only the validate_clusters: false line removed. After you start with missing.yaml, send curl localhost:10083/v3 once and write three lines to /root/ist2-name/06-missing.txt — status= (the HTTP code), no_cluster= (the value of the statistic http.outbound_0.0.0.0_9080.no_cluster) and strict_rc= (the exit code of envoy --mode validate on missing-strict.yaml).
From localhost:9983/clusters of the running Envoy, count how many endpoints each of the three clusters of reviews has and write three lines to /root/ist2-name/07-endpoints.txt in the form <클러스터 이름> <개수> (the placeholders are the cluster name and the count).
In /root/ist2-name/08-report.md, write four lines, default_cluster=, inbound_cluster=, v2_stat_name= and missing_subset_code= (respectively the name of the outgoing cluster with no subset, the name of the incoming cluster, the statistics name of v2 that you changed in step 5, and the HTTP code you received in step 6), and below them write explanations starting with - in at least four lines.

Notes

This Pod has neither a real istiod nor a real sidecar. So you cannot see the actual generated output with istioctl proxy-config; instead you know the translation rules and build the equivalent Envoy configuration by hand to confirm the behavior. The same rules show up as they are in the proxy-config output of a production cluster.
From step 4, where you count the distribution, start Envoy with --concurrency 1. With several workers, the weight experiment wobbles more.
The lines of /stats and /clusters contain |. In grep, escape it with \| or use grep -F.
When you start Envoy, detach it completely from the shell with setsid --fork nohup envoy -c <파일> --log-level warn > <로그> 2>&1 </dev/null (the placeholders are the file and the log). Before you start it again, clean up with pkill -x envoy (pkill -f 'envoy -c' also kills the shell itself that contains that string).
A server for imitating an upstream is already in the image: python3 /opt/lab/envoy/upstream.py <포트> ok|fail|slow (the placeholder is the port). The response body is <모드>:<포트> <경로> (the placeholders are the mode, the port and the path).
After you edit the configuration, first filter it with envoy --mode validate -c <파일> before you start it. The cluster name contains |, so in YAML you must always wrap it in quotes.

Write a DestinationRule with two subsets

Write a DestinationRule to /root/ist2-name/dr.yaml — name reviews, namespace default, host reviews.default.svc.cluster.local, subsets v1 (label version: v1) and v2 (label version: v2). Put the output and exit code of istioctl validate -f /root/ist2-name/dr.yaml in /root/ist2-name/01-validate.txt (the last line is rc=0).

A subset is a named selector meaning "among the Pods of the same service, the ones carrying this label". By itself it does not change traffic, and it is used only when a VirtualService points to it with subset: v1. host can be a short name too, but Istio in the end resolves it to an FQDN — if you write the FQDN from the start here, deriving the names in the next step is easy. istioctl validate checks the schema from the file alone, without a cluster.

Derive four cluster names by rule

The service reviews uses port 9080. From /root/ist2-name/dr.yaml, write the four cluster names the sidecars will have to /root/ist2-name/02-names.txt, one per line — three used when other Pods go out to reviews (one with no subset and one for each subset), and one used when the reviews Pod itself passes incoming requests to the app.

The name has four slots, 방향|포트|서브셋|호스트 (direction, port, subset, host). The direction is outbound or inbound, the port is the service port, and if there is no subset, that slot is left empty and becomes ||. The incoming side sends to its own Pod's app, so it needs neither a subset nor a host, and both of the last two slots are empty. The host slot is the host of the DestinationRule written as an FQDN. If you pull .spec.subsets[].name with yq and print in a loop, there are no typos.

Set up Envoy with exactly those names and send to v1

Write an Envoy configuration to /root/ist2-name/name.yaml — admin port 9983, listener 127.0.0.1:10083 (HTTP), route_config name 9080, sending every path to outbound|9080|v1|reviews.default.svc.cluster.local. There are three outgoing clusters — outbound|9080||reviews.default.svc.cluster.local (two endpoints, 127.0.0.1:8103 and 127.0.0.1:8104), outbound|9080|v1|reviews.default.svc.cluster.local (8103 only) and outbound|9080|v2|reviews.default.svc.cluster.local (8104 only). Start two upstreams on 8103 and 8104 as ok and start Envoy, then write the result of curl localhost:10083/ to /root/ist2-name/03-v1.txt as two lines, status= and body=.

For each service, istiod always creates a "cluster with no subset", and creates one more cluster for each subset of the DestinationRule. The endpoints of a subset cluster are picked, from all the endpoints of the service, only those whose labels match — here we imitate that with ports (8103 = a Pod with version v1, 8104 = a Pod with version v2). The cluster name contains |, so wrap it in quotes, and the route_config name also looks like a number but is a string, so attach quotes.

Count the subset weights over forty requests

Copy /root/ist2-name/name.yaml to /root/ist2-name/split.yaml and change the route to a weighted route — outbound|9080|v1|reviews.default.svc.cluster.local 75 and outbound|9080|v2|reviews.default.svc.cluster.local 25. Start Envoy again with split.yaml and --concurrency 1, send curl localhost:10083/ exactly 40 times, count from the response bodies which subset each went to, and write three lines to /root/ist2-name/04-split.txt: v1=, v2= and total=.

The route[].weight of a VirtualService becomes Envoy's weighted_clusters. A weight is a die rolled for every request, so out of forty it does not come out exactly 30 to 10 — it is normal for the numbers to be a little off. --concurrency 1 ties the workers down to one and keeps the experiment steady. Tell the subsets apart by the port in the body (whether it is ok:8103 or ok:8104).

The statistics name is the cluster name — change it with alt_stat_name

Add alt_stat_name: outbound_9080_v2_reviews to the outbound|9080|v2|reviews.default.svc.cluster.local cluster in /root/ist2-name/split.yaml (leave everything else as it is), start it again and send 20 or more requests. Copy two lines as they are from /stats on the admin port and save them to /root/ist2-name/05-stats.txt — the upstream_rq_200 line of the v1 cluster, and the upstream_rq_200 line of the v2 cluster (which now comes out under the changed name).

Envoy accumulates cluster statistics as cluster.<클러스터 이름>.<지표> (the placeholders are the cluster name and the metric). Istio cluster names contain |, which is troublesome to handle when carrying them over to Prometheus. So if you give a pattern to outboundClusterStatName in the mesh configuration, istiod attaches an alt_stat_name to each cluster and changes only the statistics name — routing is still done with the original name. If you filter like /stats?filter=upstream_rq_200, the two lines are easy to find.

Point at a subset that does not exist and you get a 503

Copy /root/ist2-name/split.yaml to /root/ist2-name/missing.yaml and change two things — add validate_clusters: false to route_config, and add before the existing routes a route that sends the path prefix /v3 to outbound|9080|v3|reviews.default.svc.cluster.local (you do not create this cluster). Also make /root/ist2-name/missing-strict.yaml, a copy of missing.yaml with only the validate_clusters: false line removed. After you start with missing.yaml, send curl localhost:10083/v3 once and write three lines to /root/ist2-name/06-missing.txt — status= (the HTTP code), no_cluster= (the value of the statistic http.outbound_0.0.0.0_9080.no_cluster) and strict_rc= (the exit code of envoy --mode validate on missing-strict.yaml).

A VirtualService pointing to a subset that is not in the DestinationRule is a very common mistake in Istio. istiod sends the route down with RDS, and a dynamically received route is accepted even if the cluster it points to does not exist (the default of validate_clusters is false when dynamic). Then a 503 comes only when the request arrives, and the response flag NC (no cluster) is printed in the access log. A static configuration has the default true, so it rejects the same configuration outright — this step puts the two shapes side by side.

A subset is a subset of the same service's endpoints

From localhost:9983/clusters of the running Envoy, count how many endpoints each of the three clusters of reviews has and write three lines to /root/ist2-name/07-endpoints.txt in the form <클러스터 이름> <개수> (the placeholders are the cluster name and the count).

/clusters prints several lines of 이름::주소::지표::값 (the placeholders are the name, the address, the metric and the value) for each endpoint. Pick lines by a metric that appears only once per endpoint (for example cx_active) and count by cluster name. The key point is that the cluster with no subset holds the endpoints of both subsets — a subset is not a new service but the same endpoint list divided by labels. That is why a Pod without the label is in no subset and goes only to the cluster with no subset.

Summarize it as how to read the names

In /root/ist2-name/08-report.md, write four lines, default_cluster=, inbound_cluster=, v2_stat_name= and missing_subset_code= (respectively the name of the outgoing cluster with no subset, the name of the incoming cluster, the statistics name of v2 that you changed in step 5, and the HTTP code you received in step 6), and below them write explanations starting with - in at least four lines.

Copy from the files of the earlier steps. In the explanation lines, it is good to write as if you were teaching a colleague who is seeing proxy-config clusters output for the first time how to read the names.