Translate a VirtualService into a Route Table and Check Order and Retries

Goal

Following the rules by which a VirtualService is translated into an Envoy route table, set up the table by hand and confirm with requests the Host header, first-match priority, weights and retries.

Why it matters

A VirtualService mistake usually comes not from a single rule but from the order and overlap between rules. Static analysis cannot catch that, so you have to know the translation rules and be able to draw how it will behave in Envoy in order to filter it out in change review.

Steps

Write /root/ist2-route/dr.yaml (DestinationRule reviews, host reviews.default.svc.cluster.local, subsets v1, v2 and broken — the label version has the same name as each) and /root/ist2-route/vs.yaml (VirtualService reviews, hosts [reviews]). The http of vs.yaml has four entries in this order — jason (if the header end-user is exactly jason, v2), api (if the path prefix is /api, v1), flaky (if the path prefix is /flaky, broken, retries: {attempts: 3, retryOn: 5xx}, timeout: 2s) and default (v1 with weight 75 and v2 with 25). Put the output and exit code of istioctl analyze --use-kube=false vs.yaml dr.yaml in /root/ist2-route/01-analyze.txt (the last line is rc=0).
The service reviews is in the namespace default on port 9080. Write the names in the route table this VirtualService makes at the sidecar to /root/ist2-route/02-names.txt as three lines — route_config= (the route table name), virtual_host= (the virtual host name) and domains= (the four short names among that virtual host's domains, separated by commas, in any order).
Write an Envoy configuration to /root/ist2-route/route.yaml — admin port 9984, listener 127.0.0.1:10084, and the route table name, virtual host name and four domains from step 2. There are three routes, named, in this order — jason (path / + header end-user exactly jason → outbound|9080|v2|reviews.default.svc.cluster.local), api (prefix /api → outbound|9080|v1|reviews.default.svc.cluster.local) and default (prefix / → outbound|9080|v1|reviews.default.svc.cluster.local). There are two clusters, v1 (127.0.0.1:8105) and v2 (127.0.0.1:8106). Start the two upstreams as ok and start Envoy, then call / with three Host headers and write the HTTP codes to /root/ist2-route/03-hosts.txt as three lines, reviews=, reviews.default.svc.cluster.local= and ratings=.
Send three requests with Host reviews to the Envoy running with route.yaml, tell the subset by the port in the response body, and write it to /root/ist2-route/04-match.txt — anonymous= (/ with no header), jason= (/ with end-user: jason) and api_as_jason= (/api/list with end-user: jason). The value is v1 or v2.
Make /root/ist2-route/vs-bad.yaml, with the default rule moved to the very front of vs.yaml, and check the exit code of istioctl analyze --use-kube=false vs-bad.yaml dr.yaml. The same mistake in Envoy: start Envoy again with /root/ist2-route/route-bad.yaml, in which the default route is moved to the very front of route.yaml, and call / with end-user: jason. In /root/ist2-route/05-order.txt, write two lines: analyze_rc= (the exit code of the vs-bad analysis) and jason_after= (the subset jason went to in route-bad). After you check, go back and start it with route.yaml.
Copy route.yaml to /root/ist2-route/route-split.yaml and change only the default route to weighted_clusters (v1 75, v2 25 — the default rule of the VirtualService as it is). After you start it again with --concurrency 1, send / exactly 40 times with Host reviews and write three lines to /root/ist2-route/06-split.txt: v1=, v2= and total=.
Copy route-split.yaml to /root/ist2-route/route-retry.yaml and add two things — the cluster outbound|9080|broken|reviews.default.svc.cluster.local (127.0.0.1:8114, an upstream that always returns 503), and a flaky route before default (prefix /flaky → that cluster, timeout: 2s, retry_policy: {retry_on: 5xx, num_retries: 3}). Start an upstream on 8114 as fail, start Envoy again and call /flaky just once. In /root/ist2-route/07-retry.txt, write status= (the HTTP code), upstream_rq_retry= and upstream_rq_total= (the two statistic values of the broken cluster).
In /root/ist2-route/08-report.md, write four lines — route_config=, virtual_host=, first_match_wins= (yes if only the first matching route is used, as you saw in step 5) and retry_attempts= (the attempts value of the VirtualService) — and below them write explanations starting with - in at least four lines.

Notes

This Pod has neither a real istiod nor a real sidecar. So you cannot see the actual generated output with istioctl proxy-config; instead you know the translation rules and build the equivalent Envoy configuration by hand to confirm the behavior. The same rules show up as they are in the proxy-config output of a production cluster.
Attach -H 'Host: reviews' to every request. The virtual host is chosen by the Host header.
In steps 6 and 7, which count distribution and retries, start Envoy with --concurrency 1. Statistics are cumulative, so count right after you start it again.
When you start Envoy, detach it completely from the shell with setsid --fork nohup envoy -c <파일> --log-level warn > <로그> 2>&1 </dev/null (the placeholders are the file and the log). Before you start it again, clean up with pkill -x envoy (pkill -f 'envoy -c' also kills the shell itself that contains that string).
A server for imitating an upstream is already in the image: python3 /opt/lab/envoy/upstream.py <포트> ok|fail|slow (the placeholder is the port). The response body is <모드>:<포트> <경로> (the placeholders are the mode, the port and the path).
After you edit the configuration, first filter it with envoy --mode validate -c <파일> before you start it. The cluster name contains |, so in YAML you must always wrap it in quotes.

Write a VirtualService with four routes and pass it through analysis

Write /root/ist2-route/dr.yaml (DestinationRule reviews, host reviews.default.svc.cluster.local, subsets v1, v2 and broken — the label version has the same name as each) and /root/ist2-route/vs.yaml (VirtualService reviews, hosts [reviews]). The http of vs.yaml has four entries in this order — jason (if the header end-user is exactly jason, v2), api (if the path prefix is /api, v1), flaky (if the path prefix is /flaky, broken, retries: {attempts: 3, retryOn: 5xx}, timeout: 2s) and default (v1 with weight 75 and v2 with 25). Put the output and exit code of istioctl analyze --use-kube=false vs.yaml dr.yaml in /root/ist2-route/01-analyze.txt (the last line is rc=0).

A VirtualService is "where to send", and a DestinationRule is "how to treat it after sending and the definition of subsets". analyze reads the two files together and looks at reference relationships, such as whether the subset the VS points to is in the DR. If you attach a name to each http entry, the same name is attached to the route on the Envoy side, which makes it easier to find later.

Derive the names of the route table, virtual host and domains

The service reviews is in the namespace default on port 9080. Write the names in the route table this VirtualService makes at the sidecar to /root/ist2-route/02-names.txt as three lines — route_config= (the route table name), virtual_host= (the virtual host name) and domains= (the four short names among that virtual host's domains, separated by commas, in any order).

On the outgoing side, the sidecar keeps one route table per port. If several services use the same port 9080, one virtual host per service goes into one table, and it is chosen by the request's Host header. So the table name is the port and the virtual host name is FQDN:포트 (the placeholder is the port). An app in the same namespace also calls it by a short name like reviews or reviews.default, so domains holds together the names made by cutting the FQDN from the back (see the proxy-config example in the official documentation).

Set up the route table with those names and see it chosen by the Host header

Write an Envoy configuration to /root/ist2-route/route.yaml — admin port 9984, listener 127.0.0.1:10084, and the route table name, virtual host name and four domains from step 2. There are three routes, named, in this order — jason (path / + header end-user exactly jason → outbound|9080|v2|reviews.default.svc.cluster.local), api (prefix /api → outbound|9080|v1|reviews.default.svc.cluster.local) and default (prefix / → outbound|9080|v1|reviews.default.svc.cluster.local). There are two clusters, v1 (127.0.0.1:8105) and v2 (127.0.0.1:8106). Start the two upstreams as ok and start Envoy, then call / with three Host headers and write the HTTP codes to /root/ist2-route/03-hosts.txt as three lines, reviews=, reviews.default.svc.cluster.local= and ratings=.

What chooses the route table is the listener (the port), and what chooses the virtual host within the table is the Host header. If a request comes with a Host that is not in the domains of any virtual host, no route is found and you get a 404 — that is what it looks like when a sidecar is asked for a service name it does not know. Call with changing headers, as in curl -H 'Host: reviews' localhost:10084/. Write a header match in match.headers as string_match: {exact: …}.

When a header match and a path match overlap, who wins

Send three requests with Host reviews to the Envoy running with route.yaml, tell the subset by the port in the response body, and write it to /root/ist2-route/04-match.txt — anonymous= (/ with no header), jason= (/ with end-user: jason) and api_as_jason= (/api/list with end-user: jason). The value is v1 or v2.

Envoy looks at routes from the top and uses only the first one that matches. api_as_jason matches both the jason rule and the api rule, and which one is written first decides the answer. If the body is ok:8105, it is v1, and if ok:8106, it is v2.

Put the catch-all in front — analysis is silent and traffic is wrong

Make /root/ist2-route/vs-bad.yaml, with the default rule moved to the very front of vs.yaml, and check the exit code of istioctl analyze --use-kube=false vs-bad.yaml dr.yaml. The same mistake in Envoy: start Envoy again with /root/ist2-route/route-bad.yaml, in which the default route is moved to the very front of route.yaml, and call / with end-user: jason. In /root/ist2-route/05-order.txt, write two lines: analyze_rc= (the exit code of the vs-bad analysis) and jason_after= (the subset jason went to in route-bad). After you check, go back and start it with route.yaml.

default has no conditions, so it matches every request. If it is at the very front, the rules behind it are never used. But analyze only looks at whether what each rule references exists and does not look at rules shadowing one another, so it says it is clean. That is why, when you edit a VirtualService, a person has to keep "broad ones to the back". You can change the list order like yq '.spec.http |= [.[3], .[0], .[1], .[2]]'.

weight becomes weighted_clusters — count over forty requests

Copy route.yaml to /root/ist2-route/route-split.yaml and change only the default route to weighted_clusters (v1 75, v2 25 — the default rule of the VirtualService as it is). After you start it again with --concurrency 1, send / exactly 40 times with Host reviews and write three lines to /root/ist2-route/06-split.txt: v1=, v2= and total=.

A weight rolls a die for every request, so out of forty it does not come out exactly 30 to 10. If the sum is 40 and v1 has more, it is working properly. The same reason explains why, if you set a 1% canary ratio in Istio and check with a few dozen requests, v2 may never appear.

retries and timeout become retry_policy — count how many times it is hit

Copy route-split.yaml to /root/ist2-route/route-retry.yaml and add two things — the cluster outbound|9080|broken|reviews.default.svc.cluster.local (127.0.0.1:8114, an upstream that always returns 503), and a flaky route before default (prefix /flaky → that cluster, timeout: 2s, retry_policy: {retry_on: 5xx, num_retries: 3}). Start an upstream on 8114 as fail, start Envoy again and call /flaky just once. In /root/ist2-route/07-retry.txt, write status= (the HTTP code), upstream_rq_retry= and upstream_rq_total= (the two statistic values of the broken cluster).

The retries.attempts of a VirtualService becomes Envoy's num_retries, retryOn becomes retry_on, and timeout becomes the route's timeout. Retries are attached to one original request up to attempts times, so the upstream is hit 1+attempts times — the reason retries put several times the load on a service that is down. Statistics are cumulative, so the numbers are clean only if you call just once right after starting again. Pick the two values of the broken cluster with /stats?filter=.

Summarize it as a VirtualService translation table

In /root/ist2-route/08-report.md, write four lines — route_config=, virtual_host=, first_match_wins= (yes if only the first matching route is used, as you saw in step 5) and retry_attempts= (the attempts value of the VirtualService) — and below them write explanations starting with - in at least four lines.

Copy the values from the files of the earlier steps. If in the explanation lines you write "what I will check when I edit a VirtualService", this table will be useful at the next change review.