Istio Deep Dive — Why It Flows That Way
Why Requests Go Astray When Every Rule Looks Right
In one line
The http[] of a VirtualService becomes routes[] of an Envoy route table — with the order kept as it is. The table name is the service port, the virtual host name is FQDN:포트 (where the placeholder is the port), weight is weighted_clusters, and retries is retry_policy. The translation is this straight, so a VirtualService mistake is reproduced in Envoy as it is.
Why this was needed
After fixing a VirtualService, many people cannot explain "why does this request go to v1?" Looking at the YAML alone, all the rules look right. The problem is usually the relationship between rules. A broad rule in front shadows a narrow rule behind it, or two rules match the same request and the one written first is not the one intended. And istioctl analyze does not catch this kind of mistake. analyze only checks whether the subsets and gateways each rule points to exist; it does not look at rules shadowing one another.
That is why you need to know the translation rules. If you know them, you can draw the Envoy route table in your head as you read a VirtualService, and in production you can line up the istioctl proxy-config routes output and the VirtualService line by line.
How it works
The outgoing sidecar keeps one route table per port. If several services use the same 9080, one virtual host per service goes into one table, and it is chosen by the request's Host header.
| VirtualService | Envoy |
|---|---|
| Service port 9080 | route_config.name: "9080" |
hosts: [reviews] |
The virtual host reviews.default.svc.cluster.local:9080, with reviews, reviews.default, reviews.default.svc and the FQDN in domains |
http[i].match |
routes[i].match (prefix, headers …) |
http[i].route[].destination.subset |
route.cluster: outbound|9080|v1|… |
weight |
route.weighted_clusters |
retries.attempts / retryOn |
retry_policy.num_retries / retry_on |
timeout |
route.timeout |
A request whose Host is not in the domains of any virtual host gets a 404. Within a table, from the top, only the first route that matches is used. If a rule with no conditions (a catch-all) is in front, the rules behind it are never used.
Weights roll a die for every request. It is normal that sending 75 to 25 forty times does not come out exactly 30 to 10. Retries are added to the one original request up to attempts more times, so the upstream is hit 1+attempts times. Even if you do not write retries in a VirtualService, Istio puts in a default retry (2 times, for connection failures, refusals and so on), and also attaches a previous_hosts predicate that avoids the same host.
What it looks like in the field
I appended a new rule to the end of the list and it has no effect. That is because a catch-all was already at the end. In reviews, put "is the new rule in front of the catch-all?" as a one-line checklist item.
During an outage, upstream requests jump fourfold. If several tiers of a call chain each set retries, the load multiplies. Set retries at only one place in the chain, together with a short perTryTimeout.
Someone who has used Envoy directly expects a 15-second timeout. The default timeout of an Envoy route is 15 seconds, but if a VirtualService has no timeout, Istio puts 0s (off) in the route. So a slow call behind a sidecar waits forever. If you need an upper bound on waiting, you have to write it in the VirtualService, and the value you write shows up as it is as the route's timeout.
I cannot see the 1% canary. If you check with a few dozen requests, v2 may never show up. Look at the ratio in statistics, over a sufficient number of requests.
Official documentation: VirtualService · Debugging Envoy and Istiod
What you will do in the next lab
You write a VS and DR and pass them through analysis, derive the route table, virtual host and domain names by rule and set up Envoy. You see the table being chosen by the Host header, and that when a header match and a path match overlap, the one written first wins, and you confirm that the mistake of putting a catch-all in front is not caught by analyze and Envoy follows it as it is. Finally you count weights and retries.