VirtualService and DestinationRule — Why the Roles Split
In one line
A VirtualService decides which request goes where, and a DestinationRule decides how to treat it once it arrives there.
Why this was needed
A Kubernetes Service goes only as far as throwing requests at a set of Pods. There is nothing beyond that. Demands such as "only requests with this header go to the new version," "give up on this service if there is no response within 3 seconds," and "take an instance that returns 5xx consecutively out for 30 seconds" all had to be shouldered by the application or a front gateway.
Istio splits these demands into two resources. The reason for the split matters. Routing rules change often and differ by department, but the nature of the destination (which label is which version, how large the connection pool should be) is relatively stable. So the deployment pipeline only has to touch a single VirtualService at each deployment, and a DestinationRule, once the service owner decides it, lasts a long time.
How it works
If you look at where the two resources are translated in Envoy, the division of roles becomes clear.
| Istio resource | Envoy counterpart | What it decides |
|---|---|---|
| VirtualService | RouteConfiguration | Match conditions, weights, rewrites, timeouts, retries, fault injection, mirroring |
| DestinationRule | Cluster | Subsets (targets split by label), load balancing, connection pool, outlier detection, client TLS |
An Envoy cluster name has the form 방향|포트|서브셋|FQDN (direction, port, subset, FQDN), so it looks like outbound|9080|v2|reviews.mesh-lab.svc.cluster.local. Just remembering that the subset name is embedded in that spot as it is makes reading logs much easier.
Matching has an order. spec.http is an array, and Envoy goes down from the top and uses the first one that matches. So if you put a catch-all route with no conditions at the very top, the rules below never run. There is one practical rule — the narrower the conditions, the higher up; the default path with no conditions must always go at the very bottom. And if there is no default path at all, a request that fails to match becomes a 503 with the NR (no route) flag.
A subset selects Pods not by name but by label. If you write name: v2, labels: {version: v2} in a DestinationRule, that subset points only to Pods that carry the label version=v2. If you confuse the name with the label and leave out the label or make a typo, that subset has no endpoints, and traffic becomes a UH (no healthy upstream) 503. This label mismatch is the most common cause of 503 in a mesh.
Protocol detection is also easy to miss. A mesh decides whether to do L7 handling by a Service's port name. The port name must start like http, http-api, grpc, or tcp for HTTP routing to apply. If the name is missing or off, it is treated as TCP and header matching is ignored entirely.
Resilience settings must be set with numerical grounds.
perTryTimeoutis 1.5 to 2 times the destination's normal p99 latency.timeout(overall) is at leastperTryTimeout × (attempts + 1). Otherwise retries cannot even start and run into the overall deadline.- Be generous with
attemptsonly for idempotent requests (GET). A POST retry without an idempotency key is a duplicate-processing accident. - Always set
maxEjectionPercentinoutlierDetection. If you set it to 100% on a service with 3 instances, all of them are ejected and you get a total outage.
What it looks like in the field
First, retry storms. In an A → B → C chain, if each stage has 2 retries, C can receive up to 9 times the requests in the worst case. With four stages it is 27 times. It is piling on a service that is already dying, so put retries in only one place in the chain (preferably the outside) and keep the inner ones at 0 to 1.
Second, nested timeouts. If the outer timeout is shorter than the inner one, the outer cuts off and retries while the inner one is diligently working. The inner service repeats "work that can never complete." Deadlines must get shorter from the outside toward the inside.
Third, for a 503, look at the flag first. UH means no endpoints (check the subset labels), UO means connection pool overflow (the circuit breaker is acting), UF means connection failure (an mTLS mismatch where only one side is STRICT is a regular culprit), NR means no route (check the catch-all), and URX means retries exhausted. Knowing just these five halves your debugging time.
What you will do in the next lab
You put up the fixture's reviews v1/v2 workloads and define subsets, and then create a VirtualService that keeps the default path on v1 and sends only requests with a particular header to v2. For ratings, you apply path prefix matching and rewriting and add a timeout and retries, and then layer on outlier detection and fault injection. Finally you verify the configuration with istioctl analyze and summarize the route order as JSON. Traffic does not actually flow, but the order of the route array and the label match of subsets can be judged exactly by static analysis alone.