Writing Routing Rules and Resilience Settings

Goal

You get a feel for splitting requests by condition with a VirtualService, and for deciding the nature and resilience of the destination with a DestinationRule.

Why it matters

The separation of roles between the two resources is not a matter of taste but an operating structure. What the deployment pipeline touches every time is routing (the VirtualService), and what the service owner decides once and leaves for a long time is the nature of the destination (the DestinationRule). If you blur this boundary, the connection pool settings get shaken along with every deployment. And the matching order and the label match of subsets are half of mesh troubleshooting. If you put the default path with no conditions on top, the rules below it die, and if a subset's label does not match the Pod labels, the endpoints are empty and you get a 503 (UH). Resilience settings must not be decided by gut feeling either — if the overall timeout is shorter than the sum of the attempt times, retries end without a single one happening, and without an ejection limit the circuit breaker creates a total outage.

In this environment packets do not flow, so grading looks at the structure of the manifests and the results of istioctl analyze.

Steps

Preparation before starting. Lab Pods come up fresh for each lab, so the mesh configuration you installed in the earlier lab does not remain. If kubectl get crd virtualservices.networking.istio.io is empty, first create a manifest with istioctl manifest generate --set profile=minimal > /root/istio/manifest.yaml and run kubectl apply -f /root/istio/manifest.yaml twice (the rest attaches only after the CRDs are registered first). Then prepare the namespace with kubectl create ns mesh-lab && kubectl label ns mesh-lab istio-injection=enabled.

Apply /opt/lab/fixtures/istio/workload-v1v2.yaml to the mesh-lab namespace. The first port name of the Service reviews must be http, and the Pod labels of the Deployments reviews-v1 and reviews-v2 must be app: reviews with version: v1/version: v2 respectively.
Create a DestinationRule reviews in mesh-lab. spec.host is reviews, and in spec.subsets put two subsets named v1 (labels: {version: v1}) and v2 (labels: {version: v2}).
Create a VirtualService reviews in mesh-lab. Put reviews in spec.hosts, and do not put a match in the last item of spec.http; set the destination to host: reviews, subset: v1.
Add a conditional route at the front of the same VirtualService. The condition is the case where the header end-user is exactly labhub-tester, and the destination is subset: v2. This route must be above the last default path.
Create the ratings workload in mesh-lab — a Service ratings (port name http, port 9080) and a Deployment ratings-v1 (Pod labels app: ratings, version: v1). Then create a VirtualService ratings with uri.prefix of match as /api/ratings, rewrite.uri as /ratings, and the destination as host: ratings.
Attach timeout: 3s and retries to the route of the same VirtualService ratings. Include attempts: 3, perTryTimeout: 1s, and 5xx and connect-failure in retryOn. And in /root/istio/traffic/out/retry-note.txt, write why retrying non-idempotent requests (such as POST) is dangerous.
Create a DestinationRule ratings in mesh-lab and put consecutive5xxErrors: 5, interval: 10s, baseEjectionTime: 30s, and maxEjectionPercent: 50 in trafficPolicy.outlierDetection. Also add fault.delay to the VirtualService ratings with percentage.value: 10 and fixedDelay: 2s.
Save the result of istioctl analyze -n mesh-lab to /root/istio/traffic/out/analyze.txt (Error [ must not remain). And create /root/istio/traffic/out/routes.json. The length of the routes array must equal the actual number of spec.http items in the VirtualService reviews, and do not put a match key in the last element of the array. The value of the default_subset key is v1. mesh-lab must have at least 2 VirtualServices.

Notes

Lab Pods come up fresh for each lab, so the cluster state from the earlier lab does not remain. That is why keeping mesh configuration as manifests is itself reproducibility. It is why the value of declarative configuration is felt directly in this lab and is not an abstract principle.
You can apply directly from standard input with kubectl apply -f - -n mesh-lab. After applying, check what was actually stored with kubectl get virtualservice reviews -n mesh-lab -o yaml.
For the route count in step 8, extracting it with kubectl get virtualservice reviews -n mesh-lab -o json | jq '.spec.http | length' avoids mistakes.
In step 5, if you create only the VirtualService without creating the target workload, static analysis gives a "referenced host not found" error. Step 8 catches that error.
Common mistake 1: putting the conditional route at the end of the array. The default path grabs it first and the condition becomes meaningless.
Common mistake 2: leaving out labels in a subset and writing only the name. The name is used only in the Envoy cluster name, and what selects Pods is the label.

Put up the target workloads and version labels

Apply /opt/lab/fixtures/istio/workload-v1v2.yaml to the mesh-lab namespace. The first port name of the Service reviews must be http, and the Pod labels of the Deployments reviews-v1 and reviews-v2 must be app: reviews with version: v1/version: v2 respectively.

Just apply the fixture as it is. But check that the Service port has a name and that the Pod template has both the app and version labels. A mesh decides the protocol by the port name.

Define subsets by version

Create a DestinationRule reviews in mesh-lab. spec.host is reviews, and in spec.subsets put two subsets named v1 (labels: {version: v1}) and v2 (labels: {version: v2}).

A subset does not select Pods by name alone. For each subset you must write by which label to select Pods, and that label must be the same as the real Pod label.

Send the default path to v1

Create a VirtualService reviews in mesh-lab. Put reviews in spec.hosts, and do not put a match in the last item of spec.http; set the destination to host: reviews, subset: v1.

There must be one route with no conditions so that requests that fail to match have somewhere to go. Put that route at the very end of the array.

Send to v2 by a header condition

Add a conditional route at the front of the same VirtualService. The condition is the case where the header end-user is exactly labhub-tester, and the destination is subset: v2. This route must be above the last default path.

Matching goes from top to bottom and uses the first one that matches. If a conditional route is below the default path, it never runs.

Path prefix matching and rewriting

Create the ratings workload in mesh-lab — a Service ratings (port name http, port 9080) and a Deployment ratings-v1 (Pod labels app: ratings, version: v1). Then create a VirtualService ratings with uri.prefix of match as /api/ratings, rewrite.uri as /ratings, and the destination as host: ratings.

You use it when the path exposed to the outside differs from the path the back end knows. If the target service does not exist in the cluster, static analysis gives an error, so create the workload too.

Attach a timeout and retries

Attach timeout: 3s and retries to the route of the same VirtualService ratings. Include attempts: 3, perTryTimeout: 1s, and 5xx and connect-failure in retryOn. And in /root/istio/traffic/out/retry-note.txt, write why retrying non-idempotent requests (such as POST) is dangerous.

For retries to have meaning, the attempts must fit inside the overall timeout. And first weigh whether the request is safe to retry.

Outlier detection and fault injection

Create a DestinationRule ratings in mesh-lab and put consecutive5xxErrors: 5, interval: 10s, baseEjectionTime: 30s, and maxEjectionPercent: 50 in trafficPolicy.outlierDetection. Also add fault.delay to the VirtualService ratings with percentage.value: 10 and fixedDelay: 2s.

Always put a limit on ejection. If you pull everything out, the service dies entirely. Fault injection too becomes an experiment only when you specify a percentage.

Verify the configuration and summarize the route order

Save the result of istioctl analyze -n mesh-lab to /root/istio/traffic/out/analyze.txt (Error [ must not remain). And create /root/istio/traffic/out/routes.json. The length of the routes array must equal the actual number of spec.http items in the VirtualService reviews, and do not put a match key in the last element of the array. The value of the default_subset key is v1. mesh-lab must have at least 2 VirtualServices.

Do not count the number of routes by hand; extract it from the real resource. The fact that the default path has no conditions must also show in the summary file.