A Pod That Passes Readiness Yet Returns 500

Goal

In a service mixed with a Pod that always returns 500, confirm from response counts and Envoy's endpoint state that outlier detection actually ejects that Pod, and then add mirroring and fault injection to see what those settings become in Envoy.

Why it matters

Kubernetes' readiness check looks at the state a Pod reports about itself. A Pod that returns a 500 on every request but passes the readiness check stays in the endpoints. Outlier detection judges by the responses it actually received, so it can pull that Pod out, but to the same degree it also hides the failure. Only when you know how to confirm an ejection by eye and count it with numbers can you use this feature with confidence.

Steps

Put up the materials with kubectl apply -f /opt/fixtures/istlab/resilience-app.yaml and wait until all the Pods are ready. Then call http://api/ 30 times from the client and write ok= (the number of 200s) and err= (the number of 500s) into /root/istlab-resilience/01-baseline.txt.
Write the client Pod again into /root/istlab-resilience/client.yaml with proxyStatsMatcher.inclusionRegexps: [".*outlier_detection.*"] in the annotation proxy.istio.io/config, delete the existing client, and recreate it from this file.
Write the DestinationRule api (host api.pay.svc.cluster.local) into /root/istlab-resilience/dr.yaml and apply it — with outlierDetection, 2 consecutive 5xx (consecutive5xxErrors: 2), a check interval of 2 seconds, a base ejection time of 60 seconds, and a maximum ejection percent of 50%. Then call 30 times, twice, and write err_first= (the number of 500s in the first 30) and err_second= (the number of 500s in the second 30) into /root/istlab-resilience/03-outlier.txt.
While the ejection is in effect (within 60 seconds after step 3), save the output of istioctl proxy-config endpoints client.pay --cluster 'outbound|80||api.pay.svc.cluster.local' -o json into /root/istlab-resilience/04-endpoints.json. If 60 seconds have passed, call 30 times as in step 3 to eject it again and then save.
Write the VirtualService api (host api.pay.svc.cluster.local) into /root/istlab-resilience/vs.yaml and apply it — the destination stays api.pay.svc.cluster.local, mirror is api-v2.pay.svc.cluster.local, and mirrorPercentage is 100. After applying, call three times with x-request-id set to mirror-1, mirror-2, and mirror-3, and write two lines into /root/istlab-resilience/05-mirror.txt: mirrored= (the number of those three ids left in the api-v2 sidecar log) and client_saw_v2= (yes if v2 was in the body the client received).
Add two rules to the front of the VirtualService in /root/istlab-resilience/vs.yaml — for header x-chaos: abort, put a 503 at 100% with fault.abort, and for x-chaos: delay, put 2 seconds at 100% with fault.delay, with the destination unchanged (put the default rule with the mirror at the very end). After applying, write three lines into /root/istlab-resilience/06-fault.txt: abort_code= (the status code of the abort request), abort_flag= (the response flag in the client sidecar access log for that request), and delay_seconds= (the time the delay request took, in seconds with the decimals dropped).
Pull two things out of the configuration the client sidecar received and save them as a JSON object in /root/istlab-resilience/07-envoy.json — outlier: the outlierDetection object of the first cluster of istioctl proxy-config cluster client.pay --fqdn api.pay.svc.cluster.local -o json as it is, and mirror_cluster: the requestMirrorPolicies[0].cluster value of the last route of the api.pay.svc.cluster.local virtual host in istioctl proxy-config routes client.pay --name 80 -o json.
Write five lines into /root/istlab-resilience/08-report.md — errors_before= (the step 1 err), errors_after_ejection= (the step 3 err_second), ejected_ip= (the address with failedOutlierCheck true in the step 4 file), mirror_reached_client= (the step 5 client_saw_v2), and abort_flag= (step 6) — and write what you learned below that in at least four lines.

Notes

This VM takes 2–4 minutes to prepare. On the client you type commands with kubectl -n pay exec client -c curl -- ….
You look at the sidecar statistics with kubectl -n pay exec client -c istio-proxy -- pilot-agent request GET stats.
The ejection is released after 60 seconds. If you do not see an ejection at step 4, call 30 times again to eject and then save.
If you recreate the client in step 2, that Pod's sidecar log starts fresh.
Common mistake — putting the fault injection rules after the default rule with the mirror. The first matching rule from the top is used, so the header-conditioned rules are never applied.

A service mixed with one broken Pod

Put up the materials with kubectl apply -f /opt/fixtures/istlab/resilience-app.yaml and wait until all the Pods are ready. Then call http://api/ 30 times from the client and write ok= (the number of 200s) and err= (the number of 500s) into /root/istlab-resilience/01-baseline.txt.

Behind the api service there are three Pods, and api-bad among them always returns 500. There is no readiness check, so Kubernetes sees all three as Ready and puts them in the endpoints — to Kubernetes's eye it is a healthy Pod. So about a third of the requests fail. To repeat in a shell inside the Pod, use sh -c 'for i in $(seq 1 30); do …; done'.

Turn on the statistics that were not visible

Write the client Pod again into /root/istlab-resilience/client.yaml with proxyStatsMatcher.inclusionRegexps: [".*outlier_detection.*"] in the annotation proxy.istio.io/config, delete the existing client, and recreate it from this file.

Istio filters out most of the Envoy statistics the sidecar produces by default — because Prometheus cannot cope if thousands of Pods export them all. So even if you turn on outlier detection, numbers such as the ejection count are not visible. proxyStatsMatcher is a device that revives only the statistics you need through a Pod annotation, and since the proxy reads it when it starts, you have to recreate the Pod. You can check that it went in properly in the stats_config of the proxy bootstrap.

Pull out the broken Pod with outlier detection

Write the DestinationRule api (host api.pay.svc.cluster.local) into /root/istlab-resilience/dr.yaml and apply it — with outlierDetection, 2 consecutive 5xx (consecutive5xxErrors: 2), a check interval of 2 seconds, a base ejection time of 60 seconds, and a maximum ejection percent of 50%. Then call 30 times, twice, and write err_first= (the number of 500s in the first 30) and err_second= (the number of 500s in the second 30) into /root/istlab-resilience/03-outlier.txt.

Outlier detection judges each upstream by the responses the sidecar actually received. If the same Pod returns 5xx repeatedly, it pulls that Pod out of the load-balancing targets for a while (ejection). In the first round you see a few 500s before the ejection, and in the second round it should be 0. You count the number of ejections with the statistic you revived in step 2, …outlier_detection.ejections_enforced_total — kubectl -n pay exec client -c istio-proxy -- pilot-agent request GET stats | grep outlier.

How did Envoy write down that Pod

While the ejection is in effect (within 60 seconds after step 3), save the output of istioctl proxy-config endpoints client.pay --cluster 'outbound|80||api.pay.svc.cluster.local' -o json into /root/istlab-resilience/04-endpoints.json. If 60 seconds have passed, call 30 times as in step 3 to eject it again and then save.

An ejected Pod is not removed from the Kubernetes endpoints. It is removed only from this sidecar's load-balancing table. So kubectl get endpoints still shows all three, and in Envoy's endpoint list, failedOutlierCheck: true is attached to that Pod's healthStatus. Seen in table form, the OUTLIER CHECK column is FAILED. An ejection is not permanent; it comes back in after the base ejection time, and if it fails again it is out for longer.

Reflect production traffic on the new version — mirroring

Write the VirtualService api (host api.pay.svc.cluster.local) into /root/istlab-resilience/vs.yaml and apply it — the destination stays api.pay.svc.cluster.local, mirror is api-v2.pay.svc.cluster.local, and mirrorPercentage is 100. After applying, call three times with x-request-id set to mirror-1, mirror-2, and mirror-3, and write two lines into /root/istlab-resilience/05-mirror.txt: mirrored= (the number of those three ids left in the api-v2 sidecar log) and client_saw_v2= (yes if v2 was in the body the client received).

Mirroring copies the request, sends it to the other side, and discards that response. So you test the new version with production traffic without affecting the user. The copy has the same x-request-id as the original request, so you can find it by id in the mirror target's sidecar log — kubectl -n pay logs deploy/api-v2 -c istio-proxy. The copy goes asynchronously, so it may be printed in the log a little late.

Inject a failure on purpose — only when there is a header

Add two rules to the front of the VirtualService in /root/istlab-resilience/vs.yaml — for header x-chaos: abort, put a 503 at 100% with fault.abort, and for x-chaos: delay, put 2 seconds at 100% with fault.delay, with the destination unchanged (put the default rule with the mirror at the very end). After applying, write three lines into /root/istlab-resilience/06-fault.txt: abort_code= (the status code of the abort request), abort_flag= (the response flag in the client sidecar access log for that request), and delay_seconds= (the time the delay request took, in seconds with the decimals dropped).

Fault injection is done by the client-side sidecar before it sends the request to the upstream. So an abort does not reach the upstream and the response flag is printed as FI (fault injected). If you put a header condition, you can leave normal traffic as it is and break only the request you are testing. The http rules of a VirtualService use the first match from the top, so put the catch-all rule at the very end. You measure the time taken with curl -w '%{time_total}'.

What did the DestinationRule and VirtualService become in Envoy

Pull two things out of the configuration the client sidecar received and save them as a JSON object in /root/istlab-resilience/07-envoy.json — outlier: the outlierDetection object of the first cluster of istioctl proxy-config cluster client.pay --fqdn api.pay.svc.cluster.local -o json as it is, and mirror_cluster: the requestMirrorPolicies[0].cluster value of the last route of the api.pay.svc.cluster.local virtual host in istioctl proxy-config routes client.pay --name 80 -o json.

istiod moves the DestinationRule's consecutive5xxErrors, interval, baseEjectionTime, and maxEjectionPercent into the Envoy cluster's outlierDetection, and the VirtualService's mirror into the route's requestMirrorPolicies. The names differ slightly (consecutive5xx), and the times turn into strings such as "2s". When you changed the configuration and the behavior is odd, you end up looking at this translation result.

Write down what the mesh did on your behalf

Write five lines into /root/istlab-resilience/08-report.md — errors_before= (the step 1 err), errors_after_ejection= (the step 3 err_second), ejected_ip= (the address with failedOutlierCheck true in the step 4 file), mirror_reached_client= (the step 5 client_saw_v2), and abort_flag= (step 6) — and write what you learned below that in at least four lines.

Copy the values from the earlier step files. In the explanation lines, write 'what each of Kubernetes' readiness check and the mesh's outlier detection looks at to judge' — one looks at the state the Pod reports, and the other looks at the responses actually received.