Move Traffic Policy into a Cluster and Push It Over the Limit

Goal

Translate the traffic policy of a DestinationRule into Envoy cluster fields, and send requests to count how outlier detection and the circuit breaker actually behave.

Why it matters

Traffic policies cause incidents not because the configuration is wrong but because the expectations about the behavior are wrong. Limits are counted per sidecar, outlier detection is an after-the-fact measure, and a subset policy overwrites the parent as a whole block. Once you have seen the translated cluster and the actual behavior, you can point out these three right away in configuration review.

Steps

Write a DestinationRule to /root/ist2-dr/dr.yaml — name ratings, namespace default, host ratings.default.svc.cluster.local. The trafficPolicy has loadBalancer.simple: LEAST_REQUEST, in connectionPool tcp.maxConnections: 3, http.http1MaxPendingRequests: 2 and http.http2MaxRequests: 5, and in outlierDetection consecutive5xxErrors: 2, interval: 5s, baseEjectionTime: 30s and maxEjectionPercent: 50. There is one subset, v2 (label version: v2), and only that subset gets trafficPolicy.connectionPool.tcp.maxConnections: 1. Put the output and exit code of istioctl validate in /root/ist2-dr/01-validate.txt (the last line is rc=0).
In /root/ist2-dr/02-map.txt, write as seven lines which field of the Envoy cluster each DestinationRule field becomes. Each line has the form <trafficPolicy 아래 경로>=<클러스터 아래 경로> (the placeholders are the path under trafficPolicy and the path under the cluster), and paths joined with dots do not use array indexes. The left sides are these seven — loadBalancer.simple, connectionPool.tcp.maxConnections, connectionPool.http.http1MaxPendingRequests, connectionPool.http.http2MaxRequests, outlierDetection.consecutive5xxErrors, outlierDetection.baseEjectionTime and outlierDetection.maxEjectionPercent.
Write an Envoy configuration to /root/ist2-dr/pool.yaml — sending every path of the listener 127.0.0.1:10085 (admin port 9985) to the cluster outbound|9080||ratings.default.svc.cluster.local. That cluster has three endpoints (127.0.0.1:8107, 127.0.0.1:8108 and 127.0.0.1:8115) and the fields (lb_policy, one entry of circuit_breakers.thresholds, and outlier_detection — including the interval) that carry over the parent traffic policy of step 1 according to the step 2 correspondence table. Put the output and exit code of envoy --mode validate in /root/ist2-dr/03-validate.txt (the last line is rc=0).
Start three upstreams (8107 and 8115 are ok, 8108 is fail), start Envoy with pool.yaml, pull the values of this cluster from localhost:9985/config_dump?resource=static_clusters and write them on one line to /root/ist2-dr/04-dump.txt — lb=<lb_policy> maxconn=<max_connections> pend=<max_pending_requests> req=<max_requests> c5xx=<consecutive_5xx> base=<base_ejection_time> pct=<max_ejection_percent>.
Right after you start Envoy again with pool.yaml, send 30 requests one after another, and write four lines to /root/ist2-dr/05-eject.txt — failed_requests= (the number of 503s among the 30), ejected= (the 주소:포트 of the endpoint whose health_flags is /failed_outlier_check in /clusters, where the placeholders are the address and the port), ejections_enforced_total= (the value of the statistic outlier_detection.ejections_enforced_total) and ejections_active= (the value of the statistic outlier_detection.ejections_active).
Copy pool.yaml to /root/ist2-dr/pool-subset.yaml and add the cluster of subset v2, outbound|9080|v2|ratings.default.svc.cluster.local (endpoint 127.0.0.1:8115). The same as what Istio makes from the DestinationRule of step 1 — lb_policy and outlier_detection are the same as the parent's, and circuit_breakers.thresholds is made from only the subset's connectionPool: max_connections: 1, and for max_pending_requests, max_requests and max_retries, which the subset does not set, Istio's default 4294967295. The route sends the prefix /v2 to this cluster and leaves the rest as it is. After you start it again, read from the default_priority lines of /clusters and write three lines to /root/ist2-dr/06-subset.txt: v2_max_connections=, v2_max_pending_requests= and default_max_pending_requests= (the value of the parent cluster).
Copy pool.yaml to /root/ist2-dr/pool-cb.yaml and add the cluster outbound|9080|slow|ratings.default.svc.cluster.local — endpoint 127.0.0.1:8116 (an upstream that takes 3 seconds to answer), circuit_breakers.thresholds of max_connections: 1 and max_pending_requests: 1 (in Istio terms tcp.maxConnections: 1 and http.http1MaxPendingRequests: 1), and no outlier detection. The route sends the prefix /slow to this cluster. Start an upstream on 8116 as slow, start Envoy again with --concurrency 1, and send five /slow requests at the same time (start them with & and wait). In /root/ist2-dr/07-overflow.txt, write three lines: ok= (the number of 200s), overflow_503= (the number of 503s) and pending_overflow= (the value of the slow cluster statistic upstream_rq_pending_overflow).
In /root/ist2-dr/08-report.md, write four lines — lb_policy=, failures_before_eject= (the number of 503s taken before the ejection in step 5), v2_pending_limit= (the queue limit of v2 you saw in step 6) and overflow_status= (the code the overflowed requests received in step 7) — and below them write explanations starting with - in at least four lines.

Notes

This Pod has neither a real istiod nor a real sidecar. So you cannot see the actual generated output with istioctl proxy-config; instead you know the translation rules and build the equivalent Envoy configuration by hand to confirm the behavior. The same rules show up as they are in the proxy-config output of a production cluster.
From step 4, start Envoy with --concurrency 1. The experiments that count the circuit breaker and outlier detection then do not wobble.
Statistics and outlier state accumulate while the process is alive. Do the counting steps right after you start it again.
The names in /clusters and /stats contain |. Use grep -F or escape with \|.
When you start Envoy, detach it completely from the shell with setsid --fork nohup envoy -c <파일> --log-level warn > <로그> 2>&1 </dev/null (the placeholders are the file and the log). Before you start it again, clean up with pkill -x envoy (pkill -f 'envoy -c' also kills the shell itself that contains that string).
A server for imitating an upstream is already in the image: python3 /opt/lab/envoy/upstream.py <포트> ok|fail|slow (the placeholder is the port). The response body is <모드>:<포트> <경로> (the placeholders are the mode, the port and the path).
After you edit the configuration, first filter it with envoy --mode validate -c <파일> before you start it. The cluster name contains |, so in YAML you must always wrap it in quotes.

Write a DestinationRule with a traffic policy

Write a DestinationRule to /root/ist2-dr/dr.yaml — name ratings, namespace default, host ratings.default.svc.cluster.local. The trafficPolicy has loadBalancer.simple: LEAST_REQUEST, in connectionPool tcp.maxConnections: 3, http.http1MaxPendingRequests: 2 and http.http2MaxRequests: 5, and in outlierDetection consecutive5xxErrors: 2, interval: 5s, baseEjectionTime: 30s and maxEjectionPercent: 50. There is one subset, v2 (label version: v2), and only that subset gets trafficPolicy.connectionPool.tcp.maxConnections: 1. Put the output and exit code of istioctl validate in /root/ist2-dr/01-validate.txt (the last line is rc=0).

A DestinationRule is "how to treat it after sending". The traffic policy has three blocks — whom to send to (loadBalancer), how much to accept at once (connectionPool) and when to take a sick endpoint out (outlierDetection). You can put a trafficPolicy of the same shape under a subset too, and how it combines with the parent is the subject of step 6.

Build a correspondence table of seven fields

In /root/ist2-dr/02-map.txt, write as seven lines which field of the Envoy cluster each DestinationRule field becomes. Each line has the form <trafficPolicy 아래 경로>=<클러스터 아래 경로> (the placeholders are the path under trafficPolicy and the path under the cluster), and paths joined with dots do not use array indexes. The left sides are these seven — loadBalancer.simple, connectionPool.tcp.maxConnections, connectionPool.http.http1MaxPendingRequests, connectionPool.http.http2MaxRequests, outlierDetection.consecutive5xxErrors, outlierDetection.baseEjectionTime and outlierDetection.maxEjectionPercent.

You can find fields with similar names in Envoy's cluster configuration documentation (cluster.proto). Connection pool limits go into Envoy as a per-priority list under circuit_breakers.thresholds — what Istio calls a "connection pool" and what Envoy calls a "circuit breaker" are the same numbers. The field names of outlier detection get a little shorter as they change from camel case to underscore case, as in consecutive5xxErrors.

Set up the cluster according to the correspondence table

Write an Envoy configuration to /root/ist2-dr/pool.yaml — sending every path of the listener 127.0.0.1:10085 (admin port 9985) to the cluster outbound|9080||ratings.default.svc.cluster.local. That cluster has three endpoints (127.0.0.1:8107, 127.0.0.1:8108 and 127.0.0.1:8115) and the fields (lb_policy, one entry of circuit_breakers.thresholds, and outlier_detection — including the interval) that carry over the parent traffic policy of step 1 according to the step 2 correspondence table. Put the output and exit code of envoy --mode validate in /root/ist2-dr/03-validate.txt (the last line is rc=0).

circuit_breakers.thresholds is a list — because you can give limits separately per priority (DEFAULT, HIGH). Istio uses only the one DEFAULT entry. Write time values as strings with units attached, like 30s. 8108 is the place for the endpoint that always returns 503 (step 5).

Read back six values from config_dump

Start three upstreams (8107 and 8115 are ok, 8108 is fail), start Envoy with pool.yaml, pull the values of this cluster from localhost:9985/config_dump?resource=static_clusters and write them on one line to /root/ist2-dr/04-dump.txt — lb=<lb_policy> maxconn=<max_connections> pend=<max_pending_requests> req=<max_requests> c5xx=<consecutive_5xx> base=<base_ejection_time> pct=<max_ejection_percent>.

This step puts what you wrote in the file side by side with what Envoy read in. What istioctl proxy-config cluster <파드> --fqdn … -o json shows in production is exactly this dump (the placeholder is the Pod). Pick with .configs[].cluster | select(.name==…) and then make one line with jq string interpolation. The dump omits fields that are at their default, so if a value you expected is missing, it did not go into the configuration.

Outlier detection removes it after suffering the failures

Right after you start Envoy again with pool.yaml, send 30 requests one after another, and write four lines to /root/ist2-dr/05-eject.txt — failed_requests= (the number of 503s among the 30), ejected= (the 주소:포트 of the endpoint whose health_flags is /failed_outlier_check in /clusters, where the placeholders are the address and the port), ejections_enforced_total= (the value of the statistic outlier_detection.ejections_enforced_total) and ejections_active= (the value of the statistic outlier_detection.ejections_active).

Outlier detection is a mechanism that takes an endpoint out after getting hit. When one endpoint returns 5xx consecutive5xxErrors times in a row, it is taken out of load balancing for baseEjectionTime. So users take that many 503s first. After it is out, only the other two receive, so the 503s stop. maxEjectionPercent: 50 means that at most one of three can be taken out. The /clusters lines have the form 클러스터::주소::health_flags::값 (the placeholders are the cluster, the address, health_flags and the value).

A subset policy overwrites as a whole block

Copy pool.yaml to /root/ist2-dr/pool-subset.yaml and add the cluster of subset v2, outbound|9080|v2|ratings.default.svc.cluster.local (endpoint 127.0.0.1:8115). The same as what Istio makes from the DestinationRule of step 1 — lb_policy and outlier_detection are the same as the parent's, and circuit_breakers.thresholds is made from only the subset's connectionPool: max_connections: 1, and for max_pending_requests, max_requests and max_retries, which the subset does not set, Istio's default 4294967295. The route sends the prefix /v2 to this cluster and leaves the rest as it is. After you start it again, read from the default_priority lines of /clusters and write three lines to /root/ist2-dr/06-subset.txt: v2_max_connections=, v2_max_pending_requests= and default_max_pending_requests= (the value of the parent cluster).

The official documentation says only that "the traffic policy at the subset level overrides the corresponding setting at the DestinationRule level". The unit of overriding is a block such as connectionPool, loadBalancer, outlierDetection or tls. So if you write only tcp.maxConnections on a subset, the whole connectionPool block is replaced, and the parent's http limits (2 and 5) do not follow to v2. The blank is the default Istio puts in (4294967295, effectively unlimited) — looser even than Envoy's own default of 1024. In /clusters, find the line <이름>::default_priority::max_pending_requests::<값> (the placeholders are the name and the value).

Count the circuit breaker actually opening

Copy pool.yaml to /root/ist2-dr/pool-cb.yaml and add the cluster outbound|9080|slow|ratings.default.svc.cluster.local — endpoint 127.0.0.1:8116 (an upstream that takes 3 seconds to answer), circuit_breakers.thresholds of max_connections: 1 and max_pending_requests: 1 (in Istio terms tcp.maxConnections: 1 and http.http1MaxPendingRequests: 1), and no outlier detection. The route sends the prefix /slow to this cluster. Start an upstream on 8116 as slow, start Envoy again with --concurrency 1, and send five /slow requests at the same time (start them with & and wait). In /root/ist2-dr/07-overflow.txt, write three lines: ok= (the number of 200s), overflow_503= (the number of 503s) and pending_overflow= (the value of the slow cluster statistic upstream_rq_pending_overflow).

If the upstream is slow, one connection is tied up for 3 seconds. Requests that come in the meantime wait in the queue, and when the queue is full too, Envoy does not even go to the upstream and immediately returns a 503 and the header x-envoy-overloaded: true. So the number it accepts is "number of connections + queue length". If you send them one after another, they finish one at a time and nothing overflows, so you must send them at the same time. Statistics are cumulative, so count only once, right after you start it again.

Summarize it as a DestinationRule translation table

In /root/ist2-dr/08-report.md, write four lines — lb_policy=, failures_before_eject= (the number of 503s taken before the ejection in step 5), v2_pending_limit= (the queue limit of v2 you saw in step 6) and overflow_status= (the code the overflowed requests received in step 7) — and below them write explanations starting with - in at least four lines.

Copy the values from the files of the earlier steps. In the explanation lines, it is good to write "what I will be careful about when I edit a DestinationRule" — in particular, the way a subset policy overwrites the parent is easy to miss in review.