Build a Bootstrap and Separate Two Ways a Proxy Fails to Start

Goal

Pull pilot-agent's ingredients out of the injection output, write an istiod-style bootstrap by hand and validate it, and create and tell apart two cases in which the sidecar does not start (configuration rejected and no configuration server).

Why it matters

Half of sidecar outages come as two symptoms — the container is in CrashLoop, or it is up but not Ready. The former means the bootstrap is wrong and the latter means it could not receive configuration from istiod. If you know what the bootstrap is made of, you can tell the two apart with one line of log and one look at the admin port.

Steps

In /root/ist2-boot, use istioctl kube-inject to put a sidecar into /opt/lab/fixtures/istio/inject-target.yaml and save it as /root/ist2-boot/inject.yaml (pass all three injection configuration files). Then read the args of the istio-proxy container and write four lines to /root/ist2-boot/01-args.txt — role= (the second argument), domain= (the argument after --domain, exactly as it is), proxy_log_level= (the value of --proxyLogLevel=) and component_log_level= (the value of --proxyComponentLogLevel=).
From the environment variables of istio-proxy in /root/ist2-boot/inject.yaml, pick seven and write them to /root/ist2-boot/02-env.txt as 이름=값 (the placeholders are the name and the value) — for CA_ADDR, PILOT_CERT_PROVIDER, ISTIO_META_CLUSTER_ID, TRUST_DOMAIN and ISTIO_META_WORKLOAD_NAME, write the value as it is, and for POD_NAMESPACE and SERVICE_ACCOUNT, whose values come from the Pod, write them in the form fieldRef:<fieldPath> (for example fieldRef:metadata.name).
Read /opt/istio/mesh-config.yaml and write four lines to /root/ist2-boot/03-mesh.txt — discovery_address= (defaultConfig.discoveryAddress), root_namespace=, trust_domain=, and on ca_addr_same= write yes if that discoveryAddress is character for character the same as the CA_ADDR from step 2, otherwise no.
Write a bootstrap to /root/ist2-boot/bootstrap.yaml — admin port 9982; node.id is the four fields sidecar, 10.0.0.7, payments-6d5f9.default and default.svc.cluster.local, joined in this order with tilde (~); node.cluster is payments.default, and node.metadata has the four keys CLUSTER_ID, MESH_ID, NAMESPACE and WORKLOAD_NAME (the values are the ISTIO_META_ variables from step 2 and the namespace default); dynamic_resources receives cds_config and lds_config over ADS (gRPC) with initial_fetch_timeout: 0s; the static cluster name ADS points to is xds-grpc, which points at the discovery_address from step 3 with STRICT_DNS and uses HTTP/2. Put the output and exit code of envoy --mode validate in /root/ist2-boot/04-validate.txt (the last line is rc=0).
Copy /root/ist2-boot/bootstrap.yaml to /root/ist2-boot/boot-typo.yaml, then change only the ADS side (cluster_name of ads_config) to istiod-xds (leave the static cluster name xds-grpc as it is). Check it with envoy --mode validate and put the output and exit code in /root/ist2-boot/05-typo.txt (with rc= on the last line).
Start Envoy with /root/ist2-boot/bootstrap.yaml (there is no istiod in this Pod). From /config_dump on the admin port, pull out only the three fields id, cluster and metadata of the bootstrap's node and save them to /root/ist2-boot/06-node.json, and write three lines to /root/ist2-boot/06-ready.txt — ready_code= (the HTTP code of /ready), ready_body= (its body) and connected_state= (the value of the statistic control_plane.connected_state).
Find the ingredients of identity in /root/ist2-boot/inject.yaml and write five lines to /root/ist2-boot/07-identity.txt — token_volume= (the name of the volume that projects serviceAccountToken), token_audience= (the audience of that token), token_mount= (the path where that volume is attached to istio-proxy), ca_configmap= (the name of the ConfigMap the volume istiod-ca-cert reads) and ca_mount= (the path where istiod-ca-cert is attached to istio-proxy).
In /root/ist2-boot/08-report.md, write four lines, xds_address=, xds_cluster=, typo_rc= and ready_without_istiod= (respectively the address that receives xDS, the name of the static cluster ADS must point to, the exit code of step 5, and the /ready body you saw in step 6), and below them write explanations starting with - in at least four lines.

Notes

This Pod has neither a real istiod nor a real sidecar. So you cannot see the actual generated output with istioctl proxy-config; instead you know the translation rules and build the equivalent Envoy configuration by hand to confirm the behavior. The same rules show up as they are in the proxy-config output of a production cluster.
Because of the empty document at the end of the injection output, always filter yq with select(.kind=="Deployment") | ….
It is normal that the Envoy in step 6 does not become LIVE. When you wait for startup, wait not for /ready to be LIVE but for the admin port to respond.
When you start Envoy, detach it completely from the shell with setsid --fork nohup envoy -c <파일> --log-level warn > <로그> 2>&1 </dev/null (the placeholders are the file and the log). Before you start it again, clean up with pkill -x envoy (pkill -f 'envoy -c' also kills the shell itself that contains that string).

The proxy container starts pilot-agent, not Envoy

In /root/ist2-boot, use istioctl kube-inject to put a sidecar into /opt/lab/fixtures/istio/inject-target.yaml and save it as /root/ist2-boot/inject.yaml (pass all three injection configuration files). Then read the args of the istio-proxy container and write four lines to /root/ist2-boot/01-args.txt — role= (the second argument), domain= (the argument after --domain, exactly as it is), proxy_log_level= (the value of --proxyLogLevel=) and component_log_level= (the value of --proxyComponentLogLevel=).

The entrypoint of the image is pilot-agent, and the first two words of the arguments, proxy sidecar, are a subcommand meaning "start a proxy in the sidecar role". $(POD_NAMESPACE) in the value of --domain is not shell but Kubernetes syntax that the kubelet replaces with the environment variable of the same name when it starts the container — it stays literally in the manifest, so copy it as it is. For the form --x=값 (where the Korean word stands for the value), you can strip off just the value with sed 's/^--x=//'.

Environment variables become the proxy's label and address book

From the environment variables of istio-proxy in /root/ist2-boot/inject.yaml, pick seven and write them to /root/ist2-boot/02-env.txt as 이름=값 (the placeholders are the name and the value) — for CA_ADDR, PILOT_CERT_PROVIDER, ISTIO_META_CLUSTER_ID, TRUST_DOMAIN and ISTIO_META_WORKLOAD_NAME, write the value as it is, and for POD_NAMESPACE and SERVICE_ACCOUNT, whose values come from the Pod, write them in the form fieldRef:<fieldPath> (for example fieldRef:metadata.name).

The environment variables are of two kinds. There are values already decided at injection time (value) and values the kubelet fills in when the Pod starts (valueFrom.fieldRef). The Pod name, namespace and service account cannot be known at the moment of injection, so they are the latter. Those that start with ISTIO_META_ have their prefix removed by pilot-agent and are moved into Envoy's node.metadata — the label by which istiod recognizes this proxy. In yq, pick them one by one with .env[] | select(.name=="CA_ADDR").

Cross-check the mesh configuration defaults against the CA address

Read /opt/istio/mesh-config.yaml and write four lines to /root/ist2-boot/03-mesh.txt — discovery_address= (defaultConfig.discoveryAddress), root_namespace=, trust_domain=, and on ca_addr_same= write yes if that discoveryAddress is character for character the same as the CA_ADDR from step 2, otherwise no.

defaultConfig is the mesh-wide default proxy configuration, and within it discoveryAddress is the address to fetch xDS from. istiod is both the configuration server and the certificate authority (CA), so in a default installation, the two addresses point to the same port 15012. They differ when you attach an external CA. Pull the values out with yq and compare them in the shell with [ "$a" = "$b" ].

Write an istiod-style bootstrap by hand and filter it

Write a bootstrap to /root/ist2-boot/bootstrap.yaml — admin port 9982; node.id is the four fields sidecar, 10.0.0.7, payments-6d5f9.default and default.svc.cluster.local, joined in this order with tilde (~); node.cluster is payments.default, and node.metadata has the four keys CLUSTER_ID, MESH_ID, NAMESPACE and WORKLOAD_NAME (the values are the ISTIO_META_ variables from step 2 and the namespace default); dynamic_resources receives cds_config and lds_config over ADS (gRPC) with initial_fetch_timeout: 0s; the static cluster name ADS points to is xds-grpc, which points at the discovery_address from step 3 with STRICT_DNS and uses HTTP/2. Put the output and exit code of envoy --mode validate in /root/ist2-boot/04-validate.txt (the last line is rc=0).

In the bootstrap pilot-agent makes, the static cluster is effectively just xds-grpc. Listeners and service clusters are all received through ADS via that cluster. gRPC runs over HTTP/2, so you must turn on http2_protocol_options on the cluster. The four fields of node.id are in the order role, Pod IP, 파드이름.네임스페이스 and 네임스페이스.svc.cluster.local (the placeholders are the Pod name and the namespace). initial_fetch_timeout: 0s means "wait indefinitely until configuration is received", which is the value Istio actually uses.

One character of a cluster name makes a CrashLoop

Copy /root/ist2-boot/bootstrap.yaml to /root/ist2-boot/boot-typo.yaml, then change only the ADS side (cluster_name of ads_config) to istiod-xds (leave the static cluster name xds-grpc as it is). Check it with envoy --mode validate and put the output and exit code in /root/ist2-boot/05-typo.txt (with rc= on the last line).

The cluster_name of ADS means "open a gRPC connection to the static cluster with this name". If that name is not in the static cluster list, Envoy has no way to reach a configuration server, so it rejects it before it starts. In Kubernetes the container ends right away and starts again, over and over — that is CrashLoopBackOff. The wrong name is printed as it is in the rejection message, so you can pin down the cause from one line of the log.

A proxy that cannot reach istiod is not ready

Start Envoy with /root/ist2-boot/bootstrap.yaml (there is no istiod in this Pod). From /config_dump on the admin port, pull out only the three fields id, cluster and metadata of the bootstrap's node and save them to /root/ist2-boot/06-node.json, and write three lines to /root/ist2-boot/06-ready.txt — ready_code= (the HTTP code of /ready), ready_body= (its body) and connected_state= (the value of the statistic control_plane.connected_state).

The readiness state of the sidecar is in the end Envoy's /ready (pilot-agent answers it on 15021 on its behalf). An Envoy started with initial_fetch_timeout: 0s does not finish initialization and waits until it receives the first CDS and LDS. Even in this state the admin port is open, so you can see what was applied with config_dump. Pick just the three fields with jq '.configs[0].bootstrap.node | {id, cluster, metadata}'. Get the code of /ready with curl -o /dev/null -w '%{http_code}'.

Identity starts from two volumes

Find the ingredients of identity in /root/ist2-boot/inject.yaml and write five lines to /root/ist2-boot/07-identity.txt — token_volume= (the name of the volume that projects serviceAccountToken), token_audience= (the audience of that token), token_mount= (the path where that volume is attached to istio-proxy), ca_configmap= (the name of the ConfigMap the volume istiod-ca-cert reads) and ca_mount= (the path where istiod-ca-cert is attached to istio-proxy).

As soon as it starts, pilot-agent sends a certificate signing request to istiod (= CA_ADDR). What proves "I am this service account" at that point is the projected token whose audience is narrowed to istio-ca, and what confirms that the other side is the real istiod is the root certificate distributed through a ConfigMap. Volumes are in .volumes[], and the attach paths are in .volumeMounts[] of istio-proxy. For the projected volume, look under .projected.sources[].serviceAccountToken.

Summarize it as a checklist for when the sidecar does not start

In /root/ist2-boot/08-report.md, write four lines, xds_address=, xds_cluster=, typo_rc= and ready_without_istiod= (respectively the address that receives xDS, the name of the static cluster ADS must point to, the exit code of step 5, and the /ready body you saw in step 6), and below them write explanations starting with - in at least four lines.

Copy the values from the files of the earlier steps. This table organizes where to look first when you meet "the sidecar is in CrashLoop" or "the Pod does not become Ready" — it is good to pair symptoms with causes in the explanation lines.