What happens after the proxy is attached: resources, startup order and Jobs that never finish

Goal

You run istioctl kube-inject offline and check which field of the injection result each annotation goes down to. You look at the field level at resource requests and limits, proxy configuration (concurrency and drain time), startup order guarantee, excluding outbound ports, opting out of injection, and the classic incident of batch jobs that never finish and its fix.

Why it matters

Attaching a sidecar is one line, but the problems that arise after attaching usually come from one cell of configuration. The default proxy requests CPU 100m and memory 128Mi — with 2,000 Pods that alone is 200 CPU cores. Startup order is a problem too. If the app comes up before the proxy, requests going out in the first few seconds simply fail. The oldest trap is batch jobs. If the proxy goes in as an ordinary container, the Pod cannot move on to completed even after the job finishes, and in the past people worked around it by having the app ask the proxy to shut down as it finished. Now Kubernetes supports init containers with a restart policy, so you fundamentally solve it by moving the proxy there. This lab is the place to learn how to check all of these knobs directly in the injected manifest. If you can check before putting anything on the cluster, configuration changes become a subject of review.

Steps

Create /root/ist-lifecycle and in /root/ist-lifecycle/dep.yaml write the Deployment orders in the lifecycle namespace — replicas 2, label app: orders, one container (app, image nginx:1.27, container port 8080), and no annotations. Inject this file and save the result to /root/ist-lifecycle/baseline.yaml, and write the four things you read from it in /root/ist-lifecycle/baseline.tsv as four lines: containers, initContainers, proxyCpuRequest, and proxyMemRequest (join the container names with commas in the order they appear).
/root/ist-lifecycle/dep-resources.yaml is the step 1 Deployment copied with the name orders-res and four Pod template annotations added — sidecar.istio.io/proxyCPU is 250m, sidecar.istio.io/proxyMemory is 192Mi, sidecar.istio.io/proxyCPULimit is 1500m, and sidecar.istio.io/proxyMemoryLimit is 512Mi. Save the injection result to /root/ist-lifecycle/resources.yaml.
/root/ist-lifecycle/dep-proxyconfig.yaml is a copy with the name orders-pc with one proxy.istio.io/config annotation put in — the value is multi-line YAML with two entries, concurrency: 3 and terminationDrainDuration: 45s. Save the injection result to /root/ist-lifecycle/proxyconfig.yaml, and save the value of the environment variable PROXY_CONFIG of istio-proxy from the result to /root/ist-lifecycle/proxy-config.json.
/root/ist-lifecycle/dep-hold.yaml is a copy with the name orders-hold with only holdApplicationUntilProxyStarts: true put in the proxy.istio.io/config annotation. Save the injection result to /root/ist-lifecycle/hold.yaml, and write the container order and the lifecycle hook that appeared on the proxy in /root/ist-lifecycle/hold.tsv as two lines, containerOrder and postStart (in the postStart column, write the command the hook runs joined by spaces).
/root/ist-lifecycle/dep-exclude.yaml is a copy with the name orders-exc with the annotation traffic.sidecar.istio.io/excludeOutboundPorts put in as "3306,5432". Save the injection result to /root/ist-lifecycle/exclude.yaml, and write the arguments of the istio-init container joined by spaces on one line in /root/ist-lifecycle/init-args.txt.
/root/ist-lifecycle/dep-nosidecar.yaml is a copy with the name orders-off with the annotation sidecar.istio.io/inject put in as "false". Save the injection result to /root/ist-lifecycle/nosidecar.yaml — there must be no proxy in the result.
In /root/ist-lifecycle/job.yaml, write the Job nightly-settle in the lifecycle namespace — Pod label app: nightly-settle, restartPolicy: Never, and one container (worker, image busybox:1.36). /root/ist-lifecycle/job-native.yaml is a copy with only the name changed to nightly-settle-native and the Pod annotation sidecar.istio.io/nativeSidecar added as "true". Inject each and save them to /root/ist-lifecycle/job-injected.yaml and /root/ist-lifecycle/job-native-injected.yaml, and write the difference in /root/ist-lifecycle/job-compare.tsv as two lines, plain and native, in the format <이름>\t<containers>\t<initContainers>\t<프록시의 restartPolicy> (the placeholders are the name, containers, initContainers, and the proxy's restartPolicy; write - if there is no restartPolicy).
Create /root/ist-lifecycle/inject-summary.sh so that it re-injects in turn the eight original manifests in this directory (dep.yaml, dep-resources.yaml, dep-proxyconfig.yaml, dep-hold.yaml, dep-exclude.yaml, dep-nosidecar.yaml, job.yaml, and job-native.yaml) and prints, for each file, <파일이름>\t<containers>\t<initContainers>\t<프록시 CPU 요청> (the placeholders are the file name, containers, initContainers, and the proxy CPU request) in that order only to standard output (use - for missing cells). Save the output to /root/ist-lifecycle/inject-summary.tsv.

Notes

The injection command needs three configuration files — --injectConfigFile, --meshConfigFile, and --valuesFile. In this lab image they are under /opt/istio/.
Attach annotations to the Pod template. If you attach them to the Deployment metadata, nothing happens.
proxy.istio.io/config is an annotation that holds YAML inside its value, so write it as a multi-line block (|).
Common mistake: looking for the proxy only in containers. A native sidecar is in initContainers.
Common mistake: leaving out quotes on an annotation value. "false" must be a string.
Reference: https://istio.io/v1.24/docs/reference/config/annotations/
Reference: https://istio.io/v1.24/docs/setup/additional-setup/sidecar-injection/
Reference: https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/

What shape the proxy goes in when there are no annotations

Create /root/ist-lifecycle and in /root/ist-lifecycle/dep.yaml write the Deployment orders in the lifecycle namespace — replicas 2, label app: orders, one container (app, image nginx:1.27, container port 8080), and no annotations. Inject this file and save the result to /root/ist-lifecycle/baseline.yaml, and write the four things you read from it in /root/ist-lifecycle/baseline.tsv as four lines: containers, initContainers, proxyCpuRequest, and proxyMemRequest (join the container names with commas in the order they appear).

There is no istiod, so you must give the three injection configuration files yourself — --injectConfigFile /opt/istio/inject-config.yaml --meshConfigFile /opt/istio/mesh-config.yaml --valuesFile /opt/istio/values-config.yaml. To see the order in the result, yq -N e '.spec.template.spec.containers[].name' is convenient.

Adjust the proxy's resources with annotations

/root/ist-lifecycle/dep-resources.yaml is the step 1 Deployment copied with the name orders-res and four Pod template annotations added — sidecar.istio.io/proxyCPU is 250m, sidecar.istio.io/proxyMemory is 192Mi, sidecar.istio.io/proxyCPULimit is 1500m, and sidecar.istio.io/proxyMemoryLimit is 512Mi. Save the injection result to /root/ist-lifecycle/resources.yaml.

Annotations must be attached to the Pod template (spec.template.metadata.annotations). If you attach them to the Deployment's metadata, nothing happens — because injection targets Pods. See how much it differs from the defaults by putting it side by side with the step 1 result.

Put the proxy's own configuration in one annotation as YAML

/root/ist-lifecycle/dep-proxyconfig.yaml is a copy with the name orders-pc with one proxy.istio.io/config annotation put in — the value is multi-line YAML with two entries, concurrency: 3 and terminationDrainDuration: 45s. Save the injection result to /root/ist-lifecycle/proxyconfig.yaml, and save the value of the environment variable PROXY_CONFIG of istio-proxy from the result to /root/ist-lifecycle/proxy-config.json.

This annotation's value is a string but holds YAML inside. Write it as a multi-line block (|). The injector converts it to JSON and puts it in one container environment variable — the key point is that the format a person writes and the format the proxy reads differ.

Prevent the race in which the app starts before the proxy

/root/ist-lifecycle/dep-hold.yaml is a copy with the name orders-hold with only holdApplicationUntilProxyStarts: true put in the proxy.istio.io/config annotation. Save the injection result to /root/ist-lifecycle/hold.yaml, and write the container order and the lifecycle hook that appeared on the proxy in /root/ist-lifecycle/hold.tsv as two lines, containerOrder and postStart (in the postStart column, write the command the hook runs joined by spaces).

When you turn this value on, the injector changes two things — where it puts the proxy in the container list, and what it attaches to the proxy. You can see it right away by comparing the container order with the step 1 result. The hook is in .lifecycle.postStart.exec.command.

Leave out ports that must not pass through the proxy

/root/ist-lifecycle/dep-exclude.yaml is a copy with the name orders-exc with the annotation traffic.sidecar.istio.io/excludeOutboundPorts put in as "3306,5432". Save the injection result to /root/ist-lifecycle/exclude.yaml, and write the arguments of the istio-init container joined by spaces on one line in /root/ist-lifecycle/init-args.txt.

Redirecting outgoing traffic to the proxy is done by the init container with iptables rules. The annotation goes down as an argument of that command line — find which flag those ports attach to. It is the knob you use to leave out ports the proxy may misinterpret, such as database protocols.

Make only this Pod not receive a proxy

/root/ist-lifecycle/dep-nosidecar.yaml is a copy with the name orders-off with the annotation sidecar.istio.io/inject put in as "false". Save the injection result to /root/ist-lifecycle/nosidecar.yaml — there must be no proxy in the result.

The knobs that turn injection on and off are in two layers, the namespace label and the Pod annotation, and the Pod side wins. It is the place you use to make exceptions for workloads that must not pass through the proxy (bulk transfers, protocols the proxy does not understand). Check whether the init container also disappears.

The batch job that never finishes and its fix

In /root/ist-lifecycle/job.yaml, write the Job nightly-settle in the lifecycle namespace — Pod label app: nightly-settle, restartPolicy: Never, and one container (worker, image busybox:1.36). /root/ist-lifecycle/job-native.yaml is a copy with only the name changed to nightly-settle-native and the Pod annotation sidecar.istio.io/nativeSidecar added as "true". Inject each and save them to /root/ist-lifecycle/job-injected.yaml and /root/ist-lifecycle/job-native-injected.yaml, and write the difference in /root/ist-lifecycle/job-compare.tsv as two lines, plain and native, in the format <이름>\t<containers>\t<initContainers>\t<프록시의 restartPolicy> (the placeholders are the name, containers, initContainers, and the proxy's restartPolicy; write - if there is no restartPolicy).

A proxy that went in as an ordinary container stays alive even after the work ends, so the Job cannot move on to completed. From Kubernetes 1.28 on, you can give an init container a restart policy to make a container that "starts first, lives to the end, and is cleaned up together when the main container finishes." See which list the proxy moves to when you turn on the annotation.

Re-inject eight manifests at once and harden the result into a table

Create /root/ist-lifecycle/inject-summary.sh so that it re-injects in turn the eight original manifests in this directory (dep.yaml, dep-resources.yaml, dep-proxyconfig.yaml, dep-hold.yaml, dep-exclude.yaml, dep-nosidecar.yaml, job.yaml, and job-native.yaml) and prints, for each file, <파일이름>\t<containers>\t<initContainers>\t<프록시 CPU 요청> (the placeholders are the file name, containers, initContainers, and the proxy CPU request) in that order only to standard output (use - for missing cells). Save the output to /root/ist-lifecycle/inject-summary.tsv.

You must find the proxy both when it is in an ordinary container and when it is in an init container — if you scan the two lists chained together, one expression does it. If you keep such a table in the repository, the difference shows immediately on the day the injection configuration changes.