TT Lab
Get started
Learn Learning paths Courses

CCA — Cilium Certified Associate

Taking Components Down One by One: What Stopped and What Held

Continue in TT Lab

Goal

On a real Cilium 1.20.1, you take down one component at a time to confirm what the cilium agent, operator, cilium-envoy, and hubble-relay each handle. First read where cluster-pool IPAM and CiliumIdentity are recorded, then record what happens to new Pods, L7 requests, flow queries, and existing traffic while a component is missing, and then restore it.

Why it matters

A report that "Cilium is down" is a completely different incident depending on which component stopped. The Operator handles work that only needs to happen once per cluster, so it does not take part in forwarding or policy decisions even if it is briefly absent. Without envoy, only the paths with L7 rules stop. Without relay, only the path for querying multiple nodes from one place is cut. Without the agent, the datapath that is already loaded keeps running, but new Pods cannot receive networking, so scheduling is blocked.

Every step that takes a component down must be followed by restoring it once you have recorded your observations (only the agent is taken down in step 6 and restored in step 7). The full grading after recovery compares those records against the Pod uid, identity, flow timestamps, and the creation time of the recovered Pod, so be sure to make the records while the component is down.

Environment preparation takes about 5 minutes. Between step 6 and step 7 there is no agent, so agent commands do not work. When the session ends, the files in /root/cca-parts disappear.

Steps

  1. Run kubectl apply -f /opt/fixtures/cca-components/workload.yaml to start app (2 replicas), api-l7, and client in cca-parts. Record the following in /root/cca-parts/ipam.json — ipam_mode, cluster_pool, and mask_size (the ipam, cluster-pool-ipv4-cidr, and cluster-pool-ipv4-mask-size entries of cilium-config; the last one is a number), node_pod_cidrs (the list of spec.ipam.podCIDRs on the CiliumNode), status_pool and capacity (from the IPAM line of the agent's cilium-dbg status: the range after "allocated from" and the number after the slash), router_ip (the IPv4 address of the cilium_host device on the VM), and pods (a dictionary mapping the names of the four cca-parts Pods to their IPs).
  2. Compare the CiliumEndpoint identity numbers of the two app replica Pods and of client, and read the CiliumIdentity object of app. In /root/cca-parts/identity.json, record app_identity, app_endpoints (the two app Pod names), client_identity, security_labels (the security-labels of that CiliumIdentity as a sorted list of strings in 키=값 form, that is, key=value), and allocation_mode (identity-allocation-mode in cilium-config).
  3. Scale cilium-operator down to replicas 0 and wait until all Operator Pods are gone. In that state, create a Pod born-no-operator in cca-parts (the same curl image as client, label born=no-operator, command sleep 86400), and once it is Ready, record the following in /root/cca-parts/operator-down.json: operator_replicas (the spec.replicas of the Deployment at that moment, as a number), pod_uid, pod_ip, and identity (the CiliumEndpoint identity of that Pod). After recording, scale the Operator back to 1 and wait until it is available.
  4. In cca-parts, create a CiliumNetworkPolicy l7-get — endpointSelector app=api-l7, with one ingress entry: fromEndpoints app=client, and under toPorts, TCP "8080" with rules.http [{method: GET, path: /allowed}]. Confirm that from client, /allowed on api-l7 returns 200 and /secret returns 403, then remove the envoy Pods by adding a nodeSelector cca-lab/envoy: "off", which matches no node, to the cilium-envoy DaemonSet in kube-system. While they are gone, request api-l7 /allowed (5-second limit) and the app Service (http://app:8080/) from client, read the client→api-l7 flows with hubble inside the agent, and record l7_code, plain_code (HTTP codes as strings; "000" for no response), and flows (the list of compact flow lines for client→api-l7) in /root/cca-parts/envoy-down.json. Then remove that nodeSelector key to restore envoy, and wait until /allowed returns 200 again.
  5. Scale hubble-relay in kube-system down to replicas 0. From the VM shell, capture the one-line error from hubble status --server <hubble-relay 서비스 ClusterIP>:80 failing (the placeholder is the hubble-relay Service ClusterIP), and inside the agent read the flow lines from hubble observe --last 5 -o compact and the Current/Max Flows line from hubble status; record them in /root/cca-parts/relay-down.json as relay_error, local_flows (a list), and local_status. Then scale relay back to 1 and wait until status through the relay succeeds.
  6. Remove the agent Pods by adding the nodeSelector cca-lab/agent: "off" to the cilium DaemonSet in kube-system (do not restore it in this step). Once the Pods are gone, check the node taint, request the app Service from client, then create a curl Pod orphan (command sleep 86400) in cca-parts and wait for a FailedScheduling event. In /root/cca-parts/agent-down.json, record existing_code (the client→app code string), taint (the key:effect of the cilium taint on the node), orphan_uid, orphan_phase, and reason and message (from the FailedScheduling event of orphan).
  7. Restore the agent by removing the nodeSelector key cca-lab/agent from the cilium DaemonSet. Poll until the agent is Ready, the agent-not-ready taint on the node is gone, and orphan has been scheduled (PodScheduled=True), then record taint_removed (true/false), orphan_scheduled_at (the lastTransitionTime of the PodScheduled condition of orphan), and agent_pod (the name of the new agent Pod) in /root/cca-parts/restore.json.
  8. In /root/cca-parts/report.txt, write eight lines in 키=값 form (key=value) — ipam_mode, node_pod_cidr, identity_allocation_mode, without_operator_new_pod (the current phase of born-no-operator), without_envoy_l7 and without_envoy_plain (the two codes from envoy-down.json), and without_agent_existing and without_agent_new_pod (existing_code and orphan_phase from agent-down.json). The values must match the record files and the current configuration.

Notes

Where do Pod addresses get carved from?

Run kubectl apply -f /opt/fixtures/cca-components/workload.yaml to start app (2 replicas), api-l7, and client in cca-parts. Record the following in /root/cca-parts/ipam.json — ipam_mode, cluster_pool, and mask_size (the ipam, cluster-pool-ipv4-cidr, and cluster-pool-ipv4-mask-size entries of cilium-config; the last one is a number), node_pod_cidrs (the list of spec.ipam.podCIDRs on the CiliumNode), status_pool and capacity (from the IPAM line of the agent's cilium-dbg status: the range after "allocated from" and the number after the slash), router_ip (the IPv4 address of the cilium_host device on the VM), and pods (a dictionary mapping the names of the four cca-parts Pods to their IPs).

In cluster-pool mode, the cluster-wide pool is cut into fixed-size pieces per node and written to the CiliumNode, and each node's agent hands out Pod addresses from within its piece. Check that the pool, the piece, and the actual addresses each contain the next. cilium_host, which acts as the node's gateway, also receives its address from the same piece.

Two replicas share one identity

Compare the CiliumEndpoint identity numbers of the two app replica Pods and of client, and read the CiliumIdentity object of app. In /root/cca-parts/identity.json, record app_identity, app_endpoints (the two app Pod names), client_identity, security_labels (the security-labels of that CiliumIdentity as a sorted list of strings in 키=값 form, that is, key=value), and allocation_mode (identity-allocation-mode in cilium-config).

An identity exists per set of security-relevant labels, not per Pod. With the crd allocation mode, that set is stored as a cluster-scoped object called CiliumIdentity, and its name is its number. Check that values such as the hash that is appended to a Pod name do not go into the security labels.

A Pod born while the Operator is gone

Scale cilium-operator down to replicas 0 and wait until all Operator Pods are gone. In that state, create a Pod born-no-operator in cca-parts (the same curl image as client, label born=no-operator, command sleep 86400), and once it is Ready, record the following in /root/cca-parts/operator-down.json: operator_replicas (the spec.replicas of the Deployment at that moment, as a number), pod_uid, pod_ip, and identity (the CiliumEndpoint identity of that Pod). After recording, scale the Operator back to 1 and wait until it is available.

The official documentation describes the Operator as the component that handles "work that only needs to happen once per cluster rather than once per node," and says it is not on the path of forwarding or policy decisions. Observe who hands out the address when the node already has a podCIDR piece, and who creates the identity object for a label combination that has not been seen before. Check the new identity number with kubectl get ciliumidentity.

Removing envoy stopped only the L7 path

In cca-parts, create a CiliumNetworkPolicy l7-get — endpointSelector app=api-l7, with one ingress entry: fromEndpoints app=client, and under toPorts, TCP "8080" with rules.http [{method: GET, path: /allowed}]. Confirm that from client, /allowed on api-l7 returns 200 and /secret returns 403, then remove the envoy Pods by adding a nodeSelector cca-lab/envoy: "off", which matches no node, to the cilium-envoy DaemonSet in kube-system. While they are gone, request api-l7 /allowed (5-second limit) and the app Service (http://app:8080/) from client, read the client→api-l7 flows with hubble inside the agent, and record l7_code, plain_code (HTTP codes as strings; "000" for no response), and flows (the list of compact flow lines for client→api-l7) in /root/cca-parts/envoy-down.json. Then remove that nodeSelector key to restore envoy, and wait until /allowed returns 200 again.

In Cilium without sidecars, the node's envoy decides HTTP rules. After the L3/L4 decision, eBPF hands that connection over to the proxy, so what happens when there is no envoy to receive it? A Service with no L7 rules does not go through the proxy. If the nodeSelector key contains a slash, write it as ~1 in the JSON patch path. Read the flows with hubble observe --since <시각> --namespace cca-parts -o compact (the placeholder is the start time).

Even with the relay down, the nodes were still collecting flows

Scale hubble-relay in kube-system down to replicas 0. From the VM shell, capture the one-line error from hubble status --server <hubble-relay 서비스 ClusterIP>:80 failing (the placeholder is the hubble-relay Service ClusterIP), and inside the agent read the flow lines from hubble observe --last 5 -o compact and the Current/Max Flows line from hubble status; record them in /root/cca-parts/relay-down.json as relay_error, local_flows (a list), and local_status. Then scale relay back to 1 and wait until status through the relay succeeds.

Hubble collects flows into a ring buffer on each agent, and relay is the intermediary that lets you query the buffers of multiple nodes from one place. Compare what you lose when each of the two stops. The error of a failed command goes to standard error, so capture it with 2>&1.

With the agent removed, existing requests still work

Remove the agent Pods by adding the nodeSelector cca-lab/agent: "off" to the cilium DaemonSet in kube-system (do not restore it in this step). Once the Pods are gone, check the node taint, request the app Service from client, then create a curl Pod orphan (command sleep 86400) in cca-parts and wait for a FailedScheduling event. In /root/cca-parts/agent-down.json, record existing_code (the client→app code string), taint (the key:effect of the cilium taint on the node), orphan_uid, orphan_phase, and reason and message (from the FailedScheduling event of orphan).

The agent is the side that loads and updates BPF programs and maps; it is not a process that carries packets itself. What happens to what is already loaded while the agent is absent? A new Pod cannot have networking attached by the CNI, so the Operator puts a taint on the node to block scheduling. View the event with kubectl get events --field-selector involvedObject.name=orphan.

When the agent returns, the taint is lifted

Restore the agent by removing the nodeSelector key cca-lab/agent from the cilium DaemonSet. Poll until the agent is Ready, the agent-not-ready taint on the node is gone, and orphan has been scheduled (PodScheduled=True), then record taint_removed (true/false), orphan_scheduled_at (the lastTransitionTime of the PodScheduled condition of orphan), and agent_pod (the name of the new agent Pod) in /root/cca-parts/restore.json.

The Operator both applies and lifts the taint by watching the state of the agent Pod. It can take longer after scheduling for the container to become Running (sandbox retries), so in this step wait only until scheduling. Right after the agent is replaced, hubble-relay also briefly loses Ready while it reattaches to its peers. Before the final full grading, check once that orphan is Running.

When something stops, what stops?

In /root/cca-parts/report.txt, write eight lines in 키=값 form (key=value) — ipam_mode, node_pod_cidr, identity_allocation_mode, without_operator_new_pod (the current phase of born-no-operator), without_envoy_l7 and without_envoy_plain (the two codes from envoy-down.json), and without_agent_existing and without_agent_new_pod (existing_code and orphan_phase from agent-down.json). The values must match the record files and the current configuration.

The report is a table that pairs, for each component, one line for "what kept working while it was gone" and one for "what stopped." Read the record files with jq, and read the configuration from cilium-config and the CiliumNode.