Declaring a Cluster Without kube-proxy

Goal

Declare the Cilium settings that replace kube-proxy precisely in a values file, confirm how backends get picked up with a real workload and Service, and build a script that proves for yourself that "there really is no kube-proxy."

Why it matters

It looks as if a single line, kubeProxyReplacement: true, would finish the job, but in the field it does not. If you leave out k8sServiceHost, Cilium cannot find the API server during boot and the whole cluster does not come up, and if you switch over by removing kube-proxy from an existing cluster, the KUBE- chains left on the nodes conflict with the eBPF path. So the textbook practice is not to install it in the first place, and whether that actually happened has to be confirmed by counting the chains.

This lab teaches the process in three strands. Writing the values file covers "what you turn on," actually applying resources covers "how labels and selectors create backends," and the verification script covers "how you prove it." The third one matters especially. What the documentation says a feature does and what actually works on this node are different claims.

Write the Cilium Helm values and the script under /root/cca-config/, and actually apply the basic Kubernetes resources.

Steps

1. Create the namespace cca-net.
2. Create /root/cca-config/cilium-values.yaml and write kubeProxyReplacement: true, k8sServiceHost: 10.0.0.120, k8sServicePort: 6443, and ipam.mode: kubernetes at the top level.
3. In the same file, add routingMode: tunnel, tunnelProtocol: vxlan, bpf.masquerade: true, l7Proxy: true, encryption.enabled: true, and encryption.type: wireguard.
4. Put the label cca.homelab/tier=gpu on the nodes lab-node-0 and lab-node-1, and the label cca.homelab/tier=cpu on the node lab-node-2.
5. In the namespace cca-net, deploy the Deployment web with 3 replicas. The Pod label is app=web, the image is nginx:1.27-alpine, and the nodeSelector of the Pod spec is cca.homelab/tier: gpu.
6. In the same namespace, create the Service web. The type is ClusterIP, port is 80, targetPort is 8080, and the selector is app=web.
7. Write /root/cca-config/verify-kubeproxy-free.sh and make it executable. The script must (1) count the kube-proxy Pods in kube-system with kubectl, (2) count the KUBE- chains in the output of iptables-save, and (3) check the KubeProxyReplacement value in the cilium status. If a count is not 0, print a message and finish with exit 1.

Notes

• Do not write nesting in the values file with dots as in ipam.mode; write it as real YAML hierarchy. That is mode: kubernetes under ipam:.
• When a label key contains a slash, you can write it as is, as in kubectl label node lab-node-0 cca.homelab/tier=gpu.
• If a Pod is Pending, check the events of kubectl describe pod for a nodeSelector mismatch.
• Common mistake 1: omitting the Service's targetPort so that it takes the same value as port. The port the container listens on and the Service port are separate.
• Common mistake 2: a verification script that only prints the count and does not make a judgment. It is only verification if it compares against 0 and produces a failure.

Create the lab namespace

Create the namespace cca-net.

This is the simplest first step. Just get the name exactly right.

Write the core values for kube-proxy replacement

Create /root/cca-config/cilium-values.yaml and write kubeProxyReplacement: true, k8sServiceHost: 10.0.0.120, k8sServicePort: 6443, and ipam.mode: kubernetes at the top level.

Without kube-proxy, Cilium itself has to resolve the API server VIP, but at bootstrap time it is not up yet. That is why you have to tell it the real address separately. For IPAM, choose the mode that uses the node's PodCIDR as is.

Fill in the datapath options

In the same file, add routingMode: tunnel, tunnelProtocol: vxlan, bpf.masquerade: true, l7Proxy: true, encryption.enabled: true, and encryption.type: wireguard.

Continue writing in the same file. This is a homelab whose underlay does not know the PodCIDR routes, so you need encapsulation, and SNAT must be handed to eBPF rather than iptables. There is one more switch you must turn on to use policies at the HTTP method level.

Put tier labels on the nodes

Put the label cca.homelab/tier=gpu on the nodes lab-node-0 and lab-node-1, and the label cca.homelab/tier=cpu on the node lab-node-2.

From Kubernetes' point of view, every GPU is just one GPU. To place workloads meaningfully, a person has to plant that meaning with labels. Pay attention to the slash in the label key.

Deploy a workload with a placement constraint

In the namespace cca-net, deploy the Deployment web with 3 replicas. The Pod label is app=web, the image is nginx:1.27-alpine, and the nodeSelector of the Pod spec is cca.homelab/tier: gpu.

Put the nodeSelector in the Pod template. The label key and value must match exactly what you attached in the previous step, and if they do not, the Pods stay Pending.

Group them with a Service and check the backends

In the same namespace, create the Service web. The type is ClusterIP, port is 80, targetPort is 8080, and the selector is app=web.

The Service port and the container port can differ. If the selector does not match the Pod labels, the Service is created but has 0 backends. This state is the most common reason traffic does not flow, whether it is kube-proxy or eBPF.

Write the verification script

Write /root/cca-config/verify-kubeproxy-free.sh and make it executable. The script must (1) count the kube-proxy Pods in kube-system with kubectl, (2) count the KUBE- chains in the output of iptables-save, and (3) check the KubeProxyReplacement value in the cilium status. If a count is not 0, print a message and finish with exit 1.

It has to be a measurement, not an assertion. Count the kube-proxy Pods and the KUBE- chains on the node and check that they are 0, and look at the Cilium-side status as well. On failure, return a non-zero exit code, and give the file execute permission too.