TT Lab
Get started
Learn Learning paths Courses

CCA — Cilium Certified Associate

Gateway API Enabled, but No GatewayClass Appears

Continue in TT Lab

Goal

On the real k3s + Cilium 1.20.1 inside the VM, you turn on the Gateway API in the wrong order and then fix it, and then use Gateway and HTTPRoute to confirm path, header, and weighted splitting, and the namespace boundary (ReferenceGrant), with real requests. At the end, you find in the Service list that what actually receives the requests is the node's cilium-envoy.

Why it matters

Ingress standardized only things like paths and hosts and left the rest to implementation-specific annotations. The Gateway API separates roles — the infrastructure owner owns GatewayClass and Gateway, and the app team owns HTTPRoute — features such as header matching and weighted splitting are in the spec, and to reference another namespace, the receiving side must permit it with a ReferenceGrant.

Cilium implements this API without sidecars. L7 processing is handled by the envoy that runs once per node, and eBPF hands traffic that arrives at the gateway address to that envoy. If you do not know this structure, you waste time searching for "where is the gateway Pod?"

The order in which you turn things on also matters. The official documentation tells you to install the CRDs first. In this lab you see for yourself what stays empty when you reverse the order, and why a restart alone does not fill it in.

Environment preparation takes about 5 minutes. The Gateway API CRDs are downloaded from github raw. When the session ends, the files in /root/cca-gateway disappear.

Steps

  1. Turn on the Gateway API feature with the cilium CLI (cilium upgrade --version 1.20.1 --reuse-values --set gatewayAPI.enabled=true). Then restart the cilium-operator Deployment and the cilium DaemonSet in the order of the official documentation. Do not install the Gateway API CRDs yet. Record that moment in /root/cca-gateway/no-crd.json — enable_gateway_api (the enable-gateway-api string in cilium-config), gatewayclass_api (true if the API has a gatewayclasses resource, false if not), and operator_log (the exact log line the new Operator left saying the CRDs are missing).
  2. Install the 7 standard CRDs of Gateway API v1.6.1 (gatewayclasses gateways httproutes referencegrants grpcroutes backendtlspolicies tlsroutes) with kubectl apply --server-side -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.6.1/config/crd/standard/gateway.networking.k8s.io_<자원>.yaml (the placeholder is the resource name). After that, restart cilium-operator again, wait at least 20 seconds after the new Operator Pod is up, and then count the GatewayClasses. In /root/cca-gateway/crd-late.json, record crds (the list of all 7 installed CRD names), operator_started (the status.startTime of the restarted Operator Pod), gatewayclasses (the count at that time), and recorded_at (the UTC time of the moment you counted, from date -u +%Y-%m-%dT%H:%M:%SZ). Do not re-apply the Cilium configuration yet.
  3. With the CRDs present, run the cilium upgrade command of step 1 one more time so that the GatewayClass cilium is created and becomes Accepted=True. In /root/cca-gateway/gatewayclass.json, record name, controller (spec.controllerName), accepted (the status of the Accepted condition), and managed_by (the value of the label app.kubernetes.io/managed-by).
  4. Start the backends (store-v1, store-v2, and admin in cca-gw, and payments in cca-shop) with kubectl apply -f /opt/fixtures/cca-gateway/backends.json. In the cca-gw namespace, create the Gateway shop-gw — gatewayClassName cilium, one listener (name http, protocol HTTP, port 80). Once it is Programmed=True, record in /root/cca-gateway/gateway.json address (the value in status.addresses), programmed, service (the name of the automatically created Service), service_type, proxy_backend (the backend ip:port of that address:80 LoadBalancer frontend in the agent's cilium-dbg service list), root_code (the HTTP code of curl http://<주소>/ from the node, as a number; the placeholder is the address), and server (the value of the server header of that response).
  5. In cca-gw, create the HTTPRoute store. parentRefs is just shop-gw, and there are two rules — (1) requests with path PathPrefix /store and the header x-canary: yes go to store-v2:8080, and (2) the rest with path PathPrefix /store go to store-v1:8080. From the node, request /store/list, /store/list with the header attached, and /storefront at the gateway address, and record in /root/cca-gateway/routes.json store (the app in the response JSON of the first request), store_canary (the app of the second), and storefront (the HTTP code of the third, as a number).
  6. In cca-gw, create the HTTPRoute split. parentRefs shop-gw, one rule — path PathPrefix /checkout, backendRefs store-v1:8080 with weight 80 and store-v2:8080 with weight 20. After the route takes effect, send 100 requests to /checkout at the gateway address, count the app in the responses, and record in /root/cca-gateway/split.json path, requests (the number sent), and counts ({store-v1: n, store-v2: m}). Both versions must be observed.
  7. In cca-gw, create the HTTPRoute pay — parentRefs shop-gw, path PathPrefix /pay, and backendRefs payments:8080 in namespace cca-shop. First apply it without permission and observe the route's ResolvedRefs condition (status and reason) and the HTTP code of /pay. Then create a ReferenceGrant in cca-shop that permits the HTTPRoute in cca-gw to reference only the Service payments, and observe again. In /root/cca-gateway/refgrant.json, record resolved_refs, reason, and code (a number) for each of before and after.
  8. In /root/cca-gateway/report.txt, write seven lines in 키=값 form (key=value) — gatewayclass_owner (the managed-by label of the GatewayClass cilium), l7_proxy_daemonset (the name of the DaemonSet in kube-system that actually handles gateway requests), gateway_frontend (address:80), gateway_proxy_backend (the backend you read just now in the same way as step 4), envoy_config (the name of the CiliumEnvoyConfig automatically created in cca-gw), split_v2_count (the store-v2 count in split.json), and cross_namespace_without_grant (the HTTP code when there is no permission). The values must match the real state and the records.

Notes

The day the gateway feature was turned on without the CRDs

Turn on the Gateway API feature with the cilium CLI (cilium upgrade --version 1.20.1 --reuse-values --set gatewayAPI.enabled=true). Then restart the cilium-operator Deployment and the cilium DaemonSet in the order of the official documentation. Do not install the Gateway API CRDs yet. Record that moment in /root/cca-gateway/no-crd.json — enable_gateway_api (the enable-gateway-api string in cilium-config), gatewayclass_api (true if the API has a gatewayclasses resource, false if not), and operator_log (the exact log line the new Operator left saying the CRDs are missing).

The feature switch only changes the ConfigMap, and the Operator checks at startup whether the Gateway API resources exist. Look for the level=error line in the restarted Operator's log. Check whether the resource exists with kubectl api-resources --api-group=gateway.networking.k8s.io. If you build the file with the --arg/--argjson options of jq -n, even log lines with mixed quotes are held safely.

The CRDs were installed late and the restart was done, yet it is still empty

Install the 7 standard CRDs of Gateway API v1.6.1 (gatewayclasses gateways httproutes referencegrants grpcroutes backendtlspolicies tlsroutes) with kubectl apply --server-side -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.6.1/config/crd/standard/gateway.networking.k8s.io_<자원>.yaml (the placeholder is the resource name). After that, restart cilium-operator again, wait at least 20 seconds after the new Operator Pod is up, and then count the GatewayClasses. In /root/cca-gateway/crd-late.json, record crds (the list of all 7 installed CRD names), operator_started (the status.startTime of the restarted Operator Pod), gatewayclasses (the count at that time), and recorded_at (the UTC time of the moment you counted, from date -u +%Y-%m-%dT%H:%M:%SZ). Do not re-apply the Cilium configuration yet.

The Operator recognizing the CRDs and a GatewayClass object being created may not be the same thing. If the count is 0, you will wonder who originally creates that GatewayClass — you check that with labels in the next step. If you see two Pods, the one with the latest creationTimestamp is the new Pod. The grader compares recorded_at, not the file modification time, with the restart time and the creation time of the GatewayClass that will appear later, so write the time of the moment you recorded.

The chart creates the GatewayClass

With the CRDs present, run the cilium upgrade command of step 1 one more time so that the GatewayClass cilium is created and becomes Accepted=True. In /root/cca-gateway/gatewayclass.json, record name, controller (spec.controllerName), accepted (the status of the Accepted condition), and managed_by (the value of the label app.kubernetes.io/managed-by).

The labels and annotations of the GatewayClass say who owns this object. A Helm chart can include or omit templates depending on which APIs the cluster has when it renders. Re-read at intervals of a few seconds until the condition becomes True.

What stands behind the gateway address is envoy

Start the backends (store-v1, store-v2, and admin in cca-gw, and payments in cca-shop) with kubectl apply -f /opt/fixtures/cca-gateway/backends.json. In the cca-gw namespace, create the Gateway shop-gw — gatewayClassName cilium, one listener (name http, protocol HTTP, port 80). Once it is Programmed=True, record in /root/cca-gateway/gateway.json address (the value in status.addresses), programmed, service (the name of the automatically created Service), service_type, proxy_backend (the backend ip:port of that address:80 LoadBalancer frontend in the agent's cilium-dbg service list), root_code (the HTTP code of curl http://<주소>/ from the node, as a number; the placeholder is the address), and server (the value of the server header of that response).

When you create a Gateway, the controller creates a LoadBalancer Service, and in this k3s, servicelb provides the node IP. Find that Service's frontend in the agent's Service list (-o json) and see whether the backend is a Pod IP or a node-local address. Also check who answers with what code when there are no path rules at all. Right after the address is first attached, the connection may fail briefly, so poll.

Even with the same path, a different header goes to a different version

In cca-gw, create the HTTPRoute store. parentRefs is just shop-gw, and there are two rules — (1) requests with path PathPrefix /store and the header x-canary: yes go to store-v2:8080, and (2) the rest with path PathPrefix /store go to store-v1:8080. From the node, request /store/list, /store/list with the header attached, and /storefront at the gateway address, and record in /root/cca-gateway/routes.json store (the app in the response JSON of the first request), store_canary (the app of the second), and storefront (the HTTP code of the third, as a number).

If you put path and headers together inside a single matches entry of one rule, both must be satisfied. The Gateway API specifies that the more specific condition (the one with the header) takes precedence. PathPrefix compares by path element split by /, not by character. The backends return their own names as JSON.

Count the promise that only 20% flows to the new version

In cca-gw, create the HTTPRoute split. parentRefs shop-gw, one rule — path PathPrefix /checkout, backendRefs store-v1:8080 with weight 80 and store-v2:8080 with weight 20. After the route takes effect, send 100 requests to /checkout at the gateway address, count the app in the responses, and record in /root/cca-gateway/split.json path, requests (the number sent), and counts ({store-v1: n, store-v2: m}). Both versions must be observed.

The weights divide each individual request according to the ratio; it is not a rule that sends exactly every fifth one. So the sample counts fluctuate around 80/20. Right after you create the route, 404s may get mixed in while the envoy configuration spreads, so start counting after each of the two versions has been seen once. The grader compares the record against the weights and also draws its own sample again.

The payment Service in the next namespace can be attached only with permission

In cca-gw, create the HTTPRoute pay — parentRefs shop-gw, path PathPrefix /pay, and backendRefs payments:8080 in namespace cca-shop. First apply it without permission and observe the route's ResolvedRefs condition (status and reason) and the HTTP code of /pay. Then create a ReferenceGrant in cca-shop that permits the HTTPRoute in cca-gw to reference only the Service payments, and observe again. In /root/cca-gateway/refgrant.json, record resolved_refs, reason, and code (a number) for each of before and after.

A ReferenceGrant goes in the namespace of the side being referenced. In from, write the group, kind, and namespace of the resource that references; in to, write the group and kind of the resource being referenced (and the name if you want to narrow it). The group of the core API is an empty string. Read the condition by picking it by type from status.parents[0].conditions.

Go-live report: who created what, and who received the requests

In /root/cca-gateway/report.txt, write seven lines in 키=값 form (key=value) — gatewayclass_owner (the managed-by label of the GatewayClass cilium), l7_proxy_daemonset (the name of the DaemonSet in kube-system that actually handles gateway requests), gateway_frontend (address:80), gateway_proxy_backend (the backend you read just now in the same way as step 4), envoy_config (the name of the CiliumEnvoyConfig automatically created in cca-gw), split_v2_count (the store-v2 count in split.json), and cross_namespace_without_grant (the HTTP code when there is no permission). The values must match the real state and the records.

In Cilium without sidecars, L7 is handled by the envoy that runs once per node. Look for the traces the gateway left with kubectl -n kube-system get ds and kubectl get ciliumenvoyconfig -A, and connect them to the 127.0.0.1 backend in the Service list.