CKA — Kubernetes Administrator
Kustomize Overlays and the Resource Metrics Pipeline
Summary
Kustomize is a tool that overlays per-environment differences without changing the original YAML, and kubectl top is a command that reads the values metrics-server collects from each node's kubelet and serves through the Metrics API. For both, you need to know "where is this result produced" to avoid getting stuck, in the exam room and in the field.
Why this matters
When you deploy the same application to development and production, copies of the YAML pile up with only the name and the replica count changed. Once there are more than three copies, nobody knows which is the original, and a change made to only one side quietly misses the other. Helm solves this problem with a template language, but there is a cost to learning templates. Kustomize chose an approach with no templates, leaving the original as is and layering patches on top of it, and according to the document Declarative Management of Kubernetes Objects Using Kustomize, since kubectl 1.14 you can use it with kubectl apply -k without any separate installation.
The problem on the resource usage side is different. If you run kubectl top nodes on a newly created cluster, the answer that comes back is "Metrics API not available." The kubelet knows only the usage of its own node, and the component that gathers that across the whole cluster and serves it through an API is not in the default installation. The Resource metrics pipeline document states firmly that to access the Metrics API you must deploy metrics-server or an adapter that stands in for it. So when kubectl top does not work, it is not that the command is broken but that a piece of the pipeline is missing.
How it works
Kustomize — base and overlay
A base is a directory that contains a kustomization.yaml and resource files. An overlay is a directory that references other kustomization directories through resources and layers its own changes on top of the referenced resources. The property the document stresses is one thing — the base does not know the overlay exists. So several overlays can share one base.
# base/kustomization.yaml
resources:
- deployment.yaml
- service.yaml
# overlays/dev/kustomization.yaml
resources:
- ../../base
namePrefix: dev-
patches:
- path: replicas.yaml
Patches are written in the patches field. According to the document, Kustomize supports two patch methods, StrategicMerge and Json6902; a patch can be a file or an inline string, and patches are applied in the order they are listed. The patch target is selected by group, version, kind, name, namespace, labelSelector, and annotationSelector. The document recommends "small patches that do only one thing" — that is, keep a patch that raises the replica count and a patch that sets a memory limit separate. A StrategicMerge patch overlays a YAML fragment of the same shape as the original, so it is easy to read, and a Json6902 patch specifies a path and does add, replace, or remove, so it is used where overlaying is hard to express, such as a specific item inside a list.
There are other commonly used fields besides patches. images changes only the image name, tag, and digest without a patch, namePrefix and nameSuffix attach a string before and after the names of all resources, and configMapGenerator and secretGenerator create a ConfigMap or Secret from files or literals. An object that a generator creates gets a content hash appended to its name. When the content changes, the name changes, and the spec of the Deployment that references it changes too, so a rollout happens by itself. To turn this behavior off, use the generatorOptions field and set disableNameSuffixHash in it.
To see the result before applying, print the rendered YAML with kubectl kustomize <디렉터리> (where the placeholder is the directory), and when you are satisfied, apply it with kubectl apply -k <디렉터리>. kubectl diff -k and kubectl delete -k work the same way. -k must point to a directory that contains a kustomization.yaml, not to a file.
Resource usage — the path metrics take
The flow the Resource metrics pipeline document draws is this.
| Stage | Component | What it does |
|---|---|---|
| 1 | cAdvisor | A daemon included in the kubelet. Collects, aggregates, and exposes container metrics |
| 2 | kubelet | Provides a node-level summary through the /metrics/resource and /stats endpoints |
| 3 | metrics-server | A cluster add-on that pulls metrics from each kubelet and aggregates them |
| 4 | Metrics API | The metrics.k8s.io group. The API server serves it as an extension API |
| 5 | Consumers | HPA and VPA, and kubectl top |
metrics-server is the reference implementation of the Metrics API. To attach to the API server, the aggregation layer must be enabled and an APIService for metrics.k8s.io must be registered. The metrics-server repository lists further requirements — Webhook authentication and authorization must be enabled on the nodes' kubelets, the kubelet certificates must be signed by the cluster CA (otherwise turn off verification with --kubelet-insecure-tls), the control plane must be able to reach the metrics-server Pod, and metrics-server must be able to reach the kubelet port of every node. The collection interval is 15 seconds, and metrics-server v0.6.0 and later read the kubelet's /metrics/resource.
You should also know what the values mean. CPU is the average core usage computed from the rate of change of a cumulative counter that the kernel provides, and the interval used for the calculation appears in the window field of the Metrics API response. Memory is the working set at the time of collection, which is in-use memory that cannot be released even under memory pressure. So the memory value of kubectl top pod is different from RSS and also different from a value that includes all the cache.
The repository documentation has one caution. metrics-server is for autoscaling only, and you should not use it as a data source for a monitoring system. If you need accurate usage records, it advises having a monitoring tool such as Prometheus scrape the kubelet's /metrics/resource directly.
What it looks like in the field
I applied the overlay but the object doesn't show up. If you put namePrefix: dev- in the overlay, the Deployment name that is created is not my-nginx but dev-my-nginx. If you look for it with kubectl get deploy my-nginx, it says it does not exist. The document's example output is also deployment.apps/dev-my-nginx created. The habit of looking at the rendered result once with kubectl kustomize before applying removes this confusion.
I deployed metrics-server but kubectl top keeps failing. You see this often on clusters built with kubeadm. According to the Certificate management with kubeadm document, the kubelet serving certificate that kubeadm deploys is self-signed by default, so when an external service such as metrics-server connects to the kubelet over TLS, verification fails. On a practice cluster you get past it with --kubelet-insecure-tls, and on a production cluster you follow the "enabling signed kubelet serving certificates" procedure in the same document so that the kubelets get certificates signed by the cluster CA.
What to look at in the next reading
The next reading covers name resolution. When a Pod calls my-svc, which server answers and in what order, what structure CoreDNS's Corefile has, and where to start looking when resolution is blocked — you learn these by following the official debugging procedure.