Actually Dig Into Cluster DNS

This lab runs on real Kubernetes

A real single k3s node is running inside a VM. CoreDNS really runs, it writes queries to its log, and the Pods are real containers. So you can count things like "how many queries go out to find one name".

It takes about 2 minutes to come up the first time, because the VM boots and installs k3s.

Goal

Follow the path a Pod takes to resolve a name from start to finish, count the hidden cost that ndots:5 creates to confirm it, and then fix it with dnsConfig.

Why it matters

A large share of "it's slow" reports in a cluster are DNS. Yet it rarely shows up in application metrics. Even if one request sends five queries to find one name and four of them are NXDOMAIN, from the application's point of view it is just a "slightly slow request". The load on CoreDNS rises quietly, and one day, when the number of Pods grows, it blows up.

The cause is one line in resolv.conf. ndots:5 means a name with fewer than 5 dots is tried first with the search domains appended. example.com has one dot, so the first question asked is example.com.default.svc.cluster.local. The convenience that exists so you can write in-cluster names short becomes a cost, as is, for names that go outside.

Steps

1. Find out where CoreDNS is and what it does, and save the result to /root/k8sdns/coredns.txt. It must contain the ClusterIP of the kube-dns Service and the full Corefile.
2. Save the /etc/resolv.conf inside a Pod, as it is, to /root/k8sdns/resolv.txt. All three lines, nameserver, search and options, must be there.
3. Create a Deployment named web (2 replicas, nginx:1.27-alpine, port name http) and a Service with the same name, look up both the long name and the short name, and save the result to /root/k8sdns/svc-a.txt. You must get the same ClusterIP.
4. Turn on the log plugin in the CoreDNS Corefile, look up example.com with two different tools, count the queries in the CoreDNS log for each, and save the result to /root/k8sdns/ndots.txt. Start the file with the two lines queries_nslookup= and queries_getaddrinfo=, then append the log below them.
   • nslookup from busybox, once
   • curl, once (curlimages/curl:8.10.1) The two numbers differ greatly. Why they differ is the point of this step.
5. Create a headless Service named web-h (clusterIP: None) and save the lookup result to /root/k8sdns/headless.txt. The Pod IPs must appear as they are.
6. Look up the SRV record of web and save it to /root/k8sdns/srv.txt.
7. Start a Pod named lowdots with ndots: 1 set through dnsConfig, count the queries again with the same tool as the curl in step 4, and save the result to /root/k8sdns/fixed.txt (include a queries= line). The count must be 3 or fewer. If you change the tool, the comparison is not valid.
8. In /root/k8sdns/report.md, write the three lines dns_ip=, queries_before= and queries_after=, together with why ndots was a problem and why the problem is not visible when you measure with nslookup.

Notes

• Run the lookup tools inside a Pod. The simplest is kubectl run q --rm -it --image=busybox:1.36 --restart=Never -- nslookup <이름> (the placeholder is the name to look up).
• You can view the Corefile with kubectl -n kube-system get cm coredns -o yaml. To change it, run kubectl -n kube-system edit cm coredns and then kubectl -n kube-system rollout restart deploy coredns.
• View the log with kubectl -n kube-system logs deploy/coredns. If you want to clear the log once before counting, restart CoreDNS.
• The SRV name is _<포트이름>._<프로토콜>.<서비스>.<네임스페이스>.svc.cluster.local (the placeholders are the port name, protocol, Service and namespace). If the port has no name, no SRV record is created.
• Why the two numbers in step 4 differ: nslookup from busybox speaks the DNS protocol directly, so it does not read search or ndots. A normal application calls getaddrinfo, which follows resolv.conf as it is and asks for A and AAAA separately.
• When you count the log, restart CoreDNS just before counting. A restart is how you clear the log, and it keeps the queries from earlier steps from getting mixed in.
• Common mistake 1: putting a trailing dot on the name (example.com.). It is an absolute name, so it skips the search domains and what you wanted to learn disappears.
• Common mistake 2: leaving out clusterIP: None in step 5. Then it is an ordinary Service and only one ClusterIP comes back.

Where CoreDNS is

Find out where CoreDNS is and what it does, and save the result to /root/k8sdns/coredns.txt. It must contain the ClusterIP of the kube-dns Service and the full Corefile.

Find the Service named kube-dns in kube-system, and include the Corefile from the coredns ConfigMap as well. The Service is called kube-dns even though the software is CoreDNS because it inherited the name of the earlier implementation.

What a Pod looks at to resolve names

Save the /etc/resolv.conf inside a Pod, as it is, to /root/k8sdns/resolv.txt. All three lines, nameserver, search and options, must be there.

Start a Pod and read the /etc/resolv.conf inside it as it is. It must be the one inside the Pod, not the VM's own resolv.conf — the two are completely different.

The moment a Service name becomes an address

Create a Deployment named web (2 replicas, nginx:1.27-alpine, port name http) and a Service with the same name, look up both the long name and the short name, and save the result to /root/k8sdns/svc-a.txt. You must get the same ClusterIP.

After you create the Deployment and the Service, look up web and web.default.svc.cluster.local from a Pod in the same namespace. The short name works thanks to the search domains.

How many queries for one name

Turn on the log plugin in the CoreDNS Corefile, look up example.com with two different tools, count the queries in the CoreDNS log for each, and save the result to /root/k8sdns/ndots.txt. Start the file with the two lines queries_nslookup= and queries_getaddrinfo=, then append the log below them.

• nslookup from busybox, once
• curl, once (curlimages/curl:8.10.1) The two numbers differ greatly. Why they differ is the point of this step.

Add one log line to the Corefile and restart CoreDNS. Then look up example.com just once from a Pod and count the log lines. Do not add a trailing dot (example.com.), because that skips the search domains.

What a headless Service returns

Create a headless Service named web-h (clusterIP: None) and save the lookup result to /root/k8sdns/headless.txt. The Pod IPs must appear as they are.

Create a Service with clusterIP: None and look it up. Unlike an ordinary Service, several addresses come back.

Finding a port number by name

Look up the SRV record of web and save it to /root/k8sdns/srv.txt.

The SRV name is _<포트이름>._tcp.<서비스>.<네임스페이스>.svc.cluster.local (the placeholders are the port name, Service and namespace). In step 3 you gave the port the name http.

Fix it by lowering ndots

Start a Pod named lowdots with ndots: 1 set through dnsConfig, count the queries again with the same tool as the curl in step 4, and save the result to /root/k8sdns/fixed.txt (include a queries= line). The count must be 3 or fewer. If you change the tool, the comparison is not valid.

Put {name: ndots, value: "1"} into dnsConfig.options in the Pod spec. Then a name with only one dot no longer goes through the search domains.

What you learned

In /root/k8sdns/report.md, write the three lines dns_ip=, queries_before= and queries_after=, together with why ndots was a problem and why the problem is not visible when you measure with nslookup.

Write the three lines dns_ip=, queries_before= and queries_after=, and in the body explain why ndots was a problem.