A Short Name Is Not Free

In one line

The convenience of short names is paid for when you look up names that leave the cluster. Every name with fewer than 5 dots goes through the search list first.

In Kubernetes, writing just web is enough to find a Service in the same namespace. That is convenient. But the convenience has a price, and the price is charged when you look up a name that goes outside the cluster.

search and ndots

The /etc/resolv.conf of a Pod looks like this.

search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.43.0.10
options ndots:5

search lists the suffixes that are tried, in order, after a short name, and ndots:5 means a name with fewer than 5 dots may not be a complete name yet, so try the search list first.

web has 0 dots, so of course it uses the search list. The problem is example.com. It has only one dot, which is also fewer than 5, so the lookups go out in this order.

example.com.default.svc.cluster.local   → NXDOMAIN
example.com.svc.cluster.local           → NXDOMAIN
example.com.cluster.local               → NXDOMAIN
example.com.                            → 드디어 응답

One request costs four round trips, and three of them are thrown away.

Why it does not show up in metrics

A DNS lookup happens not in application code but in the library underneath it. So it often leaves no span in the application trace, and users only see a "slightly slow request". Only the load on CoreDNS quietly rises, until one day the number of Pods grows and it reaches its limit.

You can fix it, but it is not free

You can lower ndots per Pod with dnsConfig. But that Pod can no longer find cluster Services by short name. So it is right to apply it only to Pods that make many outbound requests. Applying it to a Pod that mostly talks inside the cluster makes things worse.

Why there are four round trips

The /etc/resolv.conf of a Pod looks like this.

search labhub-prod.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5

ndots:5 means a name with fewer than 5 dots is not treated as a complete name, so the search suffixes are appended and tried first. api.example.com has 2 dots, so it is caught by this rule.

1. api.example.com.labhub-prod.svc.cluster.local  → NXDOMAIN
2. api.example.com.svc.cluster.local              → NXDOMAIN
3. api.example.com.cluster.local                  → NXDOMAIN
4. api.example.com                                → 응답    ← 네 번째에 성공

If IPv6 is enabled, each query asks for both A and AAAA, so the real number of queries is 8. For a service that calls outside APIs often, this accounts for most of the CoreDNS load.

Three responses and what each one costs

Method	Effect	Cost
Trailing dot on the name (example.com.)	Skips search entirely	You have to change the code
ndots: 1 on the Pod	Applies to every name in that Pod	Short names stop working
NodeLocal DNSCache	Caches on the node, sharply reduces CoreDNS load	Installation and operations burden

The first row is the cheapest and safest. You can check why ndots is not lowered globally with kubectl — all the code that calls Services by short name breaks.

# 파드 단위로만, 그 파드가 짧은 이름을 안 쓴다는 것을 확인하고
spec:
  dnsConfig:
    options:
      - name: ndots
        value: "1"

Commands for diagnosing DNS problems

# 파드 안에서 — 어느 단계에서 실패하는지 본다
nslookup api.example.com
nslookup api.example.com.        # 끝점을 찍으면 되는가?

# search 가 실제로 어떻게 붙는지
dig +search +short api.example.com

# CoreDNS 가 무엇을 받고 있나
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=50 | grep NXDOMAIN

If the CoreDNS log is empty, the Corefile has no log plugin. Turn it on only while you investigate and turn it off afterwards — leaving it on produces a lot of log volume.

By symptom, it goes like this. Intermittent failures are usually conntrack problems or UDP packet loss, total failure is CoreDNS or a network policy, and only slow is the search round trips described above.

What really matters in practice

Put a trailing dot on outside domains you call often. A trailing dot, as in example.com., makes the name be treated as complete and skips search entirely. One character of code turns four round trips into one, and because you do not touch ndots, short names keep working.

Look for the signal in CoreDNS's query count. A wasted query still gets a normal response (NXDOMAIN), so the error rate does not rise. If the number of queries is several times the number of Service calls, that means you are paying this cost.

Never lower ndots globally. All the code that calls in-cluster Services by short name breaks. If you want to change it, do it per Pod, after confirming that the Pod does not use short names.

Knowing the trade-off is the whole point of this topic. In the next lab you count the queries yourself, and this story becomes concrete in numbers.