Configure NodeLocal DNS Cache in a Managed Kubernetes cluster

To reduce the load on DNS in a Managed Kubernetes cluster and speed up response times from external and internal resources, you can use NodeLocal DNS Cache.

NodeLocal DNS Cache is a DNS query caching mechanism that runs on all Managed Kubernetes nodes and provides a local cache for the DNS resolver. When user pods send requests, they first pass through the local DNS cache. If these requests match previously saved records, additional requests to an external DNS server are avoided. This allows for reduced DNS request processing time and saves network traffic in the cluster.

Read more about NodeLocal DNS Cache in the Kubernetes documentation.

The setup and verification process for NodeLocal DNS Cache depends on the CNI (Container Network Interface) plugin used in the cluster: Calico or Cilium. You can check the CNI plugin in the cluster in the control panel: in the top menu, click Products → Managed Kubernetes → cluster page → Settings tab → CNI.

1. Configure NodeLocal DNS Cache.
2. Verify NodeLocal DNS Cache operation.

1. Configure NodeLocal DNS Cache

Calico CNI

1. Create a nodelocaldns.yaml YAML file with the manifest:

   Файл nodelocaldns.yaml

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: node-local-dns
     namespace: kube-system
     labels:
       kubernetes.io/cluster-service: "true"
       addonmanager.kubernetes.io/mode: Reconcile
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: kube-dns-upstream
     namespace: kube-system
     labels:
       k8s-app: kube-dns
       kubernetes.io/cluster-service: "true"
       addonmanager.kubernetes.io/mode: Reconcile
       kubernetes.io/name: "KubeDNSUpstream"
   spec:
     ports:
     - name: dns
       port: 53
       protocol: UDP
       targetPort: 53
     - name: dns-tcp
       port: 53
       protocol: TCP
       targetPort: 53
     selector:
       k8s-app: kube-dns
   ---
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: node-local-dns
     namespace: kube-system
     labels:
       addonmanager.kubernetes.io/mode: Reconcile
   data:
     Corefile: |
       cluster.local:53 {
           errors
           cache {
                   success 9984 30
                   denial 9984 5
           }
           reload
           loop
           bind 169.254.20.25 10.96.0.10
           forward . __PILLAR__CLUSTER__DNS__ {
                   force_tcp
           }
           prometheus :9253
           health 169.254.20.25:8080
           }
       in-addr.arpa:53 {
           errors
           cache 30
           reload
           loop
           bind 169.254.20.25 10.96.0.10
           forward . __PILLAR__CLUSTER__DNS__ {
                   force_tcp
           }
           prometheus :9253
           }
       ip6.arpa:53 {
           errors
           cache 30
           reload
           loop
           bind 169.254.20.25 10.96.0.10
           forward . __PILLAR__CLUSTER__DNS__ {
                   force_tcp
           }
           prometheus :9253
           }
       .:53 {
           errors
           cache 30
           reload
           loop
           bind 169.254.20.25 10.96.0.10
           forward . __PILLAR__UPSTREAM__SERVERS__
           prometheus :9253
           }
   ---
   apiVersion: apps/v1
   kind: DaemonSet
   metadata:
     name: node-local-dns
     namespace: kube-system
     labels:
       k8s-app: kube-dns
       kubernetes.io/cluster-service: "true"
       addonmanager.kubernetes.io/mode: Reconcile
   spec:
     updateStrategy:
       rollingUpdate:
         maxUnavailable: 10%
     selector:
       matchLabels:
         k8s-app: kube-dns
     template:
       metadata:
         labels:
           k8s-app: kube-dns
         annotations:
           prometheus.io/port: "9253"
           prometheus.io/scrape: "true"
       spec:
         priorityClassName: system-node-critical
         serviceAccountName: node-local-dns
         hostNetwork: true
         dnsPolicy: Default  # Do not use cluster DNS.
         tolerations:
         - key: "CriticalAddonsOnly"
           operator: "Exists"
         - effect: "NoExecute"
           operator: "Exists"
         - effect: "NoSchedule"
           operator: "Exists"
         containers:
         - name: node-cache
           image: registry.k8s.io/dns/k8s-dns-node-cache:1.22.20
           resources:
             requests:
               cpu: 25m
               memory: 5Mi
           args: [ "-localip", "169.254.20.25,10.96.0.10", "-conf", "/etc/Corefile", "-upstreamsvc", "kube-dns-upstream" ]
           securityContext:
             capabilities:
               add:
               - NET_ADMIN
           ports:
           - containerPort: 53
             name: dns
             protocol: UDP
           - containerPort: 53
             name: dns-tcp
             protocol: TCP
           - containerPort: 9253
             name: metrics
             protocol: TCP
           livenessProbe:
             httpGet:
               host: 169.254.20.25
               path: /health
               port: 8080
             initialDelaySeconds: 60
             timeoutSeconds: 5
           volumeMounts:
           - mountPath: /run/xtables.lock
             name: xtables-lock
             readOnly: false
           - name: config-volume
             mountPath: /etc/coredns
           - name: kube-dns-config
             mountPath: /etc/kube-dns
         volumes:
         - name: xtables-lock
           hostPath:
             path: /run/xtables.lock
             type: FileOrCreate
         - name: kube-dns-config
           configMap:
             name: kube-dns
             optional: true
         - name: config-volume
           configMap:
             name: node-local-dns
             items:
               - key: Corefile
                 path: Corefile.base

2. Apply the manifest:

   kubectl apply -f nodelocaldns.yaml

   The response will show a list of the created objects:

   serviceaccount/node-local-dns created
   service/kube-dns-upstream created
   configmap/node-local-dns created
   daemonset.apps/node-local-dns created

Cilium CNI

1. Create a nodelocaldns.yaml YAML file with the manifest:

   Файл nodelocaldns.yaml

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: node-local-dns
     namespace: kube-system
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: kube-dns-upstream
     namespace: kube-system
   labels:
     k8s-app: kube-dns
     kubernetes.io/name: "KubeDNSUpstream"
   spec:
     ports:
     - name: dns
       port: 53
       protocol: UDP
       targetPort: 53
     - name: dns-tcp
       port: 53
       protocol: TCP
       targetPort: 53
     selector:
       k8s-app: kube-dns
     ---
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: node-local-dns
     namespace: kube-system
   data:
     Corefile: |
         cluster.local:53 {
             errors
             cache {
                 success 9984 30
                 denial 9984 5
         }
         reload
         loop
         bind 0.0.0.0
         forward . __PILLAR__CLUSTER__DNS__ {
                 prefer_udp
         }
         prometheus :9253
         health
         }
         in-addr.arpa:53 {
             errors
             cache 30
             reload
             loop
             bind 0.0.0.0
             forward . __PILLAR__CLUSTER__DNS__ {
                     prefer_udp
             }
             prometheus :9253
         }
         ip6.arpa:53 {
             errors
             cache 30
             reload
             loop
             bind 0.0.0.0
             forward . __PILLAR__CLUSTER__DNS__ {
                     prefer_udp
             }
             prometheus :9253
         }
         .:53 {
             errors
             cache 30
             reload
             loop
             bind 0.0.0.0
             forward . __PILLAR__UPSTREAM__SERVERS__ {
                     prefer_udp
             }
             prometheus :9253
         }
   ---
   apiVersion: apps/v1
   kind: DaemonSet
   metadata:
     name: node-local-dns
     namespace: kube-system
     labels:
       k8s-app: node-local-dns
   spec:
   updateStrategy:
   rollingUpdate:
     maxUnavailable: 10%
   selector:
     matchLabels:
       k8s-app: node-local-dns
   template:
     metadata:
       labels:
         k8s-app: node-local-dns
       annotations:
         prometheus.io/port: "9253"
         prometheus.io/scrape: "true"
     spec:
       priorityClassName: system-node-critical
       serviceAccountName: node-local-dns
       dnsPolicy: Default # Do not use cluster DNS.
       tolerations:
       - key: "CriticalAddonsOnly"
         operator: "Exists"
       - effect: "NoExecute"
         operator: "Exists"
       - effect: "NoSchedule"
         operator: "Exists"
       containers:
       - name: node-cache
       image: registry.k8s.io/dns/k8s-dns-node-cache:1.22.20
       resources:
         requests:
           cpu: 25m
           memory: 5Mi
         args: [ "-localip", "169.254.20.25,10.96.0.10", "-conf", "/etc/Corefile", "-upstreamsvc", "kube-dns-upstream", "-skipteardown=true", "-setupinterface=false", "-setupiptables=false" ]
         securityContext:
           privileged: true
         ports:
         - containerPort: 53
           name: dns
           protocol: UDP
         - containerPort: 53
           name: dns-tcp
           protocol: TCP
         - containerPort: 9253
           name: metrics
           protocol: TCP
         livenessProbe:
           httpGet:
             path: /health
             port: 8080
           initialDelaySeconds: 60
           timeoutSeconds: 5
         volumeMounts:
         - mountPath: /run/xtables.lock
           name: xtables-lock
           readOnly: false
         - name: config-volume
           mountPath: /etc/coredns
         - name: kube-dns-config
           mountPath: /etc/kube-dns
         volumes:
         - name: xtables-lock
           hostPath:
             path: /run/xtables.lock
             type: FileOrCreate
         - name: kube-dns-config
           configMap:
             name: kube-dns
             optional: true
         - name: config-volume
           configMap:
             name: node-local-dns
             items:
               - key: Corefile
                 path: Corefile.base

2. Apply the manifest:

   kubectl apply -f nodelocaldns.yaml

   The response will show a list of the created objects:

   serviceaccount/node-local-dns created
   service/kube-dns-upstream created
   configmap/node-local-dns created
   daemonset.apps/node-local-dns created

3. Create a nodelocaldnslrp.yaml YAML file with the manifest:

   Файл nodelocaldnslrp.yaml

   ---
     apiVersion: "cilium.io/v2"
     kind: CiliumLocalRedirectPolicy
     metadata:
     name: "nodelocaldns"
     namespace: kube-system
     spec:
     redirectFrontend:
         serviceMatcher:
         serviceName: kube-dns
         namespace: kube-system
     redirectBackend:
         localEndpointSelector:
         matchLabels:
             k8s-app: node-local-dns
         toPorts:
         - port: "53"
             name: dns
             protocol: UDP
         - port: "53"
             name: dns-tcp
             protocol: TCP

4. Apply the manifest:

   kubectl apply -f nodelocaldnslrp.yaml

   The response will show a list of the created objects:

   ciliumlocalredirectpolicy.cilium.io/NodeLocal DNS created 

2. Verify NodeLocal DNS Cache operation

Calico CNI

1. Enable logging for NodeLocal DNS pods. To do this, in the nodelocaldns.yaml manifest, add the ConfigMap object, add the log:

   data:
     Corefile: |
         cluster.local:53 {
           log
           errors
           cache {
                   success 9984 30
                   denial 9984 5
           }

2. Apply the manifest:

   kubectl apply -f nodelocaldns.yaml

3. Run a pod with DNS troubleshooting utilities:

   kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml

   The response will confirm that the pod has been created:

   pod/dnsutils created

4. Make sure the pod status is Running:

   kubectl get pods dnsutils

5. Connect to the pod:

   kubectl exec -i -t dnsutils -- bash

6. Run a test query to the DNS server:

   dig @169.254.20.25 servercore.com

   The response will show that the query was successful:

   ;; ANSWER SECTION:
   servercore.com.           30      IN      A       85.119.149.3
   
   ;; Query time: 437 msec
   ;; SERVER: 169.254.20.25#53(169.254.20.25)
   ;; WHEN: Wed Sep 20 15:43:46 UTC 2023
   ;; MSG SIZE  rcvd: 67

7. Check the logs:

   kubectl logs --namespace=kube-system -l k8s-app=kube-dns -f

   The response will show A and AAAA DNS records with the code NOERROR:

   [INFO] 10.10.71.3:45929 - 55864 "A IN ru-2.cloud.api.selcloud.ru.kube-system.svc.cluster.local. udp 85 false 1232" NOERROR qr,aa,rd,ra 212 3.975488442s
   [INFO] 10.10.71.3:36099 - 21119 "AAAA IN ru-2.cloud.api.selcloud.ru.kube-system.svc.cluster.local. udp 85 false 1232" NOERROR qr,aa,rd 74 3.976010334s

Cilium CNI

1. Run a pod with DNS troubleshooting utilities:

   kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml

   The response will confirm that the pod has been created:

   pod/dnsutils created

2. Make sure the pod status is Running:

   kubectl get pods dnsutils

3. Find the name of the cluster node where the dnsutils: pod is deployed:

   kubectl get pod dnsutils -o wide

   The response will show information about the dnsutils pod. For example:

   NAME     READY  STATUS   RESTARTS  AGE  IP         NODE        NOMINATED NODE  READINESS GATES
   dnsutils  1/1   Running  0         23h  10.1.0.68  node-01     <none>          <none>

   Copy the value from the NODE. column.

4. Get the IP address of the pod where NodeLocal DNS is deployed:

   kubectl get pod -o wide -n kube-system | grep 'node-local.*<node_name>'

   Specify <node_name> — the cluster node name you copied in step 3.

   The response will show information about the pod where NodeLocal DNS is deployed; copy its IP address. For example:

   node-local-dns-gv68c  1/1    Running  0         26m  10.0.0.5   node-01     <none>          <none>

   Here 10.0.0.5 is the pod IP address.

5. View the metric values for coredns_dns_requests_total:

   kubectl exec -i -t dnsutils -- curl http://<ip_address>:9253/metrics | grep coredns_dns_requests_total

   Specify <ip_address> — the pod IP address you copied in step 4.

   The response will show the number of DNS requests executed. For example:

   # HELP coredns_dns_requests_total Counter of DNS requests made per zone, protocol and family.
   # TYPE coredns_dns_requests_total counter
   coredns_dns_requests_total{family="1",proto="udp",server="dns://0.0.0.0:53",type="other",zone="."} 1
   coredns_dns_requests_total{family="1",proto="udp",server="dns://0.0.0.0:53",type="other",zone="cluster.local."} 1
   coredns_dns_requests_total{family="1",proto="udp",server="dns://0.0.0.0:53",type="other",zone="in-addr.arpa."} 1
   coredns_dns_requests_total{family="1",proto="udp",server="dns://0.0.0.0:53",type="other",zone="ip6.arpa."} 1

6. Execute test DNS queries:

   kubectl exec -i -t dnsutils -- nslookup kubernetes && \
   kubectl exec -i -t dnsutils -- nslookup kubernetes.default && \
   kubectl exec -i -t dnsutils -- nslookup servercore.com

   The response will show the result of resolving domain names for both an internal cluster service and an external resource. For example:

   Server:		10.96.0.10
   Address:	10.96.0.10#53
   
   Name:	kubernetes.default.svc.cluster.local
   Address: 10.96.0.1
   
   Server:		10.96.0.10
   Address:	10.96.0.10#53
   
   Name:	kubernetes.default.svc.cluster.local
   Address: 10.96.0.1
   
   Server:		10.96.0.10
   Address:	10.96.0.10#53
   Name:	servercore.com
   Address: 85.119.149.3
   Name:	servercore.com
   Address: 2a00:ab00:2000:2::1

7. Make sure the metric values have changed:

   kubectl exec -i -t dnsutils -- curl http://<ip_address>:9253/metrics | grep coredns_dns_requests_total

   Specify <ip_address> — the pod IP address you copied in step 4.

   The metric values in the output should increase. For example:

   # HELP coredns_dns_requests_total Counter of DNS requests made per zone, protocol and family.
   # TYPE coredns_dns_requests_total counter
   coredns_dns_requests_total{family="1",proto="udp",server="dns://0.0.0.0:53",type="A",zone="."} 1
   coredns_dns_requests_total{family="1",proto="udp",server="dns://0.0.0.0:53",type="A",zone="cluster.local."} 6
   coredns_dns_requests_total{family="1",proto="udp",server="dns://0.0.0.0:53",type="AAAA",zone="."} 1
   coredns_dns_requests_total{family="1",proto="udp",server="dns://0.0.0.0:53",type="AAAA",zone="cluster.local."} 2
   coredns_dns_requests_total{family="1",proto="udp",server="dns://0.0.0.0:53",type="other",zone="."} 1
   coredns_dns_requests_total{family="1",proto="udp",server="dns://0.0.0.0:53",type="other",zone="cluster.local."} 1
   coredns_dns_requests_total{family="1",proto="udp",server="dns://0.0.0.0:53",type="other",zone="in-addr.arpa."} 1
   coredns_dns_requests_total{family="1",proto="udp",server="dns://0.0.0.0:53",type="other",zone="ip6.arpa."} 1