Also at Deasil Works · txn2 · Plexara
Profiles GitHub · X · LinkedIn
Theme Light · Auto · Dark
imti.co
Professional notes by Craig Johnston
long-form, short-form, working drafts · since 2008
VOL. XIX · MMXXVI
116 NOTES IN PRINT
FOLIO XVII 2018-05-17 · 2 MIN · SHORT-FORM

Kubernetes Custom Upstream DNS

Customize upstream DNS resolution.

Diagram · folio xvii
flowchart TB
  POD([Pod resolves api.example.com])
  POD --> KD[kube-dns / CoreDNS]
  KD --> CHECK{cluster-local name?}
  CHECK -->|yes| SVC[(K8s Service ClusterIP)]
  CHECK -->|no, external| UP[upstream nameservers]
  UP --> G1[8.8.8.8 Google DNS]
  UP --> G2[8.8.4.4 Google DNS]
  G1 --> ANS[/A record or CNAME/]
  G2 --> ANS
  ANS -.->|return| POD

Customize the Upstream Nameservers that kube-dns uses when Pods look up external hostnames from within a Kubernetes cluster. I found that adding custom upstream nameservers to my kube-dns solved many issues I had encountered in the past with external hostname resolution on individual Pods.

§2026 Update

The mechanism here still exists, but the component changed. CoreDNS replaced kube-dns as the default cluster DNS back in Kubernetes 1.13, and kube-dns is gone from current clusters. The kube-dns ConfigMap with upstreamNameservers that this post applies no longer applies to anything.

The CoreDNS equivalent is to edit the coredns ConfigMap in the kube-system namespace and point the forward plugin at the upstreams you want:

kubectl -n kube-system edit configmap coredns

By default the Corefile has a line like forward . /etc/resolv.conf, which forwards external lookups to whatever the node uses. Change it to your nameservers and CoreDNS reloads on its own within a couple of minutes:

forward . 8.8.8.8 8.8.4.4

The official Customizing DNS Service doc covers stub domains and per-domain forwarding if you need more than a flat upstream. The rest of this post is the kube-dns version, kept as a snapshot.

Archived: the original 2018 post follows. The configuration below targets kube-dns, which CoreDNS replaced. It is kept for the archives and no longer applies to current clusters. Use the 2026 Update above for the working setup.

If you want to experiment on a production-like cluster, I suggest reading my article “Production Hobby Cluster” for a guide on setting up a fun, cheap-yet-robust experimental cluster.

The following configuration sets the upstream nameservers to use Google’s DNS servers 8.8.8.8 and 8.8.4.4.

You can apply the above configuration with the following command:

kubectl apply -f https://gist.githubusercontent.com/cjimti/3a500e8efffa1fcaedda8b844c7d6aa7/raw/ae7329733452dda8cce573fb78f33c22c65cb3fa/00-kube-dns-upstream.yml

If you are having trouble with DNS on your Kubernetes cluster, you may want to read the official documentation on Debugging DNS Resolution.

§Port Forwarding / Local Development

Check out kubefwd for a simple command line utility that bulk forwards services of one or more namespaces to your local workstation.

If in a few days you find yourself setting up a cluster in Japan or Germany on Linode, and another two in Australia and France on vultr, then you may have just joined the PHC (Performance Hobby Clusters) club. Some people tinker late at night on their truck, we benchmark and test the resilience of node failures on our overseas, budget kubernetes clusters. It’s all about going big, on the cheap.

· 2018-05-17 ← back to all notes