Also at Deasil Works · txn2 · Plexara
Profiles GitHub · X · LinkedIn
Theme Light · Auto · Dark
imti.co
Professional notes by Craig Johnston
long-form, short-form, working drafts · since 2008
VOL. XIX · MMXXVI
109 NOTES IN PRINT
FOLIO XII 2018-05-10 · 2 MIN · SHORT-FORM

Kubectl x509 Unable to Connect

Kubernetes remote access and TLS certs.

Diagram · folio xii
flowchart TB
  ERR[/"x509: certificate signed by unknown authority"/]
  ERR --> Q1{kubeconfig has the cluster CA?}
  Q1 -->|no| F1[copy CA bundle from cluster]
  Q1 -->|yes| Q2{server URL matches CA SAN?}
  Q2 -->|no| F2[regenerate cert with correct SAN]
  Q2 -->|yes| Q3{cert expired?}
  Q3 -->|yes| F3[rotate cluster certs]
  Q3 -->|no| F4[check client clock / NTP]
  F1 --> OK[/connection works/]
  F2 --> OK
  F3 --> OK
  F4 --> OK

Just set up a brand new cluster? Changed the domain or IP of your admin node? Then you may have encountered the error Unable to connect to the server: x509: certificate is valid for …. The following is a fix for this common issue. However, there are often other reasons to rebuild your cluster cert, and it’s relatively easy.

TL;DR: “I don’t care about the fix, I need to remote control my cluster. Security? What’s that?”:

kubectl --insecure-skip-tls-verify --context=some-context get pods

§2026 Update

The error and the quick escape hatch are unchanged: kubectl --insecure-skip-tls-verify still works exactly as shown, for when you just need in and will deal with TLS later. The real fix, though, used a couple of commands that have since been removed, so the steps below are updated to current kubeadm.

Two things changed. First, kubeadm alpha phase certs ... is gone. Certificate operations graduated out of alpha. Use kubeadm init phase certs apiserver to regenerate the API server certificate with new SANs, and kubeadm certs renew to rotate certs in place. To see what is actually expiring, reach for kubeadm certs check-expiration, which also answers the “cert expired?” branch in the diagram above.

Second, docker rm no longer kills the API server on most clusters. Kubernetes dropped Docker as a runtime in v1.24, so on a current containerd cluster you use crictl instead, or just let the kubelet recreate the static pod after you touch its manifest.

The article continues below. The 2026 Update above summarizes what changed; the walkthrough that follows has been updated to current commands.

Let’s say you want to fix the issue and not just skip-tls-verify. SSH to the control-plane node and run the following:

# remove the existing API server cert and key
rm /etc/kubernetes/pki/apiserver.*

# regenerate the API server cert with the new SAN
# (kubeadm skips generation if the files still exist, hence the rm above)
kubeadm init phase certs apiserver --apiserver-cert-extra-sans=new.example.com

# restart the API server static pod (containerd, via crictl)
crictl ps --name kube-apiserver -q | xargs -r crictl stop

# restart the kubelet
systemctl restart kubelet

The kubelet recreates the API server pod from its static manifest once the container stops, picking up the new certificate.

§Port Forwarding / Local Development

Check out kubefwd for a simple command line utility that bulk forwards services of one or more namespaces to your local workstation.

§Resources

· 2018-05-10 ← back to all notes