Also at Deasil Works · txn2 · Plexara
Profiles GitHub · X · LinkedIn
Theme Light · Auto · Dark
imti.co
Professional notes by Craig Johnston
long-form, short-form, working drafts · since 2008
VOL. XIX · MMXXVI
82 NOTES IN PRINT
FOLIO XII 10 MAY 2018 · 1 MIN · SHORT-FORM

Kubectl x509 Unable to Connect

Kubernetes remote access and TLS certs.

Diagram · folio xii
flowchart TB
  ERR[/"x509: certificate signed by unknown authority"/]
  ERR --> Q1{kubeconfig has the cluster CA?}
  Q1 -->|no| F1[copy CA bundle from cluster]
  Q1 -->|yes| Q2{server URL matches CA SAN?}
  Q2 -->|no| F2[regenerate cert with correct SAN]
  Q2 -->|yes| Q3{cert expired?}
  Q3 -->|yes| F3[rotate cluster certs]
  Q3 -->|no| F4[check client clock / NTP]
  F1 --> OK[/connection works/]
  F2 --> OK
  F3 --> OK
  F4 --> OK

Just set up a brand new cluster? Changed the domain or IP of your admin node? Then you may have encountered the error Unable to connect to the server: x509: certificate is valid for …. The following is a fix for this common issue. However, there are often other reasons to rebuild your cluster cert, and it’s relatively easy.

TL;DR: “I don’t care about the fix I need to remote control my cluster. Security? Whats that?”:

kubectl --insecure-skip-tls-verify --context=some-context get pods

Let’s say you want to fix the issue and not just skip-tls-verify. Ssh to the admin node and run the following (assuming Kubernetes 1.8 or greater):

# remove the certs
rm /etc/kubernetes/pki/apiserver.*

# re-create with updated --apiserver-cert-extra-sans
kubeadm alpha phase certs all --apiserver-advertise-address=0.0.0.0 --apiserver-cert-extra-sans=new.example.com

# remove the kubernetes api server container
docker rm -f `docker ps -q -f 'name=k8s_kube-apiserver*'`

# restart the kublet
systemctl restart kubelet

§Port Forwarding / Local Development

Check out kubefwd for a simple command line utility that bulk forwards services of one or more namespaces to your local workstation.

§Resources

· 10 May 2018 ← back to all notes