Kubectl x509 Unable to Connect

Kubernetes remote access and TLS certs.

Posted by Craig Johnston on Thursday, May 10, 2018

Just set up a brand new cluster? Changed the domain or IP of your admin node? Then you may have encountered the error Unable to connect to the server: x509: certificate is valid for …. The following is a fix for this common issue. However, there are often other reasons to rebuild your cluster cert, and it’s relatively easy.

TL;DR: “I don’t care about the fix I need to remote control my cluster. Security? Whats that?”:

kubectl --insecure-skip-tls-verify --context=some-context get pods

Let’s say you want to fix the issue and not just skip-tls-verify. Ssh to the admin node and run the following (assuming Kubernetes 1.8 or greater):

# remove the certs
rm /etc/kubernetes/pki/apiserver.*

# re-create with updated --apiserver-cert-extra-sans
kubeadm alpha phase certs all --apiserver-advertise-address=0.0.0.0 --apiserver-cert-extra-sans=new.example.com

# remove the kubernetes api server container
docker rm -f `docker ps -q -f 'name=k8s_kube-apiserver*'`

# restart the kublet
systemctl restart kubelet

Port Forwarding / Local Development

Check out kubefwd for a simple command line utility that bulk forwards services of one or more namespaces to your local workstation.

Resources

Note: This blog is a collection of personal notes. Making them public encourages me to think beyond the limited scope of the current problem I'm trying to solve or concept I'm implementing, and hopefully provides something useful to my team and others.

This blog post, titled: "Kubectl x509 Unable to Connect: Kubernetes remote access and TLS certs." by Craig Johnston, is licensed under a Creative Commons Attribution 4.0 International License. Creative Commons License