The platform so far runs on a set of cloud virtual machines, and that is the right default. But some of the most useful hardware you can put a workload on is not in a cloud: a GPU box in your office, a server in a colo, devices at the edge near where data is produced, machines you already own and are not paying a cloud margin on. The cloud answer to mixing these is a separate managed cluster in each place, stitched together with more managed services. The alternative is one Kubernetes cluster that spans all of them, encrypted over the public internet, and k3s with WireGuard makes it practical.

This series rebuilds my 2020 Apress book, Advanced Platform Development with Kubernetes, for 2026. The approach behind it comes from building and running data platforms in production for more than twenty years.

§One Cluster, Many Places

The reason to want a single hybrid cluster is that it lets one control plane, one set of manifests, and one scheduler place work wherever it belongs, without you managing the seams. The training job goes to the on-prem GPU because that is where the GPU is and the cloud rate for one would be brutal. The sensitive data stays on a node in your own building for residency reasons. The lightweight collector runs on a device at the edge, near the sensors. The stateless web service runs on cheap cloud workers. All of it is the same cluster, so a pod is scheduled to the right place by a label, not deployed to a different system with its own tooling.

A cluster per location multiplies your operational surface by the number of places you run, and then you spend your time keeping them in sync. One cluster across many places keeps the platform single while the hardware underneath it is wherever it makes sense, cloud or yours. This is the lock-in argument applied to geography: you are not tied to one provider’s region because your cluster is not one provider’s cluster.

§How Hard This Used to Be

A cluster like this used to be an expert’s project. In 2020 I built and documented one at full scale: eight nodes, a control plane on DigitalOcean in New York, three workers on Hetzner in Nuremberg, and an on-premises region with a GPU workstation and three Raspberry Pis collecting sensor data. One cluster, three regions, two clouds, and a garage, running real pipelines end to end. It worked, and getting it working meant days of version-sensitive steps that each had to be done by hand, in order, on every node.

Six years later the architecture holds up and almost every painful step has dissolved.

• WireGuard was an installation project. In 2020 every node needed a PPA, and the Raspberry Pis needed packages pinned from Debian unstable plus a DKMS build. WireGuard merged into the Linux kernel at 5.6, so in 2026 every mainstream distribution ships it. The prep step is now a version check.
• The k3s flags changed. The 2020 install used --no-flannel. The current spelling is --flannel-backend=none, plus --disable-network-policy so k3s does not run a policy controller against a CNI it no longer owns.
• The shared secret is generated for you. In 2020 you rolled your own cluster secret with head -c48 /dev/urandom | base64 and kept it somewhere safe. A modern k3s server writes its join token to /var/lib/rancher/k3s/server/node-token, and agents present it as K3S_TOKEN.
• The worst step is gone entirely. Kilo, the WireGuard mesh this build uses, needs a kubeconfig on every node, which in 2020 meant copying a modified k3s.yaml to each machine by hand. The current kilo-k3s.yaml manifest runs an init container on every node that generates the kubeconfig from the kubelet’s own credentials. Nothing to distribute.

That is the trajectory this whole series keeps finding: the architecture that was right in 2020 gets easier to build every year, because the ecosystem keeps sanding off the rough edges. The expertise a hybrid cluster demanded is now mostly encoded in the tools.

§The Plan

This build is the same shape at postcard size: four nodes, three regions, and every cross-region packet encrypted.

Two DNS A records make the cluster reachable: master.hc2.apk8s.dev pointing at the DigitalOcean node’s public IP, and lab.hc2.apk8s.dev pointing at the public IP of the on-prem internet router. If your home IP changes, put the second record on dynamic DNS; nothing else in the build cares.

Small machines are fine. The control plane is comfortable on 2 vCPUs and 4 GB, and k3s is the reason: it is a certified Kubernetes distribution compiled into a single binary, with a footprint that suits everything from a cloud VM to a Raspberry Pi. That range is exactly what a hybrid cluster needs, because the nodes are deliberately not uniform.

§Prepare Every Node

Each node needs three things: WireGuard in the kernel, a route to the control plane, and an open UDP port for the mesh.

# WireGuard ships in-kernel since Linux 5.6; confirm the module loads
modprobe wireguard && echo ok

If that prints ok, the node can encrypt. On any current Ubuntu, Debian, or Raspberry Pi OS it will. The 2020 ritual of PPAs and kernel headers is only needed on kernels older than 5.6, which at this point means something is wrong.

Firewall rules, on every location’s edge: TCP 6443 inbound to the control-plane node (the Kubernetes API, which agents join through), and UDP 51820 inbound wherever a WireGuard endpoint listens. For the cloud nodes that is their own public IPs. For the garage, forward UDP 51820 on the router to the GPU box’s LAN address; that single port forward is the entire on-prem network requirement.

§The Control Plane

Install the k3s server on the New York node. Each flag is a decision:

curl -sfL https://get.k3s.io | sh -s - server \
  --flannel-backend=none \
  --disable-network-policy \
  --tls-san master.hc2.apk8s.dev \
  --node-label topology.kubernetes.io/region=nyc3 \
  --node-taint dedicated=master:NoSchedule

--flannel-backend=none withholds k3s’s default flannel network, because flannel assumes nodes reach each other directly and nodes scattered across clouds and a home LAN cannot, not safely and often not at all. Kilo will own pod networking instead. --disable-network-policy turns off the kube-router policy controller k3s would otherwise run, since the CNI it polices is not there; leaving it on buys nothing and can conflict. --tls-san adds the public DNS name to the API server’s certificate, so kubectl from your workstation and agents joining by that name get a certificate that actually matches. --node-label stamps the region at registration time using topology.kubernetes.io/region, the standard Kubernetes topology label, and it matters here beyond convention because it is the label Kilo reads to learn the cluster’s geography. --node-taint keeps ordinary workloads off the control plane; only pods that explicitly tolerate dedicated=master may land there, which on a four-node cluster preserves the small head node for its actual job.

The server generates the join token; you will need it for every other node:

cat /var/lib/rancher/k3s/server/node-token
# K10a1b2c3...::server:9f8e7d6c...

Then put the cluster on your workstation. The kubeconfig k3s writes refers to 127.0.0.1, so rewrite it for the public name while copying:

ssh [email protected] \
  "sed 's/127.0.0.1/master.hc2.apk8s.dev/' /etc/rancher/k3s/k3s.yaml" \
  > ~/.kube/hc2
export KUBECONFIG=~/.kube/hc2
kubectl get nodes

This works because of the --tls-san above; without it, the API server’s certificate covers only its internal names and kubectl refuses the connection.

§Workers in a Second Cloud

Join the two Hetzner nodes as agents. Same installer, different subcommand, and the region label is the only thing that distinguishes a Nuremberg node from any other:

export K3S_URL="https://master.hc2.apk8s.dev:6443"
export K3S_TOKEN="<the node-token from the server>"

curl -sfL https://get.k3s.io | sh -s - agent \
  --node-label topology.kubernetes.io/region=nbg1

K3S_URL tells the agent where the control plane lives, by the public DNS name, and its presence is what makes the installer run k3s as an agent rather than a server. K3S_TOKEN is the join credential. Agents do not repeat the flannel flags; networking is the server’s decision, and these nodes will receive Kilo like everyone else.

Check the state of things from your workstation:

kubectl get nodes --label-columns topology.kubernetes.io/region

NAME     STATUS     ROLES                  AGE     VERSION        REGION
master   NotReady   control-plane,master   10m     v1.33.3+k3s1   nyc3
nbg-w1   NotReady   <none>                 2m      v1.33.3+k3s1   nbg1
nbg-w2   NotReady   <none>                 90s     v1.33.3+k3s1   nbg1

NotReady is correct. Three nodes on two continents have registered, but there is no pod network, because you deliberately withheld flannel, and a node with no CNI reports itself unfit for pods. Kubernetes is fine with nodes it cannot yet connect; the network is a pluggable decision you have not made yet.

§Install Kilo

Kilo is the piece built for exactly this cluster shape. It reads the topology labels you just applied, elects a leader node in each location, and builds WireGuard tunnels between the leaders, so nodes inside a location talk over their fast local network while traffic between locations rides encrypted tunnels over the public internet. That location-aware default is the difference between Kilo and a generic mesh: two Nuremberg nodes on the same private LAN should not pay for encryption and a tunnel hop to exchange packets, and with Kilo they do not. (A --mesh-granularity=full flag encrypts node-to-node everywhere, for clusters whose “locations” do not have trustworthy internal networks.)

Kilo installs as a DaemonSet from plain manifests, the same declarative shape as everything else in this series:

kubectl apply -f https://raw.githubusercontent.com/squat/kilo/main/manifests/crds.yaml
kubectl apply -f https://raw.githubusercontent.com/squat/kilo/main/manifests/kilo-k3s.yaml

The first manifest installs Kilo’s custom resource definitions, which exist mostly for one good reason: a Peer resource that can join machines that are not cluster nodes, a laptop for instance, to the same mesh. The second is the k3s-specific DaemonSet. Inside it, an init container generates each node’s kubeconfig from the kubelet’s own credentials, the automation that replaced 2020’s copy-a-file-to-every-node step, and the DaemonSet tolerates every NoSchedule and NoExecute taint, which is why the tainted control plane still gets the mesh. On each node the agent creates a kilo0 WireGuard interface, generates the node’s key pair, and writes the CNI config.

Within a minute or two, the mesh is up:

kubectl -n kube-system get pods -l app.kubernetes.io/name=kilo -o wide
kubectl get nodes

NAME     STATUS   ROLES                  AGE   VERSION
master   Ready    control-plane,master   14m   v1.33.3+k3s1
nbg-w1   Ready    <none>                 6m    v1.33.3+k3s1
nbg-w2   Ready    <none>                 5m    v1.33.3+k3s1

Every node Ready, with pod traffic between New York and Nuremberg now ciphertext on the public internet.

§The Garage Node, Behind NAT

The on-prem GPU box is where the build gets interesting, because it has no public IP of its own. It sits behind a home router doing NAT, which is the situation Kubernetes networking normally cannot tolerate and the one Kilo has annotations for.

Join it like any agent, with its own region:

export K3S_URL="https://master.hc2.apk8s.dev:6443"
export K3S_TOKEN="<the node-token from the server>"

curl -sfL https://get.k3s.io | sh -s - agent \
  --node-label topology.kubernetes.io/region=lab

The node registers, Kilo lands on it, and cross-location traffic will not flow yet, because the mesh cannot reach it. Kilo advertises each location’s WireGuard endpoint from what it can observe, and what it observes on this node is a private LAN address like 192.168.1.40, useless to a peer in New York. Two annotations fix it:

kubectl annotate node lab-gpu \
  kilo.squat.ai/force-endpoint="lab.hc2.apk8s.dev:51820"

kubectl annotate node lab-gpu \
  kilo.squat.ai/persistent-keepalive="10"

force-endpoint overrides the advertised endpoint with the routable truth: the DNS name of the home router, on the port the router forwards inward to this box. Every peer now dials the router, and the port forward delivers the packets. persistent-keepalive deals with the second half of the NAT problem: the router’s address mapping for an idle UDP flow evaporates, often in well under a minute on consumer gear, and once it does, nothing outside can initiate contact. A keepalive every ten seconds means the node itself keeps the mapping warm, so inbound traffic always finds an open path. Only NAT’d nodes need it; the cloud nodes, with real public addresses, do not.

Confirm from the garage side that the tunnel is alive, with WireGuard’s own tooling:

wg show kilo0

interface: kilo0
  public key: hK2f...Rk0=
  listening port: 51820

peer: 9xTz...mE4=
  endpoint: 164.90.xxx.xxx:51820
  allowed ips: 10.42.0.0/24, 10.42.1.0/24, 10.42.2.0/24, 10.4.0.1/32
  latest handshake: 8 seconds ago
  transfer: 1.21 MiB received, 890.44 KiB sent
  persistent keepalive: every 10 seconds

latest handshake a few seconds old is the proof. WireGuard rekeys constantly and silently; a recent handshake means both directions work, keys agree, and the NAT path is open. The allowed ips are the pod subnets of the other locations, which is Kilo doing its actual job: routing the cluster’s pod network through the tunnel.

The 2020 version of this cluster also held three Raspberry Pis running sensor collectors in its on-prem region, and the pattern for edge devices is unchanged: join with the same lab region label plus a taint like --node-taint dedicated=pi:NoSchedule, and only the workloads built for them, like the MQTT collectors from earlier in this series, tolerate their way on. Kilo routes their traffic through the location’s leader, so devices with no port forward of their own still participate.

§Prove the Mesh

A recent handshake proves the tunnel; it does not yet prove that Kubernetes networking works across it. Prove that the way the storage post proved persistence: end to end, with the actual thing the cluster claims to do. Run a web server on the garage node, and fetch from it out of a pod in Nuremberg.

# pin a pod to the on-prem region
kubectl run web --image=nginx --restart=Never \
  --overrides='{"spec":{"nodeSelector":{"topology.kubernetes.io/region":"lab"}}}'

kubectl get pod web -o wide
# NAME   READY   STATUS    IP           NODE
# web    1/1     Running   10.42.3.7    lab-gpu

# fetch it from a pod pinned to a Hetzner worker
kubectl run probe --image=busybox --restart=Never -it --rm \
  --overrides='{"spec":{"nodeSelector":{"topology.kubernetes.io/region":"nbg1"}}}' \
  -- wget -qO- 10.42.3.7 | head -4

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>

That response left a rented machine in Nuremberg, crossed the public internet as WireGuard ciphertext, entered a garage through one forwarded UDP port, and came back. No load balancer, no VPN appliance, no per-location cluster. One flat pod network that happens to span the planet.

Kilo also draws you a picture. Its kgctl command-line tool reads the mesh and emits the topology as a graph:

go install github.com/squat/kilo/cmd/kgctl@latest

kgctl graph | circo -Tsvg > hc2.svg

The output draws locations as boxes, leaders with their endpoints, and tunnels as edges. Regenerate it whenever the cluster changes and the network documents itself.

§Place Work Where It Belongs

A hybrid cluster pays off only if workloads actually land on the right hardware, and Kubernetes already owns the vocabulary: labels state what a node is, taints state who may use it. The GPU box should refuse general workloads, since a stray web pod has no business competing with training jobs for the one machine with a GPU:

kubectl label node lab-gpu node-role.platform/gpu="true"
kubectl taint node lab-gpu dedicated=gpu:NoSchedule

Then a training job states its requirements and its permission in the same manifest:

apiVersion: batch/v1
kind: Job
metadata:
  name: train
spec:
  template:
    spec:
      restartPolicy: Never
      nodeSelector:
        node-role.platform/gpu: "true"   # require the GPU node
      tolerations:
        - key: dedicated                 # accept its taint
          value: gpu
          effect: NoSchedule
      containers:
        - name: train
          image: pytorch/pytorch:2.7.0-cuda12.8-cudnn9-runtime
          command: ["python", "-c", "import torch; print(torch.cuda.is_available())"]

The nodeSelector is the requirement: only nodes carrying the GPU role label qualify, and there is exactly one. The toleration is the permission: it matches the taint, so the scheduler will actually place the pod there. The two work as a pair, and the asymmetry is deliberate. A selector without the taint would let every other pod pile onto the GPU box too; the taint without a selector would let this job land on a cloud worker with no GPU. Together they make placement exact, and the same pair generalizes: dedicated=pi for edge collectors, region selectors for residency, everything else defaulting to the cheap cloud workers.

Now the scheduler does what you would otherwise do by hand across separate clusters: GPU work to the GPU, edge collection to the edge, sensitive data to nodes in your own building, everything else to rented capacity. One cluster and one scheduler place the work.

§The Alternatives

Kilo is not the only way to do this, and the right pick depends on your situation. Kilo is purpose-built, Kubernetes-native, Apache 2.0, and my default for a self-contained hybrid cluster; this post is the argument. Tailscale has the easiest NAT traversal of any of them, punching through routers you cannot configure, with the tradeoff that its coordination plane is a hosted service unless you run Headscale, the open reimplementation. NetBird is the fully self-hostable WireGuard mesh, the pick when you want Tailscale’s ease without anyone else’s coordination server. And Cilium ClusterMesh is the answer when you would rather run a cluster per location and connect them, a legitimate architecture that trades one scheduler for blast-radius isolation. The shared foundation under all of them is WireGuard; what differs is who coordinates and how NAT gets traversed.

One more combination worth knowing exists: Kilo ships kilo-k3s-cilium.yaml and kilo-kubeadm-cilium.yaml manifests that run it purely as the inter-location VPN layer underneath an existing Cilium CNI. If you wanted to stretch the main platform cluster, with its Cilium networking, across locations rather than build a second cluster, that is the supported path.

§Operating the Hybrid Cluster

Watch the handshakes, not the nodes. The cluster-level view can lag reality. wg show kilo0 latest-handshakes on any node lists every peer with the age of its last handshake, and anything over three minutes means that tunnel is down now, whatever kubectl get nodes still says. This one command is the mesh’s health check.

Respect the MTU. WireGuard spends about 60 bytes per packet on encapsulation, so Kilo runs kilo0 at an MTU of 1420. Trouble arrives when something else in the path, a cloud private network, a PPPoE home link, shrinks the budget further: small packets flow, large packets vanish, and the symptom is connections that open fine and then hang mid-transfer. Test the real path with a forced-size ping (ping -M do -s 1380 <peer pod IP>) and lower Kilo’s MTU if it fails.

Place chatty workloads within a location. The tunnels are encrypted, not fast. New York to Nuremberg is 80 to 90 milliseconds no matter how good the software is, and a database chattering with its client across that link will feel every round trip. The region labels that steer the mesh also steer the fix: co-locate the pairs that talk constantly, and let the cross-location links carry the traffic that tolerates them. The monitoring stack sees all regions, so the inter-location latency is a graph you can watch rather than a guess.

Keep the labels deliberate. The entire structure of this cluster, mesh topology and workload placement alike, lives in node labels and a handful of annotations. They are the configuration. A mislabeled region silently reshapes the mesh; treat labels with the same care as the manifests in git.

§When Something Is Wrong

A NAT’d node joins but stays unreachable. Almost always the endpoint. Check the annotation is present and correct (kubectl get node lab-gpu -o jsonpath='{.metadata.annotations.kilo\.squat\.ai/force-endpoint}'), confirm the router still forwards UDP 51820 to the right LAN address, and check wg show kilo0 on that node: no latest handshake line at all means no peer has ever gotten through.

Tunnels drop after working for a minute. The NAT mapping is expiring between packets, which is what persistent-keepalive exists for. Confirm the annotation on the NAT’d node, and confirm wg show reports persistent keepalive: every 10 seconds; if it does not, the annotation did not take.

Connections open, then hang on real data. MTU, nearly every time. The handshake and small requests fit under the broken threshold and bulk transfer does not. Run the forced-size ping test from the operations section and lower the MTU to what the path proves it can carry.

A node flaps NotReady intermittently. Suspect the internet path before the node. Residential links and cheap cloud egress flap in ways a data-center network does not. wg show kilo0 latest-handshakes from another location tells you whether the mesh sees the same gaps; if it does, the problem is the path, and the fix is expectations or a better ISP, not Kubernetes.

Workloads land on the wrong hardware. The labels and taints, always. A pod on a cloud worker that wanted the GPU means the selector did not match a label that exists; a pod stuck Pending means it matched a node whose taint it does not tolerate. kubectl describe pod says which, in the events at the bottom.

§An Agent Can Run This Network

Notice what this cluster’s entire configuration amounts to: two curl installers with a handful of flags, two applied manifests, a region label per node, and two annotations on the NAT’d box. Its entire live state is readable as text: kubectl get nodes, wg show, kgctl graph. That makes a hybrid mesh, historically the kind of infrastructure that justified a network engineer on retainer, something an AI agent can genuinely operate. An agent with cluster access and this post as context can read handshake ages, spot the missing keepalive on a NAT’d node, and produce the exact annotation to fix it, because every diagnostic and every remedy here is a text command away. That is not hypothetical; it is how I run mine, with the agent kept on a leash and the verification habits this blog keeps returning to. And if this post is longer than your patience, that is fine too: hand it to your agent as the frame of reference and ask for the build. It is written to work as one.

§What You Have

A single Kubernetes cluster spanning two clouds and your own hardware, joined by a topology-aware WireGuard mesh, encrypted over the public internet, with workloads placed onto the right machines by label and taint, proven end to end by a packet that entered a garage. The build is four nodes, and nothing about it changes at fourteen: more regions are more labels, more NAT’d sites are more annotations, and the scheduler absorbs all of it.

The clearest payoff of owning hardware in the cluster is the GPU sitting in that garage, and that is the final post: adding on-prem GPUs to Kubernetes for self-hosted AI that is actually affordable.