Professional notes by Craig Johnston
long-form, short-form, working drafts · since 2008
long-form, short-form, working drafts · since 2008
VOL. XIX · MMXXVI
140 NOTES IN PRINT
140 NOTES IN PRINT
One Cluster Across Clouds and Your Garage: Hybrid k3s With WireGuard
A single Kubernetes cluster spanning cloud regions and your own hardware, encrypted over the public internet
→ Read this noteLatest note · folio diagram
flowchart TB M["control plane<br>cloud region A"] -->|WireGuard mesh| W["workers<br>cloud region B"] M -->|WireGuard mesh| ONP["on-prem GPU box"] M -->|WireGuard mesh| EDGE["edge devices"] KILO["Kilo: topology-aware<br>WireGuard"] -.-> M
The platform so far runs on a set of cloud virtual machines, and that is the right default. But some of the most useful hardware you can put a workload on is not in a cloud: a GPU box in your office, a server in a colo, …
Recent Notes
CXXXIX
What Happened to the Blockchain Chapter
Why the chapter is cut from this rebuild, and where tamper-evidence lives in 2026
2026-08-08
CXXXVIIIServerless Without the Cloud: Knative on Your Cluster
Scale-to-zero functions you own, after OpenFaaS moved the good parts behind a paywall
2026-08-06
CXXXVIIWrite-Heavy and Always On: Cassandra and Scylla on Kubernetes
Running Cassandra and Scylla on K8ssandra and the Scylla Operator after Rook dropped its own
2026-08-04
CXXXVIEvent-Driven Object Processing: Bucket Events to Jobs
Object storage that triggers work, a self-hosted answer to S3 plus Lambda on your own cluster
2026-08-02
CXXXVAn Instrument for MCP Agent Studies
The harness behind two DOI-archived benchmark reports, and why a study premise can be killed in one working day
2026-08-01
CXXXIVA Multi-User Data Lab: JupyterHub on Your Platform
Self-hosted notebooks for the whole team, on Keycloak SSO, wired to the lakehouse and every data source you own
2026-07-31
CXXXIIIIoT Ingest You Own: MQTT With Mosquitto on Kubernetes
Lightweight device messaging into the platform, bridged to Kafka, with the Mosquitto 2.0 change the old guides miss
2026-07-29
CXXXIISee the Whole Platform: Prometheus, Grafana, Loki, and OpenTelemetry
Self-hosted observability for everything you built, on the open standard instead of a per-host APM bill
2026-07-27
CXXXIWhen Do Agents Use Stored Knowledge?
A benchmark that killed its own hypothesis: strong models re-derive what they can check, and a weak model trusts a stale note over the evidence in front of it
2026-07-26
CXXXCilium: One eBPF Data Plane for Networking, Security, and Encryption
The CNI that replaced Weave Net, kube-proxy, and a hand-rolled WireGuard overlay with a single component
2026-07-25
CXXIXcontainerd Without Docker: the Runtime Under Your Cluster
The container runtime the kubelet actually uses, and how to configure and debug it now that the Docker shim is gone
2026-07-23
CXXVIIIDoes a Semantic Knowledge Layer Make an Agent Measurably Better?
A reproducible benchmark: 42.7% to 98.7% on knowledge-trap questions, and a platform that learns from empty
2026-07-22
CXXVIIGitOps: Git as the Source of Truth for Your Cluster
Argo CD and Flux, the deployment half the 2020 book left out, with manifests reconciled from a repo you own
2026-07-21
CXXVIBuild Images Inside Your Cluster, Without the Docker Socket
Rootless container builds in CI with BuildKit, and where supply-chain guardrails belong
2026-07-19
CXXVYour Git Forge Belongs on Your Cluster: Forgejo
Self-hosted source, CI/CD, and a registry, on the lightweight fork that left its for-profit parent
2026-07-17
CXXIVThe Agent Gateway to Your Platform: mcp-data-platform
An Apache-2.0 MCP server that gives frontier models access and grounded context to everything you built
2026-07-15
CXXIIIA Data Catalog Your Agents Can Read: DataHub
Metadata, lineage, and a business glossary over the whole platform
2026-07-13
CXXIISelf-Hosted BI with Apache Superset
Dashboards and SQL exploration over the lakehouse and Postgres, a Tableau replacement on the platform's SSO
2026-07-11
CXXISelf-Hosted Integration with Apache NiFi
Visual dataflow and ETL on your own cluster, instead of Workato and the iPaaS vendors
2026-07-09
CXXSelf-Hosted SSO with Keycloak
One login for every user-facing tool on the platform, instead of renting Okta or Auth0
2026-07-07
CXIXYour Own Snowflake: A Trino and Iceberg Lakehouse
SQL analytics over open table formats on the object storage you own
2026-07-05
CXVIIISelf-Hosted S3 with SeaweedFS
S3-compatible object storage for the data lake, backups, and the lakehouse
2026-07-03
CXVIIA Search Layer You Own: OpenSearch on Kubernetes
The index for the platform, on the Apache-licensed fork of Elasticsearch
2026-07-01
CXVIKafka Without ZooKeeper: KRaft on Kubernetes with Strimzi
A real-time event backbone, without the separate ZooKeeper ensemble to operate
2026-06-29
CXVWhen Your Tools Lie to the Agent
I drove my own platform as a real client and told it to find every place a tool said one thing and did another. A case study in MCP SDLC.
2026-06-28
CXIVA Database You Own: Postgres with CloudNativePG
High-availability PostgreSQL on your own storage, without the managed bill
2026-06-27
CXIIIWhere Tribal Knowledge Goes
An insight an analyst shares mid-session becomes a reviewed, signed-off change to the catalog rather than vanishing when the conversation ends.
2026-06-26
CXIIIngress-Nginx Retired. Meet the Gateway API.
Exposing services and automating TLS in 2026, with Cilium and cert-manager
2026-06-25
CXISteering Agents Without Caging Them
Steering agents from the server middleware instead of the tool descriptions.
2026-06-24
CXStorage the Cluster Owns: Rook and Ceph
Portable block and shared-filesystem persistence for stateful services, no managed disks required
2026-06-23