(I Make Things Interesting) Architecting, Developing, nixCraft, DevOps, AI/ML, Blockchain

Kafka on Kubernetes

Deploy a highly available Kafka cluster on Kubernetes.

Kafka is a fast, horizontally scalable, fault-tolerant, message queue service. Kafka is used for building real-time data pipelines and streaming apps. There are a few Helm based installers out there including the official Kubernetes incubator/kafka. However, in this article, I walk through applying a surprisingly small set of Kubernetes configuration files needed to stand up high performance, highly available Kafka. Manually applying Kubernetes configurations gives you a step-by-step understanding of the system you are deploying and limitless opportunities to customize.