When a platform needs to move events in real time, many producers, many consumers, durable and replayable, Kafka is the backbone. The 2020 edition ran it the only way you could back then: a Kafka cluster sitting on top of a separate Apache ZooKeeper ensemble that held its metadata. That whole second cluster is gone now. I run Kafka in KRaft mode on the storage set up earlier, managed by an operator the same way the Postgres cluster is, with no ZooKeeper to deploy.

This series rebuilds my 2020 Apress book, Advanced Platform Development with Kubernetes, for 2026. The approach behind it comes from building and running data platforms in production for more than twenty years.

§Kafka, and When You Do Not Need It

Kafka is a distributed commit log. Producers append events to topics, the log is partitioned and replicated across brokers, and any number of consumers read from it independently, replaying from any point. That durability and fan-out is what makes it the right tool for high-throughput event streaming, change data capture, and pipelines feeding many downstream systems at once.

It is also more machinery than a lot of jobs need. As I mentioned with Postgres, if you only want a modest work queue, LISTEN/NOTIFY on the database you already run does the job without standing up a streaming cluster. Reach for Kafka when you genuinely need a durable, replayable, high-volume log with independent consumers. That is the case this post sets up. Run without that need, Kafka is overhead you pay for nothing.

§ZooKeeper Is Gone, and Why That Matters

This is the big change since the book. Kafka used to keep its cluster metadata, the brokers, topics, partitions, and leadership, in a separate ZooKeeper ensemble you deployed, secured, upgraded, and kept alive alongside Kafka. That meant two distributed consensus systems to operate instead of one, with their own failure modes and security surfaces, and a class of outages where Kafka was fine but lost its connection to ZooKeeper and seized up anyway. The 2020 streaming chapter spent its first effort standing up ZooKeeper precisely because you could not have Kafka without it.

KRaft, from KIP-500, moved that metadata inside Kafka itself, managed by a Raft quorum of controller nodes. It was declared production-ready in Kafka 3.3, and Kafka 4.0 removed ZooKeeper entirely in 2025. So a Kafka cluster today is just two kinds of node: brokers that handle the message data, and controllers that run the metadata quorum, frequently the same nodes doing both. Beyond removing a whole system to operate, KRaft fails over faster and scales to far more partitions, because metadata no longer round-trips through an external service. For this platform that means one fewer distributed system to own.

§Install the Strimzi Operator

Strimzi is the CNCF operator for running Kafka on Kubernetes, and like CloudNativePG it encodes the operational expertise as controllers watching custom resources. You declare the cluster, the topics, and the users you want, and Strimzi makes Kafka match. It installs from a single manifest bundle; the namespace parameter templates it for the target namespace.

kubectl create namespace kafka
kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka

kubectl -n kafka rollout status deployment/strimzi-cluster-operator

§Declare the Cluster in KRaft Mode

A KRaft cluster is two resources. A KafkaNodePool describes a set of nodes, here three dual-role nodes that are both brokers and controllers, each claiming a 50 GiB volume from the rook-ceph-block class. Combined-role nodes are right for a development cluster; in production you typically split them into a small pool of dedicated controllers and a larger pool of brokers, which the node-pool model makes a one-line change.

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: pool-a
  namespace: kafka
  labels:
    strimzi.io/cluster: platform-kafka
spec:
  replicas: 3
  roles:
    - controller
    - broker
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 50Gi
        class: rook-ceph-block
        deleteClaim: false

The Kafka resource ties it together. The two annotations are what put it in KRaft mode with node pools; the node pool owns the replica count and storage, so they are not repeated here. The replication settings ensure every internal and user topic survives losing a broker.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: platform-kafka
  namespace: kafka
  annotations:
    strimzi.io/kraft: enabled
    strimzi.io/node-pools: enabled
spec:
  kafka:
    version: 4.0.0      # a version your Strimzi release supports
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: scram
        port: 9094
        type: internal
        tls: true
        authentication:
          type: scram-sha-512
    authorization:
      type: simple
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
  entityOperator:
    topicOperator: {}
    userOperator: {}

kubectl apply -f kafka-nodepool.yaml -f kafka-cluster.yaml

The operator provisions three Kafka nodes on Ceph-backed volumes, forms the KRaft metadata quorum among them, exposes a plain listener for trusted in-cluster traffic and an authenticated one for everything that should prove who it is, and turns on the entity operator that reconciles topics and users.

§Verify

kubectl -n kafka get pods

NAME                                            READY   STATUS    RESTARTS   AGE
platform-kafka-pool-a-0                         1/1     Running   0          3m
platform-kafka-pool-a-1                         1/1     Running   0          3m
platform-kafka-pool-a-2                         1/1     Running   0          3m
platform-kafka-entity-operator-7c9f...          2/2     Running   0          2m
strimzi-cluster-operator-6d8b...                1/1     Running   0          6m

Three Kafka nodes, the entity operator, and the cluster operator. Notice what is not there: no ZooKeeper pods. The cluster reports ready and names its metadata state plainly:

kubectl -n kafka get kafka platform-kafka

NAME             DESIRED KAFKA REPLICAS   READY   METADATA STATE   WARNINGS
platform-kafka   3                        True    KRaft

§Topics and Users as Resources

Running Kafka under an operator makes topics and the credentials that use them Kubernetes resources, version-controlled and declarative, rather than commands you run against a live cluster and hope you remember.

A KafkaTopic reconciles into a real topic with the partitions and replication you ask for.

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: events
  namespace: kafka
  labels:
    strimzi.io/cluster: platform-kafka
spec:
  partitions: 3
  replicas: 3

A KafkaUser reconciles into credentials with exactly the access you grant and nothing more. Because the cluster turned on simple authorization, this user can be scoped with ACLs to the topics it needs, and Strimzi writes its generated password into a Kubernetes secret.

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: events-writer
  namespace: kafka
  labels:
    strimzi.io/cluster: platform-kafka
spec:
  authentication:
    type: scram-sha-512
  authorization:
    type: simple
    acls:
      - resource:
          type: topic
          name: events
        operations: [Write, Describe]

kubectl apply -f topic-events.yaml -f user-events-writer.yaml

Granting a producer write access to one topic is now a manifest in a pull request, reviewable and revertible, instead of a kafka-acls.sh invocation lost to shell history, the same declarative operational surface the rest of the platform uses.

§Produce and Consume

Prove the log works. Run a throwaway producer pod against the plain in-cluster listener and type a few messages. Strimzi exposes the cluster at the bootstrap service platform-kafka-kafka-bootstrap.

kubectl -n kafka run kafka-producer -ti \
  --image=quay.io/strimzi/kafka:latest-kafka-4.0.0 \
  --rm=true --restart=Never -- \
  bin/kafka-console-producer.sh \
  --bootstrap-server platform-kafka-kafka-bootstrap:9092 --topic events

In another terminal, consume them from the beginning:

kubectl -n kafka run kafka-consumer -ti \
  --image=quay.io/strimzi/kafka:latest-kafka-4.0.0 \
  --rm=true --restart=Never -- \
  bin/kafka-console-consumer.sh \
  --bootstrap-server platform-kafka-kafka-bootstrap:9092 --topic events --from-beginning

Everything typed into the producer appears in the consumer, replayed from the start of the log. Real applications connect to the same bootstrap service from inside the cluster, and you reach it from your workstation with kubefwd exactly as with Postgres, the bootstrap name resolving locally so a producer running on your laptop writes to the real cluster.

§Operating Kafka

Managed Kafka services, MSK and Confluent Cloud, charge for the operations a cluster needs: rolling upgrades without data loss, rebalancing partitions when you add capacity, restarting failed brokers, and surfacing metrics. Own the cluster and Strimzi does all of it, most of it without you noticing.

Self-healing and rolling changes. If a broker pod dies, the operator brings it back, and it rejoins the quorum and catches up its replicas. When you change the Kafka version or a config value, Strimzi performs a controlled rolling update, moving partition leadership off each broker before it restarts so producers and consumers never see an outage, one broker at a time. The upgrade you trigger by editing version is the same deliberate, supervised pattern as the database and the cluster nodes.

Scaling and rebalancing. Adding capacity is bumping the node pool’s replicas, but new brokers join empty; existing partitions do not move to them on their own. Strimzi ships Cruise Control for exactly this. Enable it on the cluster, then a KafkaRebalance resource computes and applies an optimal partition reassignment onto the new brokers.

# in the Kafka spec
  cruiseControl: {}

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
  name: add-brokers-rebalance
  namespace: kafka
  labels:
    strimzi.io/cluster: platform-kafka
spec:
  mode: add-brokers
  brokers: [3, 4]

Strimzi proposes the reassignment; you approve it with an annotation, and Cruise Control moves the data with throttling so it does not swamp the cluster. Partition rebalancing is one of the operations most dreaded by hand on Kafka, and here it is a resource and an approval.

Metrics. Turn on the Kafka exporter and a metrics config, and Strimzi publishes Prometheus metrics for throughput, consumer lag, and under-replicated partitions, which the monitoring stack later in this series scrapes. Consumer lag in particular is the number that tells you whether your pipeline is keeping up, and you want it on a dashboard before you need it.

§When Something Is Wrong

Pods sit Pending. Storage again. The persistent-claim volumes cannot bind; check the PVCs and the Rook cluster behind them. KRaft controllers need their disks to form the quorum.

Producers fail with “not enough replicas.” You set min.insync.replicas: 2 and fewer than two brokers are in sync, usually because a broker is down or still catching up. The setting is doing its job, refusing to accept writes it cannot durably replicate. Restore the broker rather than lowering min.insync.replicas.

A KafkaTopic or KafkaUser does not appear in the cluster. The entity operator is not running or the resource is missing the strimzi.io/cluster label that ties it to your Kafka. Check kubectl -n kafka get pods for the entity-operator pod and confirm the label.

A client cannot connect. Wrong listener. The plain listener on 9092 takes no credentials and is for trusted in-cluster use; the authenticated listener on 9094 requires the SCRAM credentials from the KafkaUser secret. A client pointed at the authenticated port without credentials is refused, by design.

§What You Have

A three-node Kafka cluster in KRaft mode, replicated across Ceph volumes, with topics and scoped users managed as resources, self-healing brokers, rebalancing on demand, and metrics, all declared in a few manifests. No ZooKeeper to stand up, secure, or reason about. The streaming backbone is in place, and producers across the platform can write to it on storage you own.

Next I make the data queryable, standing up OpenSearch as the platform’s search and index layer for the events and documents flowing through here, including why its aggregation at scale matters as much as its full-text search.