A data platform that takes in device and sensor data needs a way to receive it that suits how those devices actually work: small, intermittent, often on flaky networks, sometimes a microcontroller with kilobytes of memory. That is what MQTT is for, and the Kafka backbone from earlier, while excellent for the platform’s internal event log, is the wrong shape for talking to a thousand sensors directly. So the platform runs both: MQTT at the edge where the devices are, Kafka as the durable spine, bridged together. The MQTT broker is Eclipse Mosquitto, and standing it up has one trap common to older guides.

This series rebuilds my 2020 Apress book, Advanced Platform Development with Kubernetes, for 2026. The approach behind it comes from building and running data platforms in production for more than twenty years.

§MQTT, and When It Beats Kafka

MQTT and Kafka both move messages with a publish-and-subscribe model, and it is easy to think you must choose. You do not; they solve different ends of the same pipeline. MQTT is a lightweight protocol designed for constrained devices and unreliable networks. A client opens one small connection to a broker, publishes to topics, and subscribes to others, with tiny overhead, quality-of-service levels for delivery guarantees, and a last-will message so the broker knows when a device drops off. It runs happily on a battery-powered sensor or a Raspberry Pi.

Kafka is the opposite end: a heavy, durable, replicated commit log built for high-throughput streams and many independent consumers replaying history. You would not put a Kafka client on a microcontroller, and you would not use MQTT as your system of record. The right architecture uses each for what it is good at. Devices speak MQTT to a broker near them, and a bridge moves that data into Kafka, where it becomes part of the durable platform stream the rest of your systems consume.

§The Same Pattern, Six Years Easier

This pattern is not new; what has changed is how little of it you build yourself now. In 2020 I ran and documented the full version: three Raspberry Pis joined the cluster as tainted sensor nodes, a DaemonSet dropped a shell-script collector onto each of them, and every fifteen seconds each Pi published its CPU temperature and load as JSON to a sensor/<device> topic, with NiFi subscribing, enriching each reading, and loading it into a search index. Devices to broker to pipeline to index, all on one cluster. Getting there took work that has since evaporated: a custom-built MQTT client image, a collector running as root, and version hand-holding at several steps.

The shape survives unchanged in 2026, minus the hand-holding. The pieces around it moved: Elasticsearch is now OpenSearch for reasons this series has already covered, the desktop client of choice is MQTTX since MQTT.fx went commercial, and Mosquitto itself crossed a major version that breaks every config from that era. That last one deserves its own section.

§Mosquitto, and the 2.0 Trap

The trap is the reason an old tutorial will leave you with a broker that refuses every connection. Mosquitto 1.x allowed anonymous connections by default and listened on all interfaces, so a minimal config worked. Mosquitto 2.0 changed both defaults: it now denies anonymous access by default, and a listener with no explicit bind address listens only on localhost. A config that worked in 2019 produces a broker that accepts nothing in 2026, with no obvious error explaining why.

This is secure-by-default, and it means you configure the broker explicitly. You must declare a listener with an address, and you must either allow anonymous access for a closed development setup or, for anything real, point it at a password file.

§Deploy Mosquitto

Mosquitto’s configuration is one small file, so it goes in a ConfigMap, with the broker as a Deployment and its persistence on a Ceph volume so retained messages and queued QoS messages survive a restart.

apiVersion: v1
kind: ConfigMap
metadata:
  name: mosquitto-config
  namespace: iot
data:
  mosquitto.conf: |
    listener 1883 0.0.0.0
    allow_anonymous true
    persistence true
    persistence_location /mosquitto/data/

listener 1883 0.0.0.0 declares the MQTT port and binds it to all interfaces; omit the address and a 2.0 broker binds to localhost, unreachable from any other pod. allow_anonymous true is the development setting that gets the broker testable before credentials exist; the next section turns it off. persistence true tells Mosquitto to write its in-memory state, retained messages, subscriptions, and queued QoS 1 and 2 messages, to disk so they survive a restart, and persistence_location points that file at the mounted volume rather than the container’s throwaway filesystem.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mosquitto-data
  namespace: iot
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: rook-ceph-block
  resources:
    requests:
      storage: 5Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mosquitto
  namespace: iot
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mosquitto
  template:
    metadata:
      labels:
        app: mosquitto
    spec:
      securityContext:
        fsGroup: 1883         # mosquitto's uid/gid in the official image
      containers:
        - name: mosquitto
          image: eclipse-mosquitto:2.0   # pin a current 2.0.x version
          ports:
            - { name: mqtt, containerPort: 1883 }
          volumeMounts:
            - { name: config, mountPath: /mosquitto/config }
            - { name: data, mountPath: /mosquitto/data }
      volumes:
        - name: config
          configMap: { name: mosquitto-config }
        - name: data
          persistentVolumeClaim: { claimName: mosquitto-data }
---
apiVersion: v1
kind: Service
metadata:
  name: mqtt
  namespace: iot
spec:
  selector:
    app: mosquitto
  ports:
    - { name: mqtt, port: 1883, targetPort: 1883 }

The fsGroup is the field people learn the hard way. The official image runs Mosquitto as user mosquitto, uid 1883, and a freshly provisioned Ceph volume mounts owned by root. Without fsGroup: 1883, the broker starts, fails to open its persistence file, logs one line about it, and runs on happily in memory-only mode; you discover the difference after the first restart eats your retained messages. replicas: 1 is not an oversight either: Mosquitto is a single-node broker, and two replicas behind one Service would be two separate brokers with separate subscription state, silently splitting your topics. One replica, with its state on replicated storage and Kubernetes restarting it on failure, is the correct deployment.

kubectl create namespace iot
kubectl apply -f mosquitto.yaml
kubectl -n iot rollout status deploy/mosquitto

Test it with the Mosquitto command-line clients, which ship in the same image: subscribe inside the broker pod, publish from your workstation through a port-forward.

# terminal 1: subscribe to the device topic tree
kubectl -n iot exec -it deploy/mosquitto -- mosquitto_sub -v -t 'sensor/#'

# terminal 2: publish through a port-forward
kubectl -n iot port-forward svc/mqtt 1883:1883 &
mosquitto_pub -h localhost -t sensor/bench1 -m '{"temp":21.4,"load":0.12}'

sensor/bench1 {"temp":21.4,"load":0.12}

The -v flag prints the topic alongside the payload, which matters once wildcards are involved: sensor/# matched a topic one level down, and the output shows exactly which one. You have a working broker.

§Authentication and TLS

Anonymous access is fine on a closed test and unacceptable for anything that touches a real network, because an open MQTT broker is an open door into your platform. For real use, create a password file with mosquitto_passwd, store it as a Kubernetes Secret, and turn anonymous access off.

# generate a hashed password file using the tools in the image
kubectl -n iot exec -it deploy/mosquitto -- sh -c \
  "touch /tmp/passwords && \
   mosquitto_passwd -b /tmp/passwords sensor-fleet 'a-strong-generated-secret' && \
   mosquitto_passwd -b /tmp/passwords nifi-bridge 'another-generated-secret' && \
   cat /tmp/passwords"

# save the printed file as ./passwords, then store it as a Secret
kubectl -n iot create secret generic mosquitto-passwords \
  --from-file=passwords=./passwords

mosquitto_passwd hashes each password before writing, so the file never contains plaintext, and -b takes the password on the command line for scripting. Two accounts is the meaningful minimum: one identity for the device fleet, one for the pipeline that consumes from the broker, so revoking either side later does not break the other. Mount the secret into the pod (a secret volume alongside the existing two, defaultMode: 0440 so it is not world-readable, which Mosquitto 2.0 checks and warns about) and change the config:

listener 1883 0.0.0.0
allow_anonymous false
password_file /mosquitto/secrets/passwords
persistence true
persistence_location /mosquitto/data/

Republish the test message without credentials and the broker now says Connection error: Connection Refused: not authorised., which is the correct answer. With -u sensor-fleet -P <secret> it flows again.

For finer control, Mosquitto’s dynamic security plugin manages users, groups, and per-topic access rules at runtime, so a sensor account can publish only to its own topic and nothing else. And any broker reachable beyond your trusted network must use TLS, a second listener on 8883 with a certificate from the platform’s cert-manager, so device credentials and data are encrypted in transit:

listener 8883 0.0.0.0
certfile /mosquitto/tls/tls.crt
keyfile /mosquitto/tls/tls.key

Never expose an unauthenticated, unencrypted MQTT broker to the internet; it is one of the most-scanned-for open services there is.

§A Collector on Every Edge Node

A collector publishing real readings is the template for any device-side integration: a shell script in a ConfigMap, deployed by a DaemonSet to exactly the nodes that have sensors. The 2026 version is also a small demonstration of how much simpler this got: no custom image, because the eclipse-mosquitto image carries the client tools, and no root privileges, because reading a temperature is not a privileged operation.

apiVersion: v1
kind: ConfigMap
metadata:
  name: smon
  namespace: iot
data:
  collect.sh: |-
    while true; do
      load=$(cut -d' ' -f1 /proc/loadavg)
      temp=$(cat /sensor/temp)
      mosquitto_pub -h mqtt.iot -u "$MQTT_USER" -P "$MQTT_PASS" \
        -t "sensor/$DEVICE" \
        -m "{\"device\":\"$DEVICE\",\"temp\":$temp,\"load\":$load}"
      sleep 15
    done
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: smon
  namespace: iot
spec:
  selector:
    matchLabels:
      name: smon
  template:
    metadata:
      labels:
        name: smon
    spec:
      nodeSelector:
        node-role.platform/sensor: "true"
      tolerations:
        - key: dedicated
          value: pi
          effect: NoSchedule
      containers:
        - name: smon
          image: eclipse-mosquitto:2.0
          command: ["/bin/sh", "/scripts/collect.sh"]
          env:
            - name: DEVICE
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: MQTT_USER
              valueFrom:
                secretKeyRef: { name: fleet-credentials, key: user }
            - name: MQTT_PASS
              valueFrom:
                secretKeyRef: { name: fleet-credentials, key: pass }
          volumeMounts:
            - { name: scripts, mountPath: /scripts }
            - { name: sensor, mountPath: /sensor, readOnly: true }
          resources:
            requests: { cpu: 10m, memory: 16Mi }
            limits: { memory: 32Mi }
      volumes:
        - name: scripts
          configMap: { name: smon }
        - name: sensor
          hostPath:
            path: /sys/class/thermal/thermal_zone0/

The interesting fields are the ones that aim it. The nodeSelector restricts the DaemonSet to nodes labeled as sensor carriers, and the toleration lets it onto nodes tainted dedicated=pi:NoSchedule, the taint that keeps every other workload off constrained edge hardware; a DaemonSet places one pod per matching node, which is exactly the cardinality a per-device collector wants. The DEVICE variable comes from fieldRef: spec.nodeName, so every pod knows which physical device it speaks for and publishes to its own subtopic without any per-node configuration. The credentials arrive from a Secret rather than being baked into the script, so rotating the fleet password is one secret update and a rollout. The hostPath mounts the kernel’s thermal zone read-only, which is everything this pod needs from the host and nothing more; the 2020 version of this collector ran privileged: true, and dropping that is six years of container security hygiene showing up in a ten-line manifest. And the resource block states what a shell loop deserves: almost nothing, with a memory cap so a leak on an edge device cannot matter.

Label a node and watch readings arrive:

kubectl label node <some-node> node-role.platform/sensor="true"

kubectl -n iot exec -it deploy/mosquitto -- \
  mosquitto_sub -v -t 'sensor/#' -u nifi-bridge -P '<secret>'

sensor/lab-d1 {"device":"lab-d1","temp":47234,"load":0.08}
sensor/lab-d1 {"device":"lab-d1","temp":47180,"load":0.06}

Any node you label starts reporting fifteen seconds later. Later in this series the cluster grows real edge hardware in an on-premises region, and this DaemonSet lands there without modification; that is the point of writing collectors this way.

§Bridge to Kafka

The broker receives device data; the platform consumes it from Kafka. The bridge between them is a job the NiFi you already run does cleanly. A ConsumeMQTT processor subscribes to the broker with the pipeline account, Broker URI tcp://mqtt.iot:1883, Topic Filter sensor/+, Quality of Service 1 so readings are not dropped between broker and pipeline, and a bounded Max Queue Size so backpressure holds rather than grows. Each message becomes a flow file. A Jolt transform stamps it with a collection timestamp, because the moment of measurement and the moment of arrival diverge on flaky edge links and analysts need both. Then PublishKafka writes it to a sensor-readings topic on the platform Kafka, where it is durable, replayable, and consumable by everything downstream: the OpenSearch index for dashboards, the lakehouse for history, whatever comes later.

This is also where the broker-bridging architecture from industrial setups fits: a Mosquitto broker on a factory floor or a remote site bridges to the cluster broker over an authenticated connection, so local devices keep working when the link to the platform is down, and their data syncs when it returns. That link is Mosquitto’s own bridge feature, configured on the edge broker:

connection to-platform
address mqtt.example.com:8883
bridge_cafile /mosquitto/tls/ca.crt
remote_username factory-7
remote_password <generated-secret>
topic sensor/# out 1 "" factory-7/

connection names the bridge; address points at the platform broker’s TLS listener, and bridge_cafile verifies it. The topic line is the whole policy in one directive: forward everything under sensor/#, outbound only (out), at QoS 1 so readings survive the flaky link, and prefix the remote topic with factory-7/ so the platform broker sees factory-7/sensor/press-3 and every site lands in its own namespace instead of colliding. The path runs from devices to edge broker to cluster broker to Kafka, and every hop is authenticated.

§Operating the Broker

Manage access as accounts, not one shared key. The dynamic security plugin lets each device or service have its own credentials and its own topic permissions, so a compromised sensor cannot read or write anything but its own data. That is the MQTT version of the least-privilege posture the rest of the platform takes.

Tune delivery to the use case. MQTT’s quality-of-service levels and retained messages are operational choices: QoS 0 for high-volume telemetry where a dropped reading does not matter, QoS 1 or 2 for commands that must arrive, retained messages so a subscriber gets the last known value the moment it connects. Persistence on the Ceph volume keeps queued and retained messages across restarts.

Scale beyond one broker when you need to. Mosquitto is a single-node broker, which handles a great many clients and is right for most platforms. When you genuinely need a clustered, highly available MQTT layer for very large fleets, EMQX and VerneMQ are the clustered brokers to reach for, speaking the same MQTT protocol so devices and bridges do not change. Start with Mosquitto; scale the broker only when the device count demands it.

§When Something Is Wrong

The broker refuses every connection. The 2.0 trap. Either the listener has no address and is bound to localhost, or anonymous access is denied and you sent no credentials. kubectl -n iot logs deploy/mosquitto says which: a listener line showing 127.0.0.1, or Client <id> disconnected, not authorised.

Clients connect but receive nothing. Topic mismatch or QoS. Confirm the subscriber’s topic filter matches the publisher’s topic, remembering MQTT wildcards (+ for one level, # for the rest), and that the subscriber is connected before a non-retained message is published. mosquitto_sub -v -t '#' with the pipeline account shows every topic actually flowing, which settles the argument fast.

Retained and queued messages vanish on restart. Persistence is not actually writing. Check the broker log for a failed open on /mosquitto/data/mosquitto.db; the usual cause is a missing fsGroup leaving the volume owned by root, and the fix is the securityContext from the deployment above.

A device gets old or no last value on connect. Retained messages. Only a retained message (mosquitto_pub -r) is delivered to a subscriber that connects after it was published; without the retain flag, a late subscriber waits for the next publish.

The bridge to Kafka stalls. NiFi backpressure, which is doing its job: if Kafka or a downstream step slows, the ConsumeMQTT queue fills to its bound and holds rather than dropping. Check the NiFi flow’s queues to see where it is waiting.

§The Fleet Is Legible to an Agent

A running MQTT layer is unusually easy to hand to an AI agent, because its entire live state is one subscription away: mosquitto_sub -v -t '#' is the whole truth of what every device is saying right now, and the config is five lines plus a password file. An agent operating this platform can watch the topic stream to spot the sensor that went quiet, cross-check the DaemonSet for the node that stopped scheduling, and draft the bridge stanza for a new site by pattern-matching the one above. When I add a device type, the working session is exactly that: the agent gets this post and a live mosquitto_sub feed as context, and the collector script, credentials, and NiFi property changes come back as a reviewable diff. The protocol was designed in 1999 for machines with kilobytes of memory, and that same austerity is what makes it perfect context for a model.

§What You Have

A self-hosted MQTT broker taking in device and sensor data on the lightweight protocol those devices actually use, secured with real authentication and TLS, persisting across restarts on your own storage, a collector template that lands on any node you label, and a bridge into the Kafka backbone so device data becomes part of the durable platform stream. It is the front door for the physical world, owned by you, with no per-device cloud IoT bill and no telemetry routed through someone else’s broker.

Next I give the people on the platform a place to explore all of this data directly, a multi-user JupyterHub data lab on the platform’s single sign-on.