Every component in this series has been exposing metrics: Postgres, Kafka, OpenSearch, Cilium, the cluster itself, each one had a line about the monitoring stack scraping it “later.” This is later. The 2020 book treated monitoring as an afterthought, a single page pointing at a project’s README. Observability is the difference between operating a platform and hoping it is fine. I stand up the open stack that has become the standard: Prometheus, Grafana, Loki, and OpenTelemetry, on the storage and object store you already own.

This series rebuilds my 2020 Apress book, Advanced Platform Development with Kubernetes, for 2026. The approach behind it comes from building and running data platforms in production for more than twenty years.

§You Cannot Operate What You Cannot See

Observability is usually described as three pillars, and the description is useful. Metrics are numbers over time: CPU, memory, request rate, queue depth, replication lag, the dials and gauges of the platform. Logs are the discrete events: what happened, when, with what detail. Traces follow a single request as it moves through multiple services, so you can see where the time went. Together they answer the questions you ask when something is wrong: is it healthy, what did it do, and where is it slow.

The SaaS answer to this is Datadog, New Relic, and the rest. They bill per host and per gigabyte ingested, which means the cost grows with exactly the thing you want, more visibility into more of your platform. You end up rationing observability to control the bill, which is backward. And your telemetry, the operational record of your entire platform, lives in their cloud. The open stack costs you compute and storage you already run, keeps the data on infrastructure you own, and is built on OpenTelemetry, a vendor-neutral standard, so you are never instrumenting your code for one vendor’s agent.

§Prometheus and the Operator

Prometheus is the metrics engine, and the canonical way to run it on Kubernetes is the kube-prometheus-stack Helm chart, which is one of the few genuine Helm-is-the-sane-install cases in this series because it assembles a coordinated set of pieces: the Prometheus Operator, Prometheus itself, Alertmanager, Grafana, and the exporters that surface node and cluster metrics. Install it into a monitoring namespace with its time-series data on a Ceph volume.

kubectl create namespace monitoring
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install kube-prometheus prometheus-community/kube-prometheus-stack \
  -n monitoring \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.storageClassName=rook-ceph-block

The operator is what makes this pay off. It watches for ServiceMonitor and PodMonitor resources and configures Prometheus to scrape whatever they describe, which is why every component in this series turned on a monitor as it was deployed. Postgres set enablePodMonitor, OpenSearch and Kafka exposed their exporters, Cilium published Hubble metrics. All of that now flows into one Prometheus without any further wiring, because the targets declared themselves and the operator found them.

§Grafana, on Your Single Sign-On

The chart includes Grafana, the dashboard layer over all of this. Point it at the platform’s Keycloak so it logs in through single sign-on like every other tool, and expose it through the gateway at a hostname. Grafana ships with a deep library of prebuilt dashboards for Kubernetes, Postgres, Kafka, and the rest, so the components you stood up have dashboards almost immediately, and dashboards are themselves JSON you keep in version control and provision as config, not click together by hand.

§Logs With Loki

Prometheus handles metrics; Loki handles logs, and it does so cheaply by design. Rather than indexing the full text of every log line the way a heavy log system does, Loki indexes only labels and stores the compressed log content in object storage, which means it costs a fraction of the alternatives to retain a lot of logs. Its backend is S3-compatible storage, so it writes to the SeaweedFS you already run.

# loki values: store logs in the platform's object store
loki:
  storage:
    type: s3
    s3:
      endpoint: http://seaweedfs.storage:8333
      bucketnames: loki
      s3forcepathstyle: true

Collection is Grafana Alloy, the agent that replaced Promtail, running as a DaemonSet that tails container logs on every node and ships them to Loki with Kubernetes labels attached. In Grafana, logs and metrics sit side by side, so you can pivot from a spike on a graph to the exact log lines from that moment.

§Traces and the OpenTelemetry Standard

The third pillar is tracing. The choice that matters here is the instrumentation standard rather than the backend. OpenTelemetry is the CNCF project that standardized how applications emit telemetry, and it has become the default; instrumenting your code for OpenTelemetry means your traces, and increasingly your metrics and logs too, are independent of any vendor’s agent or format. You run an OpenTelemetry Collector to receive that telemetry and route it where you want, and for trace storage Grafana Tempo, like Loki, keeps its data cheaply in the SeaweedFS object store. Your applications speak OpenTelemetry, and the backend behind it is swappable, so you are never re-instrumenting code to escape a monitoring vendor.

§Alerting

Visibility without notification means staring at dashboards, so the stack alerts. Alertmanager, included with the Prometheus chart, takes alert rules defined against the metrics, disk filling, replication lag climbing, a service down, and routes them to wherever your team lives, with grouping and silencing so a single condition does not page you repeatedly. Rules are declarative resources, version-controlled like everything else, so your definition of “something is wrong” is reviewed and tracked.

§Operating the Platform, and Handing It to an Agent

This stack is what lets you operate everything else, and it is built from the same parts as the rest: Ceph block storage for Prometheus, SeaweedFS object storage for Loki and Tempo, Keycloak for login. Protecting the platform’s storage protects its observability data along with it.

There is a 2026 dimension worth naming. All of this telemetry, the metrics, the logs, the traces, the alerts, is the context an AI agent needs to diagnose the platform. Through the MCP gateway from the platform finale, an agent can read the same Prometheus metrics and Loki logs you do, correlate a spike with the deploy that caused it, and explain what happened from the telemetry rather than guessing. The observability you build for humans also makes the platform legible to an agent operating it, given the right access.

§When Something Is Wrong

A target shows as down in Prometheus. The ServiceMonitor or PodMonitor selector does not match the service, or the metrics port is wrong. Prometheus’s targets page names every scrape and why it failed, which is the first place to look.

Prometheus runs out of disk. Its time-series database grows with cardinality and retention; size the Ceph volume for your retention window, or shorten the window. High-cardinality labels are the usual cause of unexpected growth.

Loki accepts no logs, or queries return nothing. The object store. Confirm the SeaweedFS endpoint, bucket, and path-style settings, the same S3 details every object-store consumer needs, and that Alloy is running on the nodes.

Grafana login fails. The usual Keycloak OIDC suspects: the redirect URI and the realm discovery URL, which must match exactly.

§What You Have

A complete observability stack on infrastructure you own: Prometheus collecting metrics from every component that declared itself, Grafana visualizing them behind single sign-on, Loki holding logs cheaply in object storage, OpenTelemetry keeping your instrumentation vendor-neutral, and Alertmanager telling you when something breaks. No per-host APM bill, no telemetry living in someone else’s cloud, and a platform that is now legible to both your team and the agents that help run it.

That completes the foundation layer the data spine quietly assumed. From here the series fills in the capabilities that ride on top, starting with the real-time intake the platform has been waiting for: MQTT ingest for IoT and device data.