The platform now holds data in a lot of places: tables in Postgres, topics in Kafka, indices in OpenSearch, and Iceberg tables in the lakehouse. No one, human or machine, can hold the meaning of all of it in their head: what each table is, what a column actually represents, where a dataset came from, who owns it. A metadata catalog answers those questions, and the best open one is DataHub. It is genuinely useful for people, and it turns out to be even more useful for the AI tier this part of the series is about.

This series rebuilds my 2020 Apress book, Advanced Platform Development with Kubernetes, for 2026. The approach behind it comes from building and running data platforms in production for more than twenty years.

§The AI Tier Is About Context, Not Custom Agents

Running your own models in-cluster and hand-building bespoke agents has its place at small scale, but it is not where the leverage is. The leverage is handing a frontier model access to your platform and the context to use it well. A capable model with good context outperforms a hand-built agent on almost everything, and it keeps improving without you rebuilding it, because the model improves on someone else’s schedule and your job is only to keep feeding it access and context. The way you deliver that access is the Model Context Protocol, which the next post covers. This post builds the context.

Context is the hard part, harder than access. Pointing a model at a database is easy; pointing it at a database it understands is the harder problem. A model dropped onto a schema it has never seen guesses at what cust_ltv_q means and gets it wrong, exactly as a new engineer would, except the model guesses with confidence. Give it the catalog, the documented meaning of every table and column, the lineage showing where a number came from, the ownership and the warnings, and it stops guessing and starts reasoning from what is actually true. That catalog is what I build here. DataHub for humans is data discovery; DataHub for an agent is the map it needs before it touches anything, and a wrong map is worse than none.

§DataHub Reuses the Platform You Built

DataHub is a large, multi-service system: a metadata service, a frontend, change-event consumers, and setup jobs. Assembling it by hand is not worth doing. Its Helm chart is the canonical install, one of the few places this series reaches for one. What matters is where you point it.

DataHub needs exactly three backing services: a SQL store, a search index, and a Kafka for its metadata change log. You already run all three. So skip DataHub’s bundled dependencies and aim it at the platform’s own Postgres, OpenSearch, and Kafka. Give it a database first, the usual way:

apiVersion: postgresql.cnpg.io/v1
kind: Database
metadata:
  name: datahub
  namespace: data
spec:
  cluster:
    name: platform-pg
  name: datahub
  owner: app

Then install the chart with values that override every dependency to the running services rather than new copies. The exact keys live in the chart’s values.yaml; the shape is this:

# values-datahub.yaml: point DataHub at the platform, not at fresh dependencies
global:
  sql:
    datasource:
      host: "platform-pg-rw.data:5432"
      url: "jdbc:postgresql://platform-pg-rw.data:5432/datahub"
      driver: "org.postgresql.Driver"
      username: "app"
      password:
        secretRef: platform-pg-app
        secretKey: password
  elasticsearch:
    host: "platform-os.search"
    port: "9200"
    useSSL: "true"
  kafka:
    bootstrap:
      server: "platform-kafka-kafka-bootstrap.kafka:9092"

helm repo add datahub https://helm.datahubproject.io/
helm install datahub datahub/datahub -n data -f values-datahub.yaml

The chart’s setup jobs create the OpenSearch indices and Kafka topics DataHub needs, then bring up the service and frontend.

DataHub authenticates against Keycloak rather than keeping its own users, the same as NiFi and Superset. Create a datahub OIDC client in the platform realm with redirect URI https://catalog.apk8s.dev/*, put its secret in a datahub-oidc secret, and turn on OIDC for the frontend through the chart values:

datahub-frontend:
  oidcAuthentication:
    enabled: true
    provider: keycloak
    clientId: datahub
    clientSecretRef:
      secretRef: datahub-oidc
      secretKey: client-secret
    discoveryUri: "https://auth.apk8s.dev/realms/platform/.well-known/openid-configuration"
    baseUrl: "https://catalog.apk8s.dev"

Expose the frontend through the gateway with an HTTPRoute at catalog.apk8s.dev, and the catalog sits behind the platform’s single sign-on like everything else.

§Fill the Catalog

An empty catalog helps no one. DataHub learns your platform through ingestion recipes, small YAML files that tell it how to read metadata from a source. This one catalogs the Postgres databases, schemas, tables, and columns:

source:
  type: postgres
  config:
    host_port: platform-pg-rw.data:5432
    database: app
    username: app
    password: "${POSTGRES_PASSWORD}"
sink:
  type: datahub-rest
  config:
    server: "http://datahub-datahub-gms.data:8080"

Run it with the datahub CLI, or better, as a Kubernetes CronJob so the catalog stays current as schemas change. Add a recipe per source, Trino for the lakehouse tables, Kafka for the topics, OpenSearch for the indices, and DataHub assembles a single searchable map of the whole platform, with column-level lineage showing how a field in a dashboard traces back through the lakehouse to the topic it arrived on.

§Meaning, Not Just Schema

Schema is the easy half, and ingestion fills it for free. The half that makes the catalog worth having is meaning, and that is the part people add: a business glossary defining terms like “active customer” or “recognized revenue,” documentation on tables and columns, ownership so there is a name to ask, tags for sensitivity, and deprecation markers for the table everyone should stop using. DataHub gives your team one place to write that down, searchable and attached to the actual data rather than rotting in a wiki. That is what turns cust_ltv_q into “customer lifetime value, trailing four quarters, in USD,” for the analyst who finds it and for the model that reads it.

Three kinds of meaning earn their keep especially. Ownership tells a person, or an agent, who is accountable for a dataset and whether it is trustworthy. PII and sensitivity tags mark the columns that must be handled carefully, which is exactly the warning you want an agent to see before it queries a table of personal data. And data products group related datasets into a thing a consumer actually wants, “the customer 360,” rather than a scatter of tables. This is governance, and it is the same metadata that makes the catalog legible to humans and to models at once.

§Lineage Is the Trust Layer

The capability that justifies a catalog on its own is lineage, and DataHub does it at the column level. A field on a Superset dashboard traces back through the Trino query and the Iceberg table to the Kafka topic it arrived on, every hop recorded. For a human, that is impact analysis: change this column and see everything downstream that breaks first. For an agent, it is trust. A model asked where a number came from can answer from recorded lineage instead of inventing a plausible story. Lineage turns “the model says” into “the model says, and here is the path the data took to get there.”

§Operating the Catalog

A catalog is only as good as it is current, and a stale catalog actively misleads, so the operating concern is freshness. Run each ingestion recipe as a scheduled CronJob, and the catalog tracks the platform as schemas change, tables appear, and lineage shifts, with no one remembering to refresh it. The glossary and ownership, the human-added meaning, accrue over time and persist because they live in the catalog, not in someone’s notes.

Two things make this cheap to operate. It reuses the platform’s own Postgres, OpenSearch, and Kafka, so there is no new datastore to run or back up; protecting the spine protects the catalog. And it sits behind the same Keycloak single sign-on as every other tool, so access is governed once. The payoff is a single map of the whole platform that serves two audiences from the same metadata: people discovering data, and the model that is about to be given access to it.

When something is wrong: if the chart’s setup jobs fail, they could not create their OpenSearch indices or Kafka topics, so confirm the external endpoints in the values are reachable and the credentials are right. If ingestion runs but nothing appears, the recipe’s datahub-rest sink cannot reach the GMS service, or its source credentials are wrong; the recipe logs the failed source plainly.

§Where MCP Comes In

DataHub ships an MCP server, so an agent can query the catalog directly: what tables exist, what a column means, where a dataset came from, answered from this metadata instead of guessed. That is one slice of context.

The broader move, and the next post, is an MCP layer over the entire platform, its data and its operations, not only its metadata, so a frontier model can work across everything you have built. The open project for that is txn2/mcp-data-platform; the managed, supported build for teams that would rather buy it is Deasil Works’ Plexara. Either way the pattern holds: give the model access and grounded context, and let it do the work, rather than hand-building an agent for every task. The catalog you just stood up is the first and most important layer of that context.