A platform spends most of its life moving data between systems: pulling from an API, reshaping a payload, draining a queue into a table, fanning one feed out to three destinations. The SaaS answer to that is an integration platform, Workato, MuleSoft, Boomi, Fivetran, billed per task or per connector and holding your flows in their cloud. The open answer is Apache NiFi, which runs on your own cluster and wires together everything this series has built: Kafka topics in, Postgres and the lakehouse out.

This series rebuilds my 2020 Apress book, Advanced Platform Development with Kubernetes, for 2026. The approach behind it comes from building and running data platforms in production for more than twenty years.

§Why NiFi and Not an iPaaS

Workato and the integration vendors sell a fraction of what NiFi already does, and they rent it back to you with a meter running. The iPaaS pricing model is per-task or per-connector, which means the cost scales with exactly the thing you want to grow, the volume of data you move, and the flows themselves live in the vendor’s cloud, so the integration layer that ties your whole stack together is the part you least control.

NiFi is a visual dataflow engine with hundreds of built-in processors, and three properties that the vendors charge a premium for are built into its core. Backpressure: every connection between processors has a bounded queue, so a slow destination cannot drown a fast source; the flow throttles itself instead of losing data or falling over. Guaranteed delivery: a write-ahead flowfile repository means a record in flight survives a restart, so the engine does not silently drop work when a pod cycles. And provenance: NiFi records the lineage of every single record, where it came from, every processor it passed through, what it looked like at each step, so when something looks wrong months later you can replay exactly what happened to one record. You drag boxes onto a canvas, connect them, and a production dataflow with all of that is running. No per-task billing, no connector catalog to pay into, your flows on your cluster.

Version 2 added the change that closes the last gap. You can now write processors in Python. When NiFi’s built-in components do not cover a transform, you drop a small Python class into its extensions directory and it shows up on the canvas like any other processor, no Java, no build. For a platform whose data people already live in Python, that turns NiFi from a tool the data team tolerates into one they extend.

§Deploy NiFi

NiFi runs as a StatefulSet with its repositories on persistent volumes, and as the platform’s first user-facing tool it carries no login of its own; it authenticates against the Keycloak realm from the last post. It needs two secrets: the OIDC client secret Keycloak generated for the nifi client, and a sensitive-properties key that encrypts secrets stored inside flows.

kubectl create namespace flow

kubectl -n flow create secret generic nifi-oidc \
  --from-literal=client-secret='the-nifi-client-secret-from-keycloak' \
  --from-literal=sensitive-key='change-this-sensitive-key'

NiFi keeps several repositories, content, flowfile, provenance, database, plus its config and state, and each should survive a restart. The StatefulSet claims a rook-ceph-block volume for each, enables HTTPS, sets the proxy host so NiFi accepts requests arriving through the gateway, and points its OpenID Connect settings at the Keycloak realm.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nifi
  namespace: flow
spec:
  serviceName: nifi
  replicas: 1
  selector:
    matchLabels:
      app: nifi
  template:
    metadata:
      labels:
        app: nifi
    spec:
      containers:
        - name: nifi
          image: apache/nifi:2.5.0   # pin a current 2.x version
          ports:
            - { name: https, containerPort: 8443 }
          env:
            - { name: NIFI_WEB_HTTPS_PORT, value: "8443" }
            - { name: NIFI_WEB_PROXY_HOST, value: "nifi.apk8s.dev:443" }
            # OpenID Connect against Keycloak (maps to nifi.security.user.oidc.*)
            - { name: NIFI_SECURITY_USER_OIDC_DISCOVERY_URL, value: "https://auth.apk8s.dev/realms/platform/.well-known/openid-configuration" }
            - { name: NIFI_SECURITY_USER_OIDC_CLIENT_ID, value: "nifi" }
            - name: NIFI_SECURITY_USER_OIDC_CLIENT_SECRET
              valueFrom:
                secretKeyRef: { name: nifi-oidc, key: client-secret }
            - name: NIFI_SENSITIVE_PROPS_KEY
              valueFrom:
                secretKeyRef: { name: nifi-oidc, key: sensitive-key }
          volumeMounts:
            - { name: conf,       mountPath: /opt/nifi/nifi-current/conf }
            - { name: state,      mountPath: /opt/nifi/nifi-current/state }
            - { name: content,    mountPath: /opt/nifi/nifi-current/content_repository }
            - { name: flowfile,   mountPath: /opt/nifi/nifi-current/flowfile_repository }
            - { name: provenance, mountPath: /opt/nifi/nifi-current/provenance_repository }
            - { name: database,   mountPath: /opt/nifi/nifi-current/database_repository }
  # one volume per repository; all use the same rook-ceph-block claim
  volumeClaimTemplates:
    - metadata: { name: conf }
      spec: { accessModes: [ReadWriteOnce], storageClassName: rook-ceph-block, resources: { requests: { storage: 5Gi } } }
    - metadata: { name: state }
      spec: { accessModes: [ReadWriteOnce], storageClassName: rook-ceph-block, resources: { requests: { storage: 5Gi } } }
    - metadata: { name: content }
      spec: { accessModes: [ReadWriteOnce], storageClassName: rook-ceph-block, resources: { requests: { storage: 20Gi } } }
    - metadata: { name: flowfile }
      spec: { accessModes: [ReadWriteOnce], storageClassName: rook-ceph-block, resources: { requests: { storage: 10Gi } } }
    - metadata: { name: provenance }
      spec: { accessModes: [ReadWriteOnce], storageClassName: rook-ceph-block, resources: { requests: { storage: 10Gi } } }
    - metadata: { name: database }
      spec: { accessModes: [ReadWriteOnce], storageClassName: rook-ceph-block, resources: { requests: { storage: 5Gi } } }

A headless Service governs the StatefulSet and gives the UI a name to reach.

apiVersion: v1
kind: Service
metadata:
  name: nifi
  namespace: flow
spec:
  clusterIP: None
  selector:
    app: nifi
  ports:
    - { name: https, port: 8443 }

kubectl apply -f nifi-statefulset.yaml -f nifi-service.yaml
kubectl -n flow rollout status statefulset/nifi

This is a single NiFi node, which handles a great deal and is right for development. A production NiFi runs as a cluster, where nodes share the load and elect a coordinator, and that coordination still uses ZooKeeper, the one place it survives in this stack. Rather than hand-wire a NiFi cluster, the NiFiKop operator manages clustered NiFi on Kubernetes, including the ZooKeeper it needs; reach for it when one node is no longer enough. The single node here keeps the moving parts visible.

§Expose the Canvas Through the Gateway

NiFi’s UI is HTTPS on 8443. An HTTPRoute puts it behind the gateway at a hostname, and because NiFi terminates its own TLS, the route uses an HTTPS backend.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: nifi
  namespace: flow
spec:
  parentRefs:
    - name: platform-gateway
      namespace: gateway
  hostnames:
    - "nifi.apk8s.dev"
  rules:
    - backendRefs:
        - name: nifi
          port: 8443

With DNS pointed at the gateway, open https://nifi.apk8s.dev. NiFi redirects you to Keycloak to sign in, then back to the canvas, with no NiFi-specific password to manage. Set the initial admin identity to your Keycloak user so the first single sign-on login has full control.

§A Flow Across the Platform

NiFi flows are built by dragging processors onto the canvas and wiring them, so this is a description rather than a manifest, but the shape is concrete. A typical platform flow drains a Kafka topic into the lakehouse:

• ConsumeKafka reads the events topic from the Kafka cluster, each message becoming a flow file.
• A transform reshapes or enriches it. NiFi ships processors for JSON, Avro, CSV, schema validation, and routing on content, and where those fall short, a Python processor handles it.
• PutDatabaseRecord writes structured rows into Postgres, or a sink writes Parquet into the lake bucket for Trino to query as an Iceberg table.

Backpressure and provenance come for free. If Postgres slows down, NiFi holds the queue rather than dropping data, and every record carries a full lineage you can inspect when something looks wrong months later. That observability is what the integration vendors charge a premium for.

§A Processor in Python

When you need custom logic, NiFi 2 lets you write it in Python. A processor is a small class; drop the file into NiFi’s python/extensions directory and it appears on the canvas.

from nifiapi.flowfiletransform import FlowFileTransform, FlowFileTransformResult
import json

class UppercaseMessage(FlowFileTransform):
    class Java:
        implements = ['org.apache.nifi.python.processor.FlowFileTransform']

    class ProcessorDetails:
        version = '0.0.1'
        description = 'Uppercase the message field of a JSON record'

    def transform(self, context, flowfile):
        record = json.loads(flowfile.getContentsAsBytes())
        record['message'] = record['message'].upper()
        return FlowFileTransformResult('success', contents=json.dumps(record))

No Java, no Maven, no build step.

§Flows as Code

A flow clicked together on a canvas is the same trap as a realm clicked together in Keycloak: it works, but it is not the source of truth. NiFi versions flows through a registry. You run NiFi Registry alongside NiFi, place a process group under version control, and each change is a commit; the registry can be backed by a git repository, so the flow definitions live in version control like every other manifest on the platform. Promotion from a development cluster to production becomes importing a known flow version rather than rebuilding it by hand.

Parameter contexts are the other half of this. Instead of baking a hostname or a credential into a processor, you reference a parameter, and the parameter context supplies the value per environment. The same versioned flow runs against the dev database and the prod database by pointing at a different context. Connections to systems are controller services, a DBCPConnectionPool for Postgres or a Kafka client service, defined once and shared by every processor that needs them, so a credential lives in one place rather than scattered across the canvas.

§Operating NiFi

A managed iPaaS hides its operations inside the meter. Running NiFi yourself makes them visible.

Provenance is your audit trail. Every record’s lineage is queryable from the UI: pick a flowfile, see its full history, and replay it from any point. When a downstream system reports bad data, you look at exactly what NiFi received and did. This is the observability the vendors sell as a premium tier, on by default.

Backpressure is your safety valve. Each connection has object and size thresholds; when a destination slows, the upstream queue fills to its bound and the source stops pulling rather than overrunning memory. Tuning those thresholds is how you shape a flow’s behavior under load, so it slows rather than running out of memory.

Scale, and watch it. When one node is not enough, the NiFiKop operator runs a balanced cluster. NiFi exposes metrics through a Prometheus reporting task, so throughput, queue depth, and back-pressure events land on the same dashboards as the rest of the platform. The repositories sit on Ceph volumes that get snapshotted, and the flows live in the registry’s git, so both the engine’s state and its definitions are backed up.

§When Something Is Wrong

Login fails with an “untrusted proxy” error. NiFi rejects requests whose host header it does not recognize. Set NIFI_WEB_PROXY_HOST to the external hostname and port the gateway presents (nifi.apk8s.dev:443); without it, NiFi refuses requests arriving through the gateway.

You log in but have no permissions. The OIDC login succeeded but NiFi does not know that identity is an administrator. The initial admin identity must match the claim Keycloak sends, the user’s email or username, so the first single-sign-on user is granted control. A mismatch logs you in as a user NiFi authorizes for nothing.

After a restart, flows show sensitive values as unreadable. The sensitive-properties key changed. That key encrypts secrets inside flows, and if it is not stable across restarts the encrypted values cannot be decrypted. Keep it in the persisted secret, as the deployment does, and never rotate it casually.

The redirect after Keycloak login fails. The nifi client’s redirect URI in Keycloak does not match. It must include NiFi’s callback path under https://nifi.apk8s.dev, exact as every OIDC redirect URI must be.

§What You Have

A self-hosted integration and ETL engine that moves data between every component on the platform, with backpressure, provenance, and guaranteed delivery built in, extensible in Python, behind your own gateway with TLS, on your own storage. It does what Workato and the iPaaS vendors sell, more of it, without the per-task meter or the flows living in someone else’s cloud.

The data platform is now complete end to end, ingest, store, search, warehouse, and move, with NiFi the first tool behind the platform’s single sign-on. Next I add the front end for exploring all of it, Apache Superset, a Tableau replacement that has matured enormously, on the same Keycloak login and querying the lakehouse and Postgres directly.