A common cloud pattern is a file landing in a bucket that automatically kicks off some work: a CSV arrives and gets validated, an image gets a thumbnail, a log file gets compressed. The cloud does this with S3 events wired to Lambda functions, which works but locks you to one vendor. The object storage you already run can do the same thing, and the compute that responds runs as ordinary Kubernetes Jobs on your own cluster. This is the self-hosted version of S3-plus-Lambda, the processing half of the data lake the storage post set up.

This series rebuilds my 2020 Apress book, Advanced Platform Development with Kubernetes, for 2026. The approach behind it comes from building and running data platforms in production for more than twenty years.

§Event-Driven, Without the Cloud Functions

The appeal of the bucket-event pattern is that the storage tells you when something happened, so you do not poll. An object lands, a notification fires, and a piece of code runs in response. The cloud’s version, S3 to Lambda, ties the storage, the event mechanism, and the function runtime to one provider, each with its own limits and billing.

The self-hosted version uses pieces you already own. SeaweedFS publishes a notification when anything in the store changes. The notification lands on a Kafka topic, the platform’s durable backbone. The work that responds is a Kubernetes Job, the cluster’s native unit for run-to-completion tasks. There is no proprietary function runtime, no per-invocation meter, and no fifteen-minute execution limit, only storage, a queue, and the cluster’s own scheduler.

§From a Clock to an Event

In 2020, the practical version of this on a self-hosted platform ran on a clock, not an event: a Python script in a ConfigMap, run hourly as a Kubernetes CronJob, pulling the previous hour of data out of the search index, serializing it to CSV, and loading the file into object storage. That was the pattern I ran and documented then, because it was the right available tool, and there is still nothing wrong with it; a CronJob remains correct when the trigger really is time, and this platform runs them for periodic aggregation. What was missing in 2020 was the storage answering back. Reacting to an individual object landing meant polling, and polling a bucket is the kind of glue code that quietly becomes a liability. That capability gap has since closed, which is this post.

One thing changed alongside it. MinIO, the object store of that era, implements S3-style bucket notifications; MinIO also left this platform when its license did, and SeaweedFS took its place. SeaweedFS does the notification job differently, and the difference is worth understanding rather than papering over, because guides that assume the S3 event API will point you at configuration SeaweedFS does not have.

§What SeaweedFS Actually Emits

AWS S3 defines an event taxonomy (s3:ObjectCreated:* and its relatives) that you configure per bucket, with prefix and suffix filters, delivered as JSON. SeaweedFS does not implement that API. What it has instead is broader: the filer, the component that owns all metadata, can publish every change in the store to a message queue. Not just one bucket you remembered to configure, every create, update, rename, and delete, cluster-wide, onto a topic. S3 buckets in SeaweedFS are directories under /buckets/, so an object landing in the upload bucket is a file created at /buckets/upload/<key>, and the event stream carries it like any other change.

Enable it with the notification config, generated by weed scaffold -config=notification and mounted into the filer:

# notification.toml, mounted into the filer pod
[notification.kafka]
enabled = true
hosts = [
  "platform-kafka-kafka-bootstrap.kafka:9092"
]
topic = "filer-events"

hosts points at the platform Kafka’s bootstrap service, and topic is a dedicated topic for the stream, declared as a KafkaTopic resource like the platform’s other topics so its retention and partitions are versioned config rather than accidents. Restart the filer and the store starts announcing itself.

Two facts about the messages matter to everything downstream, and they come straight from the filer source rather than any S3 document. The message key is the full path of the changed entry, /buckets/upload/report-2026-08-01.csv, which means cheap filtering: a consumer can decide by key alone whether it cares, without deserializing anything. And the message value is protobuf, SeaweedFS’s own EventNotification from filer.proto, not JSON. Its shape encodes the event type by presence: a create has NewEntry and no OldEntry, a delete has OldEntry and no NewEntry, an update or rename has both. A consumer expecting S3’s JSON will read garbage; a consumer that knows the proto gets more than S3 offers, including both sides of every rename.

§See the Stream First

Before building anything, look at the events, and the filer ships the tool: weed filer.meta.tail subscribes to the same change stream and prints it as JSON.

kubectl -n data exec -it deploy/seaweedfs-filer -- \
  weed filer.meta.tail -pathPrefix=/buckets/upload -pattern='*.csv'

Upload a file to the bucket from a data lab notebook, and the event prints:

{
  "directory": "/buckets/upload",
  "eventNotification": {
    "newEntry": {
      "name": "report-2026-08-01.csv",
      "attributes": { "fileSize": 48231, "mime": "text/csv" }
    }
  }
}

That is the payload you will be reacting to, seen before a line of controller code exists. The notebook is also where the transform itself gets prototyped: pull one object with boto3, run the logic against it, check the output, all interactively. The previous post promised this workflow, prototype in the lab and promote to a Job, and this is the promotion path in practice: the logic proves out in a notebook cell, then ships as a small compiled worker.

§A Worker in a 10MB Container

The work itself wants to be a small, fast, self-contained program, and Go is the right fit here for the same reasons it is everywhere else I reach for it. The concrete example: a compressor that reads an object, gzips it as a stream, and writes the result back to a processed bucket, without ever holding the whole file in memory. An io.Pipe connects the read side to the write side, and errors propagate through the pipe so a failed read fails the upload rather than silently truncating it.

// stream GetObject -> gzip -> PutObject, no full-file buffering
pr, pw := io.Pipe()
go func() {
    obj, err := s3c.GetObject(ctx, &s3.GetObjectInput{Bucket: &src, Key: &key})
    if err != nil {
        pw.CloseWithError(err)
        return
    }
    defer obj.Body.Close()
    gz, _ := gzip.NewWriterLevel(pw, gzip.BestSpeed) // errs only on invalid level
    if _, err := io.Copy(gz, obj.Body); err != nil {
        pw.CloseWithError(err)
        return
    }
    pw.CloseWithError(gz.Close())
}()
if _, err := s3c.PutObject(ctx, &s3.PutObjectInput{
    Bucket: &dst, Key: &outKey, Body: pr,
}); err != nil {
    log.Fatalf("put %s: %v", outKey, err)
}

CloseWithError is the line that makes the streaming safe: whatever kills the producer side, the PutObject reading from the pipe sees the same error and aborts, so the destination bucket never holds a half-compressed object presented as complete. gzip.BestSpeed is a stated choice, not a default: for pipeline compression the goal is throughput, and the few percent of ratio that BestCompression buys costs multiples of the CPU.

The packaging matters as much as the code. A multistage Dockerfile builds a static binary and copies it into a scratch image, so the final container is the binary and nothing else: a few megabytes, no shell, no package manager, running as an unprivileged user.

FROM golang:1.24 AS build
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o /compressor ./cmd/compressor

FROM scratch
COPY --from=build /compressor /compressor
USER 65534:65534
ENTRYPOINT ["/compressor"]

CGO_ENABLED=0 is what makes scratch possible: the binary links nothing dynamic, so it needs no libc and no base image to carry one. USER 65534 runs it as nobody, because a batch worker has no reason to be root anywhere. A ten-megabyte single-binary container is easy to scan, fast to pull, and exposes very little, the same minimal-attack-surface discipline that runs through the platform’s security posture.

§The Controller

The controller is the piece that turns events into work: a small Go service that consumes the topic, filters for the objects it owns, and creates a Job per object. The core loop is short enough to read whole:

r := kafka.NewReader(kafka.ReaderConfig{
    Brokers: []string{"platform-kafka-kafka-bootstrap.kafka:9092"},
    GroupID: "object-processor",
    Topic:   "filer-events",
})
for {
    m, err := r.ReadMessage(ctx)
    if err != nil {
        log.Fatalf("read: %v", err)
    }
    path := string(m.Key) // e.g. /buckets/upload/report-2026-08-01.csv
    if !strings.HasPrefix(path, "/buckets/upload/") ||
        !strings.HasSuffix(path, ".csv") {
        continue
    }
    var ev filer_pb.EventNotification
    if err := proto.Unmarshal(m.Value, &ev); err != nil {
        log.Printf("skip undecodable event for %s: %v", path, err)
        continue
    }
    if ev.NewEntry == nil || ev.NewEntry.IsDirectory || ev.OldEntry != nil {
        continue // only fresh file creations
    }
    if err := createJob(ctx, clientset, path); err != nil {
        log.Printf("job for %s: %v", path, err)
    }
}

The filtering happens in two stages that mirror the message design. The key check is nearly free and eliminates everything outside the upload bucket’s CSVs before any decoding; the protobuf check then keeps only genuine creations, skipping directories, updates (OldEntry present), and deletes (NewEntry absent). The consumer GroupID gives the controller Kafka’s usual guarantees: restart it and it resumes from its committed offset, run it during a burst and nothing is missed, because the events are as durable as any other platform stream. This is where the architecture beats a webhook: the queue remembers, so the controller does not have to be up when the object lands.

createJob builds the same Job you would write by hand in YAML:

job := &batchv1.Job{
    ObjectMeta: metav1.ObjectMeta{GenerateName: "compress-"},
    Spec: batchv1.JobSpec{
        BackoffLimit:            ptr.To(int32(2)),
        TTLSecondsAfterFinished: ptr.To(int32(300)),
        Template: corev1.PodTemplateSpec{
            Spec: corev1.PodSpec{
                RestartPolicy: corev1.RestartPolicyNever,
                Containers: []corev1.Container{{
                    Name:  "compress",
                    Image: "registry.apk8s.dev/platform/compressor:v1",
                    Args:  []string{"--src=" + path, "--dest=processed"},
                }},
            },
        },
    },
}
_, err := clientset.BatchV1().Jobs("lake").Create(ctx, job, metav1.CreateOptions{})

GenerateName lets Kubernetes append a unique suffix, so concurrent objects never collide on a name. BackoffLimit: 2 retries a failed worker twice before the Job is marked failed and a human, or an agent, looks at it. TTLSecondsAfterFinished: 300 is the janitor: five minutes after completion, the Job and its pod are deleted, so a pipeline processing thousands of objects a day does not accumulate thousands of dead objects in etcd. RestartPolicy: Never makes each attempt a fresh pod, which keeps the retry semantics in the Job where they are visible, not in the kubelet where they are not.

The controller’s permissions are exactly one verb wide:

apiVersion: v1
kind: ServiceAccount
metadata: { name: object-processor, namespace: lake }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: job-creator, namespace: lake }
rules:
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: ["create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: object-processor-creates-jobs, namespace: lake }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: Role, name: job-creator }
subjects: [{ kind: ServiceAccount, name: object-processor }]

A Role with create on jobs in one namespace is the entire blast radius of a compromised controller. It cannot read secrets, cannot see other namespaces, cannot delete anything. Least privilege is cheapest when the component genuinely needs almost nothing.

§Operating the Pipeline

Scale is bounded by the cluster. Each object becomes a Job, and the scheduler runs as many concurrently as the nodes allow, so throughput is your own capacity rather than a provider’s concurrency limit. A burst of uploads becomes a burst of Jobs that drains as fast as the cluster can.

Make the work idempotent and retryable. Events can arrive more than once, and Jobs can fail and retry, so a worker should produce the same result if it runs twice on the same object. Writing the output under a deterministic key derived from the input path, together with the BackoffLimit, makes the pipeline safe to re-run, which is what you want from anything event-driven.

You can see all of it. The events flow through Kafka, so consumer lag tells you if processing is keeping up; the Jobs are Kubernetes objects, so their success and failure show in the same monitoring as everything else. Compared to a cloud function, where observability is whatever the provider exposes, the whole pipeline is in the open.

§When Something Is Wrong

No events arrive on the topic. The notification config. Confirm notification.toml is actually mounted where the filer reads it and that the filer restarted after the change; weed filer.meta.tail proving events exist while the topic stays empty points squarely at the notification config rather than the store.

The consumer prints binary garbage. It expected JSON. The payload is protobuf, decoded with filer_pb.EventNotification; this is the single most common surprise for anyone arriving from MinIO or AWS, and it is the reason the taxonomy section above exists.

The controller cannot create Jobs. RBAC. Its ServiceAccount’s Role must grant create on jobs; a controller that can read events but not create Jobs fails at the last step, and the API server’s Forbidden in the controller log says exactly which verb is missing.

Jobs fail to reach the object store. S3 credentials or endpoint, the same SeaweedFS details every consumer needs, passed to the worker as environment from a mounted secret.

Finished Jobs pile up. TTLSecondsAfterFinished is missing or was removed. Set it on the Job spec, and Kubernetes deletes completed Jobs after the window; without it, a busy pipeline leaves thousands of finished Job objects behind.

§A Pipeline an Agent Can Extend

This pipeline has a property worth designing for on purpose: every part of it fits in an AI agent’s working context. The controller is a hundred lines of Go, the worker is one function, the contract between storage and controller is one protobuf message, and this post plus filer.proto is the full specification. When the platform needs a second processor, images to thumbnails, logs to Parquet, the working session is short: the agent gets the controller source and the new requirement, and returns a new filter clause and a new worker, which then face the same gates all Go on this platform faces, the linters, the tests, and the review discipline of AI on a leash. Compare that with extending a cloud-function pipeline, where the agent would be reasoning about IAM policies, event source mappings, and per-provider limits it cannot see. Small, inspectable, self-hosted systems are not just cheaper to run; they are the systems agents extend reliably.

§What You Have

An event-driven processing pipeline on infrastructure you own: object storage that announces every change onto a durable queue, a hundred-line controller that turns the changes you care about into Kubernetes Jobs, and a hardened ten-megabyte worker per object doing the actual work. It is the capability of S3 plus Lambda, prototyped in a notebook and shipped as a Job, without a proprietary function runtime, a per-invocation bill, or any execution limit beyond your own cluster’s capacity.

Next I add the database for the data that does not fit Postgres, the write-heavy, always-available workloads that call for Cassandra.