Running on Kubernetes

StoatFlow runs as exactly one active JVM process, so on Kubernetes you deploy it as a single-replica StatefulSet — not a Deployment. A StatefulSet gives you a stable identity, a per-pod persistent volume for the state directory, and an update strategy that tears the old pod down before the new one starts, which is exactly what running one active instance requires. (For a hot-standby cluster, see High availability — a readiness-gated StatefulSet of two or more replicas.)

Why a StatefulSet, not a Deployment

A Deployment with replicas: 1 and the default RollingUpdate strategy will, during a rollout, start the new pod before terminating the old one (maxSurge: 1). For a moment two processes are alive and pointed at the same source topics — which is precisely the data-corruption scenario the engine cannot tolerate. (Hot standby is the supported way to run more than one replica: the standbys are passive, and a RollingUpdate is readiness-gated so they never process concurrently — see High availability.)

A StatefulSet with replicas: 1 does the opposite: on an update it terminates the old pod and waits for it to fully stop before creating the replacement. Combined with podManagementPolicy: OrderedReady, you get an at-most-one-running guarantee across rollouts. That, plus a stable PVC for the state directory, is why StatefulSet is the right primitive here.

The state volume

State stores are RocksDB-backed on local disk under stoatflow.state.dir (default ${java.io.tmpdir}/stoatflow). The state directory is durable, not authoritative — the changelog topics are the source of truth, and on restart the runtime rebuilds from them. A persistent volume doesn't change correctness; it changes restart speed. With a surviving state directory the runtime replays only the changelog records since the last commit (a delta restore) instead of reading the whole changelog from scratch.

Mount a PVC at a fixed path and point stoatflow.state.dir at it:

# In the StatefulSet pod spec
volumeMounts:
  - name: state
    mountPath: /var/lib/stoatflow/state

# In the StatefulSet spec
volumeClaimTemplates:
  - metadata:
      name: state
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: fast-ssd   # SSD/NVMe — RocksDB is latency-sensitive
      resources:
        requests:
          storage: 100Gi

Then set the directory to that mount, either in application.yaml:

stoatflow:
  state:
    dir: /var/lib/stoatflow/state

…or via the env-var override (double underscore separates path segments):

STOATFLOW__STATE__DIR=/var/lib/stoatflow/state

Graceful shutdown and the grace period

On SIGTERM (which Kubernetes sends on pod termination) the runtime stops accepting new records, drains in-flight work, injects one final commit barrier, commits it, and closes the consumer and producer cleanly. That final commit is what keeps the shutdown exactly-once — interrupting it just means the next start resumes from the previous barrier, but letting it finish avoids re-processing on restart.

The runtime bounds its own graceful shutdown with stoatflow.shutdown.timeout-ms (default 25000). Kubernetes bounds it independently with terminationGracePeriodSeconds (default 30): when that elapses, Kubernetes sends SIGKILL regardless of progress.

Align the two so Kubernetes doesn't kill a still-progressing shutdown. Set terminationGracePeriodSeconds comfortably above your shutdown timeout in seconds:

spec:
  template:
    spec:
      terminationGracePeriodSeconds: 60   # > stoatflow.shutdown.timeout-ms / 1000, with margin

stoatflow:
  shutdown:
    timeout-ms: 25000   # default; raise for very large in-flight epochs

If your topology runs large epochs or slow stateful commits, raise both together — stoatflow.shutdown.timeout-ms first, then terminationGracePeriodSeconds so it stays larger. The runtime also has a hard-exit fallback that guarantees a clean process exit if shutdown itself hangs on a stuck lock (stoatflow.shutdown.hard-exit-on-shutdown-hang, default true); leave it on so Kubernetes always observes a clean termination rather than relying solely on SIGKILL. See Configuration reference for the full shutdown key set.

Secure license injection

StoatFlow validates a license at startup, so the pod needs the key — but the key is a secret and must never land in the image, a ConfigMap, or source control. Store it in a Kubernetes Secret and surface it one of two ways. Both map onto the license resolution order: an explicit key (env var) beats a key file.

First, create the Secret:

kubectl create secret generic stoatflow-license \
  --from-literal=license-key="key/...your production key..."

Option A — Secret as an environment variable

The simplest approach: project the secret value into STOATFLOW_LICENSE_KEY. The runtime reads the env var directly.

# In the container spec
env:
  - name: STOATFLOW_LICENSE_KEY
    valueFrom:
      secretKeyRef:
        name: stoatflow-license
        key: license-key
  # Production REQUIRES an explicit environment label — set it per deployment:
  - name: STOATFLOW_LICENSE_ENVIRONMENT
    value: prod-eu-west

Option B — Secret mounted as a file

Mount the secret as a file and point STOATFLOW_LICENSE_FILE at it. Useful when your platform standardises on mounted secrets over env vars.

# In the container spec
env:
  - name: STOATFLOW_LICENSE_FILE
    value: /etc/stoatflow/license/license.key
  - name: STOATFLOW_LICENSE_ENVIRONMENT
    value: prod-eu-west
volumeMounts:
  - name: license
    mountPath: /etc/stoatflow/license
    readOnly: true

# In the pod spec
volumes:
  - name: license
    secret:
      secretName: stoatflow-license
      items:
        - key: license-key
          path: license.key
      defaultMode: 0400   # owner read-only

Resource sizing

StoatFlow scales vertically — give the one process the CPU, memory, and disk it needs. As a starting point:

Memory splits roughly into JVM heap (state cache), off-heap (RocksDB block cache, write buffers, producer buffers), and OS overhead — budget for all three, not just the heap. Set container requests equal to limits for the Guaranteed QoS class, so the kernel doesn't reclaim memory from the process under node pressure. These are starting points; Tuning covers how to refine lane count, commit cadence, and memory caps against your workload.

A representative manifest

A complete single-replica StatefulSet: ConfigMap for application.yaml, Secret-injected license, persistent state volume, grace period aligned with shutdown, and both health probes. Adjust image, topic/broker config, resources, and the license-injection option to your environment.

apiVersion: v1
kind: ConfigMap
metadata:
  name: stoatflow-config
data:
  application.yaml: |
    stoatflow:
      application-id: my-stream-app
      bootstrap-servers: kafka-broker:9092
      state:
        dir: /var/lib/stoatflow/state
      shutdown:
        timeout-ms: 25000
      license:
        # The literal key is injected from a Secret via STOATFLOW_LICENSE_KEY (below);
        # the explicit env var wins over anything in this file.
        environment: prod-eu-west
    runtime:
      http:
        enabled: true
        port: 8080
      metrics:
        enabled: true
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: my-stream-app
spec:
  replicas: 1                      # exactly one — never more
  serviceName: my-stream-app
  podManagementPolicy: OrderedReady
  updateStrategy:
    type: RollingUpdate            # StatefulSet rolling update stops the old pod before starting the new
  selector:
    matchLabels:
      app: my-stream-app
  template:
    metadata:
      labels:
        app: my-stream-app
    spec:
      terminationGracePeriodSeconds: 60   # > stoatflow.shutdown.timeout-ms / 1000, with margin
      containers:
        - name: app
          image: my-registry/my-stream-app:latest
          ports:
            - name: http
              containerPort: 8080
          env:
            - name: STOATFLOW_LICENSE_KEY
              valueFrom:
                secretKeyRef:
                  name: stoatflow-license
                  key: license-key
            # application.yaml is loaded from the mounted ConfigMap:
            - name: STOATFLOW_CONFIG_FILES
              value: /etc/stoatflow/application.yaml
          readinessProbe:
            httpGet:
              path: /health/ready
              port: http
            initialDelaySeconds: 30
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health/live
              port: http
            initialDelaySeconds: 60
            periodSeconds: 30
          resources:
            requests:
              cpu: "16"
              memory: 64Gi
            limits:
              cpu: "16"
              memory: 64Gi          # match requests → Guaranteed QoS
          volumeMounts:
            - name: config
              mountPath: /etc/stoatflow
              readOnly: true
            - name: state
              mountPath: /var/lib/stoatflow/state
      volumes:
        - name: config
          configMap:
            name: stoatflow-config
  volumeClaimTemplates:
    - metadata:
        name: state
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: fast-ssd
        resources:
          requests:
            storage: 100Gi

Verify the rollout

After applying the manifests, confirm the single pod comes up and reports ready:

kubectl rollout status statefulset/my-stream-app
kubectl get pods -l app=my-stream-app          # exactly one pod, 1/1 Running

# Port-forward the HTTP port and check readiness:
kubectl port-forward statefulset/my-stream-app 8080:8080 &
curl -s localhost:8080/health/ready            # {"status":"UP", ...} once state restoration completes

During cold start /health/ready returns 503 until state restoration finishes — that's expected and is exactly why the readiness probe gates traffic. On a rolling update, watch that the old pod reaches Terminating and stops before the replacement is created; the StatefulSet guarantees this ordering, but it's worth confirming on first deploy.

The rollout you should observe, end to end — for the multi-replica hot-standby roll, where readiness additionally enforces the redundancy floor, see High availability:

Next steps

• High availability — the opt-in hot-standby cluster: a multi-replica StatefulSet for near-instant failover.
• Probes — liveness vs. readiness, the JSON each returns, and how to tune probe timings.
• Observability — scraping /metrics, the key metrics to alert on, and structured logs.
• Tuning — lane count, commit cadence, and memory caps for your workload.
• Production checklist — everything to verify before going live.
• License configuration — the full license reference, including CI/CD.