High availability

StoatFlow gives you two ways to stay available, and you choose per application. By default an application is a single instance that recovers by restarting — Kubernetes brings the pod back and state rebuilds from the changelog. For workloads that can't absorb that restart window, you can opt into a hot standby: a cluster of one active instance plus one or more warm standbys, where a standby takes over in seconds. The single-instance default is unchanged; hot standby is an opt-in upgrade on top of it.

The two HA tiers

Hot standby adds availability redundancy without changing the processing model, the exactly-once guarantees, or the single-active-instance shape. Everything the Architecture page describes still holds; you're keeping one or more warm spares ready, nothing more. The simplest cluster is one active and one standby — a two-replica StatefulSet, the default; add standbys for more redundancy (see desired-standbys).

How hot standby works

With ha.mode: active-standby you run two or more instances of the same application. At any moment one is the active — it owns the source partitions, processes records, and commits — and the rest are passive standbys. Each standby consumes the same changelog the active writes, keeping its own local state warm and a step behind, but it does not process source records or produce output. They coordinate through a compacted internal Kafka topic (auto-created; its name is configurable via stoatflow.ha.coordination-topic), exchanging a periodic heartbeat so each knows who is alive and which role they hold.

How the standby stays warm. A single changelog applier streams committed deltas into local state in real time — one consumer (in assign() mode, read_committed under exactly-once / read_uncommitted under at-least-once) and one virtual thread. It applies uniformly across RocksDB and in-memory stores and folds the derived-index rebuilds for foreign-key joins and suppression into the same loop, so the standby's state — primary stores and derived indices alike — tracks the active continuously rather than in periodic batches. Replication lag is read straight from that consumer's position, with no admin round-trip on the broker.

Promotion happens two ways:

• Graceful handoff — a rolling deploy, or an explicit POST /ha/switch. The active drains its in-flight work, commits, and hands the role over to an elected standby. This is an in-place role swap: the demoted active rebuilds its engine as a standby inside the same pod and rejoins warm — no process exit, no cold state open, no pod restart.
• Failover — the active stops heart-beating; the standbys notice, elect a successor, and it promotes itself.

Because the standby is already warm, taking over is a matter of seconds rather than a full cold restore. (A crash, a lost node, or a genuinely stuck restore still restarts the pod — the fence protects every path.)

The graceful path end to end — here triggered by an explicit POST /ha/switch; a rolling deploy runs the same sequence per pod:

Drain-to-LSO before processing. Whatever the trigger, a promoting pod first drains its applier to the committed end of the changelog — the log-stable-offset (LSO) — before it processes a single record. The departing active's producer is fenced eagerly, which freezes the LSO, so there is a fixed, finite target to drain to and no new committed deltas can appear past it. A caught-up standby drains almost nothing (it's already current); a standby that had fallen behind drains exactly the records it was missing — so it never resumes processing on stale state. This is what makes failover correct under exactly-once even when the standby was lagging at the moment the active died.

Electing the successor. With more than one standby, promotion is an election, not a free-for-all. Candidates are ranked by replication lag: a standby is eligible only while its total committed-changelog lag is at or below acceptable-recovery-lag, and the freshest eligible standby wins — an exact tie broken by a stable hash of the pod identity (nudged by failover-priority), so the choice always converges on exactly one. The winner claims a promotion token on the coordination topic before it fences the old active and restores; a standby that loses the claim simply stands down and stays a standby — no fence, no error, no restart. That ordering — elect and claim, then fence — is what stops two pods promoting at once in a multi-standby cluster. If the active is gone and no standby is within the threshold, the least-behind one is elected anyway after promotion-grace-ms (availability first) — still draining to the LSO before it processes. The producer-epoch fence stays the backstop: even if the election ever misfired, the broker still lets only one instance commit.

The same machinery under a crash — ungraceful failover from detection to takeover:

Processing guarantees under failover

How clean a failover is depends on the processing guarantee the application runs under.

• Exactly-once (the default). Failover is split-brain-proof. Kafka's transactional producer fencing — a broker-enforced guarantee — ensures at most one instance can ever commit a given transaction. If a network partition or a false failure-detection briefly leaves both instances believing they are active, only one can commit; the broker fences the other, which then shuts itself down. Downstream consumers reading read_committed see exactly-once output across the handoff: no duplicates, no lost work.
• At-least-once. Failover is bounded-duplicate. The cluster still coordinates promotion safely — the promotion token is itself the fence under at-least-once — but without transactional fencing an ungraceful failover can re-process a small, bounded window of records, so a few duplicate output records are possible.

Configuration

Hot standby is configured under stoatflow.ha. The only required key is mode; everything else has a sensible default.

stoatflow:
  application-id: my-app
  bootstrap-servers: kafka:9092
  ha:
    mode: active-standby      # off (default) | active-standby
    desired-standbys: 2       # 1 (default) = active/passive pair; here 1 active + 2 standbys
    max-standbys: 3           # ceiling on standbys (bounds changelog fan-out)
    # pod-id, coordination-topic, and the timing knobs all have sensible
    # defaults — a two-replica pair sets only mode.

Coordination topic

The cluster coordinates through one compacted internal topic (stoatflow.ha.coordination-topic, default __stoatflow_<applicationId>_ha). It carries the per-pod heartbeats and the promotion token (the compare-and-set that serialises the election). It is auto-created on startup; the application's Kafka principal needs Describe + Read + Write on it, plus Create if you rely on auto-creation.

Two hard requirements:

• Exactly one partition. The single-partition total order is load-bearing for the promotion-token compare-and-set — never create it with more than one partition.
• Exclusive to one application. The default name is applicationId-scoped for exactly this reason. If you override coordination-topic, give every application its own topic — sharing one across applications intermixes pod-status records and collides on the token, breaking the election and fencing.

Under at-least-once the topic is correctness-critical: the promotion token is itself the fence (ALO has no broker-enforced producer-epoch fence), so size its durability (replication factor, min.insync.replicas) accordingly. Under exactly-once the token is an availability layer — the election — and the fence is the producer epoch.

If your cluster denies topic auto-creation, pre-create the topic before starting the application. StoatFlow logs the exact required spec at ERROR and fails startup if the topic is missing and cannot be created:

Deploying the cluster on Kubernetes

Run the cluster as a StatefulSet with two or more replicas (not a Deployment), keeping everything from the single-replica Kubernetes manifest — the state volume, license injection, config — and adjusting these points:

• replicas: N — one active plus N-1 passive standbys. replicas: 2 is the classic pair (the default); the example below uses replicas: 3 for one active + two standbys.
• STOATFLOW__HA__DESIRED_STANDBYS / STOATFLOW__HA__MAX_STANDBYS — set alongside replicas when you run more than one standby (desired-standbys should be replicas - 1).
• updateStrategy: RollingUpdate — the roll is readiness-gated and order-independent (see below).
• volumeClaimTemplates — each pod gets its own persistent volume, so every standby holds warm state across restarts.
• Split /health/live and /health/ready probes.
• terminationGracePeriodSeconds ≥ your graceful-handoff budget (epoch drain + final commit), so a draining active finishes its handoff before SIGKILL.
• POD_NAME from the downward API → stoatflow.ha.pod-id.
• A headless Service (the StatefulSet's serviceName).

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: my-stream-app
spec:
  serviceName: my-stream-app     # headless Service
  replicas: 3                    # one active + two passive standbys (replicas: 2 = the default pair)
  updateStrategy:
    type: RollingUpdate          # readiness-gated; roll order doesn't matter
  selector:
    matchLabels:
      app: my-stream-app
  template:
    metadata:
      labels:
        app: my-stream-app
    spec:
      # Must exceed the graceful-handoff budget so a draining active
      # completes its handoff before SIGKILL.
      terminationGracePeriodSeconds: 60
      containers:
        - name: app
          image: my-registry/my-stream-app:latest
          ports:
            - name: http
              containerPort: 8080
          env:
            # Stable per-pod identity → stoatflow.ha.pod-id
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: STOATFLOW__HA__MODE
              value: "active-standby"
            - name: STOATFLOW__HA__DESIRED_STANDBYS
              value: "2"           # replicas - 1 (omit for the default pair)
            - name: STOATFLOW__HA__MAX_STANDBYS
              value: "3"
          readinessProbe:          # gates the rolling update
            httpGet: { path: /health/ready, port: http }
            periodSeconds: 5
            failureThreshold: 3
          livenessProbe:           # stays UP while a standby catches up
            httpGet: { path: /health/live, port: http }
            periodSeconds: 10
            failureThreshold: 6
          volumeMounts:
            - name: state
              mountPath: /var/lib/stoatflow/state
  volumeClaimTemplates:
    - metadata:
        name: state
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 100Gi        # ≥ active state size; the standby needs the same capacity

How readiness gates a rolling deploy

The split health probes are what make the rolling update safe and order-independent.

Because the standby reports ready only after it has caught up, Kubernetes won't move on to the next pod in the roll until the one it just restarted is a caught-up standby. And because a still-catching-up standby keeps its liveness probe UP, Kubernetes won't kill it mid-catch-up. The result is a safe rolling deploy with no custom controller — whichever pod rolls first, the cluster never cuts over to an instance that isn't ready. See Probes and Health checks.

The redundancy floor. With more than one standby, readiness is gated on redundancy as well as catch-up: a caught-up standby reports ready to roll only if rolling it would still leave the active plus at least one other caught-up standby (governed by desired-standbys). So a rolling deploy takes one pod down at a time, standbys first and the active last, and never drops below an active and a warm spare while it runs. A two-replica pair has no other spare to preserve, so this reduces to the pair's behaviour.

Operating the cluster

When HA is enabled, each pod exposes a small set of HTTP endpoints (in-cluster). They are not registered when ha.mode: off.

curl -s localhost:8080/ha/status

{
  "selfPodId": "my-stream-app-0",
  "selfRole": "ACTIVE",
  "selfState": "ACTIVE",
  "replicationLagRecords": 0,
  "replicationLagMs": 0,
  "tailFreshnessMs": 180,
  "tailCaughtUp": true,
  "restartRequired": null,
  "readyStandbyCount": 2,
  "desiredStandbys": 2,
  "redundancyBelowDesired": false,
  "tokenEpoch": 1,
  "tokenHolder": "my-stream-app-0",
  "peers": [
    {
      "podId": "my-stream-app-1",
      "role": "STANDBY",
      "state": "READY_STANDBY",
      "replicationLagRecords": 1240,
      "replicationLagMs": 85,
      "failoverPriority": 0,
      "generation": 7,
      "freshnessMs": 420
    },
    {
      "podId": "my-stream-app-2",
      "role": "STANDBY",
      "state": "READY_STANDBY",
      "replicationLagRecords": 980,
      "replicationLagMs": 70,
      "failoverPriority": 0,
      "generation": 7,
      "freshnessMs": 310
    }
  ]
}

The write endpoints are thin shims: they publish a command and return 202 Accepted with the command's offset as an idempotency token; the targeted pod observes it and acts. promote and demote require a ?pod=<id> target; switch targets whichever pod is currently active.

To prevent a self-inflicted outage, switch/promote/demote are rejected with 409 Conflict unless a caught-up (READY_STANDBY) target exists. Override with ?force=true when you deliberately want to hand off to a behind pod — a forced promotion still fully restores to the committed changelog before processing (correctness is never traded; force only skips the readiness check).

curl -X POST localhost:8080/ha/switch
# { "command": "SWITCH", "target": null, "force": false, "commandOffset": 42 }

curl -X POST 'localhost:8080/ha/promote?pod=my-stream-app-1&force=true'
# { "command": "PROMOTE", "target": "my-stream-app-1", "force": true, "commandOffset": 43 }

tailFreshnessMs / tailCaughtUp report how current this pod's view of the coordination topic is. A pod whose tail has gone stale (tailCaughtUp: false) defers failover decisions until its view is current again — correctness is unaffected (the fence is the split-brain backstop), but a persistently stale tail on a pod is worth investigating.

Every role change is recorded on the coordination topic, so its full history can be replayed with a console consumer if you need an audit trail. See REST API for the complete endpoint reference.

Metrics and what to alert on

The runtime exposes HA metrics on /metrics (Prometheus) when HA is enabled.

A healthy cluster shows exactly one pod with role = 1, its standbys' replication lag hovering near zero, peers = N-1, and redundancy.below.desired = 0. See Metrics and Observability for the full surface.

Failover characteristics

• Graceful (rolling deploy or /ha/switch): about one handoff — the active drains and commits, an elected standby promotes, and the old active rebuilds as a standby in place (no pod restart). No reprocessing under exactly-once.
• Ungraceful (the active crashes): bounded by detection (staleness-misses × heartbeat-ms) plus a near-instant promotion, since the elected standby is already warm. This is independent of state size — the cold-restore window is exactly what hot standby removes.

Segmented window/session stores run with the WAL off and would otherwise be durable only on a clean close(). A periodic checkpoint bounds how much of the changelog a pod re-applies after an ungraceful restart — and how much a promoting standby drains — to at most one checkpoint interval. Tune it with stoatflow.state.segment-checkpoint-records (default 100000) and stoatflow.state.segment-checkpoint-ms (default 30000), whichever fires first; see the configuration reference.

For measured cold-start numbers on the default (fast-restart) tier — the cost you avoid with hot standby — see Benchmarks.

Limitations and what to consider

• It's redundancy, not scale. Hot standby keeps warm spares; it does not add throughput. There is still exactly one active instance. To go faster, scale the active vertically.
• More standbys, more footprint and more changelog reads. Each standby is another always-on instance with its own persistent volume, and another consumer streaming the changelog — so K standbys multiply that read fan-out K-fold. max-standbys bounds it; pick the redundancy you need, not the maximum. The standbys run under the same application licence — they don't add seats.
• The standbys are not queryable. While passive a standby only mirrors the changelog into local state; interactive queries (readOnlyStore, the state HTTP endpoints) are served by the active alone. A standby answers reads only once it has promoted.
• Give the active CPU headroom. A busy active under a fan-out workload needs its cores; standbys co-located with the active on one node compete for them, and a starved active can miss a commit window (handled cleanly, but it's still a restart). Spread the pods across nodes, or size the node for the whole cluster.
• An extra internal topic. The coordination topic is auto-created and compacted; account for it when planning topics and ACLs. See Coordination topic for the principal's permissions, the single-partition and exclusive-per-application requirements, and the pre-create spec for locked-down clusters.
• At-least-once is bounded-duplicate, not split-brain-proof — see Processing guarantees under failover.
• Increasing a source topic's partition count requires a restart of the cluster; StoatFlow surfaces this via stoatflow.ha.restart.required and the liveness probe, so Kubernetes recycles the pod.
• Single region. Every pod must reach the same Kafka cluster; hot standby is not a multi-region or active-active mechanism.
• The default is often the right answer. If a restart-window gap is tolerable, fast restart is simpler and cheaper.

Next steps

• Kubernetes — the full single-replica manifest this page extends.
• Probes — liveness versus readiness and how the readiness gate works.
• Observability — scraping metrics and what to alert on.
• Exactly-once — the guarantee that makes failover split-brain-proof.