Hot standby for StoatFlow: failover in seconds, not a cold restore

A StoatFlow application recovers by restarting: the process dies, the orchestrator brings it back, and local state rebuilds from the Kafka changelog up to the last committed barrier — no data loss, no duplicate output. That is fast, and for most workloads it is enough. But the rebuild reads the changelog, so recovery time scales with how much state there is. At tens of gigabytes, a cold restore can run into minutes.

Hot standby removes that scaling. It is an opt-in active/passive pair where a warm standby takes over in seconds, independent of state size — and under exactly-once, with no duplicates across the handoff. The full operator guide is on High availability; here is why it exists and what it gives you.

TL;DR

• What: Hot standby — an opt-in active/passive pair. Off by default; the single-instance runtime is byte-for-byte unchanged when you don't enable it.
• Why: Recovery has always been fast restart, but a cold rebuild from the changelog scales with state size. For large state or a tight recovery-time objective, that window is too long.
• How: A passive standby continuously follows the changelog and stays warm. Promotion is seconds, independent of state size — no cold restore on the critical path.
• Guarantee: Under exactly-once, broker-enforced transactional fencing makes split-brain impossible — at most one instance ever commits. Under at-least-once, failover is bounded-duplicate.
• The catch: It is redundancy, not scale. Still one active instance, roughly double the footprint. To go faster, scale up, not out.
• Docs: High availability.

In this post

• Why this exists
• What hot standby gives you
• Fast restart vs hot standby
• How it works
• Exactly-once across failover
• Tradeoffs and limits
• Running it
• When to turn it on

Why this exists

StoatFlow runs as exactly one instance per application. That is the whole point: no consumer-group rebalancing, no state migration, no repartition topics — you stop paying the distribution tax for scale-out you don't need. Availability in that model comes from fast restart, and fast restart is a credible story because there is no rebalancing to wait out: restart cost is restoration cost, full stop.

The cost, though, is bounded by state size. On small-to-medium state the engine is processing again quickly — the benchmarks measure cold-start across representative workloads, and a persistent volume narrows it further to just the changelog gap since the last commit. But a cold start still reads the changelog, and a large store takes longer to replay than a small one. For a workload carrying tens of gigabytes of state with a tight recovery-time objective, minutes of recovery on an unplanned restart is the line between acceptable and not.

When we shipped the first alpha, failover behaviour was a fair question we said we hadn't answered yet. This is the answer: an availability tier for the workloads that can't absorb a cold-restore window — without giving up the single-instance model, and without reintroducing the distribution tax. There is still exactly one active instance. There is still no rebalancing and no horizontal scale. There is now a warm spare.

What hot standby gives you

• Failover in seconds, independent of state size. The standby is already warm, so taking over is not a restore — it is a handoff. The cold-start window that scales with your state is exactly what this removes.
• Near-zero-downtime rolling deploys. A version roll costs about one graceful handoff: the active drains, commits, and hands over to the standby. No rebalance, no cluster-wide reconvergence.
• Exactly-once preserved across the handoff. The recovery anchor is the last committed barrier whether you restart cold or fail over to a standby. Under exactly-once there are no duplicates downstream for read_committed consumers.
• An order-independent rollout, with no custom controller. Readiness gates the roll: a standby reports ready only once it has caught up, and a catching-up standby is not killed mid-catch-up. Kubernetes never advances the roll onto an instance that isn't ready — whichever pod rolls first.
• Opt-in, with the default untouched. Hot standby is off unless you turn it on. Leave it off and you get exactly today's single-instance behaviour, with none of the extra moving parts.
• Observable and operable. A /ha/status endpoint reports each pod's role, replication lag, and the peers it sees; /ha/switch, /ha/promote, and /ha/demote drive a controlled handoff; and the pair exports stoatflow.ha.* metrics for dashboards and alerts.

Fast restart vs hot standby

Neither tier involves two active instances — that remains the rule. You choose per application.

How it works

At any moment one instance is the active — it owns the source partitions, processes records, and commits — and the other is a passive standby. The standby consumes the same changelog the active writes, keeping its own local state warm and a step behind; it does not process source records or produce output. The two coordinate through a compacted internal Kafka topic, so each knows the other is alive and which role it holds.

Promotion happens two ways. A graceful handoff — a rolling deploy or an explicit /ha/switch — drains the active, commits, and hands the role over. A failover — the active crashes — is detected when the standby stops seeing the active's heartbeats, and the standby promotes itself. Because it is already warm, taking over does not wait on a cold restore.

The safety property underneath is Kafka's own. Under exactly-once, a transactional producer is fenced by its epoch: if a network partition ever left two instances briefly believing they are active, the broker lets only one commit and fences the other, which then shuts itself down. Correctness does not depend on the failure detector being perfect — it depends on the fence, which the broker enforces.

Exactly-once across failover

How clean a failover is depends on the processing guarantee the application runs under — and the two behave differently enough to call out.

Under exactly-once (the default), failover is split-brain-proof. The transactional fence guarantees at most one instance can ever commit a given transaction, so the handoff produces no duplicates and loses no committed work. There is nothing extra to do downstream.

Under at-least-once, failover is bounded-duplicate. The pair still coordinates promotion safely, but without transactional fencing an ungraceful failover can re-process a small, bounded window of records, so a few duplicate output records are possible. If you run hot standby under at-least-once, make downstream consumers idempotent or duplicate-tolerant. The guarantee model is covered on Exactly-once.

Tradeoffs and limits

Hot standby is a deliberate trade, and it has edges worth stating plainly.

• It is redundancy, not scale. A warm spare does not add throughput. There is still exactly one active instance — to go faster, give it more cores and memory.
• Roughly double the footprint. A second always-on instance, its own persistent volume, and the licence that goes with it.
• At-least-once is bounded-duplicate, not split-brain-proof. Exactly-once is the guarantee that makes a handoff clean.
• Crash failover is not zero-downtime. It is bounded by detection plus a near-instant promotion — short, but not zero. A graceful handoff (deploy or /ha/switch) is about one handoff.
• Single region. Both instances must reach the same Kafka cluster; this is not a multi-region or active-active mechanism.
• The default is often the right answer. If a cold-restore window of seconds-to-minutes is acceptable, fast restart is simpler and cheaper. Hot standby exists to remove that window, not because the default is unsafe.

Running it

You deploy the pair as a two-replica StatefulSet — the same single-replica manifest you already use, with replicas: 2, a per-pod persistent volume, and a rolling update. The split /health/live and /health/ready probes are what make the roll order-independent, and a termination grace period long enough for the active to finish its handoff keeps deploys clean.

Turning it on is one configuration knob: stoatflow.ha.mode: active-standby. The rest — the coordination topic, the per-pod identity, the staleness and lag thresholds — has sensible defaults; on Kubernetes a typical deployment sets only the mode. High availability has the manifest, the configuration reference, and the operator endpoints; Running on Kubernetes covers the deployment shape it extends.

When to turn it on

Reach for hot standby when a cold-restore window is the thing you can't afford — large state, a tight recovery-time objective, or a deploy cadence where even a fast restart's gap shows up downstream. For everything else, the single-instance default remains the simpler, cheaper choice, and it is unchanged whether hot standby exists or not.

This is the first cut of the failover story, and it is built to get smarter: promotion selection becomes lag-aware so the freshest standby wins, and the readiness threshold becomes a tunable rather than a binary. The model — one active, a warm passive spare, the fence as the backstop — is the part that stays. Start with the High availability guide, and if you're running large-state stateful Kafka Streams in production today, that is exactly the workload this was built for.