In-place engine restart: the primitive behind multi-standby HA

The failover-testing post ended on a piece of work that was still exploratory, and a constraint it told you to live with. The work: tear down and rebuild the processing engine in the same process, without exiting the JVM, and resume from the last committed transactional offsets — an in-place restart. The constraint, stated plainly in its tradeoffs: two is the supported topology; stay at two. Both have moved. The in-place restart shipped, and a real leader election shipped on top of it — so scaling a StoatFlow HA cluster past two standbys, which used to crash-loop, now elects exactly one successor and holds.

Two changes, and they belong together. This is what they are, why the pair needed them, and what a live three-pod cluster does with them — measured on StoatFlow 1.0.0-rc.1, an exactly-once word-count deployment on Kubernetes.

TL;DR

• What: An in-place engine restart — rebuild the engine inside the live JVM, no process exit — and a leader election built on it that makes one active plus K warm standbys a supported topology.
• Why: Every role change used to wrap engine teardown in a process restart, so a graceful demote bounced the pod and paid a cold store open. And scaling the pair past two crash-looped: competing standbys promoted at once and the exactly-once fence resolved the losers by killing them.
• How: The restart swaps the engine instance in place and resumes from the last committed offsets. Promotion now goes through a token claimed before the fence, as the sole authority to promote; a standby that loses the claim stands down as a return value, not a crash.
• The guarantee: Exactly-once is unchanged — the same transactional.id epoch fence that protects a cross-pod handoff protects an in-place one, and it stays the backstop behind the election.
• The catch: Still exactly one active. K standbys are redundancy, not scale — and each one is another changelog reader. To go faster, scale up, not out.
• Docs: High availability.

In this post

• The process round-trip we set out to remove
• Restarting in place
• From a pair to a cluster
• A real election
• What the cluster does now
• Tradeoffs and limits
• Running it

The process round-trip we set out to remove

Look at the four ways a StoatFlow active can lose its role — a graceful /ha/switch, a SIGTERM from a rolling deploy, a JVM crash, a lost node — and one round-trip recurs underneath all of them: the active exits the process and lets Kubernetes restart it. A graceful demote self-terminated and came back as a fresh pod. A crash was a container restart. Even an in-place recovery was the orchestrator rebuilding the whole container. The engine teardown and re-initialisation always happened, but always wrapped in a process restart.

That wrapper is expensive in exactly the case you most want to be cheap. A /ha/switch is a planned, graceful handoff — it should cost about one commit. But if the demoted active has to exit and be rescheduled, it pays the full price of a cold start on the way back: a new container, a fresh JVM, and a RocksDB store that opens and replays the changelog before the pod is a warm standby again. On large state that is minutes, for an operation whose useful work took milliseconds. The failover-testing measurements named this as the last remaining cost on the fast paths — the promotion itself was already sub-second; what was left was the process round-trip around it.

Restarting in place

The in-place engine restart removes the wrapper. Instead of exiting, the instance swaps its engine: it tears down the old StreamProcessingEngine and builds a fresh one inside the same live process, then resumes from the last committed transactional offsets. The JVM never exits, the container is never rescheduled, and — the part that matters for recovery time — the state stores are never closed and reopened. The store registry is reused across the swap, so a fresh engine attaches to the already-open, already-warm RocksDB rather than cold-starting it.

Two dispositions cover the reasons an engine gets rebuilt. A graceful restart drains the lanes, broadcasts a final commit barrier, and commits the in-flight transaction before tearing down — the new engine resumes from a clean, committed boundary with nothing to reprocess. An abort restart throws the in-flight epoch away: it aborts the open transaction, tears down, and the fresh engine reprocesses the uncommitted tail. That is an in-process crash recovery, and it is what a transient processing fault gets — the faithful analogue of Kafka Streams' REPLACE_THREAD, which recycles the failed stream thread and keeps the application up. StoatFlow has no per-partition thread to recycle, so until now a fault that Kafka Streams would have shrugged off took the whole instance down. Now it rebuilds the engine and stays running, behind a fault-only budget that escalates to a real shutdown if restarts start looping.

Exactly-once holds across the swap the same way it holds across a pod restart. The fresh engine's transactional producer is fenced by its transactional.id epoch — the identical mechanism that fences a crashed pod's producer when a standby takes over — so an aborted epoch's writes can never reach a read_committed consumer, whether the engine that wrote them died with the JVM or was torn down inside it. The restart is scheduler-agnostic: the same primitive serves a transient fault, a hot-standby demote, and — later — dynamic lane re-tuning. Build it once, cleanly, and each of those gets faster.

From a pair to a cluster

Hot standby shipped as a pair: one active, one warm standby. One spare is enough to survive a lost node, but operators wanted more — a second spare so a rolling deploy never drops to zero redundancy, and headroom to lose a node during a deploy. The obvious move is to raise the replica count. It didn't work.

Scaling the pair to three replicas crash-looped. Two standbys would decide to promote at the same moment; the exactly-once fence did its job and let only one commit — but it resolved the losers by fencing their producers, which surfaced as ProducerFencedException, flipped them to ERROR, and let Kubernetes restart them straight back into the same race. The failover-testing post measured the pair and, in its tradeoffs, said exactly this: stay at two. On the live cluster the failure was not subtle — three pods logged 52, 53, and 53 restarts in twelve hours before the fix.

The root cause is worth stating precisely, because it explains the shape of the fix. StoatFlow had a strong fencing layer and a weak election layer. Fencing — the transactional producer epoch — guarantees at most one instance ever commits, and it is airtight. But it is a safety mechanism, not a selection one: when two standbys promote together, the fence decides the winner by who called initTransactions() last, which is a function of timing, not of which standby is furthest caught up. And its way of saying no to a loser is to kill it. With one standby there is never a second promoter, so the gap never showed. With two, the election collapsed onto the fence — and the fence elects by accident and resolves by casualty.

A real election

The fix is to add the election layer the pair never needed: pick the successor deliberately, and before anyone touches the fence.

Promotion now goes through a promotion token — a single record on the same compacted coordination topic the pair already uses, claimed by a compare-and-set. Winning the claim is the sole authority to promote: a standby reads the current holder, and if its claim goes through it proceeds to fence-and-restore; if it loses, it stands down — it becomes a standby again and waits, as an ordinary return value, with no fenced producer, no ERROR, no restart. That single reordering — claim the token, then fence — is what turns the crash-loop into a clean stand-down. The loser never reaches the fence, so the fence never has to kill it.

Who claims? The token is contended by the election winner, and the winner is chosen the way the hot-standby post said it would be — lag-aware, so the freshest standby wins. Candidates are ranked by replication lag; the most caught-up wins, and an exact tie breaks on a stable hash of the pod identity, so the choice always converges on one even with no coordinator to ask. Lag narrows the field to who should take over; the hash guarantees the field narrows to one; the token compare-and-set serialises the claim, so that even if two pods pick differently under a brief split view, only one wins and the other stands down.

The load-bearing rule is when a claimant may take the token from a holder that already has it. The answer: only if the holder is unprotected — either stale, its heartbeats gone quiet past the staleness window so it has likely crashed, or gracefully draining, on its way out and saying so. Against a live, healthy holder, a claimant stands down. That one rule does double duty. It lets a genuinely dead active's token be reclaimed quickly, and it lets a graceful demote hand off immediately: the demoting active publishes that it is draining, which marks its token takeable, so the designated successor claims it at once instead of waiting out the staleness clock. This is exactly where the in-place restart plugs in — a graceful /ha/switch now demotes the old active in place: it drains, drops the token, rebuilds its engine as a standby in the same pod, and rejoins warm. No exit, no reschedule, no cold store open.

The fence does not go away. It is still there, still airtight, still the thing that makes split-brain impossible under exactly-once. The token and the fence divide the work cleanly: the token is an availability mechanism — it bounds how often two pods ever contend to promote — and the fence is the correctness mechanism — it bounds what happens if they do. Correctness never depends on the token being perfect. If the election ever misfires, the fence still guarantees one committer; the token just means the loser learns it lost by reading a record instead of by being executed.

What the cluster does now

The proof is the same three-replica deployment, on the same cluster, with only the build changed. On the old image it logged 52, 53, and 53 restarts across the three pods in twelve hours. On the fixed image it logs zero — steady state is one ACTIVE and two READY_STANDBY, a single token holder, and a token epoch that ticks up once per election rather than climbing without bound.

Every way an active can lose its role was run against the three-pod cluster, under load, exactly-once, with the three pods co-located on one worker. In each, the election picked one successor and no pod ever reached ERROR or CrashLoopBackOff.

Two honesty notes on the numbers behind that table. The promotion critical path — from deciding to promote to serving — measured between roughly 0.75 and 1.8 seconds across these runs, but this is a CPU-constrained box: three engines and a load generator sharing one eight-core worker, heavier than the two-worker pair the earlier measurements ran on. Don't read these as faster than those — it is a different, tighter test. And the wall-clock on the ungraceful paths is dominated by things that are not the election: a crashed pod carrying twelve hours of state spends most of its recovery in RocksDB restore, and a lost node spends most of its stop-the-world in the ~7-second staleness detection window. The election itself is the fast part; the numbers around it are state size and detection, as they were for the pair.

The rolling upgrade is where the extra standby earns its place. Readiness is gated not only on being caught up but on redundancy: a caught-up standby reports ready to roll only if rolling it would still leave the active plus at least one other caught-up standby. So a three-pod roll takes one pod down at a time, highest ordinal first, the active last — and never drops below an active and a warm spare while it runs. The pair had no spare to preserve during a roll; a cluster does, and the roll refuses to spend it.

Tradeoffs and limits

The election and the in-place restart change how a cluster behaves; they do not change what StoatFlow is. The edges are worth stating plainly.

• It is still exactly one active. K standbys are redundancy, not throughput — each is a warm spare, none processes source records. This is the single-instance model intact: you don't pay the distribution tax for scale-out you don't need, and to go faster you give the active more cores, not more replicas.
• Each standby is another changelog reader. A standby stays warm by streaming the changelog the active writes, so K standbys multiply that read fan-out K-fold. ha.max-standbys bounds it — pick the redundancy you need, not the maximum you can spell.
• A busy active on a shared node has less headroom than it looks. The co-located drill surfaced this: three engines and a load generator on one worker, and a fan-out workload can starve the active of CPU until a commit misses its window and the exactly-once machinery times the transaction out. That now triggers a clean in-place restart rather than a hung pod — but it is still a restart. Give the active room; the in-place restart makes the failure graceful, not free.
• Node loss on a co-located cluster is a lost active, not a lost node. If every standby shares a worker with the active, losing that worker loses all of them. A standby is insurance against losing a node only if it lives on a different one — spread the replicas across nodes for real node-loss protection.
• The token is availability; the fence is correctness. The election reduces how often two pods contend to promote; it does not, and is not relied upon to, guarantee one committer. The transactional fence does that — under a network partition and under a misfiring election alike. Under at-least-once there is no fence, so the token is the whole story; treat it as bounding contention, not eliminating it.
• A ProducerFencedException on the demoting pod during a graceful switch is expected, not a fault. The successor fences the old active's producer while it drains; the demoted pod moves through restore and back to a warm standby without ever reaching ERROR. It is the fence doing its job on a handoff, not a crash.

Running it

The topology is a configuration change on the deployment you already run. Hot standby is still one knob — stoatflow.ha.mode: active-standby — and going past two is the replica count plus a small amount of intent: ha.desired-standbys sets the redundancy the readiness gate protects, ha.max-standbys caps the changelog fan-out, and ha.failover-priority nudges which equally-caught-up standby is preferred. The election, the token, the staleness thresholds, and the in-place demote all have working defaults; a typical deployment sets the mode and the replica count and leaves the rest.

The model is the one the hot-standby post described, now without its two footnotes. One active, warm spares, the election picks the freshest, the fence backstops — and a planned role change no longer bounces a pod or waits on a cold store. The two footnotes it shipped with — promotion will become lag-aware and stay at two — are both closed. The High availability guide has the configuration reference and the operator endpoints, and Exactly-once covers the guarantee the fence enforces. The failover drill in the repository runs every scenario in the table above against a live cluster, if you want the numbers on your own hardware rather than mine.