Posts tagged “Engineering”

Jul 2, 2026

In-place engine restart: the primitive behind multi-standby HA

The failover-testing post ended on an exploratory idea — rebuild the processing engine in the same process, no JVM exit — and a constraint: stay at two standbys. Both have moved. The in-place restart shipped, and a lag-aware leader election shipped on top, so a StoatFlow HA cluster now runs one active and any number of warm standbys, scaling past two elects exactly one successor instead of crash-looping, and a graceful role swap no longer bounces the pod.

Jun 27, 2026

Measuring StoatFlow failover: four scenarios, from the logs

The obvious way to test HA failover — kubectl delete pod — is a graceful SIGTERM handoff, not a crash. Here are the four real failover scenarios, the millisecond timing measured from pod logs (because a 10-second Prometheus scrape cannot resolve a one-second event), and what the numbers revealed: a single JVM crash recovers in place without failing over, and stop-the-world is not the latency blip.

Jun 6, 2026

Compiling StoatFlow to a GraalVM native image: G1, PGO, and why JNI beats FFM under AOT

Native image is viable for a throughput-first stream processor — but only with Oracle G1 + PGO. The wrong turn (we blamed FFM; it was the GC) and the reversal (FFM is faster on the JVM, slower under AOT) are the most interesting parts.