StoatFlow

Blog

Engineering notes from the StoatFlow team — launch announcements, design choices, benchmarks, and product updates.
Engineering

Scheduled sources: a topology source that isn't a topic

Every source in a Kafka Streams topology is a topic. The calendar and your own state — a snapshot across keys, a sweep over what you hold — needed a job outside the topology or a punctuator bent into a source. StoatFlow's scheduled sources run your code on a schedule and turn what it forwards into an ordinary KStream: keyed, processed in parallel on the lanes, committed with the stream. What that opens up, how it works, two verified examples, and the catch — the emitter's view is read-only, a run does not yield to live traffic on its own, and a read from outside the engine is not in the transaction.

Announcement

From first alpha to release candidate: the road to StoatFlow 1.0.0

StoatFlow 1.0.0-rc.1 is cut and the feature set is frozen — what separates the candidate from GA is proof, not features. The twelve weeks from the first alpha: 26 releases, 1,086 commits, a compatibility matrix grown from 417 to 619 tracked entries, three full-codebase review rounds — and the framework integration that kept us honest.

Deep Dive

Three gates, one transaction: error handling in StoatFlow

A record can fail on the way in, in the middle, or on the way out. StoatFlow gives every gate the same verdicts — continue, fail, or dead-letter — and settles them all inside the exactly-once transaction: DLQ records commit on the same barrier as your output, and what cannot be handled kills the epoch, never the guarantee. The full model, from one bad record to a Kubernetes restart.

Deep Dive

Internal consistency on Kafka: emitting a correct answer at every commit

Money can only be moved, never created — so a stream that tracks balances should read total = 0 at every consistent cut. The Flink Table API gets it right 0.035% of the time; our Kafka Streams twin sends total to −1,619 … +1,792. StoatFlow holds it at exactly 0, at every one of its committed cuts. Here is why, and the measured proof.

Engineering

In-place engine restart: the primitive behind multi-standby HA

The failover-testing post ended on an exploratory idea — rebuild the processing engine in the same process, no JVM exit — and a constraint: stay at two standbys. Both have moved. The in-place restart shipped, and a lag-aware leader election shipped on top, so a StoatFlow HA cluster now runs one active and any number of warm standbys, scaling past two elects exactly one successor instead of crash-looping, and a graceful role swap no longer bounces the pod.

Engineering

Measuring StoatFlow failover: four scenarios, from the logs

The obvious way to test HA failover — kubectl delete pod — is a graceful SIGTERM handoff, not a crash. Here are the four real failover scenarios, the millisecond timing measured from pod logs (because a 10-second Prometheus scrape cannot resolve a one-second event), and what the numbers revealed: a single JVM crash recovers in place without failing over, and stop-the-world is not the latency blip.