StoatFlow

Posts tagged “Engineering”

All posts

Scheduled sources: a topology source that isn't a topic

Every source in a Kafka Streams topology is a topic. The calendar and your own state — a snapshot across keys, a sweep over what you hold — needed a job outside the topology or a punctuator bent into a source. StoatFlow's scheduled sources run your code on a schedule and turn what it forwards into an ordinary KStream: keyed, processed in parallel on the lanes, committed with the stream. What that opens up, how it works, two verified examples, and the catch — the emitter's view is read-only, a run does not yield to live traffic on its own, and a read from outside the engine is not in the transaction.

Cutting CI from 50 minutes to 16: choosing a runner platform

Our CI p90 hit 54 minutes and the constraint was RAM, not CPU — a faster runner was the wrong instinct. We built a self-hosted Hetzner fleet, validated it, chose Blacksmith on a head-to-head benchmark, then took another eight minutes off by deleting four lines of YAML. Four weeks on: the speed held, the cost model did not.

In-place engine restart: the primitive behind multi-standby HA

The failover-testing post ended on an exploratory idea — rebuild the processing engine in the same process, no JVM exit — and a constraint: stay at two standbys. Both have moved. The in-place restart shipped, and a lag-aware leader election shipped on top, so a StoatFlow HA cluster now runs one active and any number of warm standbys, scaling past two elects exactly one successor instead of crash-looping, and a graceful role swap no longer bounces the pod.

Measuring StoatFlow failover: four scenarios, from the logs

The obvious way to test HA failover — kubectl delete pod — is a graceful SIGTERM handoff, not a crash. Here are the four real failover scenarios, the millisecond timing measured from pod logs (because a 10-second Prometheus scrape cannot resolve a one-second event), and what the numbers revealed: a single JVM crash recovers in place without failing over, and stop-the-world is not the latency blip.