https://stoatflow.io/blog Engineering deep dives and product announcements from StoatFlow — stateful stream processing for Kafka. en Tue, 15 Sep 2026 00:00:00 GMT https://stoatflow.io/blog/three-clocks-stream-time-vs-watermarks https://stoatflow.io/blog/three-clocks-stream-time-vs-watermarks Tue, 15 Sep 2026 00:00:00 GMT Kafka Streams, Flink and StoatFlow answer "is this window done?" with three different clocks. StoatFlow runs two of them: it accepts a late record the way Kafka Streams does, on the operator's stream time, and closes the window the way Flink does, on a single global watermark. Part 1 of a four-part series on event time — including two corrections to a claim our own compatibility matrix made about grace. Hartmut Armbruster Deep Dive https://stoatflow.io/blog/scheduled-sources https://stoatflow.io/blog/scheduled-sources Sat, 29 Aug 2026 00:00:00 GMT Every source in a Kafka Streams topology is a topic. The calendar and your own state — a snapshot across keys, a sweep over what you hold — needed a job outside the topology or a punctuator bent into a source. StoatFlow's scheduled sources run your code on a schedule and turn what it forwards into an ordinary KStream: keyed, processed in parallel on the lanes, committed with the stream. What that opens up, how it works, two verified examples, and the catch — the emitter's view is read-only, a run does not yield to live traffic on its own, and a read from outside the engine is not in the transaction. Hartmut Armbruster Engineering https://stoatflow.io/blog/ci-runner-platform-case-study https://stoatflow.io/blog/ci-runner-platform-case-study Wed, 26 Aug 2026 00:00:00 GMT Our CI p90 hit 54 minutes and the constraint was RAM, not CPU — a faster runner was the wrong instinct. We built a self-hosted Hetzner fleet, validated it, chose Blacksmith on a head-to-head benchmark, then took another eight minutes off by deleting four lines of YAML. Four weeks on: the speed held, the cost model did not. Hartmut Armbruster Engineering https://stoatflow.io/blog/road-to-1-0-0 https://stoatflow.io/blog/road-to-1-0-0 Tue, 04 Aug 2026 00:00:00 GMT StoatFlow 1.0.0-rc.1 is cut and the feature set is frozen — what separates the candidate from GA is proof, not features. The twelve weeks from the first alpha: 26 releases, 1,086 commits, a compatibility matrix grown from 417 to 619 tracked entries, three full-codebase review rounds — and the framework integration that kept us honest. Hartmut Armbruster Announcement https://stoatflow.io/blog/error-handling-three-gates-one-transaction https://stoatflow.io/blog/error-handling-three-gates-one-transaction Mon, 27 Jul 2026 00:00:00 GMT A record can fail on the way in, in the middle, or on the way out. StoatFlow gives every gate the same verdicts — continue, fail, or dead-letter — and settles them all inside the exactly-once transaction: DLQ records commit on the same barrier as your output, and what cannot be handled kills the epoch, never the guarantee. The full model, from one bad record to a Kubernetes restart. Hartmut Armbruster Deep Dive https://stoatflow.io/blog/llms-txt-machine-readable-docs https://stoatflow.io/blog/llms-txt-machine-readable-docs Sun, 26 Jul 2026 00:00:00 GMT The StoatFlow documentation is now published as llms.txt, llms-full.txt, and raw markdown — the retrieval-side complement to the AI Assistant Skills pack. Nothing to install. Hartmut Armbruster Announcement https://stoatflow.io/blog/ai-assistant-skills https://stoatflow.io/blog/ai-assistant-skills Fri, 24 Jul 2026 00:00:00 GMT A free, open skills pack that makes AI coding assistants write correct StoatFlow instead of hallucinated Kafka Streams. Hartmut Armbruster Announcement https://stoatflow.io/blog/internal-consistency https://stoatflow.io/blog/internal-consistency Mon, 13 Jul 2026 00:00:00 GMT Money can only be moved, never created — so a stream that tracks balances should read total = 0 at every consistent cut. The Flink Table API gets it right 0.035% of the time; our Kafka Streams twin sends total to −1,619 … +1,792. StoatFlow holds it at exactly 0, at every one of its committed cuts. Here is why, and the measured proof. Hartmut Armbruster Deep Dive https://stoatflow.io/blog/in-place-restart-multi-standby https://stoatflow.io/blog/in-place-restart-multi-standby Thu, 02 Jul 2026 00:00:00 GMT The failover-testing post ended on an exploratory idea — rebuild the processing engine in the same process, no JVM exit — and a constraint: stay at two standbys. Both have moved. The in-place restart shipped, and a lag-aware leader election shipped on top, so a StoatFlow HA cluster now runs one active and any number of warm standbys, scaling past two elects exactly one successor instead of crash-looping, and a graceful role swap no longer bounces the pod. Hartmut Armbruster Engineering https://stoatflow.io/blog/obfuscation-and-the-public-api-boundary https://stoatflow.io/blog/obfuscation-and-the-public-api-boundary Wed, 01 Jul 2026 00:00:00 GMT A drop-in Kafka Streams DSL has to stay byte-stable while the engine beneath it stays hidden. Those two goals pull apart — and every interesting bug we hit while obfuscating StoatFlow lived on the seam between them. Hartmut Armbruster Deep Dive https://stoatflow.io/blog/ha-failover-testing https://stoatflow.io/blog/ha-failover-testing Sat, 27 Jun 2026 00:00:00 GMT The obvious way to test HA failover — kubectl delete pod — is a graceful SIGTERM handoff, not a crash. Here are the four real failover scenarios, the millisecond timing measured from pod logs (because a 10-second Prometheus scrape cannot resolve a one-second event), and what the numbers revealed: a single JVM crash recovers in place without failing over, and stop-the-world is not the latency blip. Hartmut Armbruster Engineering https://stoatflow.io/blog/hot-standby-high-availability https://stoatflow.io/blog/hot-standby-high-availability Wed, 17 Jun 2026 00:00:00 GMT StoatFlow recovery has always been fast restart — but restart time scales with state. Hot standby is an opt-in active/passive pair that fails over in seconds regardless of state size, with exactly-once preserved across the handoff. Redundancy, not scale-out. Hartmut Armbruster Announcement https://stoatflow.io/blog/kip-1271-record-headers-in-state-stores https://stoatflow.io/blog/kip-1271-record-headers-in-state-stores Sun, 14 Jun 2026 00:00:00 GMT KIP-1271/1285 let a state store keep a record's headers next to its value — shipped in Apache Kafka 4.3.0. What the KIP is, where Kafka Streams has got to (and what is still in flight), how StoatFlow 1.0.0 compares — and why a one-varint format change becomes an engine-wide one. Hartmut Armbruster Deep Dive https://stoatflow.io/blog/kip-1035-state-store-managed-offsets https://stoatflow.io/blog/kip-1035-state-store-managed-offsets Sat, 06 Jun 2026 00:00:00 GMT Kafka Streams kept each task's changelog offsets in a separate checkpoint file that could drift from the state it described. KIP-1035, in Kafka 4.3, moves them inside the state store — atomic with the data, and the keystone for transactional state stores. Hartmut Armbruster Deep Dive https://stoatflow.io/blog/native-image-g1-pgo-jni-vs-ffm https://stoatflow.io/blog/native-image-g1-pgo-jni-vs-ffm Sat, 06 Jun 2026 00:00:00 GMT Native image is viable for a throughput-first stream processor — but only with Oracle G1 + PGO. The wrong turn (we blamed FFM; it was the GC) and the reversal (FFM is faster on the JVM, slower under AOT) are the most interesting parts. Hartmut Armbruster Engineering https://stoatflow.io/blog/introducing-stoatflow https://stoatflow.io/blog/introducing-stoatflow Sun, 17 May 2026 00:00:00 GMT The first alpha is here. Kafka Streams DSL on a single-replica runtime with virtual-thread parallelism — measurably less CPU, memory, and latency on the same hardware. Hartmut Armbruster Announcement