From first alpha to release candidate: the road to StoatFlow 1.0.0

TL;DR

• What: StoatFlow 1.0.0-rc.1 is cut — the twenty-seventh release on the 1.0.0 line, and the one that freezes it: from here to general availability, bug fixes only.
• The journey: twelve weeks and 26 releases since the first alpha. Hot standby grew into multi-standby HA with in-place recovery, state stores became bounded, metrics became Kafka Streams–compatible, emission became internally consistent, a rejected output record stopped taking its epoch down with it, and porting gained a codemod, a state-migration tool, and an AI skills pack.
• Compatibility: KSML — an entire framework built on Kafka Streams — now has an experimental StoatFlow integration. A framework exercises what no single application does: every public class, constructor, join variant, handler, and config key. Working through its findings — and our own audits behind them — grew the compatibility matrix from 417 tracked entries to 619, almost all of it around the DSL, not in it. Credit where it is due: Jeroen van Disseldorp.
• The honest part: before freezing, we ran three full-codebase review rounds. They surfaced 4 critical and 36 high-severity findings; every one is fixed, and the later rounds re-audited the earlier fixes rather than taking them on trust.
• GA: no date. The candidate proves out against real workloads first — what remains is proof, not features.

StoatFlow 1.0.0-rc.1 is cut, and with it the 1.0 feature set is locked. From here to general availability the rule is strict: bug fixes only — no new features, no API changes, no configuration changes. The eight beta releases carried the last of it: a correctness campaign for the most part, and in the final two a handful of behaviour changes we would rather make before a freeze than after one. What separates the candidate from 1.0.0 is proof, not features.

This post is the account of the twelve weeks between the first alpha and this candidate: the journey in three acts, the framework integration that sharpened what "Kafka Streams–compatible" means here, the bugs we found because we went looking for them — and the ledger of limitations the launch post named, settled item by item.

Twelve weeks, three acts

Act one — make it port (alpha.1–alpha.13). Compatibility was the premise, not the project: at alpha.1 the compatibility matrix already tracked the DSL, the topology API and the state stores method by method, with nothing marked missing. The early alphas worked the level beneath that: the accessors, store suppliers, describe() output, interactive-query metadata, and exact overloads that real Kafka Streams code actually calls. Two milestones anchor the act. The in-memory test driver became a drop-in — constructed the Kafka Streams way, so that, as the alpha.11 notes put it, "a Kafka Streams test suite ports with an import swap". And configuration followed in alpha.13: build StoatFlow's configuration from a standard Kafka-Streams-keyed Properties or Map — "a config that works with Kafka Streams now configures StoatFlow with no changes". The same act shipped record headers in state stores with an in-place format upgrade, first-class Maven builds alongside Gradle, Flink-style watermark alignment, and GraalVM native-image support.

Act two — make it operable (alpha.14–alpha.18). With porting credible, the middle alphas turned to running the thing. Hot-standby high availability arrived as an opt-in active/passive pair, then hardened into multi-standby clusters with in-place recovery — an engine that rebuilds inside the live process instead of bouncing the pod. Windowed, session, sliding, cogroup, and join stores became bounded; the alpha.17 notes name the defect plainly — "these stores previously grew without limit" — and that is the act in one line: say what was wrong, fix it, move on. Idle exactly-once applications stopped churning empty commits (roughly 200x fewer idle transactions when quiet). RocksDB internals became observable, and an opt-in metrics mode publishes under the Kafka Streams metric names, so existing Grafana dashboards and alerts keep working. The act closed with alpha.18: barrier-consistent emission, a published OpenRewrite recipe that automates the code port, and licence validation that defers itself out of the way of CI runs.

Act three — make it correct (beta.1–beta.8). The betas were a different kind of work. The beta.1 note says it plainly: "a top-to-bottom internal review of the entire codebase drove a full pass of fixes, hardening exactly-once behaviour and Kafka Streams compatibility ahead of general availability." beta.2 is the correctness release — eleven named fixes plus a sweep of smaller ones, which the review section below returns to. beta.3 shipped the two halves of an assisted migration: the AI assistant skills pack and a data-migration tool that carries an existing application's state across the cutover. beta.4 added processor.wrapper.class — decorate every node in a topology, DSL-built and Processor API alike, through the same interface Kafka Streams uses — and published the documentation machine-readably as llms.txt. beta.5 turned an undiagnosed RocksDB startup failure into a supported, explained downgrade path for header-format stores — one Kafka Streams refuses outright — and closed a metrics audit: every meter documented, every documented meter real. And beta.6 closed the range-query surface: open-ended (null) bounds now behave across every store type the way Kafka Streams treats them, and reverseRange takes its bounds low-first the way Kafka Streams does — passed the other way round it had been quietly returning the wrong answer rather than complaining. The same release stopped the changelog topics behind headers-enabled window, session and versioned stores growing without limit; deployments already carrying one need a manual kafka-configs --alter to recover the space.

Then two more releases, and neither was polish. beta.7 stopped splitting a topology at every key change and started splitting it where Kafka Streams would — selectKey → mapValues → to() is one sub-topology now, not two — and made unprefixed Kafka client properties actually apply: a ported application's security.protocol and sasl.* had been recognised, warned about, and then ignored, so it connected unauthenticated and unencrypted. It also closed a race between closing a RocksDB store and flushing it that took the whole JVM down with a SIGSEGV — on the standby, which meant a promotion could kill the instance meant to replace the active one. beta.8 was the largest release on the line, and its centre is the poison-epoch rewrite: a record the broker rejects poisons the transaction it sits in, and StoatFlow used to abort that transaction and advance past it, losing every innocent record that happened to share the epoch. Now the source offsets are held, the epoch is replayed with the one bad record quarantined, and its epoch-mates commit normally. Alongside it, the dead letter for an oversized record became deliverable at all: it had been assembled larger than the record the broker had just refused, so a DLQ topic sized like the output topic refused it too — nothing was ever written, in precisely the case a DLQ exists for. Two more from that release deserve naming, because neither was visible from outside: an engine restart arriving mid-commit could break exactly-once, silently; and an at-least-once topology with no sink committed nothing at all — no offsets, no changelog — so every restart replayed its input from the beginning. The one deliberate tightening in the set: a Processor API processor may now only reach the stores it declared, where before it could quietly read someone else's.

The numbers underneath the acts, for the record:

One caution about reading that table: volume is evidence of effort, not of correctness. Nothing in a commit count says the engine is right. That claim belongs to the two sections that follow — the integration that tried to break compatibility from the outside, and the reviews that tried to break the engine from the inside.

The integration that kept us honest

The most useful thing that happened to StoatFlow in these twelve weeks was another engineer's test suite.

Early in the alpha, Jeroen van Disseldorp — CTO of Axual and creator of KSML — began integrating StoatFlow as an alternative engine underneath KSML. KSML is an open-source framework that builds Kafka Streams applications from YAML and Python, without writing Java. That is not porting an application: it means swapping the engine underneath an entire framework and running the framework's own test suite against the replacement. The integration is experimental — an honest effort to support StoatFlow as a second engine, and to test how far Kafka Streams portability really goes.

It is also the hardest compatibility exercise we could have asked for. A framework touches essentially the whole public surface: all the operators, not one topology's worth; every constructor and overload, because a framework cannot pick the convenient one; the exception handlers, the lifecycle state machine, the configuration surface down to individual keys, the serde boundaries, and the test harness its own suite runs on. KSML's dynamic data model reaches cases hand-written Java never produces — a key that is not null but serialises to null bytes, serdes chosen at runtime, topic names from a custom extractor. The findings arrived in rounds through June; our own porting audits kept pulling on the same threads well after.

The measure of that work is the public compatibility matrix. At alpha.1 it tracked 417 methods and behaviours across 62 API surfaces, and none was marked "not implemented" — the operator core was compatible on day one; that was the premise of the product. Twelve weeks later it pins 619 entries across 72 surfaces, and the growth sits exactly where a framework lives and an application port rarely looks: all 74 StreamsConfig keys of Kafka Streams 4.3 accounted for — mapped, passed through to the Kafka clients, or documented as moot on a single instance; the test harness pinned method by method, all green; topology description, interactive-query metadata, exception handlers, lifecycle, each from present to pinned. The early waves were about API shape — does Kafka Streams–idiom code compile? The later ones were about execution fidelity — does it behave, configure, observe, and test the same?

Some of what came out you have already met: the drop-in test driver and the Kafka Streams config model in act one both trace back to these rounds. The rest reshaped the surfaces around the DSL:

• Runtime behaviour. StreamsUncaughtExceptionHandler.REPLACE_THREAD stopped being a documented downgrade and became a real recovery — the in-place engine restart. Null keys are dropped before aggregation exactly as Kafka Streams drops them, across all nineteen aggregation operators.
• Handlers and interop. The error-handling surface follows the Kafka Streams handler shape; a custom KafkaClientSupplier uses the exact Kafka Streams signature; describe() prints the Kafka Streams topology description, character for character where tooling depends on it.
• Tooling. The port path itself got tooled: the OpenRewrite recipe rewrites the code, the migration tool carries the state, and the skills pack keeps AI assistants writing StoatFlow rather than half-remembered Kafka Streams.

The integration also exposed a process weakness we did not like: divergences were being discovered by a person, not by our CI. That is now guarded — build gates compile Kafka Streams–idiom Java against the artefacts we actually publish, so the next divergence has to get past a machine before it can reach anyone's port.

We went looking

Compatibility is one axis of confidence. The other is whether the engine is right — and for that we did not wait for reports.

Before freezing the surface we ran three full-codebase review rounds, each one reading the engine end to end against a fixed set of lenses: correctness, exactly-once semantics, thread safety, crash windows. Between them they surfaced four critical and thirty-six high-severity findings. Every one is fixed — and because a fix you have not re-verified is just a claim, the later rounds re-audited the earlier rounds' fixes rather than taking them on trust.

The beta.2 release notes are what that looked like from the outside: eleven named fixes plus a sweep. Three are worth retelling for what they teach:

• KTable-KTable join retractions. A non-materialised join did not forward the (key, null) retraction when one side was deleted — a downstream store or compacted topic kept the stale pre-delete join result forever. Correct-looking output, permanently wrong at rest.
• Versioned-store reads during commits. A versioned store read could return a stale older version for the whole commit window. The rewrite that fixed it also made the read path measurably faster — the fix and the optimisation were the same change.
• Foreign-key join consistency under exactly-once. When the two sides of a foreign-key join process concurrently, an intermediate update could — around a crash — go permanently missing from the output stream. Final values always converged, which is precisely why nobody would have noticed.

None of the three arrived as a field report. All of them came out of reviews that went looking — which is the point. A release candidate is not a claim that no bugs remain; it is the evidence of having gone looking, three times over, with the findings written down and closed.

The ledger from May

The launch post made claims, and — deliberately — named its own limitations. A release candidate should answer for both.

"State migration is a reprocess, not a restore" — closed. In May, the recommended path onto StoatFlow was to reprocess your input topics; if retention ruled that out, we asked you to get in touch. Now the migration tool translates a Kafka Streams application's changelog topics into StoatFlow's format and carries the input offsets across, so a stateful cutover resumes exactly where Kafka Streams stopped — reading the old application's topics, never modifying them.

"Failover behaviour is a fair question" — answered, with measurements. The launch post admitted it had no failover story. It now has one with numbers attached: hot standby, grown into multi-standby clusters with a lag-aware election and in-place recovery, and measured scenario by scenario from the pod logs. "The docs are still being written" no longer applies either — the documentation now runs from getting started through operations and reference.

The single-machine ceiling — stands. On the benchmarked 8-vCPU machine it is 200–300 MB/s of uncompressed throughput; higher-end hardware remains unbenchmarked. That number moved nowhere in twelve weeks because we spent them elsewhere. It is still the honest boundary of the design.

No horizontal scale-out — stands, by design. Twenty-six releases changed a great deal, and none of it touched the bet the product is built on: one instance per application, and no distribution tax for scale-out your workload does not need. Standbys are redundancy, not throughput. To go faster, scale up.

Two closed, two standing — and the two that stand are the product, not the gap.

What a release candidate means here — and what's next

Feature-locked is a discipline, not a mood. Between rc.1 and 1.0.0 the only changes are bug fixes; the API, the configuration surface, and the wire formats hold still. There is no GA date to announce, deliberately: the candidate now proves out against real workloads — ports, soak runs, failure drills — and 1.0.0 follows the evidence, not a calendar.

Locking 1.0 also frees the thinking past it. Three items lead the 1.1 line: Flink-style side outputs, so a processor can route to multiple named outputs instead of contorting a topology around one; an async I/O processor, for calling slow external services at high parallelism without giving up per-key ordering; and interactive queries over REST, reading state stores over HTTP without writing a server. Behind those sits a bench that is designed and reviewed but not scheduled: multi-table joins, CEP pattern recognition, read-only queries served from hot standbys. Directions, not commitments — the only commitment right now is the freeze.

Credits

A warm special thanks to Jeroen van Disseldorp, CTO of Axual, who built the experimental KSML integration and filed findings against us round after round. The compatibility half of this story owes its precision to those reports — the matrix is as sharp as it is because of them. If you want to see what a framework on top of Kafka Streams looks like, KSML is worth your time.

Thanks as well to everyone who tested StoatFlow, put it through an evaluation, or reviewed our work and told us what was wrong. rc.1 is better for it.

Read on:

• StoatFlow: Kafka Streams compatible engine built to scale up — not out — the first alpha, and the ledger this post settles.
• Hot standby, measuring failover, and in-place restart + multi-standby — the availability arc.
• Internal consistency on Kafka — barrier-consistent emission, measured to zero.
• Three gates, one transaction — the error-handling model.
• Migration — the tooled port path, code and state.
• Getting started.

For what happens between here and 1.0.0, reach out — and for the running commentary on how StoatFlow gets built, follow along on LinkedIn.