How StoatFlow differs from Kafka Streams

If you know Kafka Streams, the fastest way to understand StoatFlow is by the differences. The DSL you write is the same — the same StreamsBuilder, the same KStream / KTable, the same operators. What changes is the runtime model underneath: how many processes run, where state lives, how parallelism is expressed, how re-keying happens, and how exactly-once is committed. This page lays out those deltas. For the model in its own right, start with Architecture; for porting an existing topology, see Migration.

What carries over unchanged: the DSL

The thing you spend most of your time on — the topology — is the part that does not change. StoatFlow implements the Kafka Streams DSL: StreamsBuilder, KStream, KTable, KGroupedStream, the windowed and session variants, the joins, the Processor API, the Consumed / Produced / Materialized / Grouped config objects, and the KS-compatible functional interfaces (ValueMapper, KeyValueMapper, ValueJoiner, Reducer, Aggregator, …). The full method-by-method status is in the Kafka Streams compatibility matrix.

Concretely, this map-filter topology is written the same way against StoatFlow as against Kafka Streams:

val intermediate =
    stream1
        .selectKey { _, _ -> "lala" }
        .map { key, value -> KeyValue(value.substring(0, 3), "$key:$value") }
        .filter { _, value -> value.length > 5 }
        .mapValues { value -> value.uppercase() }

intermediate.to(
    "output-topic",
    Produced.`as`<String, String>("sink1")
        .withKeySerde(Serdes.String())
        .withValueSerde(Serdes.String()),
)

KStream<String, String> intermediate =
    stream1
        .selectKey((k, v) -> "lala")
        .map((key, value) -> KeyValue.pair(value.substring(0, 3), key + ":" + value))
        .filter((k, value) -> value.length() > 5)
        .mapValues(value -> value.toUpperCase());

intermediate.to(
    "output-topic",
    Produced.<String, String>as("sink1")
        .withKeySerde(Serdes.String())
        .withValueSerde(Serdes.String()));

The imports differ — StoatFlow's DSL lives under io.stoatflow.core.topology.* (and the runtime under io.stoatflow.runtime.*) rather than org.apache.kafka.streams.* — and the entry point differs (covered below). The operators in between are the same shape, which is what makes a port mechanical rather than a rewrite.

There are deliberate API-surface differences where the single-instance model makes a Kafka Streams concept unnecessary or replaces it with something Flink-shaped — for example, event-time extraction is expressed as a Flink-style WatermarkStrategy on Consumed (the KS TimestampExtractor is still accepted and adapts onto one), and the partitioner settings on TableJoined are accepted as no-ops because there are no partition-bound tasks to route between. Those are catalogued in the Kafka Streams compatibility matrix; everything below is the conceptual why.

Single instance vs. multi-instance and rebalancing

This is the root difference; the rest follow from it.

Kafka Streams is a cluster. You run N instances, each with some number of stream threads. The instances form a Kafka consumer group, the group coordinator assigns tasks (each bound to an input partition) across the live members, and that assignment changes — rebalances — whenever an instance joins, leaves, or fails. Rebalancing is the mechanism that gives Kafka Streams its horizontal scaling and its failover, and it's also the source of much of its operational complexity: state has to migrate or restore on the new owner, processing pauses during the handoff, and you tune around it (standby replicas, static membership, cooperative rebalancing).

StoatFlow runs as exactly one active JVM process, assigned every partition of every source topic, with no group to rebalance. High availability comes from fast restart by default, with an opt-in hot-standby cluster (one active + warm standbys) for near-instant failover — still a single active instance, never a second active taking over a partition (ADR-001).

The trade-off is explicit: you give up open-ended horizontal scale-out and instead scale a single instance vertically with cores and memory. You cannot run two active replicas of the same StoatFlow application against the same source topics; the supported way to run more than one instance is the passive hot-standby cluster, which coordinates a single active explicitly. The reasoning behind accepting that trade is on Motivation; the consequences for operations are on Architecture.

Lane parallelism vs. partition-bound tasks

Because there's no cluster, parallelism is expressed differently.

In Kafka Streams, processing parallelism is capped by the partition count of the input topics: one task per partition, one stream thread runs one or more tasks. To process more in parallel you add partitions — a topic-level change with downstream consequences for every consumer of that topic. The unit of concurrency is the partition, and it's fixed by your topic layout.

In StoatFlow, the single consumer reads all partitions, and the engine then distributes work across lanes — key-affinity units of concurrent processing inside the JVM. The same key always routes to the same lane (so per-key order is preserved); different keys run on different lanes in parallel. Lane count is decoupled from partition count: you set it at startup, and it scales with CPU cores, not with how many partitions the source topic happens to have. Lanes run on virtual threads, so a lane blocked on a REST call or a database query parks at near-zero cost while other lanes make progress — which makes in-line blocking enrichment natural in a way the partition-bound model isn't.

The full treatment — choosing a lane count, why your keyspace caps the benefit, and the blocking-I/O story — is on Lanes and parallelism.

In-memory re-keying vs. repartition topics

When a topology changes a record's key — selectKey, groupBy, or a key-changing join — the record may need to move to a different unit of parallelism.

Kafka Streams handles this by writing the re-keyed record to an internal repartition topic and re-reading it on the other side, so that the new key lands on the correct partition (and therefore the correct task). That's a broker round-trip plus an extra serialize/deserialize per re-keyed record. You can see it in a Kafka Streams topology as an explicit or implicit repartition() boundary:

// Kafka Streams — re-keying forces a repartition topic round-trip
stream1
    .selectKey { _, _ -> "lala" }
    .map { key, value -> KeyValue(value.substring(0, 3), "$key:$value") }
    .repartition()          // ← writes to an internal topic, re-reads on the other side
    .filter { _, value -> value.length > 5 }

StoatFlow re-hashes the new key and hands the record to the lane that owns it in-memory, between lanes in the same process. There is no internal repartition topic, no broker hop, and no extra serialization round-trip — the same selectKey / map chain needs no repartition() call at all:

// StoatFlow — re-keying is an in-process handoff; no repartition() needed
stream1
    .selectKey { _, _ -> "lala" }
    .map { key, value -> KeyValue(value.substring(0, 3), "$key:$value") }
    .filter { _, value -> value.length > 5 }

// StoatFlow — re-keying is an in-process handoff; no repartition() needed
stream1
    .selectKey((k, v) -> "lala")
    .map((key, value) -> KeyValue.pair(value.substring(0, 3), key + ":" + value))
    .filter((k, value) -> value.length() > 5);

The output is identical to what Kafka Streams produces; the path is shorter (ADR-010). repartition() still exists in StoatFlow's DSL for source compatibility, but it's an in-memory operation rather than a topic round-trip. After any re-key, the new key's lane assignment again guarantees per-(new-)key ordering — the affinity property travels with the record across the handoff. The mechanism is detailed on Lanes and parallelism.

Barrier-based exactly-once vs. per-task transactions

Both systems deliver exactly-once over Kafka transactions; the scope of the transaction is what differs.

Kafka Streams coordinates exactly-once across the cluster. Each task commits its own work, and the framework manages the transactional producers and offset commits across all the tasks and instances participating — a distributed coordination problem, and historically a fiddly one to configure correctly.

StoatFlow uses a single commit barrier that flows through the entire topology. When the barrier completes, the runtime executes one Kafka transaction that atomically commits every state-store write (via changelog topics), every sink output record, and the consumer-group offsets for every contributing input partition — all three together, or none. Because there's one process, there's one barrier and one transaction covering the whole topology: no per-task transactions to coordinate, no cross-instance two-phase commit, no external checkpoint store. Exactly-once is the default rather than something you opt into and tune (ADR-004). The protocol is in the Chandy-Lamport family of distributed-snapshot algorithms — see Exactly-once.

The full mechanism, what "exactly-once" means in concrete terms here, crash recovery, and the at-least-once trade-off are on Exactly-once semantics. The barrier scheduling cadence and the transaction protocol itself are implementation concerns and stay in the source.

Global state vs. partition-scoped state

State placement follows directly from the single-instance model.

In Kafka Streams, state is partition-scoped: each task owns the state for its partitions, in a local store, and a key is only reachable from the task that owns it. This is why KTable-KTable joins require co-partitioning — both tables must be partitioned the same way so that matching keys land in the same task — and why interactive queries across a cluster need partition routing or RPC to reach the instance that holds a given key.

In StoatFlow, state is global: every store lives in the one JVM, and any processing lane can read or write any key. There's no partition-scoped isolation, no inter-instance lookup protocol, and no replication of the same data across JVMs. Two consequences fall out:

• No co-partitioning requirement. KTable-KTable joins work without aligning the tables' partitioning, because there are no partition-bound tasks to align (ADR-007). Foreign-key joins likewise don't need a co-partitioned subscription topic.
• No partition routing for queries. Interactive Queries reach any store directly — there's no withPartition(...) and no cross-instance RPC, because all state is locally accessible (ADR-025).

Correctness under concurrency is preserved by key affinity, not by partition isolation: each key is only ever touched by the single lane that owns it, so updates to one key are serialized while different keys update in parallel. The depth — why that's safe, the one cross-key case where you coordinate yourself, and the store types available — is on State and thread safety.

The entry point and the deployment unit

Two practical differences you hit immediately when porting.

Entry point. Where Kafka Streams takes a Topology plus a Properties and a KafkaStreams object you start(), StoatFlow has two front doors:

• StoatFlow.fromBuilder(config, builder) on the :core module — the DSL-and-engine entry point, configured with a typed StreamsConfig, with start() / awaitTermination() / close().
• StoatFlowRuntime.fromConfig(...) on the :runtime module — the batteries-included wrapper that loads application.yaml, starts the HTTP admin + metrics server and health endpoints, and runs your topology until terminated. This is what Your first app uses.

Deployment unit. A Kafka Streams deployment is a fleet you size by instance count and partition count. A StoatFlow deployment is a single pod you size by cores and memory — one active replica, scaled vertically. High availability is a fast, clean restart by default, or an opt-in hot-standby cluster where a warm passive standby takes over in seconds. The operational surface for that — health probes, metrics, the debug endpoints, and the failure policies — is described on Architecture.

The deltas at a glance

The full method-by-method parity table — including the handful of KS APIs that StoatFlow deliberately drops or replaces — is the Kafka Streams compatibility matrix.

Where to go next

• Architecture — the single-instance model the deltas above all follow from
• Lanes and parallelism — lanes, key affinity, and in-memory re-keying in full
• Exactly-once — the commit barrier and crash recovery
• State and thread safety — the global state model and co-partitioning-free joins
• Comparison matrix — feature-by-feature against Kafka Streams and Flink
• Migration — porting an existing Kafka Streams topology