Migrating from Kafka Streams

Moving a Kafka Streams application to StoatFlow is mostly a port, not a rewrite: the part you spend the most time on — the topology — carries over largely unchanged, and what changes is the runtime model around it. This page frames the migration conceptually: what stays the same, what differs operationally, and how to choose between starting clean and carrying state across. The two sub-pages then walk the concrete steps for each path.

What carries over unchanged

StoatFlow implements the Kafka Streams DSL. The topology you already wrote — StreamsBuilder, KStream / KTable / KGroupedStream, the windowed and session variants, the joins, the Processor API, and the KS-compatible functional interfaces (ValueMapper, KeyValueMapper, ValueJoiner, Reducer, Aggregator, …) — is the same code on both engines. The method-by-method status is in the KS compatibility matrix.

The same map-filter chain is written the same way against either engine:

val intermediate =
    stream1
        .selectKey { _, _ -> "lala" }
        .map { key, value -> KeyValue(value.substring(0, 3), "$key:$value") }
        .filter { _, value -> value.length > 5 }
        .mapValues { value -> value.uppercase() }

intermediate.to(
    "output-topic",
    Produced.`as`<String, String>("sink1")
        .withKeySerde(Serdes.String())
        .withValueSerde(Serdes.String()),
)

KStream<String, String> intermediate =
    stream1
        .selectKey((k, v) -> "lala")
        .map((key, value) -> KeyValue.pair(value.substring(0, 3), key + ":" + value))
        .filter((k, value) -> value.length() > 5)
        .mapValues(value -> value.toUpperCase());

intermediate.to(
    "output-topic",
    Produced.<String, String>as("sink1")
        .withKeySerde(Serdes.String())
        .withValueSerde(Serdes.String()));

Two things shift even on the unchanged-code path:

• Imports. StoatFlow's DSL lives under io.stoatflow.core.topology.* (and the runtime under io.stoatflow.runtime.*) rather than org.apache.kafka.streams.*. This is a package rename across your topology source, not a logic change.
• Entry point. Where Kafka Streams takes a Topology plus a Properties and a KafkaStreams object you start(), StoatFlow has two front doors — StoatFlow.fromBuilder(config, builder) on :core (DSL and engine only) and StoatFlowRuntime.fromConfig(...) on :runtime (the batteries-included wrapper that loads application.yaml and starts the HTTP admin, metrics, and health endpoints). See Modules overview.

What changes — and where it bites

The DSL is stable; the engine underneath is a different shape. None of these require topology rewrites, but they change how you configure, deploy, and reason about the application. Each is covered in depth on its concept page — this is the orientation.

The same shift runs through parallelism (lane count is a config knob, not a topic-layout decision), re-keying (internal repartition topics disappear — repartition() becomes in-process), and state (global — no co-partitioning requirement, no partition routing for queries). The full aspect-by-aspect table, and the conceptual why behind each row, are on How StoatFlow differs from Kafka Streams; the model in its own right is on Architecture.

Config keys that go away

A subset of Kafka Streams configuration encodes assumptions that the single-instance model removes. These keys have no StoatFlow equivalent and are dropped during the port:

• Instance- and rebalancing-related — num.stream.threads, standby-replica settings, static-membership and cooperative-rebalancing tuning. There is no group to rebalance and no second active instance, so these KS knobs have nothing to act on. (StoatFlow's own opt-in hot standby is a separate mechanism under stoatflow.ha.*, not a port of these.)
• Exactly-once enablement — the KS processing.guarantee opt-in and its associated transactional tuning. StoatFlow commits the whole topology under one barrier-linked transaction by default; you choose exactly-once or at-least-once as a mode, not by assembling the per-task transactional machinery yourself.
• Partitioning that assumes a cluster — withPartition(...) on interactive queries, partitioner settings on TableJoined (withPartitioner / withOtherPartitioner), and the repartition-topic partition counts. With one instance and global state there is nothing to route across.

API-surface differences

A handful of Kafka Streams APIs deliberately diverge. Some are accepted but inert because the single-instance model gives them nothing to act on — the partitioner methods on TableJoined, for example, compile and do nothing. Some are stricter than KS — a configured processor.wrapper.class throws instead of being silently ignored, and a multicast StreamPartitioner on repartition() is rejected. And a few have small shape changes — VersionedRecord.validTo() returns an Optional, exception handlers receive a Record instead of separate key/value. These are catalogued in the KS compatibility matrix; the shape changes surface at compile time during the port, which makes them easy to find and fix.

The decision: green-field cutover vs. carrying state

The single question that determines the shape of your migration is what happens to your existing state.

StoatFlow does not read Kafka Streams' changelog topics directly. The two engines store and lay out state differently, so there is no in-place restore from a KS changelog into a StoatFlow store. That leaves two honest paths, and which one you take depends entirely on whether your topology's correctness depends on state that you cannot cheaply rebuild.

The decision, compressed into one tree:

When a green-field cutover is the right call

Choose Without data migration when any of these holds:

• The topology is stateless — map / filter / flatMap / routing, no aggregations or joins that accumulate. There is nothing to carry; reprocessing produces identical output.
• The state is derivable from input still in retention. If your source topics retain enough history to recompute every aggregate, count, and join from scratch, reprocessing rebuilds the state exactly, and the cutover is a config-and-deploy exercise.
• You can tolerate a reprocessing window. Reading the input topics from the beginning takes time proportional to their size; if you can run the new instance to catch-up before switching consumers over, this is the lowest-risk path.

This is the default recommendation. It avoids any state-format coupling between the two engines, and the result is provably correct because it is computed from the same input the KS app consumed.

When you need a state-carrying migration

Choose With data migration when reprocessing is not viable:

• Input topics have aged out. Compacted or time-limited source topics no longer hold the full history, so a reprocess would produce incomplete aggregates.
• A full reprocess is too slow or too expensive. Very large state, or long-running windowed aggregations spanning weeks, can make recomputation impractical even when the input technically survives.
• You need continuity of in-flight state — open sessions, long windows, running counters that downstream systems depend on without a recomputation gap.

This path is more constrained than the name suggests: StoatFlow cannot import Kafka Streams changelogs or store files directly, so "carrying state" is a per-topology plan rather than a switch you flip. The sub-page covers what StoatFlow's own restoration can and cannot reuse, why direct changelog reuse is off the table, and how to sequence the rebuild — and migrations of this shape are worth talking through with us.

Recommended order

A migration in practice tends to run in this order, regardless of which state path you take:

1. Port the build and the topology. Swap the dependency, rename the imports, change the entry point, and let the compiler surface the dropped APIs. Validate the ported topology with the in-memory test driver — no broker required, deterministic, and the fastest way to confirm the logic is intact.
2. Clean up the config. Remove the KS keys that have no meaning here (above) and express parallelism as lane count. See Configuration model.
3. Pick the state path — without or with data migration — and follow that page.
4. Deploy as a single pod and wire up the operational surface: health probes, metrics, and the debug endpoints described on Architecture and in Operating.

Where to go next

• Without data migration — the green-field cutover path: reprocess from input, no state carried across
• With data migration — carrying existing state across when reprocessing isn't viable
• How StoatFlow differs from Kafka Streams — the conceptual deltas the migration follows from
• Comparison matrix — feature-by-feature against Kafka Streams and Flink
• KS compatibility matrix — method-by-method DSL parity and the deliberate exceptions
• Migration in flight and want a second pair of eyes? Get in touch — real people read every email during the alpha.