Migration carrying state

This page is about stateful topologies — aggregations, joins, windows, anything with a state store — and what happens to that state when you move to StoatFlow. There are two supported ways to bring the state across: rebuild it by reprocessing the input topics, or translate it with the stoatflow-migration-tool CLI, which reads your Kafka Streams changelog topics and seeds byte-translated StoatFlow changelogs plus the carried-over input offsets. A third path — engage us — remains for the shapes the tool deliberately does not cover.

If your topology has no state stores, you don't need any of this — see Migration without state.

How StoatFlow carries its own state

StoatFlow's durability model is the same family as Kafka Streams: every state-store write is logged to a compacted changelog topic named {application-id}-{store-name}-changelog, local state lives in RocksDB (or in memory), and on every start the runtime decides per store whether to reuse, delta-restore, or full-restore. The full model — including the per-store restore decision — is on Architecture: state and durability and Lifecycle.

The property that makes a state-carrying migration possible: a first start over pre-seeded changelog topics is just a forced full restore. The engine needs no migration mode — the migration tool writes StoatFlow-format changelogs, and the first start rebuilds RocksDB from them exactly as it would after losing a disk.

The supported-path matrix

Direct reuse of a Kafka Streams changelog — pointing StoatFlow at the KS topics — remains unsupported. The two engines' on-wire changelog formats differ for several store types (window-key seqnums, versioned timestamp placement, the LEFT/OUTER join outer store, timestamped value envelopes), and the offset bookkeeping differs. That byte gap is exactly what the migration tool's translation closes — offline, verifiably, and outside the engine.

What the translate path supports

Per store type (the tool's type values in parentheses):

Classifying your stores

The tool's config declares each store's type, and the classification follows the StoatFlow-side store kind, not just the KS operation. Two things matter:

1. Name your stores explicitly on both sides (Materialized.as(...), StreamJoined.withStoreName(...), TableJoined.as(...)). KS auto-generated names (KSTREAM-AGGREGATE-STATE-STORE-0000000007) are not guaranteed to match the ported topology's generated names; the config's ksName → sfName mapping is the escape hatch, not the plan.
2. Get the kv vs kv-timestamped split right — it is byte-silent if wrong. Both engines' changelogs carry bare value bytes for KV stores (KS strips the timestamp into the record timestamp), so the type tells the tool whether the StoatFlow store expects a timestamp-wrapped value.

Source KTables resolve per the compaction of their source topic:

The tool's plan command resolves and prints this path per store, and structurally sanity-checks every declared type against sampled changelog records — see the migration tool.

Semantics caveats at the cutover boundary

• Suppress final emissions. StoatFlow's suppress buffer is changelog-backed and restorable — the boundary risk exists only because the KS buffer is not translated. Any final result still sitting in the KS suppress buffer at shutdown is lost to the StoatFlow side. Mitigation: cut over at a windows-closed point — quiesce input, let stream time advance past window-end + grace so KS emits, then stop. Note stream time only advances with records; a hard input stop before the drain leaves the buffer full forever. See the suppress notes on Windowing.
• OnWindowClose duplicate re-emission. With the {store}-emitfrontier companion left unseeded, StoatFlow's first watermark tick past a window close re-emits a final for every restored closed window within KS retention — duplicates of results KS already emitted, in one burst. Harmless for idempotent-upsert consumers; disruptive for incremental/append consumers. Fix: set emitFrontier: true in the tool config so translate seeds the frontier with the store's own max changelog timestamp.
• ALO clean-shutdown requirement. Under at_least_once, only a clean KS shutdown aligns changelog state with committed input offsets — a crashed ALO app's snapshot inherits at-least-once duplication into migrated state (double-counting on resume). Exactly-once snapshots are crash-safe. The tool warns loudly; never migrate a crashed ALO app's snapshot.
• LEFT/OUTER join boundary. Unmatched records buffered at cutover migrate via the outer-join rule and emit their null-side finals from StoatFlow's watermark scan once it passes window-end + grace. Optionally drain first (quiesce past join-window close) to minimize translated volume; INNER joins are unaffected. See Joins.
• Stream time vs watermark. StoatFlow's stream time is the watermark; at resume it rebuilds from live records with the configured strategy (e.g. bounded out-of-orderness lag) — window closure and expiry around the boundary can differ from KS by up to the out-of-orderness bound. See Event time and watermarks.
• Retention boundaries. Restored window/session records older than the StoatFlow store's retention are expired on the first post-restore watermark tick — same as native state, but worth stating.
• Logging-disabled KS stores have no changelog to translate from — reprocess or accept the loss.

What is and is not supported

Where to go next

• The migration tool — config reference, the four commands, the full cutover runbook, and the rollback line.
• Migration without state — the reprocess path when state is rebuildable.
• Architecture: state and durability — the changelog + RocksDB model and how restoration works.
• Reusing your Kafka Streams dashboards — KS-compatible metrics for watching the cutover on the dashboards you already have.
• Suppress-heavy topology, versioned stores at scale, or unsure how to classify? Get in touch — real people read every email.