Migration carrying state

This page is about stateful topologies — aggregations, joins, windows, anything with a state store — and what happens to that state when you move to StoatFlow. The short version: StoatFlow rebuilds its own local state from its own changelog topics on every start, so a StoatFlow-to-StoatFlow move carries state automatically. Carrying an existing Kafka Streams state store across the migration is a different problem, and the supported answer is to reprocess your input, not to point StoatFlow at the old changelog.

If your topology has no state stores, you don't need any of this — see Migration without state.

How StoatFlow carries its own state

StoatFlow's durability model is the same family as Kafka Streams: every state-store write is logged to a compacted changelog topic in Kafka, and local state lives in RocksDB (or in memory). On a commit barrier, the state writes, the changelog records, and the consumer offsets all commit together in one Kafka transaction — so the changelog is always consistent with what the engine has processed. See State and durability for the conceptual model and Exactly-once for the barrier.

Changelog topics follow the Kafka Streams naming convention:

{application-id}-{store-name}-changelog

So a word-counts store in an app with application-id: word-count logs to word-count-word-counts-changelog. The state-store name is whatever you gave it — Materialized.as("word-counts"), Stores.persistentKeyValueStore("..."), and so on.

What happens on start

On every start, the runtime decides per store how to bring local state up to date. There are three outcomes, fastest to slowest:

The runtime knows which case applies because it stores the committed changelog offset inside RocksDB itself, in the same atomic write as the data. There is no separate metadata file to fall out of sync. On start it compares that stored offset against the changelog's end offset in Kafka and picks delta, full, or skip. (In-memory stores have no local state to reuse, so they always full-restore.) Restoration runs in parallel across stores, so a topology with many state stores recovers concurrently rather than store-by-store.

The per-store decision, as the runtime makes it on every start:

Migrating from Kafka Streams: reprocess, don't reuse

The natural instinct when moving a stateful app is to keep the changelog topics and point the new engine at them. For a Kafka Streams → StoatFlow move, this is not supported, and it will not produce correct state. Two independent reasons:

1. The changelog encoding is not byte-compatible. StoatFlow writes its own normalized changelog key formats for window, session, and versioned stores — they do not match the segment-ID / timestamp-inverted layout Kafka Streams uses. A StoatFlow restore that read a Kafka Streams changelog for those store types would deserialize garbage.
2. The committed-offset bookkeeping differs. StoatFlow tracks the changelog position it has restored to inside its own RocksDB column family. A Kafka Streams changelog carries no StoatFlow offset metadata, so the validation that decides skip/delta/full has nothing valid to compare against.

The supported path is to let StoatFlow build the state itself by reprocessing the input.

Recommended procedure

1. Confirm the input topics retain enough history. Reprocessing rebuilds aggregate state by re-reading the source topics from the start. This only reconstructs correct state if the source topics still hold the records that built it. For a compacted source (e.g. a KTable over a compacted topic) the latest value per key is always present. For an aggregation over a non-compacted, time-retained topic, anything aged out of retention cannot be replayed — the rebuilt aggregate will reflect only what is still in the topic. If your retention is shorter than your aggregation history, treat the rebuilt state as approximate and validate before cutover.
2. Deploy StoatFlow with a new application-id (and auto.offset.reset=earliest, which is already StoatFlow's default). This makes the new consumer group start at the beginning of every source topic.

   stoatflow:
     application-id: word-count-sf   # NEW id — not the old Kafka Streams application.id
     bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVERS:-localhost:9092}
     # ...
   

   
   Set the reset on the source via Consumed:
   KotlinJava

   import io.stoatflow.core.topology.AutoOffsetReset
   import io.stoatflow.core.topology.Consumed
   
   builder.stream(
       "orders",
       Consumed.`as`<String, Order>("source")
           .withOffsetResetPolicy(AutoOffsetReset.Earliest),
   )
   

3. Let the rebuild run to completion and watch the lag. While reprocessing, the new app reads the full history of each source topic, rebuilding state and writing it to its own changelog topics as it goes. Track progress with the consumer-lag and throughput metrics on /metrics; the app is caught up when consumer lag on the source topics reaches roughly zero. If an initial state restore is running (e.g. a restart that rebuilds from the changelog rather than from the source), per-store progress surfaces through the stoatflow.restoration.* meters — stoatflow.restoration.in.progress (1 while restoring), stoatflow.restoration.stores.total / stoatflow.restoration.stores.completed, and the per-store counters stoatflow.restoration.records.restored.total and stoatflow.restoration.bytes.restored.total (both tagged store_name), plus the per-store stoatflow.restoration.duration.seconds timer. See Metrics. Committed changelog offsets per store are visible on the /offsets endpoint. The /state endpoint reports the lifecycle state (VALIDATING_STATE, RESTORING, RUNNING) and the transition history, not per-store restoration values.
4. Decide what the output should do during the rebuild. Reprocessing re-emits results for everything it replays. If the topology writes to the same sink topics your Kafka Streams app used, downstream consumers will see the historical results again. Pick one:
   • Rebuild silently, then cut over. Point the StoatFlow app at throwaway sink topics for the rebuild, verify the state, then switch the sinks (or the downstream readers) to the real topics. Cleanest for downstream systems that are not duplicate-tolerant.
   • Rebuild into the real topics. Acceptable only if downstream is idempotent or duplicate-tolerant.
5. Cut over. Once the StoatFlow app is caught up and its state is validated, stop the Kafka Streams app and let StoatFlow take over live traffic. Because the two apps use different application-ids and different consumer groups, you can run the StoatFlow rebuild alongside the still-live Kafka Streams app and keep the old one serving until the moment you switch — there is no shared offset state to corrupt.

What is and is not supported for in-place reuse

Caveats

• Reprocessing fidelity is bounded by source retention. If records that contributed to an aggregate have aged out of the source topic, they cannot be replayed and the rebuilt aggregate will differ from the original. Compacted sources are safe (latest value per key is retained); time-retained sources are only safe if retention covers the full aggregation history.
• Event-time results depend on timestamps, not arrival order. Windowed aggregations key off each record's event time. Reprocessing replays records in their original partition order with their original timestamps, so windows close the same way — provided your TimestampExtractor / watermark strategy matches what the original app used. Mismatched time handling produces different window boundaries. See Event time and watermarks.
• Cold-start time scales with state size. A full rebuild reads the entire history of the source (or changelog) topics. For large state this is minutes, not seconds. Plan the cutover window accordingly; see Benchmarks for measured cold-start numbers.
• /health/ready stays 503 until restoration completes. During a rebuild the readiness probe returns 503 so Kubernetes won't route traffic and load balancers won't mark the instance up. The 503 body carries the stoatflow health component with the lifecycle state in its details (VALIDATING_STATE or RESTORING while state is being rebuilt) — that is what identifies the in-progress restoration. Per-store progress is on the stoatflow.restoration.* meters on /metrics, not in the probe body. See Probes.
• Output during a rebuild is real output. Reprocessing re-emits results. If you rebuild into live sink topics, downstream sees the replay. Use throwaway sinks for the rebuild unless downstream is duplicate-tolerant (step 4).

Where to go next

• Migration without state — the dependency swap, config changes, and runtime swap for stateless and reprocess-from-scratch moves.
• Architecture: state and durability — the changelog + RocksDB model and how restoration works.
• Exactly-once — the commit barrier that keeps state, output, and offsets consistent.
• Migrating a large stateful topology and want a second pair of eyes on the cutover plan? Get in touch — real people read every email during the alpha.