Kafka Streams compatibility matrix
The engineering view of Kafka Streams DSL parity in StoatFlow: every operator, helper class, and store method, with its current status and notes where behaviour differs. Targets Kafka Streams 4.4.x: StoatFlow reimplements the Kafka Streams DSL surface under its own package, so most KS applications port by swapping the dependency and rewriting imports โ see Migrating from Kafka Streams, and let the automated port recipe do it for you.
For the high-level, feature-by-feature comparison against Kafka Streams and Flink, see Comparison matrix. For how-to guidance on the operators below, see Building.
1.0.0-rc.4). A small number of rows differ in signature from Kafka Streams (marked โ ๏ธ) or are intentionally absent (marked ๐) โ read the notes.Legend
| Symbol | Meaning |
|---|---|
| โ | Implemented |
| โ ๏ธ | Partial โ stub, limited, or differing signature |
| โ | Not implemented yet |
| ๐ | Incompatible, or won't be implemented (not needed in the single-instance model) |
| โ๏ธ | Deprecated in Kafka Streams |
| ๐ | StoatFlow extension โ not in Kafka Streams |
| ๐ | StoatFlow extension on an otherwise KS-compatible interface |
| ๐ก | Recognised no-op โ accepted and warned about, never an error (configuration keys only) |
| ๐งฉ | Client pass-through โ forwarded to the Kafka clients the key is valid for (configuration keys only) |
StreamsBuilder
| Method | Status | Notes |
|---|---|---|
| StreamsBuilder() | โ | No-arg |
| StreamsBuilder(TopologyConfig) | โ | KSC-77 โ seeds build-time defaults (DSL store format, FK reverse-index cache, explicit naming) and the KIP-1112 ProcessorWrapper from a TopologyConfig projection of StreamsConfig. It carries no serde defaults: default.key.serde / default.value.serde on the runtime StreamsConfig decide every serde, as in Kafka Streams. Named-topology / per-topology overrides N/A (single-instance) |
| stream(topic, consumed?) | โ | Single topic subscription; single-arg form Java-callable |
| stream(Collection<String>) | โ | Multiple topics subscription |
| stream(Pattern) | โ | Pattern-based subscription โ N patterns combine into one subscription; a topic two patterns accept is rejected at assignment; literals are quoted into the combined regex (KSC-101); no own-topic filtering. Deleting a matched topic is survived (in-place engine restart, ~0.5-1 s, no pod restart), matching KS; a deleted literal source topic stays fatal, also matching KS โ see Streams builder |
| table(topic, consumed?) | โ | |
| table(topic, materialized) | โ | Explicit skip-Consumed overload (KS parity). The Materialized serdes also read the source topic, as in Kafka Streams |
| table(topic, consumed?, materialized) | โ | With state store |
| globalTable(topic, consumed?) | โ | Wraps KTable |
| globalTable(topic, materialized) | โ | Without Consumed parameter. โ ๏ธ A versioned Materialized is accepted and materialises a versioned store โ Kafka Streams throws GlobalTables cannot be versioned. (KSC-125, more permissive) |
| build() | โ | Returns Topology |
| build(Properties) | โ ๏ธ | The Properties are ignored โ logs a warning and delegates to build(). Config carries over via StreamsConfig.fromProperties(...) / the KS-shaped StreamsConfig(Properties) ctor instead (ADR-124) |
| addStateStore(storeBuilder) | โ | Register standalone store |
| addStateStore(storeBuilder, keySerde, valueSerde) | โ | With changelog Serdes |
| addStateStore(storeSupplier, keySerde, valueSerde) | โ | Convenience overload |
| addGlobalStore(storeBuilder, topic, consumed, processor) | โ | Custom update processor |
| addGlobalStore(storeBuilder, topic, consumed) | โ | Default put/delete semantics, written in the store's shape: a timestamped builder holds ValueAndTimestamp(value, record timestamp), a versioned one a version at the record timestamp. ๐ StoatFlow convenience โ Kafka Streams always takes a processor |
| addReadOnlyStateStore(storeBuilder, topic, consumed, processor) | โ | KIP-813: custom processor |
| addReadOnlyStateStore(storeBuilder, topic, consumed) | โ | KIP-813: Default put/delete semantics, written in the store's shape: a timestamped builder holds ValueAndTimestamp(value, record timestamp), a versioned one a version at the record timestamp. ๐ StoatFlow convenience โ Kafka Streams always takes a processor |
| scheduled(interval, type, emitter) | ๐ | Interval-based periodic record source |
| scheduled(interval, type, named, emitter) | ๐ | With explicit naming |
| scheduled(interval, type, keySerde, emitter) | ๐ | With explicit key serde for lane assignment |
| scheduled(interval, type, keySerde, options, emitter) | ๐ | With per-source ScheduledSourceOptions โ when the source fires first (ScheduledStart.IMMEDIATELY / NEXT_INTERVAL); default from dsl.scheduled-source-start (A2.19) |
| scheduled(named, interval, type, keySerde, options, emitter) | ๐ | Named, with per-source ScheduledSourceOptions (A2.19) |
| scheduled(cron, emitter) | ๐ | Cron-based calendar scheduling |
| scheduled(cron, named, emitter) | ๐ | Cron with explicit naming |
| scheduled(cron, keySerde, emitter) | ๐ | Cron with explicit key serde for lane assignment |
Topology (Processor API)
Both the high-level DSL (StreamsBuilder) and the low-level Topology builder are supported.
| Method | Status | Notes |
|---|---|---|
| describe() | โ | Returns the Kafka Streams TopologyDescription (nested Source/Processor/Sink/Subtopology/GlobalStore node graph). Breaking vs older StoatFlow: previously returned the native model โ that view moved to describeStoatFlow() |
| describeStoatFlow() | ๐ | StoatFlow-native topology model (the former describe() shape) |
| addSource(name, topics...) | โ | 16 Kafka Streams-shaped overloads: offset reset, timestamp extractor, deserializers, pattern (same rules as stream(Pattern)), and ๐ watermark strategy (no KS equivalent โ the Processor API route to a per-source generator, withIdleness and alignment). The deserializers, timestamp extractor and offset reset passed in are used; null means the configured default (KS parity). |
| addSource(name, consumed, topics... | pattern) | ๐ | Takes a Consumed, mapped exactly as StreamsBuilder.stream() maps it: key/value serdes, timestamp extractor or watermark strategy, offset reset. Unlike a key deserializer, it declares the key serde, so Processor API timers and in-memory sub-topology boundaries behind the source use it instead of default.key.serde. A Consumed.as(...) name must match the node name; withMaterializeFromSourceTopic is refused |
| addProcessor(name, supplier, parents...) | โ | User-defined processor with ProcessorSupplier |
| addScheduledSource(name, interval, type, emitter) | ๐ | Processor API route to an interval scheduled source โ StoatFlow extension (ADR-039) |
| addScheduledSource(name, interval, type, options, emitter) | ๐ | With per-source ScheduledSourceOptions (A2.19) |
| addScheduledSource(name, cron, emitter) | ๐ | Cron-based scheduled source |
| addSink(name, topic, parents...) | โ | 8 overloads: serializers, partitioner, TopicNameExtractor. The serializers and partitioner passed in are used; null means the configured default (KS parity). Difference (KSC-148): a static topic must exist when the application starts โ a missing one refuses the start; a TopicNameExtractor sink is not checked. |
| addStateStore(builder, processors...) | โ | Register store and optionally connect to processors. Several processors may connect the same builder instance and share the store; a different builder under an existing name is rejected |
| addGlobalStore(...) | โ | 2 variants: with/without TimestampExtractor. The deserializers and timestamp extractor passed in are used; null means the configured default (KS parity). |
| addGlobalStore(storeBuilder, sourceName, consumed, topic, processorName, supplier) | ๐ | The Consumed form, mapped as addSource(name, consumed, โฆ). An offset reset other than earliest is refused: a store's source always reads from the beginning |
| addReadOnlyStateStore(...) | โ | 2 variants (KIP-813): source topic is changelog. The deserializers and timestamp extractor passed in are used; null means the configured default (KS parity). |
| addReadOnlyStateStore(storeBuilder, sourceName, consumed, topic, processorName, supplier) | ๐ | The Consumed form, mapped as addSource(name, consumed, โฆ). An offset reset other than earliest is refused |
| connectProcessorAndStateStores(...) | โ | Connect existing store to processor |
TopologyDescription
describe() returns the Kafka Streams TopologyDescription โ a graph of nested node interfaces.
| Member | Status | Notes |
|---|---|---|
| subtopologies() / globalStores() | โ | Sets of Subtopology / GlobalStore |
| Subtopology.id() / nodes() | โ | |
| Node.name() / predecessors() / successors() | โ | |
| Source.topicSet() / topicPattern() | โ | The matched topics of a pattern source are a runtime fact (stoatflow.source.pattern.matched.topics), not part of the description |
| Processor.stores() | โ | Connected store names |
| Sink.topic() / topicNameExtractor() | โ | Static or dynamic routing |
| GlobalStore.id() / source() / processor() | โ |
KStream โ stateless transformations
| Method | Status | Notes |
|---|---|---|
| filter(predicate) | โ | |
| filterNot(predicate) | โ | |
| map(mapper) | โ | Key-changing; the sub-topology boundary opens downstream, where an operator needs the new key's affinity (KS repartitionRequired model, ADR-138) |
| mapValues(ValueMapper) | โ | |
| mapValues(ValueMapperWithKey) | โ | Key-aware value mapper variant |
| flatMap(mapper) | โ | Key-changing; the sub-topology boundary opens downstream, where an operator needs the new key's affinity (KS repartitionRequired model, ADR-138) |
| flatMapValues(ValueMapper) | โ | |
| flatMapValues(ValueMapperWithKey) | โ | Key-aware value mapper variant |
| selectKey(keySelector) | โ | Key-changing; the sub-topology boundary opens downstream, where an operator needs the new key's affinity (KS repartitionRequired model, ADR-138) |
| peek(action) | โ | Java BiConsumer overload |
| merge(other) | โ | |
| split() | โ | Returns BranchedKStream |
| repartition() | โ | In-memory only |
| repartition(Repartitioned) | โ | Custom partitioner, naming |
| print() | โ | Configurable via Printed |
KStream โ terminal operations
| Method | Status | Notes |
|---|---|---|
| forEach(action) | โ | Java BiConsumer overload |
| to(topic, produced?) | โ | Static topic routing. Difference (KSC-148): the topic must exist when the application starts โ a missing one refuses the start with MissingOutputTopicException, where Kafka Streams finds it at the first send (opt-out validation.validate-output-topics) |
| to(TopicNameExtractor, produced?) | โ | Dynamic topic routing. Not checked at start โ its topics are chosen per record |
| toTable() | โ | Auto-generates store name; the store takes the stream's serdes, windowed keys included |
| toTable(named?, materialized?) | โ | With naming and materialization; Materialized serdes first, then the stream's (KS parity) |
KStream โ Processor API
| Method | Status | Notes |
|---|---|---|
| process(processorSupplier, named?, storeNames...) | โ | Key-changing; ProcessorSupplier. Connects the named stores (and any declared via supplier.stores()). Like KS, it does not open a sub-topology boundary of its own โ see the note below. A node connected to a store another part of the topology writes is placed with that store's writer, as KS puts processors sharing a store into one subtopology; nodes in different sub-topologies connecting one store warn (presumed-cross-sub-topology-write) โ see Sharing a store between processors |
| processValues(processorSupplier, named?, storeNames...) | โ | Non-key-changing; FixedKeyProcessorSupplier + named/declared stores. Like KS, it does not open a sub-topology boundary of its own โ see the note below. Store placement as for process |
| transform() | ๐ | Removed from KStream in KS 4.x; use process() |
| transformValues() | ๐ | Removed from KStream in KS 4.x; use processValues() |
process() / processValues(), even with connected stores, and since 1.0.0 neither does StoatFlow (topology.processor-api-key-affinity: off โ presumed restores the pre-1.0.0 boundary). After a many-to-one re-key that means per-key state at such a node is fragmented across lanes, exactly as it is across tasks in KS. StoatFlow adds a guard KS does not have: it refuses to compile the provable case โ a proven re-key into a node with a writable store (topology.validation.papi-key-affinity-presumed, default error) โ and warns where the feeder is itself a Processor API node, whose key change is only a conservative default (papi-key-affinity-presumed-chain, default warn). See Key affinity after a re-key.KStream โ grouping
| Method | Status | Notes |
|---|---|---|
| groupByKey() | โ | Returns KGroupedStream |
| groupByKey(Grouped) | โ | With Serdes configuration |
| groupBy(keySelector) | โ | Key-changing; the sub-topology boundary opens downstream, where an operator needs the new key's affinity (KS repartitionRequired model, ADR-138) |
| groupBy(keySelector, Grouped) | โ | With Serdes configuration |
KStream โ joins
| Method | Status | Notes |
|---|---|---|
| join(KStream, ValueJoiner, windows) | โ | Stream-stream windowed join |
| join(KStream, ValueJoiner, windows, streamJoined) | โ | With StreamJoined config |
| join(KStream, ValueJoinerWithKey, windows) | โ | Key-aware joiner variant |
| join(KStream, ValueJoinerWithKey, windows, streamJoined) | โ | Key-aware joiner with config |
| leftJoin(KStream, ValueJoiner, windows) | โ | Stream-stream left join. An unmatched record whose window has already closed when it arrives is emitted at once with a null side, under its own timestamp, never buffered โ KS's late branch, judged on the watermark instead of stream time (KSC-132) |
| leftJoin(KStream, ValueJoiner, windows, streamJoined) | โ | With StreamJoined config |
| leftJoin(KStream, ValueJoinerWithKey, windows) | โ | Key-aware joiner variant |
| leftJoin(KStream, ValueJoinerWithKey, windows, streamJoined) | โ | Key-aware joiner with config |
| outerJoin(KStream, ValueJoiner, windows) | โ | Stream-stream outer join. Late unmatched records on either side are emitted at once with a null side, as for leftJoin |
| outerJoin(KStream, ValueJoiner, windows, streamJoined) | โ | With StreamJoined config |
| outerJoin(KStream, ValueJoinerWithKey, windows) | โ | Key-aware joiner variant |
| outerJoin(KStream, ValueJoinerWithKey, windows, streamJoined) | โ | Key-aware joiner with config |
| join(KTable, ValueJoiner) | โ | Stream-table inner join. After a re-key the join reads the table from another lane: under exactly_once as of the stream record's commit, and an lruMap or in-memory versioned table is refused there (KSC-126, also for leftJoin); with no re-key it reads on its own lane |
| join(KTable, ValueJoinerWithKey) | โ | Key-aware joiner variant |
| leftJoin(KTable, ValueJoiner) | โ | Stream-table left join |
| leftJoin(KTable, ValueJoinerWithKey) | โ | Key-aware joiner variant |
| join(GlobalKTable, keySelector, ValueJoiner) | โ | Stream-GlobalKTable inner join. Kotlin: also a (V, GV) -> VR lambda overload, optionally with a Named before the joiner. Under exactly_once every global-table join (join and leftJoin, all overloads) reads the table as of the stream record's commit โ stronger than KS, whose global thread gives no cut against the stream tasks; same API. An lruMap or in-memory versioned global table is refused there (KSC-126) |
| join(GlobalKTable, keySelector, ValueJoinerWithKey) | โ | Key-aware joiner variant; deprecated since Kafka Streams 4.4 (KIP-1340): the joiner receives the stream key, not the join key โ use ValueJoinerWithStreamAndMappedKey |
| join(GlobalKTable, keySelector, ValueJoiner|ValueJoinerWithKey, Named) | โ | Named overloads; the ValueJoinerWithKey form is deprecated as above |
| join(GlobalKTable, keySelector, ValueJoinerWithStreamAndMappedKey, Named) | โ | KIP-1340 (Kafka Streams 4.4): the joiner receives (streamKey, mappedKey, value1, value2). Kotlin: also a (K, GK, V, GV) -> VR lambda overload, optionally with a Named before the joiner |
| leftJoin(GlobalKTable, keySelector, ValueJoiner) | โ | Stream-GlobalKTable left join. Kotlin: also a (V, GV?) -> VR lambda overload, optionally with a Named before the joiner |
| leftJoin(GlobalKTable, keySelector, ValueJoinerWithKey) | โ | Key-aware joiner variant; deprecated since Kafka Streams 4.4 (KIP-1340): the joiner receives the stream key, not the join key โ use ValueJoinerWithStreamAndMappedKey |
| leftJoin(GlobalKTable, keySelector, ValueJoiner|ValueJoinerWithKey, Named) | โ | Named overloads; the ValueJoinerWithKey form is deprecated as above |
| leftJoin(GlobalKTable, keySelector, ValueJoinerWithStreamAndMappedKey, Named) | โ | KIP-1340 (Kafka Streams 4.4): the joiner receives (streamKey, mappedKey, value1, value2); mappedKey is null when the selector returns null (the table value is then null too). Kotlin: also a (K, GK?, V, GV?) -> VR lambda overload, optionally with a Named before the joiner |
KStream โ branching (BranchedKStream)
| Method | Status | Notes |
|---|---|---|
| branch(predicate) | โ | |
| branch(predicate, branched) | โ | With Branched config |
| defaultBranch() | โ | |
| defaultBranch(branched) | โ | With Branched config |
| noDefaultBranch() | โ |
KGroupedStream
| Method | Status | Notes |
|---|---|---|
| count() | โ | With Materialized overloads |
| reduce(reducer) | โ | With Materialized overloads |
| aggregate(initializer, aggregator) | โ | With Materialized overloads |
| windowedBy(TimeWindows) | โ | Tumbling/hopping windows โ TimeWindowedKStream |
| windowedBy(SlidingWindows) | โ | Sliding windows (KIP-450) โ returns TimeWindowedKStream, exactly like Kafka Streams (no separate SlidingWindowedKStream type) |
| windowedBy(SessionWindows) | โ | Session windows |
| cogroup() | โ | Returns CogroupedKStream |
Windowed grouped streams
TimeWindowedKStream
Reached by both windowedBy(TimeWindows) (tumbling/hopping) and windowedBy(SlidingWindows) (KIP-450
sliding) โ sliding windows flow through this type, exactly as in Kafka Streams. All overloads take a
plain-key Materialized<K, V, โฆ>. For small event times โ below size โ advance for hopping windows, below
the time difference for sliding ones โ the windows differ from Kafka Streams' โ see Windows near time 0 under
the architectural differences.
| Method | Status | Notes |
|---|---|---|
| count() | โ | With Materialized overloads (plain key) |
| reduce(reducer) | โ | With Materialized overloads |
| aggregate(initializer, aggregator) | โ | With Materialized overloads |
| emitStrategy() | โ | OnWindowUpdate (default), OnWindowClose (lateness and window close follow the watermark, not per-aggregation stream time โ KSC-132) |
SessionWindowedKStream
| Method | Status | Notes |
|---|---|---|
| count() | โ | With session merger |
| reduce(reducer) | โ | Reducer doubles as merger |
| aggregate(initializer, aggregator, sessionMerger) | โ | With session merger |
| emitStrategy() | โ | OnWindowUpdate (default), OnWindowClose (lateness and window close follow the watermark, not per-aggregation stream time โ KSC-132) |
Cogrouped streams
CogroupedKStream
| Method | Status | Notes |
|---|---|---|
| cogroup(KGroupedStream, aggregator) | โ | Add another stream to the cogroup |
| aggregate(initializer) | โ | With Materialized overloads |
| windowedBy(TimeWindows) | โ | Returns TimeWindowedCogroupedKStream |
| windowedBy(SlidingWindows) | โ | Sliding windows (KIP-450) โ returns TimeWindowedCogroupedKStream |
| windowedBy(SessionWindows) | โ | Returns SessionWindowedCogroupedKStream |
TimeWindowedCogroupedKStream
| Method | Status | Notes |
|---|---|---|
| aggregate(initializer) | โ | With Materialized overloads |
| emitStrategy() | โ | OnWindowUpdate (default), OnWindowClose (lateness and window close follow the watermark, not per-aggregation stream time โ KSC-132) |
SessionWindowedCogroupedKStream
| Method | Status | Notes |
|---|---|---|
| aggregate(initializer, sessionMerger) | โ | With Materialized overloads; (initializer, sessionMerger) argument order matches Kafka Streams |
| emitStrategy() | โ | OnWindowUpdate (default), OnWindowClose (lateness and window close follow the watermark, not per-aggregation stream time โ KSC-132) |
KTable โ transformations
| Method | Status | Notes |
|---|---|---|
| filter(predicate) | โ | Derived table (no state store) |
| filter(predicate, Named) | โ | With naming |
| filter(predicate, Materialized) | โ | With materialization |
| filter(predicate, Named, Materialized) | โ | With naming and materialization |
| filterNot(predicate) | โ | Inverse of filter |
| filterNot(predicate, Named) | โ | With naming |
| filterNot(predicate, Materialized) | โ | With materialization |
| filterNot(predicate, Named, Materialized) | โ | With naming and materialization |
| mapValues(ValueMapper) | โ | Derived table (no state store); joined and grouped like a materialized table, with old values, as in KS |
| mapValues(ValueMapper, Named) | โ | With naming |
| mapValues(ValueMapper, Materialized) | โ | With materialization |
| mapValues(ValueMapper, Named, Materialized) | โ | With naming and materialization |
| mapValues(ValueMapperWithKey) | โ | Key-aware value mapper variant |
| mapValues(ValueMapperWithKey, Named) | โ | With naming |
| mapValues(ValueMapperWithKey, Materialized) | โ | With materialization |
| mapValues(ValueMapperWithKey, Named, Materialized) | โ | With naming and materialization |
| toStream() | โ | Same key type |
| toStream(KeyValueMapper) | โ | Key-changing variant; the sub-topology boundary opens downstream, where an operator needs the new key's affinity (ADR-138) |
| queryableStoreName() | โ | Returns store name if materialized; null for a derived table built without Materialized and for an untilWindowCloses / untilTimeLimit suppressed table (KS parity). An untilBarrier table keeps its source's store name |
| groupBy(keyValueSelector) | โ | Selector returns KeyValue<KR, VR>; returns KGroupedTable<KR, VR> |
| groupBy(keyValueSelector, Grouped) | โ | With Serdes configuration |
| suppress(Suppressed) | โ | untilWindowCloses, untilTimeLimit, untilBarrier. Each emission carries the buffered record's latest headers, timestamp, topic, partition and offset, as in Kafka Streams; records leaving together leave in Kafka Streams' order (buffer time, then key), and a downstream groupBy or foreign-key join gets the value from before the key's first buffering as the old value โ Kafka Streams overwrites it per update, so StoatFlow retracts what downstream actually saw. Joins read the suppressed table through its value getter: a buffered key reads its pre-buffering value. The buffer falls back to default.key.serde / default.value.serde when no serde was set upstream โ including for windowed tables, where Kafka Streams fails (KAFKA-9259) |
| transformValues(ValueTransformerWithKeySupplier, named?, storeNames...) | โ | Stateful per-key value transform; returns KTable<K, VR> |
| transformValues(ValueTransformerWithKeySupplier, Materialized, Named, storeNames...) | โ | Materializes the result into a queryable store (tombstone-aware: a null transform deletes). Supplier parameter carries Kafka Streams' ? super K, ? super V, ? extends VR wildcards, so a supplier producing a subtype of the result value type still fits |
KTable โ primary-key joins
Both tables share key type K. No co-partitioning required (global state model).
| Method | Status | Notes |
|---|---|---|
| join(KTable, joiner) | โ | Inner join. Either side may be materialized, a derived filter / mapValues / transformValues, suppressed or a windowed aggregation โ read through its value getter, as in KS; only an unmaterialized join result is refused. Each side reads the other from another lane: under exactly_once as of its own commit, and an lruMap or in-memory versioned input table is refused (KSC-126, every primary-key join overload) โ a RocksDB versioned one is accepted |
| join(KTable, joiner, Materialized) | โ | With materialization of join result |
| join(KTable, joiner, Named, Materialized) | โ | With naming and materialization |
| leftJoin(KTable, joiner) | โ | Left table always emits |
| leftJoin(KTable, joiner, Materialized) | โ | With materialization of join result |
| leftJoin(KTable, joiner, Named, Materialized) | โ | With naming and materialization |
| outerJoin(KTable, joiner) | โ | Full outer join |
| outerJoin(KTable, joiner, Materialized) | โ | With materialization of join result |
| outerJoin(KTable, joiner, Named, Materialized) | โ | With naming and materialization |
KTable โ foreign-key joins
Foreign-key extractor: a Java Function<V, KO?> (or BiFunction<K, V, KO?> for key-aware extraction) โ or a Kotlin (V) -> KO? / (K, V) -> KO? lambda.
| Method | Status | Notes |
|---|---|---|
| join(KTable, fkExtractor, joiner) | โ | Inner FK join. โ ๏ธ An lruMap or in-memory versioned input table on either side is refused under exactly_once (KSC-126, all FK join overloads); a RocksDB versioned one is accepted |
| join(KTable, fkExtractor, joiner, Materialized) | โ | With materialization of join result |
| join(KTable, fkExtractor, joiner, TableJoined, Materialized) | โ | With naming and materialization |
| leftJoin(KTable, fkExtractor, joiner) | โ | Left FK join; a null FK emits joiner(left, null) (KIP-962). FK-join retractions are forwarded materialized or not, as in Kafka Streams |
| leftJoin(KTable, fkExtractor, joiner, Materialized) | โ | With materialization of join result |
| leftJoin(KTable, fkExtractor, joiner, TableJoined, Materialized) | โ | With naming and materialization |
| outerJoin(KTable, fkExtractor, joiner) | โ ๏ธ | Not supported in KS either |
KGroupedTable
KGroupedTable is the result of KTable.groupBy() and uses changelog semantics โ moving a key from group A to group B decrements A and increments B. Every readable table tracks old values for groupBy() โ materialized, derived (filter, mapValues, transformValues), suppressed or windowed โ as in Kafka Streams; only an unmaterialized join result is refused.
| Method | Status | Notes |
|---|---|---|
| count() | โ | Changelog semantics |
| reduce(adder, subtractor) | โ | Kafka Streams compatible; both adder and subtractor are Reducer<V> |
| aggregate(init, adder, subtractor) | โ | Kafka Streams compatible; both adder and subtractor are Aggregator<K, V, VR> โ no StoatFlow-only Subtractor type |
GlobalKTable
| Method | Status | Notes |
|---|---|---|
| queryableStoreName() | โ | Returns underlying table's store name |
| asKTable() | ๐ | StoatFlow extension to unwrap |
Processor API
ProcessorContext
| Method | Status | Notes |
|---|---|---|
| forward(record) | โ | Forward to downstream processors |
| forward(record, childName) | โ | Forward to specific child |
| getStateStore(name) | โ | Access connected state stores โ declared via ProcessorSupplier.stores() or the process(supplier, "name") varargs. An undeclared store throws StoreNotConnectedException (KS parity). Global stores (addGlobalStore, globalTable) are exempt and returned read-only unless the node declares them, in which case they are writable (a StoatFlow extension โ KS makes declaring a global store a TopologyException). The read-only view covers every store family (plain, timestamped, versioned, headers-aware) and keeps the store's declared type, so the caller's getStateStore<TimestampedKeyValueStore<..>> cast still succeeds; read-only stores (addReadOnlyStateStore) are exempt too โ a deliberate divergence, KS exempts only globals |
| schedule(interval, type, punctuator) | โ | KS-compatible punctuator, KS fire times (STREAM_TIME aligned to 0, a grid, one fire per catch-up โ the first fire is at the first watermark, which in StoatFlow may be negative); inside the callback context.timestamp() is the punctuation time and headers() empty, as in KS |
| schedule(interval, type, mode, punctuator) | ๐ | StoatFlow extension with BLOCKING/NON_BLOCKING mode |
| schedule(interval, startTime, type, punctuator) | โ | KIP-1146: anchored punctuation with grid-aligned fire times |
| schedule(interval, startTime, type, mode, punctuator) | โ | KIP-1146: anchored punctuation with BLOCKING/NON_BLOCKING mode |
| recordMetadata() | โ ๏ธ | Not on the interface: topic() / partition() / offset() report the record's source coordinates directly, following the callback they are called in โ past an in-memory repartition the source record's, where Kafka Streams reports its repartition topic's; "" / -1 / -1 where Kafka Streams' recordMetadata() is empty (init(), a punctuator), in onTimer, and for what those callbacks forward |
| applicationId() | โ | The configured application.id, on every context; under the TopologyTestDriver, the driver config's id |
| currentSystemTimeMs() | โ | Wall-clock time via System.currentTimeMillis(); under the TopologyTestDriver, the driver's wall clock, as in Kafka Streams |
| currentStreamTimeMs() | โ | Delegates to currentWatermarkMs() (global watermark) |
| currentWatermarkMs() | ๐ | Global watermark in epoch millis |
| timerService() | ๐ | Key-based timer access (Flink-inspired) |
Processor interfaces
| Type | Status | Notes |
|---|---|---|
| Processor<KIn,VIn,KOut,VOut> | โ | Key-changing processor interface |
| FixedKeyProcessor<KIn,VIn,VOut> | โ | Key-preserving processor interface |
| ContextualProcessor<KIn,VIn,KOut,VOut> | โ | Abstract base managing ProcessorContext |
| ContextualFixedKeyProcessor<KIn,VIn,VOut> | โ | Abstract base managing FixedKeyProcessorContext |
| ProcessorSupplier<KIn,VIn,KOut,VOut> | โ | Factory for Processor instances |
| ProcessorSupplier.stores() | โ | KS 4.x pattern for declaring stores inline |
| FixedKeyProcessorSupplier<KIn,VIn,VOut> | โ | Factory for FixedKeyProcessor instances |
| ValueTransformerWithKey<K,V,VR> | โ | Consumed by KTable.transformValues; init binds StoatFlow's FixedKeyProcessorContext |
| ValueTransformerWithKeySupplier<K,V,VR> | โ | Factory for ValueTransformerWithKey; declares stores via stores(). As a KTable.transformValues parameter it carries Kafka Streams' ? super K, ? super V, ? extends VR wildcards |
| ValueTransformer / Transformer (+ suppliers) | โ๏ธ | Deprecated in KS and removed from KStream; kept as nameable types for source compatibility |
init() / close() lifecycle | โ | Once per task โ init() at task start, before any records (application start, an HA pod's activation, TopologyTestDriver construction), close() once at shutdown (KS contract), on every Processor API route (process / processValues / transformValues / addProcessor). A punctuator scheduled in init() starts without input. A global or read-only store's update processor follows the route that added it: through Topology.addGlobalStore / addReadOnlyStateStore it initialises at task start, like addProcessor; through StreamsBuilder.addGlobalStore / addReadOnlyStateStore it initialises on each lane's first record |
FixedKeyProcessor.onTimer + schedule() / timerService() | โ | Full Processor timer parity for FixedKeyProcessor (processValues): periodic punctuators via schedule() and keyed timers via timerService() โ onTimer(...) (key-preserving FixedKeyTimerContext, full state access, re-registration). Record timestamp is the event time |
Punctuators and timers
| Type / Method | Status | Notes |
|---|---|---|
| PunctuationType.STREAM_TIME | โ | Fires on event-time watermark advancement |
| PunctuationType.WALL_CLOCK_TIME | โ | Fires on system clock |
| PunctuatorMode.BLOCKING | ๐ | Barriers wait for punctuator completion before commit |
| PunctuatorMode.NON_BLOCKING | ๐ | Barriers proceed independently (default) |
| Punctuator.punctuate(timestamp) | โ | Callback invoked at scheduled time |
| Cancellable.cancel() | โ | Cancel scheduled punctuator or timer |
| TimerService.registerEventTimeTimer(key, timestamp) | ๐ | Fire when watermark โฅ timestamp |
| TimerService.registerProcessingTimeTimer(key, timestamp) | ๐ | Fire at wall-clock time |
| TimerService.deleteEventTimeTimer(key, timestamp) | ๐ | Cancel event-time timer |
| TimerService.deleteProcessingTimeTimer(key, timestamp) | ๐ | Cancel processing-time timer |
| TimerService.registerEventTimeTimer(key, name, timestamp) | ๐ | Named timer: one per (key, name) across both domains โ registering the name again replaces its pending timer, at another time or a processing-time one |
| TimerService.registerProcessingTimeTimer(key, name, timestamp) | ๐ | Named processing-time timer โ replaces the name's pending timer, also an event-time one |
| TimerService.deleteTimer(key, name) | ๐ | Delete a named timer by name, whatever its domain |
| TimerService.deleteAllNamedTimers(key) | ๐ | Delete every named timer of a key; unnamed timers stay |
| TimerService.namedTimer(key, name) | ๐ | The pending NamedTimer (timeNotion(), timestamp()), or null โ also for the firing timer inside its own onTimer, until that call re-arms it |
| TimerService.registerEventTimeTimer(key, name, timestamp, headers) / registerProcessingTimeTimer(key, name, timestamp, headers) | ๐ | A timer with headers of its own, upserted by key into the ones it captures from the arming record (timer headers) |
| TimerContext.timerName() / FixedKeyTimerContext.timerName() | ๐ | The fired timer's name, null for an unnamed timer |
| TimerContext.headers() / FixedKeyTimerContext.headers() | ๐ | The fired timer's headers โ the arming record's, plus those its registration passed; a fresh copy per fire; a record onTimer forwards with forward(key, value) (fixed-key: forward(value)) carries them, a forwarded Record keeps its own headers, as in process() |
| TimerService.currentWatermark() | ๐ | Current event-time watermark |
| TimerService.currentProcessingTime() | ๐ | Current wall-clock time (the driver's wall clock under the TopologyTestDriver) |
PunctuatorMode and the key-based TimerService (which fires in the key's affinity lane, also with full read/write state access) are StoatFlow extensions.Scheduled sources (StoatFlow extension)
A ScheduledEmitter โ registered via StreamsBuilder.scheduled(...) โ periodically emits records (interval- or cron-based) using the ScheduledEmitterContext below, without consuming from Kafka.
ScheduledEmitterContext
| Method | Status | Notes |
|---|---|---|
| forward(key, value) | ๐ | Emit record stamped with the fire time โ the watermark that fired a STREAM_TIME source, the wall-clock time otherwise |
| forward(key, value, timestamp) | ๐ | Emit record with explicit timestamp |
| forward(record) | ๐ | Emit record with key, value, timestamp, headers |
| getStateStore(name) | ๐ | Access a state store read-only โ writes throw UnsupportedOperationException. Every family is covered (plain, timestamped, versioned, headers-aware) and the view keeps the store's declared type, so a getStateStore<TimestampedKeyValueStore<..>> cast still succeeds |
| currentWatermarkMs() | ๐ | Current watermark (global event-time progress) |
| currentStreamTimeMs() | โ | Delegates to currentWatermarkMs() (KS compat) |
| currentSystemTimeMs() | โ | Wall-clock time via System.currentTimeMillis() (KS compat); under the TopologyTestDriver, the driver's wall clock, the same as currentWallClockTime() |
| currentWallClockTime() | ๐ | Current wall-clock time |
CronExpression
| Method | Status | Notes |
|---|---|---|
| unix(expression) | ๐ | Unix cron (5 fields: min hour dom mon dow) |
| quartz(expression) | ๐ | Quartz cron (6 fields with seconds) |
| spring(expression) | ๐ | Spring cron (6 fields with seconds) |
| parse(expression, cronType) | ๐ | Parse with explicit type |
| nextExecutionTime(afterMs, zoneId?) | ๐ | Calculate next fire time |
| timeUntilNextExecution(fromMs, zoneId?) | ๐ | Duration until next fire |
Functional interfaces
KS-compatible functional interfaces backing the Java-friendly DSL overloads.
| Interface | Status | Used in |
|---|---|---|
KeyValueMapper<K, V, VR> | โ | map, flatMap, groupBy, FK joins |
ValueMapper<V, VR> | โ | KTable.mapValues, FK join extractors |
ValueMapperWithKey<K, V, VR> | โ | mapValues with key access |
Predicate<K, V> | โ | filter, filterNot |
ForeachAction<K, V> | โ | peek, forEach |
Initializer<VA> | โ | All aggregate() |
Aggregator<K, V, VA> | โ | All aggregate() โ incl. both the adder and subtractor of KGroupedTable.aggregate() (KS uses Aggregator for both; no Subtractor type) |
Reducer<V> | โ | All reduce() |
Merger<K, V> | โ | Session window aggregate() |
ValueJoiner<V1, V2, VR> | โ | All joins |
ValueJoinerWithKey<K, V1, V2, VR> | โ | Key-aware joins. In a GlobalKTable join the key is the stream key, not the join key โ those overloads are deprecated since Kafka Streams 4.4 |
ValueJoinerWithStreamAndMappedKey<StreamKey, TableKey, StreamValue, TableValue, VOut> | โ | Kafka Streams 4.4 (KIP-1340): KStreamโGlobalKTable joins, apply(streamKey, mappedKey, value1, value2). On a left join the mapped key and the table value may be null, so a Kotlin left-join joiner types them nullable โ one declared with non-null type arguments compiles on Kafka Streams (Java platform types) but not here. A null-value record is dropped before the key selector runs (Kafka Streams runs it first) |
StreamPartitioner<K, V> | โ | Kafka Streams 4.4 shape (KIP-1321): the default partitions(topic, key, value, headers, numPartitions) is what StoatFlow calls, with the record's headers; the KIP-837 4-arg partitions(...) stays the single abstract method, deprecated for removal. Sink multicast matches KS; the in-memory repartition path throws on multicast (single-instance) and passes the lane count. Lives in the topology package |
Configuration helper classes
Consumed
| Method | Status | Notes |
|---|---|---|
| with(keySerde, valueSerde) | โ | |
| with(keySerde, valueSerde, timestampExtractor, resetPolicy) | โ | Extractor adapted to a stream-time watermark strategy |
| keySerde(serde) | โ | |
| valueSerde(serde) | โ | |
| as(name) | โ | |
| withWatermarkStrategy(strategy) | ๐ | StoatFlow extension (mutually exclusive with withTimestampExtractor) |
| withTimestampExtractor() | โ | KS TimestampExtractor adapted to a per-source watermark strategy that assigns event time and emits a monotonous watermark โ the highest timestamp seen less 1 ms, so the source's watermark follows its newest record as Kafka Streams' partition time does. Windows close and judge lateness on the global watermark, the minimum across partitions: as in KS for a single partition, and with several once the slowest has passed them. Use withWatermarkStrategy for bounded-out-of-orderness/idleness. The extractor receives the real partition time (restored from the committed offsets after a restart) |
| negative event times | ๐ | Every event time but Long.MIN_VALUE is valid end to end, as in Flink โ Kafka Streams fails the application on a negative one by default (FailOnInvalidTimestamp) and otherwise drops it. Windows, joins, onWindowClose() and the Processor API handle it; a sink writes the record without a timestamp (the producer or broker stamps the wall-clock time, counted as stoatflow.sink.records.negative.timestamp.total). A missing time โ Kafka's -1 on a source with no extractor or assigner, or Long.MIN_VALUE from one โ is skipped and counted (dropped.records{reason=no_event_time}). To keep the KS refusal, use the built-in extractors โ see Negative event time under the architectural differences (KSC-138) |
| withOffsetResetPolicy() | โ | Per-source offset reset. AutoOffsetReset: Earliest / Latest / None / byDuration(Duration) (KIP-1106 โ seek to the first offset at/after now โ duration) |
| withNumberOfLanes(int) | ๐ | Sets this source chain's lane count (parallelism) |
| withLaneQueueCapacity(int) | ๐ | Sets this source chain's per-lane queue capacity |
WatermarkStrategy (Flink-inspired)
Event-time progress is driven by a WatermarkStrategy. A KS Consumed.withTimestampExtractor(...) is supported and adapted to one under the hood; Consumed.withWatermarkStrategy(...) is the richer StoatFlow-native form (bounded out-of-orderness, idleness, alignment).
| Method | Status | Notes |
|---|---|---|
| forBoundedOutOfOrderness(maxOutOfOrderness) | ๐ | Bounded out-of-orderness |
| forMonotonousTimestamps() | ๐ | Strictly increasing timestamps |
| noWatermarks() | ๐ | Processing-time only |
| withTimestampAssigner(assigner) | ๐ | Custom event-time assignment |
| withIdleness(duration) | ๐ | Idle-source handling |
| withWatermarkAlignment(group, maxDrift, updateInterval) | ๐ | Flink FLIP-217 alignment, Flink's signature. A partition whose watermark drifts beyond the group cap is held, never paused: it keeps being fetched, its records go only in timestamp order, and while a lagging partition has nothing buffered the whole application waits (the drift bound applies against empty laggards; strict order otherwise). Requires idleness on the group's sources. See Watermark alignment |
The TimestampExtractor SAM is present with the KS-exact, non-generic signature long extract(ConsumerRecord<Object, Object>, long), and Consumed.withTimestampExtractor(...) wires it to event-time + stream-time watermarks.
KS-optional builder arguments (serdes, grace, partitioner, offset-reset, name) across Consumed/Grouped/Produced/Repartitioned/Joined/StreamJoined/Materialized/Branched accept null (= "use the default"), matching Kafka Streams; Named itself still requires a non-null name.
Built-in timestamp extractors
Kafka Streams' built-in extractors ship in io.stoatflow.core.processor under their Kafka Streams names, so a ported default.timestamp.extractor or Consumed.withTimestampExtractor(...) keeps its meaning after the import swap. default.timestamp.extractor also accepts their Kafka Streams class names (org.apache.kafka.streams.processor.*, a string or a Class) and loads StoatFlow's, so a ported Properties works unchanged. As in Kafka Streams, the first three treat every negative record timestamp as invalid โ they are how a port keeps Kafka Streams' refusal of negative times:
| Extractor | Status | Notes |
|---|---|---|
| FailOnInvalidTimestamp | โ | Fails the application on a negative record timestamp โ Kafka Streams' default default.timestamp.extractor |
| LogAndSkipOnInvalidTimestamp | โ | Skips a record with a negative timestamp, with a WARN; counted as dropped.records{reason=no_event_time} and its offset committed. Its onInvalidTimestamp returns Long.MIN_VALUE (StoatFlow's "no event time") where Kafka Streams returns the negative timestamp for its record queue to drop โ the record is skipped either way |
| UsePartitionTimeOnInvalidTimestamp | โ | Replaces a negative record timestamp with the partition time; fails while the partition time is unknown, as in Kafka Streams |
| WallclockTimestampExtractor | โ | The current wall-clock time as event time |
Every extractor receives the real partition time: the highest event time extracted from the record's partition before it, restored from the committed offsets after a restart.
Grouped
| Method | Status | Notes |
|---|---|---|
| as(name) | โ | |
| keySerde(serde) | โ | |
| valueSerde(serde) | โ | |
| with(keySerde, valueSerde) | โ | |
| with(name, keySerde, valueSerde) | โ | |
| withName(name) | โ | |
| withKeySerde(serde) | โ | |
| withValueSerde(serde) | โ | |
| withNumberOfLanes(int) | ๐ | Sets the grouped sub-topology's lane count |
| withLaneQueueCapacity(int) | ๐ | Sets the grouped sub-topology's per-lane queue capacity |
Produced
| Method | Status | Notes |
|---|---|---|
| with(keySerde, valueSerde) | โ | |
| keySerde(serde) | โ | |
| valueSerde(serde) | โ | |
| as(name) | โ | |
| streamPartitioner(partitioner) | โ |
Materialized
| Method | Status | Notes |
|---|---|---|
| as(storeName) | โ | |
| as(storeSupplier) | โ | |
| as(DslStoreSuppliers) | โ | Store type supplier. Applies only when no explicit supplier is set โ an explicit as(supplier) wins (Kafka Streams' order since KSC-100, 2026-09-09) |
| with(keySerde, valueSerde) | โ | |
| keySerde(serde) | โ | |
| valueSerde(serde) | โ | |
| withCachingEnabled() | โ | Sets the store's cache caps (default caps, or EmissionSuppression limits). Caching is always on and never changes downstream emissions โ every update is forwarded (KSC-88; consolidate with Suppressed.untilBarrier()) |
| withCachingDisabled() | โ | Accepted, changes nothing: caching cannot be turned off and the store keeps the default caps; every update is forwarded anyway (KSC-88) |
| withLoggingEnabled() | โ | Enable changelog with optional topic config |
| withLoggingDisabled() | โ | Disable changelog for this store |
| withRetention() | โ | Override computed retention for windowed stores. It is a minimum โ a persistent store keeps up to retention + segmentInterval; see window and session retention |
| withStoreType(DslStoreSuppliers) | โ | Store type selection. Keeps an explicit supplier, which wins over it (Kafka Streams' order since KSC-100); both set โ one WARN per store at build |
| withRecordHeaders() | ๐ | Persist the record's headers alongside key/value in the store (KIP-1271 / KIP-1285). Its serdes then see headers as in Kafka Streams 4.4 โ a write's key and value with the value's headers, lookups with the current record's, iterators with each entry's stored headers โ always on a copy, so the record's own headers never gain a store's schema id. A store without record headers calls its serdes with empty headers (Kafka Streams: the current record's), so a schema-registry serde in header mode needs this option |
| withoutRecordHeaders() | ๐ | Disable header persistence (default) |
| withChangelogNumPartitions(n) | ๐ | Per-store changelog partition count. No KS equivalent โ KS co-partitioning forces the count to equal the source topic's, so it is never a decision there. Create-time immutable, and bidirectional: with the default at 3, pinning a small store back to 1 is the common use. On a foreign-key join the internal -subscription-store inherits it too |
Branched
| Method | Status | Notes |
|---|---|---|
| as(name) | โ | |
| withName(name) | โ | |
| withFunction(chain) | โ | |
| withConsumer(consumer) | โ | Java Consumer overload |
Named
Named is open/subclassable, so KSML's name-validation subclasses work.
| Method | Status | Notes |
|---|---|---|
| as(name) | โ | |
| withName(name) | โ | Fluent builder form |
| name() | โ | Read the configured name |
Printed
| Method | Status | Notes |
|---|---|---|
| toSysOut() | โ | Default stdout |
| toFile(filePath) | โ | File output |
| withLabel(label) | โ | Prefix label |
| withKeyValueMapper(mapper) | โ | Custom formatting; takes a KS KeyValueMapper<K, V, String> |
| withName(processorName) | โ | Set processor name |
Joined
Joined<K, VLeft, VRight> is the Kafka Streams generic form. Its serdes and grace period are advisory โ stored and round-tripped for source compatibility, but StoatFlow resolves the effective serdes/grace from the upstream Consumed/Grouped configuration.
| Method | Status | Notes |
|---|---|---|
| as(name) | โ | |
| create() | โ | Create with defaults |
| with(keySerde, valueSerde, otherValueSerde) | โ | Static factory; serdes advisory |
| withName(name) | โ | |
| withGracePeriod(duration) | โ | Advisory grace period |
| withKeySerde(serde) | โ | Advisory |
| withValueSerde(serde) | โ | Advisory |
| withOtherValueSerde(serde) | โ | Advisory |
| name() | โ | Read the configured name |
| gracePeriod() | โ | Round-trips the advisory grace period |
| keySerde() / valueSerde() / otherValueSerde() | โ | Round-trip the advisory serdes |
TableJoined
| Method | Status | Notes |
|---|---|---|
| as(name) | โ | |
| with() | โ | Defaults |
| with(partitioner, otherPartitioner) | โ | Accepted no-op stub (partitioning is N/A single-instance) |
| withName(name) | โ | |
| withPartitioner() | โ | Accepted no-op stub for KS source compatibility |
| withOtherPartitioner() | โ | Accepted no-op stub for KS source compatibility |
JoinWindows
| Method | Status | Notes |
|---|---|---|
| of(timeDifference) | โ๏ธ | Deprecated; use ofTimeDifferenceWithNoGrace |
| ofTimeDifferenceWithNoGrace(timeDifference) | โ | Symmetric window, no grace |
| ofTimeDifferenceAndGrace(timeDifference, grace) | โ | Symmetric window with grace |
| before(timeDifference) | โ | Asymmetric window (look back) |
| after(timeDifference) | โ | Asymmetric window (look forward) |
| grace(duration) | โ๏ธ | Deprecated; use ofTimeDifferenceAndGrace |
| size() | โ | Window size in ms |
| gracePeriodMs() | โ | Grace period in ms |
beforeMs / afterMs are exposed as read-only properties (getBeforeMs() / getAfterMs() from Java; before / after return Duration). A KS port that reads the public jw.beforeMs field replaces it with the accessor.StreamJoined
| Method | Status | Notes |
|---|---|---|
| as(name) | โ | Create with processor name |
| with(keySerde, leftValueSerde, rightValueSerde) | โ | Create with Serdes |
| create() | โ | Create with defaults |
| withName(name) | โ | Set processor name |
| withKeySerde(serde) | โ | Set key Serde |
| withLeftValueSerde(serde) | โ | Set left stream value Serde |
| withRightValueSerde(serde) | โ | Set right stream value Serde |
| withThisStoreSupplier(storeSupplier) | โ | Custom left window store. Must retain duplicates โ a non-duplicate supplier is rejected when join() is called (KSC-99), as Kafka Streams rejects it. Its retention must cover the join (before + after + grace), or it is rejected there too; Kafka Streams also requires that exact retention and a window size of before + after, where StoatFlow accepts a longer retention and any window size. Any Stores window supplier is accepted, as in Kafka Streams: a timestamped or headers-aware one builds a plain window store with the same backend, name, retention and window size, since the join keeps its serializers' headers inside the stored values (KSC-129). Wins over withDslStoreSuppliers for its side and no longer clears it (Kafka Streams' order since KSC-100) |
| withOtherStoreSupplier(storeSupplier) | โ | Custom right window store; same KSC-99 guard, same supplier handling, same KSC-100 precedence |
| withLoggingEnabled() | โ | Enable changelog logging |
| withLoggingEnabled(config) | โ | Enable changelog logging with topic config |
| withLoggingDisabled() | โ | Disable changelog logging |
| withDslStoreSuppliers(suppliers) | โ | Each join store without an explicit supplier uses this type โ an explicit supplier wins for its side and neither setter clears the other (Kafka Streams' order since KSC-100, 2026-09-09). Both retain duplicates (fixed 2026-09-03 โ before that this path silently built a non-duplicate join store); a custom DslStoreSuppliers that ignores DslWindowParams.retainDuplicates is rejected at DSL build (KSC-99) |
| withStoreName(name) | โ | Set base store name |
| withChangelogNumPartitions(n) | ๐ | Changelog partition count for this join's stores. No KS equivalent. The number is multiplied โ one call sets it on 2 stores (INNER) or 3 (LEFT/OUTER, adding -outer-store), so (6) is 12 or 18 partitions |
Repartitioned
| Method | Status | Notes |
|---|---|---|
| as(name) | โ | |
| streamPartitioner(partitioner) | โ | numPartitions = numLanes |
| withStreamPartitioner(partitioner) | โ | |
| withName(name) | โ | |
| with(keySerde, valueSerde) | โ | keySerde used for lane assignment hashing |
| withKeySerde() | โ | Used for lane assignment hashing at sub-topology boundary |
| withValueSerde() | โ | Reserved for future use |
| withNumberOfLanes(int) | ๐ | Sets the repartition sub-topology's lane count |
| numberOfLanes(int) | ๐ | Static factory; sets the lane count |
| withLaneQueueCapacity(int) | ๐ | Sets the per-lane queue capacity |
| withNumberOfPartitions() | ๐ | KS-compat alias of withNumberOfLanes (sets the lane count) |
| numberOfPartitions(int) | ๐ | KS-compat static-factory alias of numberOfLanes |
EmitStrategy
| Method | Status | Notes |
|---|---|---|
| onWindowUpdate() | โ | Default; emit on every update |
| onWindowClose() | โ | Emit only when window closes; driven by the global event-time watermark |
OnWindowClose emission is watermark-driven: a window's final result is emitted when the global watermark passes the window end plus any configured grace period. See Event time and watermarks.Suppressed
| Method | Status | Notes |
|---|---|---|
| untilWindowCloses(bufferConfig) | โ | For windowed KTables; requires a StrictBufferConfig (KS parity); flush driven by the global event-time watermark. A final result keeps its latest headers and timestamp; a null final result is dropped (KS parity). The grace period comes from the window definition upstream, through any operators in between (the largest across a merge); with no windowed ancestor the build fails with Kafka Streams' "Window close time is only defined for windowed computations" |
| untilTimeLimit(duration, bufferConfig) | โ ๏ธ | Rate-limits per-key updates with KS semantics โ every update buffered, the wait counted from a key's first update. โ ๏ธ The wait is measured on the watermark, not the task's stream time: Duration.ZERO emits on the next watermark advance, and a release trails by the out-of-orderness bound |
| withName(name) | โ | KS NamedOperation; names the suppression node |
untilWindowCloses releases a key's final value once the window closes, and untilTimeLimit releases the latest buffered value once the per-key time limit elapses in event time. See Event time and watermarks. Suppress buffers (and versioned stores, which the migration tool supports as experimental) are the two store families with state-carrying migration caveats.suppress() throws a validation exception for tables backed by versioned stores โ their temporal semantics conflict with suppression.BufferConfig
| Method | Status | Notes |
|---|---|---|
| unbounded() | โ | No limits (use with caution) |
| maxRecords(count) | โ | Bounded by record count |
| maxBytes(bytes) | โ | Bounded by the buffer's serialized size, headers included (KS estimates memory); checked after each update, as in KS |
| withMaxRecords(count) | โ | Fluent record bound |
| withMaxBytes(bytes) | โ | Fluent byte bound |
| withNoBound() | โ | Remove bounds |
| emitEarlyWhenFull() | โ | Emit instead of evict on overflow |
| shutDownWhenFull() | โ | Shut down on buffer overflow |
| withLoggingEnabled(config) / withLoggingDisabled() | โ | Changelog control for the suppress buffer |
| withChangelogNumPartitions(n) / changelogNumPartitions() | ๐ | Changelog partition count for the buffer store. No KS equivalent; returns BC, preserving the self-type. The buffer follows the global changelog.num-partitions by default, like every other store; set this only on a measurement |
WindowedSerdes
| Method | Status | Notes |
|---|---|---|
| timeWindowedSerdeFrom(...) | โ ๏ธ | Takes Serde<T> instead of Class<T> (no serde registry) |
| timeWindowedSerdeFrom(..., windowSize) | โ ๏ธ | Takes Serde<T> only; window size encoded in serialized bytes |
| sessionWindowedSerdeFrom(...) | โ ๏ธ | Takes Serde<T> instead of Class<T> |
| TimeWindowedSerde(inner) | โ | WindowedSerdes.TimeWindowedSerde(serde) |
| TimeWindowedSerde(inner, windowSize) | โ ๏ธ | Window size not required (encoded in serialized bytes) |
| SessionWindowedSerde(inner) | โ | WindowedSerdes.SessionWindowedSerde(serde) |
Serde<T> rather than KS's Class<T> (there is no serde registry), and omit the window-size parameter (window bounds are encoded in the serialized bytes). TimeWindowedSerde.forChangelog() is not needed โ StoatFlow uses identical encoding for client and changelog.State stores
Record types & iterators
| Type | Status | Notes |
|---|---|---|
| Windowed<K> | โ | Windowed-key wrapper (key + Window) |
| ValueAndTimestamp<V> | โ | Timestamped-store value wrapper (KIP-258). Null semantics as in Kafka Streams: make(null, ts) returns null, makeAllowNullable an instance; a null value is written as a delete and a null-deserialized read returns null |
| VersionedRecord<V> | โ | Versioned-store record; validTo() returns Optional<Long> |
| KeyValueIterator / WindowStoreIterator | โ | Closeable iterators for range/scan queries; use after close() throws InvalidStateStoreException, as in Kafka Streams โ and so does any read on a closed store |
| ReadOnlyRecord<K, V> | โ | KIP-1356: the read-only record a headers-aware query returns (key, value, timestamp, headers), in io.stoatflow.core.processor โ the migration recipe rewrites Kafka Streams' processor.api import. Record implements it, and so does FixedKeyRecord (Kafka Streams' does not). Its timestamp may be negative (StoatFlow accepts negative event times) |
| ReadOnlyRecordIterator<K, V> | โ | KIP-1356: a closeable iterator of ReadOnlyRecords; remove() unsupported |
State store metrics
| Metric | Status | Notes |
|---|---|---|
| num-keys (KIP-1250) | โ | Entry-count gauge for in-memory KV/window/session stores |
| uncommitted-bytes (KIP-892) | โ | stoatflow.store.uncommitted.bytes, INFO level, as in Kafka Streams. Emitted for every store family, where Kafka Streams registers it on key-value stores only (KSC-118); the application-wide total is stoatflow.barrier.epoch.uncommitted.bytes |
StateStore
| Method | Status | Notes |
|---|---|---|
| approximateNumUncommittedBytes() | โ | KIP-892 (Kafka Streams 4.4); defaults to 0 for a custom store. Counts the bytes held in memory for epochs not yet written to RocksDB, including the committed epoch still being written, which READ_COMMITTED can already read โ memory, not visibility (KSC-117). A custom store a builder of your own returns is not counted towards the memory limit, whatever it reports: no commit reaches it, so its bytes could never drain |
KeyValueStore
| Method | Status | Notes |
|---|---|---|
| get(key) | โ | |
| put(key, value) | โ | |
| putIfAbsent(key, value) | โ | |
| putAll(entries) | โ | |
| delete(key) | โ | Reads and returns the previous value in both engines โ a read-then-write. Prefer put(key, null) when you do not need it: Deleting a key |
| compute(key, remappingFunction) | ๐ | Atomic read-compute-write; Kotlin lambda + Java BiFunction overloads |
| merge(key, value, remappingFunction) | ๐ | Atomic merge; Kotlin lambda + Java BiFunction overloads |
| all() | โ | Read-your-writes semantics |
| range(from, to) | โ | Read-your-writes semantics. Null bounds = open-ended (KIP-763, KSC-89); from > to returns empty + WARN (KS-lenient) |
| reverseAll() | โ | Read-your-writes semantics |
| reverseRange(from, to) | โ | Read-your-writes semantics. Null bounds = open-ended (KIP-763, KSC-89); from > to returns empty + WARN (KS-lenient); (low, high) bounds order, KS-exact (KSC-90 โ was (high, low) pre-2026-07-31) |
| approximateNumEntries() | โ | |
| prefixScan() | โ | KIP-614; read-your-writes semantics |
ReadOnlyKeyValueStore
| Method | Status | Notes |
|---|---|---|
| get(key) | โ | |
| containsKey(key) | ๐ | StoatFlow extension; bloom filter optimization for RocksDB |
| all() | โ | |
| range(from, to) | โ | Null bounds = open-ended (KIP-763, KSC-89); from > to returns empty + WARN (KS-lenient) |
| reverseAll() | โ | |
| reverseRange(from, to) | โ | Null bounds = open-ended (KIP-763, KSC-89); from > to returns empty + WARN (KS-lenient); (low, high) bounds order, KS-exact (KSC-90 โ was (high, low) pre-2026-07-31) |
| approximateNumEntries() | โ | |
| prefixScan(prefix, serializer) | โ | KIP-614 |
| readOnly(isolationLevel) | โ | KIP-892 (Kafka Streams 4.4). READ_COMMITTED reads the state of the last commit barrier; READ_UNCOMMITTED returns the store itself. Iterators over an in-memory store are weakly consistent: never an uncommitted row, but an iterator held open can see later commits. An lruMap, an in-memory versioned store, or a custom store with no caching layer keeps no commit buffer and answers live, with a warning (KSC-116, KSC-106). Isolation level |
WindowStore
Key-range bounds are compared as serialized bytes (unsigned lexicographic); null bounds are open-ended (KIP-763, KSC-91). All Instant variants delegate to the Long variants.
| Method | Status | Notes |
|---|---|---|
| put(key, value, windowStartTime) | โ | |
| fetch(key, windowStartTime) | โ | Point lookup |
| fetch(key, timeFrom, timeTo) | โ | Single key time range; Long and Instant variants |
| backwardFetch(key, timeFrom, timeTo) | โ | Reverse iteration; Long and Instant variants |
| fetch(keyFrom, keyTo, timeFrom, timeTo) | โ | Key range + time range; Long and Instant variants. Null key bounds = open-ended (KIP-763, KSC-91); keyFrom > keyTo returns empty + WARN (KS-lenient) |
| backwardFetch(keyFrom, keyTo, timeFrom, timeTo) | โ | Reverse key range; bounds stay (low, high) like fetch (KS-exact); Long and Instant variants. Null key bounds = open-ended (KIP-763, KSC-91) |
| fetchAll(timeFrom, timeTo) | โ | All keys time range; Long and Instant variants |
| backwardFetchAll(timeFrom, timeTo) | โ | Reverse all keys; Long and Instant variants |
| all() | โ | All entries |
| backwardAll() | โ | Reverse iteration |
| windowSizeMs | ๐ | StoatFlow extension for window size access |
| retentionMs | ๐ | StoatFlow extension for retention access |
| expireWindows(watermark) | ๐ | StoatFlow extension for manual expiration |
| approximateNumEntries() | ๐ | StoatFlow extension for entry count |
ReadOnlyWindowStore
| Method | Status | Notes |
|---|---|---|
| All query methods from WindowStore | โ | 16 query methods total |
| containsKey(key, windowStartTime) | ๐ | StoatFlow extension; Long and Instant variants |
| readOnly(isolationLevel) | โ | KIP-892 (Kafka Streams 4.4). READ_COMMITTED reads the state of the last commit barrier; READ_UNCOMMITTED returns the store itself. Iterators over the default segmented store, or an in-memory one, are weakly consistent: never an uncommitted row, but an iterator held open can see later commits. An lruMap, an in-memory versioned store, or a custom store with no caching layer keeps no commit buffer and answers live, with a warning (KSC-116, KSC-106). Isolation level |
SessionStore
| Method | Status | Notes |
|---|---|---|
| put(sessionKey, aggregate) | โ | Windowed key |
| remove(sessionKey) | โ | Windowed key |
| fetchSession(key, startTime, endTime) | โ | Long and Instant variants |
| findSessions(key, earliestEndTime, latestStartTime) | โ | Long and Instant variants |
| backwardFindSessions(key, earliestEndTime, latestStartTime) | โ | Long and Instant variants |
| findSessions(keyFrom, keyTo, earliestEndTime, latestStartTime) | โ | Key range; Long and Instant variants. Null key bounds = open-ended (KIP-763, KSC-91); keyFrom > keyTo returns empty + WARN (KS-lenient) |
| backwardFindSessions(keyFrom, keyTo, earliestEndTime, latestStartTime) | โ | Key range; Long and Instant variants. Null key bounds = open-ended (KIP-763, KSC-91) |
| fetch(key) | โ | All sessions for single key |
| backwardFetch(key) | โ | Reverse order |
| fetch(keyFrom, keyTo) | โ | Key range queries. Null key bounds = open-ended (KIP-763, KSC-91); keyFrom > keyTo returns empty + WARN (KS-lenient) |
| backwardFetch(keyFrom, keyTo) | โ | Reverse order; bounds stay (low, high) (KS-exact). Null key bounds = open-ended (KIP-763, KSC-91) |
| findAll() | ๐ | StoatFlow extension |
| expireSessions(watermark) | ๐ | StoatFlow extension |
| approximateNumEntries() | ๐ | StoatFlow extension |
ReadOnlySessionStore
| Method | Status | Notes |
|---|---|---|
| All query methods from SessionStore | โ | Query methods for IQ access |
| containsSession(key, startTime, endTime) | ๐ | StoatFlow extension; Long and Instant variants |
| readOnly(isolationLevel) | โ | KIP-892 (Kafka Streams 4.4). READ_COMMITTED reads the state of the last commit barrier; READ_UNCOMMITTED returns the store itself. Iterators over the default segmented store, or an in-memory one, are weakly consistent: never an uncommitted row, but an iterator held open can see later commits. An lruMap, an in-memory versioned store, or a custom store with no caching layer keeps no commit buffer and answers live, with a warning (KSC-116, KSC-106). Isolation level |
Stores factory
StoatFlow's suppliers implement the KS *BytesStoreSupplier interfaces and the *StoreBuilder(...) factories return a KS StoreBuilder<T>. Some signatures take a Serde<T> where KS takes Class<T> (no serde registry); StoreBuilder.withCaching*/withLogging* are honored, and build() throws (StoatFlow builds stores via the registry). A custom StateStoreBuilder whose build() returns a persistent supplier's get() is refused at startup (KSC-124): register the supplier through Stores.keyValueStoreBuilder(supplier, โฆ) (or the window/session equivalent) or Materialized.as(supplier).
| Method | Status | Notes |
|---|---|---|
| persistentKeyValueStore(name) | โ | RocksDB key-value store |
| inMemoryKeyValueStore(name) | โ | In-memory key-value store |
| persistentWindowStore(name, retention, windowSize, retainDuplicates) | โ | RocksDB window store. Segments are sized max(retention / 3, 60s, windowSize) (KS: max(retention / 2, 60s)), with no way to configure it, and expiry drops whole segments โ so effective retention is up to 4/3ร what you configure where the divisor governs (retention โฅ 180s), up to retention + 60s where the 60s floor governs (see the windowing docs), and up to 2ร where windowSize governs on the segmented backend, which is the common case since ofSizeWithNoGrace makes retention == windowSize (retention is a minimum bound, as in Kafka Streams, whose own slack is up to 3/2ร; in-memory stores retain exactly 1ร). Changing retention on an existing segmented store is safe โ each segment records its own time range on disk, so the change remaps rather than being reinterpreted; Kafka Streams requires a wipe. Only a hand-built non-segmented store (no Stores factory builds one) keeps the interval in every key, where a retention change that moves it needs a restore. Changing the window size, switching plainโtimestamped, or flipping retainDuplicates is detected at startup: a store whose changelog is its recovery source refuses to start, since its changelog holds the old layout and a rebuild would misread it (see Changes a changelog cannot rebuild), and a store with no changelog refuses too, or starts empty under stoatflow.state.layout-mismatch: wipe-and-restore. A stream-stream join buffer's window size is not compared: a join reads its buffers by record timestamp. retainDuplicates=true (the stream-stream join-buffer setting) uses the same segmented backend since 2026-09-03, so its expiry reclaims disk by dropping whole segments too โ before that it ran a non-segmented backend whose deleteRange expiry reclaimed nothing until a later background compaction happened to cover the tombstone |
| inMemoryWindowStore(name, retention, windowSize, retainDuplicates) | โ | In-memory window store. Not segmented: expires per record at exactly retention, and its supplier reports segmentIntervalMs() == 1, as KS's does. |
| persistentSessionStore(name, retentionPeriod) | โ | RocksDB session store. Segments are sized max(retention / 3, 60s) โ a session has no window size, so KIP-1366's >= windowSize term does not apply โ not configurable, and effective retention is up to retention + segmentInterval. On a uniform layout StoatFlow never retains more than Kafka Streams here: its divisor is /3 against KS's /2 and both are floored at 60s, so its interval is never the larger of the two โ equal while both sit on the floor, tighter once KS leaves it. (Through a retention narrowing that ordering is suspended, since the older, wider segments below are still live.) Bounds assume the default onWindowUpdate(). See the windowing docs for the regimes and worked examples. A retention change remaps rather than requiring a wipe. After a retention narrowing, existing segments keep their original span, so the store frees disk gradually rather than at once |
| inMemorySessionStore(name, retentionPeriod, expiryIndex) | โ | In-memory session store. Not segmented: expires per record at exactly retention, and its supplier reports segmentIntervalMs() == 1, as KS's does. ๐ The third parameter (default false) opts into an end-time expiry index โ no KS equivalent; the two-argument call is unchanged. See expiring sessions from an in-memory store. |
| inMemorySessionStoreWithHeaders(name, retentionPeriod, expiryIndex) | โ | In-memory headers-aware session store (KIP-1271). ๐ Same expiryIndex third parameter as inMemorySessionStore โ it builds the same store class. |
| persistentTimestampedKeyValueStore(name) | โ | Timestamped KV store (KIP-258) |
| persistentTimestampedWindowStore(name, retention, windowSize, retainDuplicates) | โ | Timestamped window store (KIP-258); 4-arg retainDuplicates overload supported. Same segment sizing as persistentWindowStore (segmented whatever the retainDuplicates value, since 2026-09-03) and the same effective-retention bounds. Switching a populated store between this and persistentWindowStore changes the value format: a store whose changelog is its recovery source refuses to start (see Changes a changelog cannot rebuild) |
| inMemoryTimestampedKeyValueStore(name) | โ | In-memory timestamped KV store (KIP-258) |
| inMemoryTimestampedWindowStore(name, ...) | โ | In-memory timestamped window store (KIP-258). Not segmented: expires per record at exactly retention, and segmentIntervalMs() reports 1, as KS's does โ the segmented persistentTimestampedWindowStore keeps up to retention + segmentInterval. |
| persistentTimestampedWindowStoreWithHeaders(name, retention, windowSize, retainDuplicates) | โ | Headers-aware timestamped window store (KIP-1271), with Kafka Streams 4.4's 4-arg retainDuplicates form; the 3-arg form (false) is StoatFlow's own. A duplicate-retaining one is accepted as a stream-stream join store |
| inMemoryTimestampedWindowStoreWithHeaders(name, retention, windowSize, retainDuplicates) | ๐ | In-memory headers-aware timestamped window store โ no Kafka Streams equivalent. Same retainDuplicates parameter |
| persistentVersionedKeyValueStore(name, historyRetention) | โ | Versioned store โ RocksDB (KIP-889). Point-in-time reads below the retention floor return the latest version if it satisfies the bound, else null โ KS-exact. Divergence: the stream time behind the grace window is shared by every lane writing the store, where KS keeps one per task; under exactly_once a write and a read are judged on it as of the lane's own commit. Divergence: retention is a minimum bound. KS drops whole segments; StoatFlow reclaims with a budgeted background sweep (stoatflow.state.versioned.sweep-period-ms), so the on-disk footprint is bounded rather than exact, and a standby applies the active's reclamation tombstones, so its footprint tracks the active within replication lag. Under exactly_once every lane reads it as of its own commit, so any join can read it |
| persistentVersionedKeyValueStore(name, historyRetention, segmentInterval) | โ | 3-arg KS overload; segmentInterval accepted but ignored (not segmented) โ logs a warning |
WindowBytesStoreSupplier.segmentIntervalMs() / SessionBytesStoreSupplier.segmentIntervalMs() | โ | Reports the real interval for the backend the supplier builds โ max(retention / 3, 60s, windowSize) for a segmented window store, max(retention / 3, 60s) for a session store, and 1 for an in-memory one, exactly as Kafka Streams' suppliers do |
| inMemoryVersionedKeyValueStore(name, historyRetention) | โ | Versioned store โ in-memory (KIP-889). ๐ no KS equivalent. Same read contract and sweep as the persistent backend; its observed stream time is not durable, so it restarts unset. No per-commit write buffer: exactly-once through an engine rebuild of an aborted commit, but its reads are live, so it answers READ_COMMITTED live, with a warning (KSC-106), and under exactly_once it is refused as a table a join reads from another lane (KSC-126) โ the TopologyTestDriver judges the declared store, so it refuses this one and accepts persistentVersionedKeyValueStore |
| lruMap(name, maxCacheSize, tombstoneOnEviction) | ๐ | StoatFlow extension: with true, each eviction also writes a changelog delete (needs logging; refused with withLoggingDisabled()). The two-argument form keeps Kafka Streams' behaviour (evictions are local only) |
| lruMap(name, maxCacheSize) | โ | LRU cache store with bounded entries. No per-commit write buffer (exactly-once through an engine rebuild of an aborted commit); reads are always live, so it answers READ_COMMITTED live, with a warning (KSC-116); refused under exactly_once as a table a join reads from another lane โ either side of a primary-key or foreign-key join, a re-keyed stream-table join, a global-table join (KSC-126) |
| keyValueStoreBuilder(supplier, keySerde, valueSerde) | โ | Returns a KS StoreBuilder |
| timestampedKeyValueStoreBuilder(supplier, keySerde, valueSerde) | โ | KIP-258 |
A store switched plain โ timestamped, or to another store type, window size or retainDuplicates, under one name | โ ๏ธ | Kafka Streams upgrades a plain RocksDB key-value store reopened as timestamped in place, and checks nothing else: a resized windowed aggregation reads its old windows with the new size. A StoatFlow store whose changelog is its recovery source refuses to start when its value format flips (plain โ timestamped), its type changes (key-value โ versioned, window โ session), its window size changes or its retainDuplicates flips: its changelog holds the old layout, so it cannot rebuild the store. The store is left untouched. Only Processor API stores can flip their value format or retainDuplicates โ every DSL store is timestamped whatever its supplier; a DSL windowed aggregation's new window size refuses too, and a stream-stream join's JoinWindows change is adopted, since a join reads its buffers by record timestamp. See Changes a changelog cannot rebuild |
| windowStoreBuilder(supplier, keySerde, valueSerde) | โ | Returns a KS StoreBuilder |
| timestampedWindowStoreBuilder(supplier, keySerde, valueSerde) | โ | KIP-258 |
| sessionStoreBuilder(supplier, keySerde, aggSerde) | โ | Returns a KS StoreBuilder |
| versionedKeyValueStoreBuilder(supplier, keySerde, valueSerde) | โ | KIP-889 |
| timestamped{KeyValue,Window} / sessionStoreWithHeadersBuilder(...) | โ | Headers-aware builders (KIP-1271); pass a *WithHeaders supplier |
| KeyValue/Window/Session/VersionedBytesStoreSupplier, StoreBuilder | โ | KS supplier/builder interfaces are present |
KeyLockManager (StoatFlow extension)
Striped lock utility for multi-key atomic sections in user-defined Processors. Not in Kafka Streams.
| Method | Status | Notes |
|---|---|---|
| withLock(key, block) | ๐ | Single-key lock scope |
| withLocks(keys, block) | ๐ | Multi-key lock scope with deadlock-free ordering |
Record headers in state stores (KIP-1271 / KIP-1285)
Opt-in persistence of a record's Headers alongside key/value, across the timestamped KV, versioned, timestamped-window, and session store families.
| Surface | Status | Notes |
|---|---|---|
| Materialized.withRecordHeaders() | ๐ | Per-store opt-in (DSL) |
| dsl.store.format = HEADERS | โ | Global default to headers-aware stores |
| Stores.*WithHeaders(...) suppliers + *WithHeadersBuilder(...) | โ | Processor-API headers stores (all four families) |
| *WithHeadersBuilder(...) over a plain or timestamped supplier | ๐ | Refused when the topology is built (KSC-145); Kafka Streams accepts it and keeps no headers. Pass a Stores.*WithHeaders(...) supplier |
| QueryableStoreTypes.*WithHeaders() | โ | Read stored headers back via IQ |
| Headers-aware IQv2 queries (KIP-1356) | โ | Read stored headers back through query() โ key-value, window and session stores; see Interactive Queries v2 below |
| SessionStoreWithHeaders.expireSessions(watermark) | ๐ | StoatFlow extension (abstract). StoatFlow expires sessions from the aggregation, not inside the store, so a custom implementation must implement it. Kafka Streams' interface has no such method |
| In-place upgrade (existing store โ headers-aware) | โ | Flip on, restart โ no wipe, no changelog replay |
| Downgrade refusal (headers-aware โ plain) | โ | Like KS, a downgrade is refused with a diagnosed, actionable message rather than a raw RocksDB error โ it is a declared format change, not corruption |
state.format.downgrade = wipe-and-restore | ๐ | StoatFlow-only: acknowledge the downgrade in config and it rebuilds that store from the changelog. KS offers no equivalent โ its only remedy is deleting local state by hand |
| Empty-keyspace downgrade (nothing written in headers mode) | ๐ | Dropped in place, free. KS refuses this case too |
Reserved header names (__stoatflow.v, __stoatflow.v.*) and the stoatflow.epoch stamp | โ ๏ธ | A headers-aware store does not keep them (KSC-149): the reserved names, kept for StoatFlow's changelog format version, are dropped from the stored headers with one WARN per store naming the header, and the engine's stoatflow.epoch stamp is dropped silently. The restore drops the same names, so a live store and a restored one, a table restored from its source topic included, hold the same headers. Every other header stays, __stoatflow.vendor and the dead-letter __stoatflow.errors.* included. Kafka Streams keeps every header |
| StreamJoined headers | โ (differs) | KS 4.3+ side buffers follow dsl.store.format: under HEADERS (or in memory) they keep the record's headers plus the serializers' and a header-mode serde works; under the default they drop them and it fails. StoatFlow's side buffers always keep what their serializers wrote, never the record's, so a header-mode serde works on every store type (KSC-129) |
Interactive Queries
StoatFlow (entry point) methods
| Method | Status | Notes |
|---|---|---|
| store(StoreQueryParameters) | โ | Returns a read-only view, as in Kafka Streams, in one of two ways. When the store holds the value shape the matcher promises, the view keeps the store's interface and every write throws UnsupportedOperationException โ including through sessionStoreWithHeaders() and versionedKeyValueStoreWithHeaders(). When the matcher converts (keyValueStore() over a timestamped store, the headers-aware twins, sessionStore() over a headers-aware session store), the view implements only the ReadOnly* interface and has no write methods: a cast to a writable store type fails with ClassCastException. Either way the handle looks the store up on every read, so it survives an in-place engine restart; QueryableStoreType.create(StateStoreProvider, String) has Kafka Streams' signature |
| storeNames() | โ | List available store names |
| query(StateQueryRequest) | โ | Interactive Queries v2 (KIP-796) โ see Interactive Queries v2 below. Gates by Kafka Streams' exclusion list, so it answers on a PAUSED/DRAINING application where store() throws; NOT_ACTIVE while stores restore (KSC-109) |
| queryMetadataForKey(store, key, serializer) | โ | Returns KeyQueryMetadata โ single-instance: the local host, no standbys, the key's faithful Kafka partition |
| queryMetadataForKey(store, key, headers, serializer) | โ | KIP-1321: the key serialized with the headers, as at the sink |
| queryMetadataForKey(store, key, headers, partitioner) | โ | KIP-1321: the partition your StreamPartitioner returns; anything but a single partition throws IllegalArgumentException. โ ๏ธ Optional.empty() also throws (KS returns metadata without a usable partition) |
| streamsMetadataForStore(store) | โ | The single local host materializing the store |
| metadataForAllStreamsClients() | โ | The one local instance |
| setUncaughtExceptionHandler(handler) | โ | KS StreamsUncaughtExceptionHandler; REPLACE_THREAD triggers an in-place engine restart (fault-storm-guarded โ repeated faults escalate to shutdown), under hot-standby HA as well as without it: the fault is absorbed in place and the pod keeps the active role; SHUTDOWN_CLIENT/SHUTDOWN_APPLICATION shut the single instance down |
| pause() | ๐ | Pause processing with drain semantics |
| unpause() | ๐ | Resume processing after pause |
| awaitState(state, timeout) | ๐ | Wait for target state with timeout |
| stateTransitionHistory() | ๐ | Recent state transitions with timestamps |
StoreQueryParameters
| Method | Status | Notes |
|---|---|---|
| fromNameAndType(name, type) | โ | Factory method |
| enableStaleStores() | โ | Allow queries during RESTORING |
| withPartition(partition) | ๐ | N/A for single-instance |
QueryableStoreTypes
| Method | Status | Notes |
|---|---|---|
| keyValueStore() | โ | Returns ReadOnlyKeyValueStore. Over a timestamped store (the default for grouped and windowed aggregations) or its headers-aware twin it answers with the plain value, as in Kafka Streams |
| windowStore() | โ | Returns ReadOnlyWindowStore. Over a timestamped window store or its headers-aware twin it answers with the plain value, as in Kafka Streams |
| sessionStore() | โ | Returns ReadOnlySessionStore. Also accepts a headers-aware session store and answers with the plain aggregate, as in Kafka Streams |
| timestampedKeyValueStore() | โ | Returns ReadOnlyKeyValueStore<K, ValueAndTimestamp<V>> (KIP-258). Also accepts the headers-aware twin and drops the headers, as in Kafka Streams. Answers every DSL key-value store โ table(), toTable(), globalTable(), KTable operators and cogroup aggregates included |
| timestampedWindowStore() | โ | Returns ReadOnlyWindowStore<K, ValueAndTimestamp<V>> (KIP-258). Also accepts the headers-aware twin and drops the headers, as in Kafka Streams |
| versionedKeyValueStore() | โ | Returns ReadOnlyVersionedKeyValueStore<K, V> (KIP-889). readOnly(isolationLevel) on the versioned store (KIP-892): READ_COMMITTED answers as of the last commit, validTo included; an in-memory versioned store answers live, with a warning (KSC-106) |
StateStoreProvider.defaultIsolationLevel() | โ | Kafka Streams 4.4: the level store() applies โ the configured default.interactive.query.isolation.level โ for a custom QueryableStoreType to apply with readOnly(level) |
withPartition() and inter-instance RPC are unnecessary.Interactive Queries v2
The org.apache.kafka.streams.query package, name for name, in io.stoatflow.core.query โ the import swap ports it. The
TopologyTestDriver has query() too, which Kafka Streams' driver does not.
| Type / member | Status | Notes |
|---|---|---|
| StateQueryRequest | โ | inStore/withQuery, withPositionBound, withAllPartitions, withPartitions, enableExecutionInfo, requireActive, withIsolationLevel (KIP-892) โ immutable, as in Kafka Streams |
| StateQueryResult | โ | One store partition, 0 (KSC-107): the store answers under partition 0, and a request naming another partition gets a DOES_NOT_EXIST entry under that partition; getGlobalResult() is always null. Read it with getPartitionResults().get(StateQueryResult.SINGLE_PARTITION) |
| StateQueryResult.SINGLE_PARTITION | ๐ | 0 โ the one store partition |
| getOnlyPartitionResult() | โ | As in Kafka Streams: null for a successful null answer; throws next to a DOES_NOT_EXIST entry |
| withPartitions(S) with a partition other than 0 | โ ๏ธ | DOES_NOT_EXIST for it โ the answer Kafka Streams documents; its code returns NOT_PRESENT, which StoatFlow never returns (KSC-107) |
| QueryResult | โ | Factories and accessors as in Kafka Streams; an UNKNOWN_QUERY_TYPE message names the store, not its class (KSC-110) |
| Position / PositionBound | โ ๏ธ | Same API. Moves once per commit, not per record (KSC-108): the last commit's source-topic offsets, the same for every store, with no repartition topics โ except a store with logging disabled and no source topic, which reports only what this instance committed since it started (as Kafka Streams' position for such a store restarts empty). A bound fails on any partition the position does not hold, and query() never waits for it |
| QueryConfig, FailureReason, ResultOrder | โ | As in Kafka Streams |
| KeyQuery, RangeQuery | โ | The bare value from timestamped and headers-aware key-value stores, as in Kafka Streams. skipCache() is accepted and changes nothing (KSC-111; Kafka Streams ignores it too) |
| TimestampedKeyQuery, TimestampedRangeQuery | โ | ValueAndTimestamp from timestamped and headers-aware stores; UNKNOWN_QUERY_TYPE on a plain store |
| WindowKeyQuery, WindowRangeQuery | โ | Value shapes as in Kafka Streams: a DSL windowed aggregation answers ValueAndTimestamp elements; a session store the aggregate. Final classes, where Kafka Streams leaves them open (KSC-110) |
| VersionedKeyQuery, MultiVersionedKeyQuery | โ | As in Kafka Streams, including the exceptions for a KeyQuery on a versioned store and an inverted time range; a RocksDB store reads one consistent snapshot, lazily |
| ReadOnlyVersionedKeyValueStore.get(key, fromTime, toTime, order) | ๐ | The versioned range read behind MultiVersionedKeyQuery, public on the store (Kafka Streams keeps it internal) |
When query() answers | โ ๏ธ | KSC-109: an unknown store throws UnknownStateStoreException first, in every state. Not started or stopped: IllegalStateException (Kafka Streams has its own exception types). An HA standby throws, as store() does. While stores restore: NOT_ACTIVE, with or without requireActive(). A retriable InvalidStateStoreException from the read answers NOT_ACTIVE (Kafka Streams: STORE_EXCEPTION) |
| TimestampedKeyWithHeadersQuery | โ | KIP-1356: a ReadOnlyRecord from a headers-aware timestamped key-value store โ key, value, timestamp and frozen headers; null for a missing key or tombstone. A negative stored timestamp is returned (Kafka Streams fails it) |
| TimestampedKeyWithHeadersQuery.skipCache() | โ ๏ธ | Accepted, changes nothing (KSC-111) โ Kafka Streams honours it; use the isolation level |
| TimestampedRangeWithHeadersQuery | โ | KIP-1356: a ReadOnlyRecordIterator over a key range of a headers-aware key-value store, tombstones omitted |
| TimestampedWindowKeyWithHeadersQuery | โ | KIP-1356: one key's windows in a closed start range, from a headers-aware window store |
| TimestampedWindowRangeWithHeadersQuery | โ | KIP-1356: withWindowStartRange from a headers-aware window store, withKey from a headers-aware session store (timestamp = the session's end); each store refuses the other form, as in Kafka Streams |
| StateStore.query(...) / custom Query types | ๐ | No per-store query hook; an unknown query type answers UNKNOWN_QUERY_TYPE |
IQ metadata (single-instance: always local)
Kafka Streams' host/partition-distribution metadata types exist so cross-instance "find the host, then query" code compiles and runs. Under StoatFlow's single-instance model the answer is always "this host, locally."
| Type | Status | Notes |
|---|---|---|
| HostInfo(host, port) | โ | The local host; sourced from application.server (synthetic localhost:0 if unset) |
| StreamsMetadata | โ | hostInfo(), stateStoreNames(), topicPartitions(); no standbys |
| KeyQueryMetadata | โ | activeHost() = local, no standby hosts, partition() = the key's faithful Kafka partition |
Exception handlers
| Handler | Status | Notes |
|---|---|---|
| Source deserialisation with headers | โ | Source deserializers receive the record headers (deserialize(topic, headers, data)), as in Kafka Streams โ header-aware serdes work at the source |
| DeserializationExceptionHandler | โ | KS-ordered handle(ErrorHandlerContext, ConsumerRecord, Exception); LogAndFail (default), LogAndContinue, DLQ |
| ProcessingExceptionHandler | โ | KS-ordered handle(ErrorHandlerContext, Record, Exception); LogAndFail (default), LogAndContinue, DLQ. A FAIL verdict a processor catches and rethrows wrapped is not offered to the handler again โ Kafka Streams re-offers it at the wrapping node |
| ProductionExceptionHandler | โ | KS-ordered handle(ErrorHandlerContext, ProducerRecord, Exception); intelligent retry. A send failure no handler answered CONTINUE fails the commit before the offsets, under both guarantees, and a handler that throws is a FAIL, as in Kafka Streams. Difference (KSC-147): Kafka Streams also asks the handler about a failed changelog record or dead letter; StoatFlow treats those as FAIL without asking |
deadLetterTopics() (on all three handler interfaces) | ๐ | StoatFlow extension, default: none, inherited from one super-interface, DeadLetterTopicDeclaring โ so one class implementing several handler interfaces still compiles. The topics a handler writes dead letters to; the start checks that each exists (KSC-148; a deserialization handler's only when the topology reads a Kafka topic, a production handler's only when it has a sink). The built-in DLQ handlers return theirs; a handler that wraps one returns its delegate's. A ported handler inherits the empty default |
DeadLetterTopicDeclaring | ๐ | The interface that declares deadLetterTopics(); the three handler interfaces extend it (KSC-148). A ported handler never names it |
| ErrorHandlerContext | โ | KS interface: topic()/partition()/offset()/timestamp()/headers()/processorNodeId()/taskId(). A processing failure's headers(), the same instance as its record.headers(), is a copy of the failing record's headers (Kafka Streams hands over the record's own); a timer failure's are the timer's; a punctuator's are empty, as in Kafka Streams |
| StreamsException | โ | Extends KafkaException; StoatFlow's fatal failures extend it, so catch (StreamsException) works |
MissingOutputTopicException | ๐ | Thrown by start() when a sink topic or a declared dead-letter topic does not exist (KSC-148); extends StreamsException, missingTopics maps each missing topic to what writes to it |
Lifecycle
| Feature | Status | Notes |
|---|---|---|
| State enum | โ | StoatFlow keeps a richer set than KS's 7 โ CREATED, STARTING, VALIDATING_STATE, RESTORING, RUNNING, DRAINING, PAUSED, STOPPING, STOPPED, ERROR. For a KS-shaped KafkaStreams.State, use the KafkaStreamsState mirror + State.toKafkaStreamsState() / kafkaStreamsState() (KSC-81) |
| StateListener | โ | KS-exact single setStateListener(StateListener) โ a Java method reference / lambda resolves with no cast (KSC-82); an existing BiConsumer<State,State> migrates via StateListener.of(...) |
| StateRestoreListener | โ | Full KS-compatible: onRestoreStart / onBatchRestored / onRestoreEnd |
| StreamsUncaughtExceptionHandler | โ | setUncaughtExceptionHandler(); REPLACE_THREAD = in-place engine restart (fault-storm-guarded; works under hot-standby HA too โ absorbed in place, no failover until the budget is spent), SHUTDOWN_* = single-instance shutdown |
| cleanUp() | โ | Deletes all local state for the application id, lock files included; valid only in CREATED/STOPPED; rebuilt from changelog on next start. Unlike Kafka Streams 4.4 no process-metadata file is retained, so a full reset needs no manual delete |
| close() | โ | Graceful shutdown; blocks until terminal. Every concurrent close caller now blocks until completion (a losing second close() no longer returns early mid-shutdown) โ KSC-83 |
| close(Duration) | โ | The timeout bounds only the caller's wait, never the shutdown work (KS-exact). Duration.ZERO = async signal-and-return; negative โ IllegalArgumentException; shutdown continues after a false return. Returns true on any terminal state incl. ERROR (divergence โ KS only NOT_RUNNING) โ KSC-83 |
| close(CloseOptions) | โ | Mirrors the top-level KIP-1153 io.stoatflow.core.CloseOptions (KS-exact defaults + mutate-and-return-this fluents + @JvmStatic factories). GroupMembershipOperation has the Kafka Streams 4.4 shape โ DEFAULT, LEAVE_GROUP, REMAIN_IN_GROUP, with DEFAULT as the default and what close() passes. An explicit LEAVE_GROUP/REMAIN_IN_GROUP always wins over the leaveGroupOnClose config; DEFAULT follows it (default: remain). DEFAULT never leaves the group unless leaveGroupOnClose is set, because StoatFlow does not use the streams group protocol โ KSC-112. No-op under HA assign(). The deprecated nested KIP-812 KafkaStreams.CloseOptions is not mirrored โ a port rewrites one line to the factories โ KSC-83 |
Streams group protocol (KIP-1071 family): group.protocol=streams, the topology-description push (KIP-1331), Admin.describeStreamsGroups | ๐ | Not applicable โ StoatFlow is a classic consumer group, so describeStreamsGroups never lists it. Pull the topology description from GET /topology/ks instead โ KSC-113 |
Configuration
| Aspect | Status | Notes |
|---|---|---|
| StreamsConfig builder | โ | Type-safe builder pattern (StreamsConfig.builder(appId, bootstrap).โฆbuild()) โ the primary config model |
Properties / *_CONFIG constants | โ | Canonical KS adapter (ADR-124). StreamsConfig.fromProperties(props) / fromMap(map) adapt a KS-keyed Properties/Map onto the typed builder with complete coverage of the KS-4.4.0 overlap surface โ all 79 keys, listed one by one in KS config-key mapping below: typed mappings, class-name pluggables (serdes, the three exception handlers, default.client.supplier, rocksdb.config.setter), client prefixes (consumer./main.consumer./restore.consumer./producer./admin.), unprefixed client keys routed by per-client validity exactly as Kafka Streams does (security.protocol reaches consumer + producer + admin, acks only the producer; a prefixed spelling wins, and StoatFlow's forced overrides win over both โ main.consumer. is a main-consumer-only scope, as in KS, not an alias for consumer.), and a lenient 3-tier unknown-key policy (mapped / recognised-no-op / unknown) + opt-in stoatflow.config.strict, which throws on truly-unknown keys instead of forwarding them to every client as custom props. The typed builder/YAML stays the recommended model; the adapter is the canonical KS-compat surface, not a stepping stone to a Map-backed swap (ADR-124). The KS *_CONFIG symbol constants and prefix helpers (consumerPrefix(...) โฆ adminClientPrefix(...)) are also shipped for symbol-level source compatibility (KSC-80). |
StreamsConfig(Map) / StreamsConfig(Properties) ctor | โ | KSC-79. The KS-faithful new StreamsConfig(props) / new StreamsConfig(map) bootstrap idiom compiles after the import-swap (Properties binds the single Map<*,*> ctor); it delegates to the same fromMap(...).build() adapter as fromProperties. A missing application.id throws at construction (IllegalArgumentException, where KS throws ConfigException). The typed builder(...) / fromProperties(...) paths stay recommended. |
toProperties() | ๐ | StreamsConfig.toProperties() emits the config back to KS-keyed Properties (resolved-view, masked) โ for /config, diffing, inspect/reconstruct |
default.timestamp.extractor | โ | StreamsConfig.defaultTimestampExtractor โ a global fallback TimestampExtractor for sources with no per-Consumed WatermarkStrategy (event-time basis only; the watermark default is unaffected). Unset, the record timestamp is the event time and a negative one is accepted; Kafka Streams' built-in extractors (FailOnInvalidTimestamp, its default, LogAndSkipOnInvalidTimestamp, UsePartitionTimeOnInvalidTimestamp, WallclockTimestampExtractor) ship and keep their meaning โ see Built-in timestamp extractors |
| Admin client config | โ | StreamsConfig.adminConfig / admin.* prefix โ passthrough merged at all AdminClient sites so a secured cluster authenticates |
| KafkaClientSupplier | โ | Custom client factory for instrumentation/testing |
| DefaultKafkaClientSupplier | โ | Standard KafkaConsumer/KafkaProducer/Admin factory |
| RocksDB config | โ | Bounded memory by default |
| Serdes | โ | Compatible |
| Default key/value Serdes | โ | defaultKeySerde/defaultValueSerde propagate through the DSL |
| Consumer/Producer config | โ | Pass-through maps |
bootstrap.resolve.timeout.ms (KIP-909) | โ | Forced to 0 (synchronous bootstrap DNS resolution) on every client StoatFlow creates; a user value is replaced with a warning. Kafka Streams 4.4 does the same: an asynchronously resolving client that fails to resolve stays broken until rebuilt |
| application.server | โ | host:port (or protocol://host:port) for the local host reported by interactive-query metadata; defaults from the runtime HTTP host:port when unset. A malformed value fails at startup, as in Kafka Streams 4.4 |
| state.cleanup.dir.max.age.ms (KIP-1259) | โ | Age-gated purge of stale local state at startup (rebuilt from changelog); default disabled |
| state.cleanup.delay.ms | โ | Orphaned-store cleanup interval |
| validation.validate-output-topics | ๐ | StoatFlow-only, default true: the start refuses, naming each missing topic, when a static sink topic or a handler's declared dead-letter topic does not exist (KSC-148). false restores Kafka Streams' behaviour, for clusters that rely on broker auto-creation |
KS config-key mapping
Full Kafka Streams 4.4.0 StreamsConfig ConfigDef coverage โ all 79 keys โ as accepted by StreamsConfig.fromProperties / fromMap (ADR-124). A build-time drift test pins this set against the real Kafka Streams ConfigDef, so a key added upstream cannot go unnoticed. Some rows below cover a family of keys.
Status in this table: โ
mapped ยท ๐ added (ADR-124) ยท ๐ก recognised-no-op (warns, never throws โ even under stoatflow.config.strict) ยท ๐งฉ client pass-through (routed to every client whose own configNames() contains it, exactly as Kafka Streams does; a consumer. / producer. / admin. prefix still wins, and StoatFlow's forced overrides win over both).
A port that references the symbol rather than the raw string is covered too (KSC-80): StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.METRIC_REPORTER_CLASSES_CONFIG, the six client-prefix constants (CONSUMER_PREFIX / MAIN_CONSUMER_PREFIX / RESTORE_CONSUMER_PREFIX / GLOBAL_CONSUMER_PREFIX / PRODUCER_PREFIX / ADMIN_CLIENT_PREFIX) and the prefix helpers (StreamsConfig.consumerPrefix(prop) โฆ adminClientPrefix(prop)) all ship, and the keys they build route or no-op exactly as the raw strings do. TOPIC_PREFIX and the topology-optimization value constants are not shipped.
| KS key | Status | StoatFlow mapping / reason |
|---|---|---|
application.id | โ | applicationId |
bootstrap.servers | โ | bootstrapServers |
state.dir | โ | stateDir (StoatFlow default /var/stoatflow/state) |
default.key.serde | โ | defaultKeySerde (Class or class-name) |
default.value.serde | โ | defaultValueSerde (Class or class-name) |
num.stream.threads | โ | โ numLanes (closest single-instance analogue). The lanes' thread type is the StoatFlow-only key lanes.thread-type (PLATFORM default, VIRTUAL) โ not a KS key |
processing.guarantee | โ | processingGuarantee (plus the exactly_once_beta alias; StoatFlow defaults to EXACTLY_ONCE where KS defaults to at_least_once โ KSC-84) |
application.server | โ | applicationServer (interactive-query metadata host:port โ KSC-47) |
deserialization.exception.handler (+ deprecated default.deserialization.exception.handler) | ๐ | deserializationExceptionHandler (class-name) |
production.exception.handler (+ deprecated default.production.exception.handler) | ๐ | productionExceptionHandler (class-name) |
processing.exception.handler | ๐ | processingExceptionHandler (class-name) |
default.timestamp.extractor | ๐ | defaultTimestampExtractor (global fallback, event-time basis). Kafka Streams' built-in extractors ship in io.stoatflow.core.processor under their Kafka Streams names; their Kafka Streams class names load StoatFlow's |
default.client.supplier | ๐ | kafkaClientSupplier (class-name) |
rocksdb.config.setter | ๐ | rocksDbConfigSetter (class-name) |
dsl.store.format | ๐ | dslStoreFormat (DslStoreFormat.fromConfigString) |
commit.interval.ms | ๐ | โ barrierIntervalMs (a semantic seed; the commit cadence is adaptive) |
buffered.records.per.partition | ๐ | bufferedRecordsPerPartition โ a pause threshold, as in Kafka Streams (any positive value) |
max.task.idle.ms | ๐ | maxTaskIdleMs (1:1; both allow -1) |
replication.factor | ๐ | changelogReplicationFactor |
state.cleanup.delay.ms / state.cleanup.dir.max.age.ms | ๐ | stateCleanupDelayMs / stateCleanupDirMaxAgeMs (KIP-1259) |
ensure.explicit.internal.resource.naming | ๐ | ensureExplicitNaming (StoatFlow defaults to true, KS to false; the mapping applies only when the key is present) |
processor.wrapper.class | โ ๏ธ | KIP-1112 โ supported. One ProcessorWrapper wraps every node, DSL and Processor API alike, at topology-compile time. Full KS fidelity for Processor API nodes; DSL-internal nodes get the record path only, and replacing an internal node is rejected at build time. A wrapper set on the runtime config alone is honoured, where KS ignores that case as too late |
num.standby.replicas | ๐ก | Single-instance: no standby replicas |
acceptable.recovery.lag / max.warmup.replicas / task.assignor.class / probing.rebalance.interval.ms | ๐ก | Single-instance: no standby, warmup, assignor or probing |
rack.aware.assignment.* / upgrade.from | ๐ก | Single-instance: no rack-aware assignment, no rolling-upgrade protocol |
group.protocol | ๐ก | StoatFlow manages consumer-group membership internally |
topology.description.push.enabled (KS 4.4) | ๐ก | Ignored with a warning: StoatFlow is not a streams group, so there is no push. The same topology description is available as a pull from GET /topology/ks โ KSC-113 |
enable.transactional.statestores (KS 4.4) | ๐ก | Accepted and ignored: StoatFlow's state stores are always transactional. true is silent; false logs a warning that it cannot be honoured (KSC-104) |
statestore.uncommitted.max.bytes (KS 4.4) | โ | Sets stoatflow.state.uncommitted-max-bytes, honoured as given. The default stays 256 MiB (Kafka Streams: 64 MiB), the budget is application-wide, under exactly-once it also counts the output held back during a commit, and -1 (disable) is refused: the budget also caps the epoch size (KSC-105) |
default.interactive.query.isolation.level (KS 4.4) | โ | Sets stoatflow.state.interactive-query-isolation-level (READ_UNCOMMITTED default, or READ_COMMITTED; case-insensitive). store() applies it to every store, global stores included, where Kafka Streams reads a global store uncommitted (KSC-144) |
statestore.cache.max.bytes / cache.max.bytes.buffering | ๐ก | StoatFlow's cache caps are per-store and commit-triggering (caching.max-entries / caching.max-estimated-bytes), not a global evict budget; global memory is bounded by state.uncommitted-max-bytes |
client.id | ๐ก | Client ids are derived from application.id; override with consumer.client.id / producer.client.id |
task.timeout.ms | ๐ก | Commit-path waits are bounded by barrierTimeoutMs instead |
topology.optimization | ๐ก | Re-keying is in-memory; KTable source reuse is autoReuseKTableSourceTopics โ on by default for a compacted source topic. A table whose local state shows it restores from its source topic is never switched to a changelog in place (the start fails, explaining how to switch; a table with no such state gets a warning instead); a versioned table always restores from its changelog |
repartition.purge.interval.ms | ๐ก | In-memory re-keying: there are no repartition topics to purge |
processing.exception.handler.global.enabled | ๐ก | The processing handler is already applied globally |
errors.dead.letter.queue.topic.name | ๐ก | The DLQ is configured per handler (DeadLetterQueue*ExceptionHandler) |
default.dsl.store / dsl.store.suppliers.class | ๐ก | Choose the store with Materialized.withStoreType |
default.list.{key,value}.serde.{inner,type} | ๐ก | No ListSerde default-type config; use explicit serdes |
windowstore.changelog.additional.retention.ms | ๐ก | Window changelogs use compaction; no extra retention |
window.size.ms / windowed.inner.class.serde | ๐ก | Deprecated in KS; use the typed WindowedSerdes |
built.in.metrics.version / metric.reporters / metrics.* / enable.metrics.push | ๐ก | Metrics go through Micrometer, not the Kafka metrics subsystem |
config.providers | ๐ก | Resolve externalised config before constructing StreamsConfig |
allow.os.group.write.access | ๐ก | StoatFlow does not relax state-dir permissions via config |
log.summary.interval.ms | ๐ก | No periodic summary-log toggle |
poll.ms | ๐ก | Use consumerPollTimeoutMs |
security.protocol / connections.max.idle.ms / metadata.max.age.ms / {receive,send}.buffer.bytes / request.timeout.ms / reconnect.backoff.* / retry.backoff.ms / metadata.recovery.* / metadata.cluster.check.enable (KS 4.4) | ๐งฉ | Client properties, routed unprefixed to every client they are valid for (KSC-92, KS parity): security.protocol reaches consumer, producer and admin; the rest go by their own ConfigDef. A consumer. / main.consumer. / producer. / admin. prefix overrides the bare form. The same routing applies to bare ssl.* and sasl.*, and to producer-only keys such as acks |
Key architectural differences
The runtime-model deltas โ single instance vs. rebalancing cluster, lanes vs. partition-bound tasks, in-memory re-keying vs. repartition topics, barrier-based exactly-once, global vs. partition-scoped state โ are summarized in the canonical deltas-at-a-glance table, with the conceptual treatment on the same page. The rows below are the operational and time-semantics differences that surface while reading this matrix:
| Aspect | Kafka Streams | StoatFlow |
|---|---|---|
| RocksDB memory | Unbounded default | Bounded default (256 MiB) |
| Window closure | Stream-time based | Watermark-based |
| Grace period | Accepts a late record while windowEnd + grace is ahead of the task's stream time (max observed timestamp, KIP-328); the same clock closes the window | One clock, the watermark (min across non-idle partitions โ out-of-orderness): a window accepts records until it passes windowEnd + grace (a session: last event + gap + grace), and then closes โ final emission, suppress release, join null side, event-time timers (KSC-132) |
| Negative event time | Never reaches the topology: the default extractor (FailOnInvalidTimestamp) fails the application on one, any other extractor's negative result is dropped with a WARN | A valid time, end to end (Flink is the reference): every Long but Long.MIN_VALUE. Windows are floor-aligned with no clamp at 0, streamโstream joins match across 0, onWindowClose() emits windows and sessions ending below 0, and the Processor API, timers and punctuators accept negative times. A sink writes the record without a timestamp; the producer or broker stamps the wall-clock time, counted as stoatflow.sink.records.negative.timestamp.total (Flink DataStream KafkaSink parity). Record, FixedKeyRecord and TestRecord refuse only Long.MIN_VALUE. To keep the KS behaviour for negative times, set default.timestamp.extractor to FailOnInvalidTimestamp or LogAndSkipOnInvalidTimestamp (KSC-138). Near Long.MAX_VALUE โ within a session gap or a join window of it โ Kafka Streams' sums wrap: its session aggregation fails the record and its join finds no partner, where StoatFlow's saturate, so the session is counted and the join emits; no setting reproduces the wrap |
| Missing event time | Kafka's -1 fails the application (default extractor) or is dropped | Skipped and counted as dropped.records{reason=no_event_time}, its offset committed: Kafka's -1 on a source with no extractor or assigner, or Long.MIN_VALUE from one. Failing is opt-in via FailOnInvalidTimestamp |
| Windows near time 0 | Hopping windows start no earlier than 0; sliding-window records below the time difference D share one [0, D] window | Floor-aligned: a hopping-window record below size โ advance also joins the windows that start before 0, and each sliding-window record below D gets its own [t โ D, t] window. Identical above those times โ see Windows near time 0. A Kafka Streams consumer of the windowed output fails on a negative window start |
| Schema Registry subjects of internal serialization | The internal topics' subjects: <app>-<name>-repartition-key, <app>-<store>-changelog-key / -value, a foreign-key join's โฆ-subscription-registration-topic-fk / -pk | The same names, plus app-scoped subjects where no topic exists. A boundary subject per key stream and input, named as in Kafka Streams only where you named the repartition and Kafka Streams creates it, after a re-key (a streamโstream join's -left / -right included), else after the merge, join or cogroup it leads into or the key's origin (<cogroup>-windowed out of a windowed cogroup); every store under its changelogFor subject (a backing source topic, else its changelog); timers <app>-<processor>-timer; scheduled sources <app>-<source>-scheduled; an outer join's right values apart. Unrelated key streams on one name are refused. Subjects StoatFlow registers |
| Watermark strategy | N/A | Flink-style WatermarkStrategy |
| Punctuators | Task-scoped, on the stream thread | Dedicated punctuation lane per sub-topology, full state access |
| Key-based timers | N/A | Flink-inspired TimerService, with named timers that a new registration replaces |
| Scheduled sources | N/A | Interval & cron-based emission |
| Runtime control | N/A | pause()/unpause() |
Test utilities
TopologyTestDriver drives a topology in-memory (no broker) for unit tests. All state stores โ including RocksDB-Materialized ones โ run in-memory in the test driver, but the exactly-once store checks judge the store you declared (KSC-126: a RocksDB versioned join input passes, an in-memory versioned or lruMap one is refused, as in production); real-engine + RocksDB end-to-end coverage uses integration tests with an embedded broker.
TopologyTestDriver
| Member | Status | Notes |
|---|---|---|
new TopologyTestDriver(topology) / (topology, Properties) / (topology, Instant) / (topology, Properties, Instant) | โ | KS-shaped constructors; Properties via StreamsConfig.fromProperties |
createInputTopic(name, Serde/Serializer, โฆ) | โ | Serde and Serializer overloads + 5-arg (โฆ, Instant, Duration) start-time/auto-advance form (serdes accepted, ignored) |
createOutputTopic(name, Serde/Deserializer, โฆ) | โ | Serde and Deserializer overloads |
getAllStateStores() / producedTopicNames() | โ | All stores by name; the set of produced topics |
advanceWallClockTime(Duration) / get*Store(name) / close() | โ | advanceWallClockTime also moves currentSystemTimeMs() and TimerService.currentProcessingTime() |
TestRecord
KS positional order (key, value, headers, timestamp); ctors (K,V), (K,V,Headers[,Long]), (K,V,Instant), (K,V,Headers,Instant), value-only (V), (ConsumerRecord), (ProducerRecord), all callable from Java (a null timestamp means unset); bean getters getKey()/getValue()/getHeaders()/getRecordTime() + KS accessors key()/value()/headers()/timestamp(). The timestamp is unset by default, as in Kafka Streams: pipeInput(TestRecord) stamps an unset record with the input topic's current time. Any timestamp but Long.MIN_VALUE is accepted, negative included, where Kafka Streams' TestRecord rejects every negative one. The driver applies the runtime's event-time rules: a record with no event time is skipped, an extractor's exception fails the pipe, and a sink stamps a negative-time record with the driver's wall clock โ see Testing.
TestInputTopic
| Method | Status | Notes |
|---|---|---|
| pipeInput(key, value , timestamp) | โ | timestamp as long or Instant |
| pipeInput(value) / pipeInput(value, Instant) | โ | Value-only (null key) โ for value-centric / Avro tests |
| pipeKeyValueList / pipeValueList / pipeRecordList | โ | Bulk piping; key-value lists use KeyValue; timed (โฆ, Instant, Duration) overloads; pipeRecordList is out-variant |
| advanceTime(Duration) | โ | Advance the input topic's record clock |
TestOutputTopic
| Method | Status | Notes |
|---|---|---|
| readKeyValue() / readKeyValuesToList() | โ | Return KeyValue<K, V> (.key / .value), not Pair |
| readKeyValuesToMap() | โ | |
| readValue() / readRecord() | โ |
KeyValue is StoatFlow's KS-shaped key-value type (the same one the DSL uses); a KS port rewrites the KeyValue import to match.
High availability (hot-standby)
Not a KS-parity surface โ Kafka Streams scales out with standby tasks; StoatFlow's opt-in hot-standby pairs one active with one or more warm standbys. The surface, and the two deliberate gaps:
| Surface | Status | Notes |
|---|---|---|
stoatflow.ha.mode = off / active-standby | ๐ | Deploy-time opt-in; off (default) is byte-identical to single-instance |
| Active/passive failover | ๐ | Crash failover in seconds; graceful deploys / /ha/switch hand off cleanly |
| Split-brain fencing | ๐ | EOS: broker-enforced producer-epoch fence; ALO: metadata-topic lease |
/ha/{status,switch,promote,demote} endpoints | ๐ | Operator shims; SWITCH/PROMOTE/DEMOTE return 409 unless a caught-up target exists (?force=true override) |
ha.acceptable-recovery-lag | ๐ | A single total lag sum (default 50000) โ not per-task like KS acceptable.recovery.lag (10000) |
| Continuous changelog replication | ๐ | The standby streams committed changelog deltas into local state in real time โ like KS standby tasks' continuous restore, for the single-active pair |
| Restore-before-process on promotion | ๐ | A promoting pod restores local state to the committed changelog end before processing |
| Queryable standby (KIP-535 IQ from standby) | โ | Standbys are non-queryable; store queries on a standby are rejected |
KS standby tasks / num.standby.replicas | โ (different model) | Single-active with warm standbys โ not N-instance standby tasks |
See High availability for the operational guide.
Related
- Comparison matrix โ feature-by-feature against Kafka Streams and self-hosted / managed Flink
- Building โ how-to guidance for the operators above
- KStream and KTable ยท Aggregations ยท Windowing ยท Joins ยท Processor API
- State stores ยท Scheduled sources
Maven reference
StoatFlow's Maven support โ the stoatflow-bom, the stoatflow-parent convention POM, and the stoatflow-maven-plugin. The three artifacts, every <properties> key, the four goals, and the pinned plugin versions.
Metrics reference
The full catalogue of Prometheus meters StoatFlow exposes on /metrics โ name, type, tags, and what each one measures, grouped by area.