Exactly-once semantics

Exactly-once is StoatFlow's default processing guarantee. This page explains the mechanism that delivers it — the commit barrier — what "exactly-once" means here in concrete terms, how recovery works after a crash, and when to switch to the cheaper at-least-once mode instead. It describes behaviour, not implementation; the Architecture page is the conceptual ceiling, and the source is the source of truth for the protocol details.

What "exactly-once" means here

Exactly-once is an end-to-end statement about the effect of each input record, not a claim that any record is physically processed only once. After a crash, StoatFlow re-reads and re-processes records — that's how recovery works. The guarantee is that the observable result is as if each input contributed exactly once:

• Every output record a downstream consumer reads (with read_committed isolation) reflects each input's effect once — no duplicates, no gaps.
• The state stores backing your aggregations, joins, and windows end up consistent with the committed output and the committed input offsets.
• A crash mid-processing leaves no partial trace: no half-applied state, no orphaned output, no advanced offsets pointing past work that was discarded.

This is the same guarantee Kafka Streams calls exactly_once_v2 and Flink calls exactly-once checkpointing. What differs is the scope and the operational model — see Comparison matrix for the feature-by-feature contrast against Kafka Streams and self-hosted / managed Flink.

The commit barrier

The whole guarantee rests on one mechanism: the commit barrier.

A commit barrier is a marker — not a data record, not user content — that the runtime periodically injects into the processing flow. As your records move through the topology (mapValues, filter, join, aggregate, custom Processor code), the barrier travels with them. When the barrier has reached the end of every processing path, the runtime executes one Kafka transaction that commits, atomically:

• State — every state-store write since the previous barrier, published to the changelog topics that back those stores.
• Output — every sink record the topology produced since the previous barrier.
• Offsets — the Kafka consumer-group offsets for every input partition that contributed records during that span.

Either all three commit together, or none of them do. There is no window in which output has been published but offsets haven't, or offsets have advanced but state hasn't been persisted. That atomicity is what makes the three facts above hold simultaneously.

The span between two successful barriers is one epoch — the unit of atomic commit. An epoch is bounded by time and by record count — and closes early under memory or state-cache pressure — so it commits on a regular cadence even when input is bursty, processing is slow, or uncommitted state grows large; the exact cadence and how it adapts are implementation concerns and stay in the source.

Why the changelog is part of the transaction

State durability in StoatFlow comes from Kafka changelog topics: every state write produces a changelog entry, and the changelog is what gets restored on restart. Because the changelog publish is committed inside the same transaction as the output and the offsets, your durable state can never disagree with what you emitted or where you resumed from. When a barrier completes, the changelog holds exactly the same data your in-memory state does. See State and thread safety for how the stores themselves work.

Crash recovery

The barrier is also the recovery anchor. Picture a crash partway through an epoch — output has been produced and state mutated in memory, but the barrier hasn't completed yet.

1. The in-flight transaction aborts. When the JVM exits non-cleanly, the open Kafka transaction is never committed; Kafka's broker-side transaction semantics abort it. The uncommitted output records, the uncommitted changelog entries, and the un-advanced offsets are all discarded together.
2. The process restarts — or a standby takes over. The last committed barrier is the recovery anchor whether the instance restarts cold (the default) or an opt-in hot standby is promoted in its place. The same transactional fencing that makes a commit atomic is what makes a standby handoff split-brain-proof: at most one instance can ever commit. See Architecture.
3. State restores to the last barrier. Local stores rebuild from their changelog topics up to the last committed barrier — never past it, because the changelog was committed transactionally with everything else.
4. The consumer resumes from the last committed offsets. Every input record after the last successful barrier is re-read and re-processed into a fresh epoch.

The records from the interrupted epoch are re-processed, but the aborted transaction means none of their earlier effects survived. Downstream consumers reading with read_committed isolation never saw the aborted output, so the re-processed epoch produces the result exactly once. No duplicate outputs, no lost state, no replayed offsets.

For a plain Kafka consumer, that's one property:

# Downstream consumer — never observe records from aborted transactions
isolation.level=read_committed

A downstream StoatFlow app sets the same property through its consumer passthrough:

stoatflow:
  kafka:
    consumer:
      isolation.level: read_committed

There's one more behaviour worth naming, because it shows up at the operational layer rather than in your topology code: a stalled commit is a fatal event, not a silent hang. If a commit transaction times out or fails, the runtime aborts the in-flight transaction and exits; the orchestrator restarts the process, and recovery proceeds exactly as above — resume from the last successful barrier, no duplicates downstream. You observe this through commit-stall metrics and the /debug/barriers endpoint before exit, and the restoration phase via /state and /health/ready on the way back up. The bounded-wait protocol behind this is an implementation concern and stays in the source.

At-least-once mode and its trade-off

Exactly-once carries a cost: every commit is a Kafka transaction, and transactions add per-commit overhead (~8–30ms in practice). For pipelines where downstream systems are already idempotent or tolerate duplicates, at-least-once mode removes that overhead.

In at-least-once mode the runtime uses a non-transactional producer and commits consumer offsets directly, on a faster cadence (~3–5ms per commit). You give up the atomic all-or-nothing boundary. On a crash:

• Records processed but not yet offset-committed are replayed — downstream consumers may see duplicate output.
• State stores may be ahead of the committed offsets, so windowed aggregations may over-count on recovery. (Plain key-value updates are typically idempotent and converge; aggregations that add per record are the case to watch.)

In exchange you get a lower commit-cadence floor on end-to-end latency and a simpler operational model. It's the right choice when the cost of an occasional duplicate is lower than the cost of the transaction overhead — and the wrong choice when downstream effects aren't idempotent.

The guarantee is selected by a single property, processing-guarantee, with two values: EXACTLY_ONCE (the default) and AT_LEAST_ONCE. It is process-wide — it applies to the whole topology, not per-operator — and switching it does not change your topology code; the same build runs under either guarantee. On the runtime it's one line of application.yaml; with :core directly it's one builder call:

# application.yaml — only needed to switch away from the EXACTLY_ONCE default
stoatflow:
  processing-guarantee: AT_LEAST_ONCE

val config = StreamsConfig.builder("order-processor", "localhost:9092")
    .processingGuarantee(ProcessingGuarantee.AT_LEAST_ONCE)
    .build()

var config = StreamsConfig.builder("order-processor", "localhost:9092")
    .processingGuarantee(ProcessingGuarantee.AT_LEAST_ONCE)
    .build();

How this compares to Kafka Streams

If you're coming from Kafka Streams, two differences matter most:

• It's the default, with no property dance. Kafka Streams ships at_least_once as the default and requires processing.guarantee=exactly_once_v2 plus a transactional broker setup to opt in. StoatFlow is exactly-once out of the box — a single instance coordinating a single transaction, with nothing extra to wire up.
• One transaction, not per-task. Kafka Streams runs a transaction per stream task and coordinates them across instances. StoatFlow's single-instance model means one barrier and one transaction cover the entire topology — there are no cross-instance two-phase commits to reason about.

The DSL semantics around exactly-once match Kafka Streams; see How StoatFlow differs from Kafka Streams for the broader behavioural map and Comparison matrix for the side-by-side EOS comparison against Kafka Streams and Flink.

Where to go next

• Architecture — the single-instance model and the commit barrier in its full context
• State and thread safety — how global state stays consistent under the commit barrier
• Event time and watermarks — why watermark progress commits alongside the barrier
• Error handling model — what the runtime does when a processor or commit fails
• Comparison matrix — EOS feature-by-feature against Kafka Streams and Flink