The error-handling model

This page is the conceptual model for how a StoatFlow application behaves when something goes wrong with a record or a commit. It describes which failures are distinct, the policies you can apply to each, and what the runtime does that you don't get to configure — the commit failure that ends in a process restart. The concrete handler classes and the configuration keys that select them live in Error handling and DLQ; this page stays at the level of behaviour.

The starting point is the same single-process model the rest of these pages build on. Because there's one instance and one transaction per commit (see Architecture and Exactly-once), every failure either gets handled inside an epoch — without disturbing the exactly-once guarantee — or it's fatal and the epoch is thrown away. There's no middle ground where a node limps along in a degraded state.

Three classes of failure

The runtime distinguishes failures by where in the record's lifecycle they happen. Each class is independent — you can log-and-continue on bad input while still failing hard on a processor bug, or vice versa.

Deserialization (on the way in)

A record arrives from Kafka as raw bytes. Before the topology can touch it, the key and value bytes must deserialize through the configured serdes. If that fails — malformed JSON, a schema the deserializer can't read, a corrupt payload — it's a deserialization failure. The topology never ran for this record; there's nothing to roll back. This is the failure class for source topics that may carry records your app can't read (a producer upstream changed format, a poison record slipped in).

Processing (inside the topology)

The record deserialized fine and entered the topology, and then a processor threw — a NullPointerException in a mapValues, a validation error in a custom Processor, an unchecked exception from an enrichment call. This is a processing failure: it happens on a processing lane, mid-DAG, after the record was already accepted. Whatever state the record touched before the throw is part of the current epoch and is governed by the same commit barrier as everything else — a skipped or dead-lettered record doesn't leak half-applied state into the committed snapshot.

Production (on the way out)

The topology produced an output record and the runtime tried to publish it. Two things can go wrong here, and the runtime treats them as one class with sub-cases:

• Serialization — the output key or value can't be turned into bytes (a serde rejects the value, a Schema Registry call fails). This is deterministic: the same record will fail the same way every time, so it's never retried.
• Send — serialization succeeded but the producer couldn't deliver (broker unreachable, request timed out, an authorization error). Some send failures are transient and worth retrying; others are permanent.

This is a production failure. It's the only class where retry is a meaningful policy, because it's the only class where the failure might be transient — see Producer error classification below.

The handler model: continue, fail, or dead-letter

Each failure class is governed by an exception handler — a policy the runtime consults when that class of failure occurs. There are three behaviours a handler can express, and the built-in handlers give you one each:

The defaults are deliberately strict. Deserialization and processing both default to log and fail — silently skipping records should be a choice you make on purpose, not a behaviour you inherit. Production defaults to a handler that retries transient send errors and fails on the rest. You opt into continue-or-dead-letter behaviour per class; the building guide shows how the handlers are named and wired.

These handler classes follow the shape of Kafka Streams' KIP-1033 (processing exception handler, Kafka 3.9.0) and KIP-1034 (dead letter queue, Kafka 4.2.0) error-handling design, so the mental model carries over directly if you know Kafka Streams. The behavioural deltas are catalogued in the Kafka Streams compatibility matrix.

Credit where it is due: both KIPs were coauthored by Damien Gasparina, Loïc Greffier and Sébastien Viale at Michelin, and grew out of Michelin's Kstreamplify. Their Current London 2025 talk Processing Exception Handling and Dead Letter Queue in Kafka Streams, and Michelin's write-ups on processing error handling and the DLQ in Spring Kafka 4.1, are the best introductions to the design this page inherits.

One escape hatch sits above the per-class policies. A failed record error (the log and fail outcome) surfaces as a fatal stream error, and by default it stops the instance. An application can register the KS-compatible StreamsUncaughtExceptionHandler; returning REPLACE_THREAD makes the runtime recover with an in-place engine restart — the processing engine is torn down and rebuilt inside the same process, resuming from the last committed barrier with the interrupted epoch aborted, exactly as a crash recovery would. It's the single-instance analogue of Kafka Streams replacing a failed stream thread, and it's bounded: a restart budget shuts the application down if faults keep recurring.

Dead-letter queue semantics

A DLQ handler doesn't publish to the dead-letter topic out of band. It hands the failed record back to the runtime, and that record is sent through the same transactional producer, on the same commit barrier as the epoch's normal output. For the deserialization, processing, and synchronous-production failure classes, this is the property that makes the DLQ trustworthy:

• No loss. The dead-lettered record commits atomically with the rest of the epoch. If the epoch commits, the DLQ record is on the topic; if the epoch aborts (a crash mid-barrier), the DLQ record is discarded along with everything else and the source record will be re-read and re-handled after restart.
• No duplicates. Because it rides the exactly-once transaction, a dead-lettered record appears on the DLQ topic exactly once for downstream consumers reading with read_committed isolation — same guarantee as your real output.

The asynchronous (broker-side) production failure path reaches the same guarantee by a different route. When the broker rejects an already-sent record (the classic case: a record exceeding the topic's max.message.bytes), the rejection poisons the in-flight transaction — nothing in it can commit, so the epoch's outputs, changelog writes and state are all aborted. Rather than advancing past them, the runtime holds the source offsets and replays the epoch: it commits a secondary transaction carrying only the poison's DLQ record, quarantines the poison's source coordinates, restarts the engine in place, and reprocesses the epoch with exactly that record skipped. Every other record in it is processed again and commits normally.

Each replay costs a full engine restart, so it is loud rather than silent: stoatflow.dlq.poison.quarantined.total counts the records being skipped, stoatflow.dlq.poison.replays.total counts the replays, and stoatflow.dlq.poison.quarantine.size shows how many offsets a replay is currently skipping (0 in steady state). The pod is briefly unready while the engine rebuilds.

Two cases cannot be replayed and stop the instance immediately, with the source offsets held so an operator restart replays the epoch:

• A fail verdict, which now means what it says — stop before further harm, rather than causing the harm and then stopping.
• A continue on a poison with no attributable source record — an output emitted by a punctuator, a watermark-driven window close, a suppression flush, a timer or a scheduled source. continue asserts the emission itself is poison, and those planes fire on their own schedule, so they would re-poison every attempt: the runtime names the plane and shuts down rather than burning the budget discovering that. A retry on one of those planes is a different claim — the send failed, not the emission — so it replays like any other retry, and the send is re-attempted.

Kafka Streams shares the failure class, and answers it differently. Verified against 4.3.1: it honours continue locally, and then its offsets ride sendOffsetsToTransaction, which throws once the producer is in ABORTABLE_ERROR, so the source position never advances — the open bug is precisely this scenario. Its own KIP-1034 dead-letter record is appended to the transaction the rejection already doomed, so it is aborted with everything else and never becomes visible to a read_committed consumer. The commit then throws, the stream thread dies, and the default SHUTDOWN_CLIENT stops the client — under a supervisor, a restart loop onto the same record, with no evidence written anywhere. StoatFlow also stops advancing, deliberately; the difference is that the dead letter survives, the innocent records survive, and the instance either recovers or stops with the poison's coordinates named.

What each DLQ record carries depends on the failure class:

• Deserialization DLQ records preserve the original raw key/value bytes — nothing was successfully deserialized, so the untouched payload is exactly what you need to diagnose or replay it.
• Processing and production DLQ records carry the record as it was at the point of failure, plus error-context metadata.

Every DLQ record is annotated with error-context headers under a StoatFlow-specific __stoatflow.errors.* namespace (KIP-1034-shaped): the failure type, the exception class and message, an optional stack trace, the originating source topic / partition / offset, and which component failed. DLQ tooling that filters out the conventional __-prefixed internal headers must opt in to surface these. The exact header set is documented in Error handling and DLQ.

Producer error classification: retriable vs fatal

Production is the one failure class where the runtime makes a finer distinction, because a send failure might be transient. When the producer reports an error, the runtime classifies it before deciding what to do:

• Retriable. Transient conditions — a request timeout, a momentarily unreachable broker — that may succeed if tried again. The default production handler returns retry. What that buys you depends on which send failed. On a synchronous send the runtime retries in place with exponential backoff for a bounded number of attempts, and writes any attached dead letter only once those are exhausted — so a retry that succeeds leaves no trace. On an asynchronous broker rejection the producer's own retries are already spent by the time the callback fires, so retry instead aborts and replays the epoch: identical to continue except that the record is not quarantined, because the verdict says the record is fine and the send wasn't. Under AT_LEAST_ONCE there is no transaction to abort and no replay to offer, so retry on that path is treated as fail.
• Fatal. Conditions that won't improve by retrying — a serialization error (deterministic, the same record fails the same way), an authentication or authorization failure, a broker-incompatibility error, or a producer-fencing condition. These can't be handled away inside the epoch, so they end it.

The serialization-vs-send split matters here: serialization failures are always classified non-retriable (retrying can't change a deterministic outcome), while send failures are classified case by case. A fatal production error that has a DLQ configured will still emit the DLQ record before the epoch ends, so even a fatal failure leaves a forensic trail.

The classification rules are stricter than Kafka Streams' in one specific way: StoatFlow has a single instance and no task to migrate, so the failure conditions Kafka Streams would resolve by moving a task elsewhere have nowhere to go and are treated as fatal. That's a direct consequence of the single-instance model, not a separate design choice.

When the commit itself fails

Everything above happens inside an epoch and leaves the exactly-once guarantee intact. There is one failure the runtime does not hand to a configurable policy: the commit barrier itself failing.

When the per-epoch Kafka transaction can't complete — it times out, the broker rejects it, the producer gets fenced — there is no safe way to continue. The runtime aborts the in-flight transaction and the process exits. As described on the architecture page, the aborted transaction's partial work — uncommitted state writes, uncommitted output, uncommitted offset advances — is discarded at the broker. Your orchestrator (Kubernetes, typically) restarts the process, which resumes from the last successful barrier and re-reads every record after it. Downstream consumers reading with read_committed isolation never see the aborted epoch's output.

This is intentional and not tunable: a single-instance, single-transaction design has no notion of "commit failed but keep going." A stalled or failed commit is treated as a fatal condition, the epoch is thrown away whole, and recovery is a clean restart from a known-good point. The same restart-on-fatal pattern covers a sustained broker outage that exhausts the retry budget — the runtime stops making progress, the process exits, and the orchestrator brings it back. The internal protocol that detects a stalled commit and converts it into this abort-and-exit behaviour stays in the source; what you observe is the restart.

What you observe

Each failure class surfaces the same way you'd expect from the architecture page's operational framing — a metric signal, a log line, and (for DLQ policies) the records themselves. The column below names the signal, not the exact Prometheus metric ID; the precise names are what you'll see exposed on /metrics.

Steady-state failure diagnosis uses the always-on admin surface — /metrics, /health/ready, and the debug endpoints — covered in the architecture page's observability section.

Where to go next

• Error handling and DLQ — the concrete handler classes, the configuration keys that select them per failure class, the full DLQ header set, and worked examples in Kotlin and Java.
• Exactly-once — why DLQ records and skipped-record offsets commit atomically with the rest of the epoch.
• Architecture — the full failure-mode and observability catalogue at the system level.
• Kafka Streams compatibility matrix — how StoatFlow's error-handling surface lines up against Kafka Streams.