The error-handling model

This page is the conceptual model for how a StoatFlow application behaves when something goes wrong with a record or a commit. It describes which failures are distinct, the policies you can apply to each, and what the runtime does that you don't get to configure — the commit failure that ends in a process restart. The concrete handler classes and the configuration keys that select them live in Error handling and DLQ; this page stays at the level of behaviour.

The starting point is the same single-process model the rest of these pages build on. Because there's one instance and one transaction per commit (see Architecture and Exactly-once), every failure either gets handled inside an epoch — without disturbing the exactly-once guarantee — or it's fatal and the epoch is thrown away. There's no middle ground where a node limps along in a degraded state.

Three classes of failure

The runtime distinguishes failures by where in the record's lifecycle they happen. Each class is independent — you can log-and-continue on bad input while still failing hard on a processor bug, or vice versa.

Deserialization (on the way in)

A record arrives from Kafka as raw bytes. Before the topology can touch it, the key and value bytes must deserialize through the configured serdes. If that fails — malformed JSON, a schema the deserializer can't read, a corrupt payload — it's a deserialization failure. The topology never ran for this record; there's nothing to roll back. This is the failure class for source topics that may carry records your app can't read (a producer upstream changed format, a poison record slipped in).

Processing (inside the topology)

The record deserialized fine and entered the topology, and then a processor threw — a NullPointerException in a mapValues, a validation error in a custom Processor, an unchecked exception from an enrichment call. This is a processing failure: it happens on a processing lane, mid-DAG, after the record was already accepted. Whatever state the record touched before the throw is part of the current epoch and is governed by the same commit barrier as everything else — a skipped or dead-lettered record doesn't leak half-applied state into the committed snapshot.

Production (on the way out)

The topology produced an output record and the runtime tried to publish it. Two things can go wrong here, and the runtime treats them as one class with sub-cases:

• Serialization — the output key or value can't be turned into bytes (a serde rejects the value, a Schema Registry call fails). This is deterministic: the same record will fail the same way every time, so it's never retried.
• Send — serialization succeeded but the producer couldn't deliver (broker unreachable, request timed out, an authorization error). Some send failures are transient and worth retrying; others are permanent.

This is a production failure. It's the only class where retry is a meaningful policy, because it's the only class where the failure might be transient — see Producer error classification below.

The handler model: continue, fail, or dead-letter

Each failure class is governed by an exception handler — a policy the runtime consults when that class of failure occurs. There are three behaviours a handler can express, and the built-in handlers give you one each:

The defaults are deliberately strict. Deserialization and processing both default to log and fail — silently skipping records should be a choice you make on purpose, not a behaviour you inherit. Production defaults to a handler that retries transient send errors and fails on the rest. You opt into continue-or-dead-letter behaviour per class; the building guide shows how the handlers are named and wired.

These handler classes follow the shape of Kafka Streams' KIP-1033 (processing) and KIP-1034 (deserialization / production / DLQ) error-handling design, so the mental model carries over directly if you know Kafka Streams. The behavioural deltas are catalogued in the Kafka Streams compatibility matrix.

One escape hatch sits above the per-class policies. A failed record error (the log and fail outcome) surfaces as a fatal stream error, and by default it stops the instance. An application can register the KS-compatible StreamsUncaughtExceptionHandler; returning REPLACE_THREAD makes the runtime recover with an in-place engine restart — the processing engine is torn down and rebuilt inside the same process, resuming from the last committed barrier with the interrupted epoch aborted, exactly as a crash recovery would. It's the single-instance analogue of Kafka Streams replacing a failed stream thread, and it's bounded: a restart budget shuts the application down if faults keep recurring.

Dead-letter queue semantics

A DLQ handler doesn't publish to the dead-letter topic out of band. It hands the failed record back to the runtime, and that record is sent through the same transactional producer, on the same commit barrier as the epoch's normal output. This is the property that makes the DLQ trustworthy:

• No loss. The dead-lettered record commits atomically with the rest of the epoch. If the epoch commits, the DLQ record is on the topic; if the epoch aborts (a crash mid-barrier), the DLQ record is discarded along with everything else and the source record will be re-read and re-handled after restart.
• No duplicates. Because it rides the exactly-once transaction, a dead-lettered record appears on the DLQ topic exactly once for downstream consumers reading with read_committed isolation — same guarantee as your real output.

What each DLQ record carries depends on the failure class:

• Deserialization DLQ records preserve the original raw key/value bytes — nothing was successfully deserialized, so the untouched payload is exactly what you need to diagnose or replay it.
• Processing and production DLQ records carry the record as it was at the point of failure, plus error-context metadata.

Every DLQ record is annotated with error-context headers under a StoatFlow-specific __stoatflow.errors.* namespace (KIP-1034-shaped): the failure type, the exception class and message, an optional stack trace, the originating source topic / partition / offset, and which component failed. DLQ tooling that filters out the conventional __-prefixed internal headers must opt in to surface these. The exact header set is documented in Error handling and DLQ.

Producer error classification: retriable vs fatal

Production is the one failure class where the runtime makes a finer distinction, because a send failure might be transient. When the producer reports an error, the runtime classifies it before deciding what to do:

• Retriable. Transient conditions — a request timeout, a momentarily unreachable broker — that may succeed if tried again. The default production handler returns retry, and the runtime retries with exponential backoff for a bounded number of attempts before giving up. A short broker hiccup self-heals; throughput dips and recovers.
• Fatal. Conditions that won't improve by retrying — a serialization error (deterministic, the same record fails the same way), an authentication or authorization failure, a broker-incompatibility error, or a producer-fencing condition. These can't be handled away inside the epoch, so they end it.

The serialization-vs-send split matters here: serialization failures are always classified non-retriable (retrying can't change a deterministic outcome), while send failures are classified case by case. A fatal production error that has a DLQ configured will still emit the DLQ record before the epoch ends, so even a fatal failure leaves a forensic trail.

The classification rules are stricter than Kafka Streams' in one specific way: StoatFlow has a single instance and no task to migrate, so the failure conditions Kafka Streams would resolve by moving a task elsewhere have nowhere to go and are treated as fatal. That's a direct consequence of the single-instance model, not a separate design choice.

When the commit itself fails

Everything above happens inside an epoch and leaves the exactly-once guarantee intact. There is one failure the runtime does not hand to a configurable policy: the commit barrier itself failing.

When the per-epoch Kafka transaction can't complete — it times out, the broker rejects it, the producer gets fenced — there is no safe way to continue. The runtime aborts the in-flight transaction and the process exits. As described on the architecture page, the aborted transaction's partial work — uncommitted state writes, uncommitted output, uncommitted offset advances — is discarded at the broker. Your orchestrator (Kubernetes, typically) restarts the process, which resumes from the last successful barrier and re-reads every record after it. Downstream consumers reading with read_committed isolation never see the aborted epoch's output.

This is intentional and not tunable: a single-instance, single-transaction design has no notion of "commit failed but keep going." A stalled or failed commit is treated as a fatal condition, the epoch is thrown away whole, and recovery is a clean restart from a known-good point. The same restart-on-fatal pattern covers a sustained broker outage that exhausts the retry budget — the runtime stops making progress, the process exits, and the orchestrator brings it back. The internal protocol that detects a stalled commit and converts it into this abort-and-exit behaviour stays in the source; what you observe is the restart.

What you observe

Each failure class surfaces the same way you'd expect from the architecture page's operational framing — a metric signal, a log line, and (for DLQ policies) the records themselves. The column below names the signal, not the exact Prometheus metric ID; the precise names are what you'll see exposed on /metrics.

Steady-state failure diagnosis uses the always-on admin surface — /metrics, /health/ready, and the debug endpoints — covered in the architecture page's observability section.

Where to go next

• Error handling and DLQ — the concrete handler classes, the configuration keys that select them per failure class, the full DLQ header set, and worked examples in Kotlin and Java.
• Exactly-once — why DLQ records and skipped-record offsets commit atomically with the rest of the epoch.
• Architecture — the full failure-mode and observability catalogue at the system level.
• Kafka Streams compatibility matrix — how StoatFlow's error-handling surface lines up against Kafka Streams.