# Getting started

StoatFlow targets **JDK 25+** and **Apache Kafka 4.x**. Working familiarity with Kafka and stream processing concepts is assumed.

## Pick a starting point

::path-cards
  :::path-card
  ---
  icon: i-lucide-rocket
  title: Starting fresh
  to: https://stoatflow.io/docs/getting-started/installation
  ---
  New project, no existing Kafka Streams code. Pull in the dependency, wire up your first topology, run it locally.
  :::

  :::path-card
  ---
  icon: i-lucide-shuffle
  title: Migrating from Kafka Streams
  to: https://stoatflow.io/docs/migration
  ---
  Existing Kafka Streams topology you want to run on StoatFlow. Dependency swap, config cleanup, what to remove and what stays.
  :::
::

## What's documented today

**Getting started** — orientation and a first running app:

- [**Modules overview**](https://stoatflow.io/docs/getting-started/modules-overview) — which artifacts to depend on; DSL-only versus the batteries-included runtime.
- [**Installation**](https://stoatflow.io/docs/getting-started/installation) — Maven repository, the dependency, and build setup (Gradle or Maven).
- [**License configuration**](https://stoatflow.io/docs/getting-started/license-configuration) — license key and environment for local development and CI/CD.
- [**Your first app**](https://stoatflow.io/docs/getting-started/first-app) — build and run a complete word-count app on the runtime.
- [**Project setup**](https://stoatflow.io/docs/getting-started/project-setup) — the StoatFlow build conventions for Gradle and Maven, and what they configure.

**Core concepts** — the model behind the runtime:

- [**Architecture**](https://stoatflow.io/docs/concepts/architecture), [**exactly-once**](https://stoatflow.io/docs/concepts/exactly-once), [**lanes and parallelism**](https://stoatflow.io/docs/concepts/lanes-and-parallelism), [**state and thread-safety**](https://stoatflow.io/docs/concepts/state-and-thread-safety), [**event time and watermarks**](https://stoatflow.io/docs/concepts/event-time-and-watermarks), the [**configuration model**](https://stoatflow.io/docs/concepts/configuration-model), the [**error-handling model**](https://stoatflow.io/docs/concepts/error-handling-model), and [**how StoatFlow differs from Kafka Streams**](https://stoatflow.io/docs/concepts/how-stoatflow-differs-from-ks).

**Building topologies** — the DSL and the Processor API:

- The [**DSL overview**](https://stoatflow.io/docs/building) plus aggregations, windowing, joins, the Processor API, serdes, error handling, state stores, and [**testing**](https://stoatflow.io/docs/building/testing).

**Configuration, runtime, and operations:**

- [**Configuration**](https://stoatflow.io/docs/configuration) — how config layers and the engine + runtime keys.
- [**Running in production**](https://stoatflow.io/docs/runtime) — the runtime, REST API, health checks, metrics, plugins, Docker, and native image.
- [**Deploying & operating**](https://stoatflow.io/docs/operating) — Kubernetes, probes, observability, tuning, and the production checklist.
- [**Migrating from Kafka Streams**](https://stoatflow.io/docs/migration) — what carries over, and cutover with or without state.
- [**Reference**](https://stoatflow.io/docs/reference) — configuration keys, the REST API, the Gradle plugin DSL, the KS compatibility matrix, and a glossary.

The docs are still maturing. [Tell us what to write next](https://stoatflow.io/contact).

## Need help?

Stuck on something the docs don't yet cover? [Get in touch](https://stoatflow.io/contact) — alpha-stage product, real people read every email. If you've hit something that probably belongs in these docs, say so and we'll write it.


# Modules overview

StoatFlow ships as a small set of modules under the `io.stoatflow` group, plus a Gradle convention plugin. This page explains what each one is for and how they fit together, so you can pick the right dependency before you start. For exact coordinates see [Installation](https://stoatflow.io/docs/getting-started/installation); for the plugin setup see [Project setup](https://stoatflow.io/docs/getting-started/project-setup).

::tldr-panel
- **`stoatflow-runtime`** — the batteries-included production wrapper (YAML config, HTTP admin + health endpoints, Prometheus metrics, graceful shutdown). It pulls in `stoatflow-core`, so it's usually the only main dependency you add.
- **`stoatflow-core`** — the DSL and the engine on their own, no runtime scaffolding. Choose this when you want to embed StoatFlow and own the lifecycle yourself.
- **`stoatflow-test-utils`** — the broker-free `TopologyTestDriver` for unit-testing topologies.
- **`stoatflow-test-runtime`** — wires that test driver up from your runtime `application.yaml`, so tests load the same config your app does.
- **The StoatFlow build conventions** — the `io.stoatflow` Gradle plugin and the `stoatflow-parent` Maven POM apply the JDK 25 toolchain, the required JVM flags, and a runnable build. Both tools, kept symmetric.
::

## The two main modules

StoatFlow separates the stream-processing library from the production wrapper around it. You depend on exactly one of them as your main dependency.

### `stoatflow-core` — DSL + engine

`stoatflow-core` is the library proper: the Kafka Streams-compatible DSL (`StreamsBuilder`, `KStream`, `KTable`, aggregations, joins, windowing), the Processor API, state stores, watermark strategies, and the single-instance engine that runs them. Everything you write a topology against lives here, under `io.stoatflow.core.*`.

It carries no opinion about *how* your process starts, gets configured, or exposes itself for operations. You construct the engine, hand it a topology and a `StreamsConfig`, and drive its lifecycle yourself. The DSL-only example wires it up by hand:

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.StoatFlow
import io.stoatflow.core.config.StreamsConfig
import io.stoatflow.core.topology.StreamsBuilder
```

```java [Java]
import io.stoatflow.core.StoatFlow;
import io.stoatflow.core.config.StreamsConfig;
import io.stoatflow.core.topology.StreamsBuilder;
```
::

Depend on `stoatflow-core` directly when you're embedding StoatFlow into an application that already owns configuration, HTTP, and observability — or when you deliberately want a minimal footprint and intend to manage the engine lifecycle yourself.

### `stoatflow-runtime` — batteries-included wrapper

`stoatflow-runtime` is a thin, customer-extensible wrapper around the core engine. It adds the operational scaffolding a production deployment needs:

- **YAML configuration** loaded from `application.yaml` (plus environment-variable and overlay-file overrides).
- **An HTTP server** exposing admin and introspection endpoints — `/info`, `/config`, `/topology`, `/state`, `/watermarks`, and more.
- **Kubernetes health probes** at `/health/live` and `/health/ready`, backed by indicators for application state, broker connectivity, Schema Registry, and license validity.
- **Prometheus metrics** at `/metrics` via Micrometer.
- **A plugin system** (`RuntimePlugin`, `PluginContext`, lifecycle listeners) for hooking into start/stop and registering your own handlers, health indicators, or plugins.
- **Graceful shutdown** wired into the JVM lifecycle.

The entry point is `StoatFlowRuntime.fromConfig(...)`, which loads your config, starts the HTTP and metrics server, and runs your topology until terminated:

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.runtime.StoatFlowRuntime

val runtime = StoatFlowRuntime.fromConfig(
    topologyBuilder = { buildTopology(it) },
)
runtime.start()
runtime.awaitTermination()
```

```java [Java]
import io.stoatflow.runtime.StoatFlowRuntime;

var runtime = StoatFlowRuntime.fromConfig(Main::buildTopology);
runtime.start();
runtime.awaitTermination();
```
::

`stoatflow-runtime` depends transitively on `stoatflow-core`, so adding it gives you the full DSL plus the wrapper in one dependency. Most applications want this — it's the default the rest of the getting-started guide builds on (see [Your first app](https://stoatflow.io/docs/getting-started/first-app)). The runtime module is published un-obfuscated precisely so you can extend it: writing a custom `RuntimePlugin`, a custom `HealthIndicator`, or an extra HTTP handler all work against types you can see by name.

## The testing modules

StoatFlow provides two testing modules. Both are test-scoped dependencies, and which one you use follows directly from which main module you chose.

### `stoatflow-test-utils` — the broker-free test driver

`stoatflow-test-utils` provides `TopologyTestDriver`, an in-memory harness that runs a topology synchronously without a Kafka broker. You pipe records into test input topics and read results from test output topics, with explicit control over timestamps and watermarks. Because processing is synchronous and deterministic, tests are fast and easy to debug. It supports the full DSL — stateful processors, windowing, joins, custom partitioners — and exposes state stores for assertions.

```kotlin
val driver = TopologyTestDriver.fromBuilder(builder)
val input = driver.createInputTopic<String, String>("input-topic", Serdes.String(), Serdes.String())
val output = driver.createOutputTopic<String, String>("output-topic", Serdes.String(), Serdes.String())

input.pipeInput("key1", "hello")
val record = output.readRecord()
```

`TopologyTestDriver` is **not** thread-safe — create a fresh instance per test case when running tests in parallel.

This is the module to add when you depend on `stoatflow-core` and build your topology in code with no `application.yaml`.

### `stoatflow-test-runtime` — the same driver, from your runtime config

`stoatflow-test-runtime` is the testing counterpart to `stoatflow-runtime`. Its `StoatFlowTestDriver.fromConfig(...)` factory loads the same `application.yaml` your production app uses and returns a configured `TopologyTestDriver`. It mirrors the production `StoatFlowRuntime.fromConfig(...)` API — you pass the same topology builder — so your tests exercise the same config your runtime resolves at startup (env vars with the `STOATFLOW_` prefix still take highest priority, and a `src/test/resources/application.yaml` shadows the main one).

```kotlin
val driver = StoatFlowTestDriver.fromConfig { builder, config ->
    MyApp.buildTopology(builder)
}
```

It depends on both `stoatflow-test-utils` and `stoatflow-runtime`, so you get the in-memory driver and config loading together. The StoatFlow example apps that ship with `application.yaml` use this module for their tests. Reach for it whenever you're on `stoatflow-runtime` and want your tests to load configuration the way your deployed app does, rather than reconstructing it in test code.

## The build conventions

The build conventions are a build-time concern, not a runtime dependency. StoatFlow requires a JDK 25 toolchain and two JVM flags (`--enable-preview` and `--enable-native-access=ALL-UNNAMED`) at both compile and run time — the conventions apply all of that for you, plus a runnable application setup and a fat-jar build, so you don't hand-maintain the toolchain or argument lists. They ship for **both build tools, kept symmetric**: the `io.stoatflow` Gradle plugin and the `stoatflow-parent` Maven POM.

On Gradle, apply the plugin and point it at your main class:

```kotlin
// build.gradle.kts
plugins {
    kotlin("jvm") version "2.4.0"   // omit for a Java-only project
    id("io.stoatflow") version "<stoatflow-version>"
}

stoatflow {
    mainClass.set("com.example.MainKt")
}
```

On Maven, inherit from `stoatflow-parent` and set `<stoatflow.mainClass>` instead. Both tools also offer opt-in Docker and GraalVM native-image builds, and bundle the same native flags and container entrypoints under the hood, so they can't drift. The full setup for both — including the `pluginManagement` repository the Gradle marker resolves from and the Maven BOM — is in [Project setup](https://stoatflow.io/docs/getting-started/project-setup).

## How they fit together

| Module             | Group / artifact                                  | Scope | Depends on          | Use when                                                                                         |
| ------------------ | ------------------------------------------------- | ----- | ------------------- | ------------------------------------------------------------------------------------------------ |
| Core               | `io.stoatflow:stoatflow-core`                     | main  | —                   | You want only the DSL + engine and will own the lifecycle, config, and observability yourself.   |
| Runtime            | `io.stoatflow:stoatflow-runtime`                  | main  | core                | The default for most apps — batteries-included config, HTTP, health, metrics, graceful shutdown. |
| Test utils         | `io.stoatflow:stoatflow-test-utils`               | test  | core                | Broker-free unit tests of a topology built in code.                                              |
| Test runtime       | `io.stoatflow:stoatflow-test-runtime`             | test  | test-utils, runtime | Tests that should load the same `application.yaml` as your runtime app.                          |
| Gradle plugin      | `io.stoatflow` (plugin id)                        | build | —                   | Gradle builds — applies the JDK 25 toolchain, required JVM flags, runnable + fat-jar build.      |
| Maven parent / BOM | `io.stoatflow:stoatflow-parent` · `stoatflow-bom` | build | —                   | Maven builds — the same conventions via `<parent>`; the BOM aligns dependency versions.          |

The same table as a graph — solid arrows are dependencies, the dashed one is the build-time conventions:

```mermaid
flowchart LR
    APP["your app"] --> RT["stoatflow-runtime"]
    RT --> CORE["stoatflow-core"]
    TESTS["your tests"] --> TRT["stoatflow-test-runtime"]
    TRT --> TU["stoatflow-test-utils"]
    TRT --> RT
    TU --> CORE
    BUILD["build conventions —<br/>io.stoatflow plugin ·<br/>stoatflow-parent POM"] -.applied to the build.-> APP
```

In practice the common path is: depend on `stoatflow-runtime` for your app, `stoatflow-test-runtime` for your tests, and apply the StoatFlow build conventions (the `io.stoatflow` Gradle plugin, or the `stoatflow-parent` POM on Maven). A DSL-only embedding swaps the first two for `stoatflow-core` and `stoatflow-test-utils`.

A minimal dependency block:

::code-tabs{group="build"}
```kotlin [Gradle]
// build.gradle.kts
dependencies {
    implementation("io.stoatflow:stoatflow-runtime:<stoatflow-version>")
    // ...or DSL only:
    // implementation("io.stoatflow:stoatflow-core:<stoatflow-version>")

    testImplementation("io.stoatflow:stoatflow-test-runtime:<stoatflow-version>")
    // ...or, for a core-only project:
    // testImplementation("io.stoatflow:stoatflow-test-utils:<stoatflow-version>")
}
```

```xml [Maven]
<!-- pom.xml -->
<dependencies>
  <dependency>
    <groupId>io.stoatflow</groupId>
    <artifactId>stoatflow-runtime</artifactId>
    <version>${stoatflow.version}</version>
  </dependency>
  <!-- ...or DSL only: stoatflow-core -->

  <dependency>
    <groupId>io.stoatflow</groupId>
    <artifactId>stoatflow-test-runtime</artifactId>
    <version>${stoatflow.version}</version>
    <scope>test</scope>
  </dependency>
  <!-- ...or, for a core-only project: stoatflow-test-utils -->
</dependencies>
```
::

::callout{color="info" icon="i-lucide-info"}
StoatFlow is in **alpha**. The current version is :stoatflow-version — substitute it for the `<stoatflow-version>` placeholder above. Full coordinates and credential setup are in [Installation](https://stoatflow.io/docs/getting-started/installation).
::

## Next steps

- **[Installation](https://stoatflow.io/docs/getting-started/installation)** — the private Maven repository, credentials, exact coordinates, and JVM toolchain.
- **[Project setup](https://stoatflow.io/docs/getting-started/project-setup)** — applying the `io.stoatflow` Gradle plugin in full.
- **[Your first app](https://stoatflow.io/docs/getting-started/first-app)** — build and run a complete word-count app on `stoatflow-runtime`.
- **[Architecture](https://stoatflow.io/docs/concepts/architecture)** — how the single-instance engine runs your topology.


# Installation

This guide gets your build ready to use StoatFlow: wire up the private Maven repository with your credentials, add the dependency, and configure the JVM toolchain. Then [configure your license](https://stoatflow.io/docs/getting-started/license-configuration) and [build your first app](https://stoatflow.io/docs/getting-started/first-app).

::tldr-panel
- **Prerequisites:** JDK 25+, a reachable Kafka 4.x cluster, and your onboarding email (Maven token + license key).
- **Repo:** `https://maven.stoatflow.io/releases`, authenticated with the read-only token from your onboarding email.
- **Dependency:** `io.stoatflow:stoatflow-runtime` (batteries-included) or `io.stoatflow:stoatflow-core` (DSL only).
- **Run:** needs the `--enable-preview` and `--enable-native-access=ALL-UNNAMED` JVM flags — the StoatFlow build conventions (Gradle plugin / Maven parent POM) set them for you.
::

## Prerequisites

- **JDK 25 or newer.** StoatFlow uses JDK preview features (virtual threads, structured concurrency), so a 25+ toolchain is mandatory — not just for building, but at runtime.
- **Apache Kafka 4.x**, reachable from where you run the app. For local development the quickest option is a single-node KRaft broker via Docker (`localhost:9092`).
- **Your onboarding email.** It contains your read-only Maven **token** (username `customer-<your-slug>` + secret) and your **license key**. Don't have one yet? [Get in touch](https://stoatflow.io/contact).

::callout{color="warning" icon="i-lucide-shield"}
Treat the Maven token and license key as secrets. The steps below keep them in your **home directory** (`~/.gradle/gradle.properties`, `~/.m2/settings.xml`), never in a file you commit.
::

## 1. Register the private Maven repository

StoatFlow artifacts are served from a private, authenticated repository at `https://maven.stoatflow.io/releases`. You can wire it up two ways — directly on each developer machine (below), or once through your corporate artifact manager.

::callout{color="info" icon="i-lucide-building-2"}
**Corporate environments — proxy it through your artifact manager.** If your organisation runs a central repository (JFrog Artifactory, Sonatype Nexus, or similar), add `https://maven.stoatflow.io/releases` as a **remote / proxy repository** and surface it through the **virtual (group) repository** your developers already resolve against. That way:

- your StoatFlow token is configured **once**, centrally — never copied onto individual machines;
- developers need **no local credentials and no build changes** — they resolve through the corporate repo exactly as they do today;
- dependencies (and the `io.stoatflow` Gradle plugin) **pull through and cache** via the proxy, with authentication handled there.
::

**Trial vs. production:** the per-developer setup below is the fastest way to start a trial. Once you've agreed a contract and are moving toward production, switch to the corporate-proxy setup.

### Store your credentials

::code-tabs{group="build"}
```properties [Gradle]
# ~/.gradle/gradle.properties   — your HOME dir, never committed
stoatflowRepoUser=customer-your-slug
stoatflowRepoToken=PASTE_YOUR_MAVEN_TOKEN_FROM_THE_ONBOARDING_EMAIL
```

```xml [Maven]
<!-- ~/.m2/settings.xml   — your HOME dir, never committed -->
<settings>
  <servers>
    <server>
      <id>stoatflow-releases</id>
      <username>customer-your-slug</username>
      <password>PASTE_YOUR_MAVEN_TOKEN_FROM_THE_ONBOARDING_EMAIL</password>
    </server>
  </servers>
</settings>
```
::

### Point your build at the repository

::code-tabs{group="build"}
```kotlin [Gradle]
// settings.gradle.kts
dependencyResolutionManagement {
    repositories {
        mavenCentral()
        maven {
            url = uri("https://maven.stoatflow.io/releases")
            credentials {
                username = providers.gradleProperty("stoatflowRepoUser").get()
                password = providers.gradleProperty("stoatflowRepoToken").get()
            }
        }
    }
}
```

```xml [Maven]
<!-- pom.xml -->
<repositories>
  <repository>
    <id>stoatflow-releases</id>
    <url>https://maven.stoatflow.io/releases</url>
    <releases><enabled>true</enabled></releases>
  </repository>
</repositories>
```
::

The `<id>` in `pom.xml` must match the `<server><id>` in `settings.xml` — that's how Maven applies your credentials.

## 2. Add the dependency

Most applications want &#x2A;*`stoatflow-runtime`** — the batteries-included wrapper (YAML config, HTTP admin + health endpoints, Prometheus metrics, graceful shutdown). It transitively pulls in `stoatflow-core&#x60;, so it's the only dependency you need. If you want just the DSL and engine with no runtime scaffolding, depend on &#x2A;*`stoatflow-core`** instead.

::code-tabs{group="build"}
```kotlin [Gradle]
// build.gradle.kts
dependencies {
    implementation("io.stoatflow:stoatflow-runtime:<stoatflow-version>")
    // ...or DSL only:
    // implementation("io.stoatflow:stoatflow-core:<stoatflow-version>")

    // In-memory topology testing (no broker required):
    testImplementation("io.stoatflow:stoatflow-test-utils:<stoatflow-version>")
}
```

```xml [Maven]
<!-- pom.xml -->
<dependencies>
  <dependency>
    <groupId>io.stoatflow</groupId>
    <artifactId>stoatflow-runtime</artifactId>
    <version>${stoatflow.version}</version>
  </dependency>
  <!-- In-memory topology testing (no broker required): -->
  <dependency>
    <groupId>io.stoatflow</groupId>
    <artifactId>stoatflow-test-utils</artifactId>
    <version>${stoatflow.version}</version>
    <scope>test</scope>
  </dependency>
</dependencies>
```
::

::callout{color="info" icon="i-lucide-info"}
StoatFlow is in **alpha**. The current version is :stoatflow-version — substitute it for the `<stoatflow-version>` placeholder (Gradle) or set it as `<stoatflow.version>` in your `pom.xml` `<properties>` (Maven), or use the exact version from your onboarding email if it differs. The published modules are `stoatflow-core`, `stoatflow-runtime`, `stoatflow-test-utils`, and `stoatflow-test-runtime`, all under group `io.stoatflow`.
::

## 3. Configure the JVM toolchain

StoatFlow needs a **JDK 25 toolchain** and two JVM flags at **both compile and run time**:

- `--enable-preview` — StoatFlow is built on JDK preview features.
- `--enable-native-access=ALL-UNNAMED` — RocksDB's state store uses the Foreign Function & Memory API.
- `-XX:+UseG1GC` keeps GC pauses short and predictable — which matters for stream processing, where a long stop-the-world pause stalls commit barriers and inflates end-to-end latency. The JVM only auto-selects G1 on "server-class" hardware (≥ 2 CPUs and ≥ \~2 GB of memory), falling back to the single-threaded Serial collector below that — so set it explicitly to get consistent pause behaviour even on a single core or a small container. (The StoatFlow build conventions already apply this flag to the local `run` task and to the Docker image they build.)

### Option A — the StoatFlow build conventions (recommended)

StoatFlow ships build conventions for **both Gradle and Maven*&#x2A; — the JDK 25 toolchain, the preview + native-access JVM args (compile, test, and run), and a runnable fat JAR — so you don't hand-maintain any of it. This page covers just the toolchain setup; the full walkthrough (container images, native image, every knob) is in &#x2A;*[Project setup](https://stoatflow.io/docs/getting-started/project-setup)**.

#### Gradle — the `io.stoatflow` plugin

Add a `pluginManagement` repository for the plugin marker in `settings.gradle.kts`:

```kotlin
// settings.gradle.kts
pluginManagement {
    repositories {
        gradlePluginPortal()
        mavenCentral()
        maven {
            url = uri("https://maven.stoatflow.io/releases")
            credentials {
                username = providers.gradleProperty("stoatflowRepoUser").get()
                password = providers.gradleProperty("stoatflowRepoToken").get()
            }
        }
    }
}
```

Then apply it in `build.gradle.kts`:

```kotlin
// build.gradle.kts
plugins {
    kotlin("jvm") version "2.4.0"   // omit for a Java-only project
    id("io.stoatflow") version "<stoatflow-version>"
}

stoatflow {
    mainClass.set("com.example.MainKt")  // your application entry point
}
```

#### Maven — the `stoatflow-parent` POM

Inherit from the parent POM; it applies the toolchain, the JVM flags, the fat JAR, and the test split. Your dependency's version comes from the bundled BOM, so you omit it:

```xml
<!-- pom.xml -->
<parent>
  <groupId>io.stoatflow</groupId>
  <artifactId>stoatflow-parent</artifactId>
  <version>REPLACE-WITH-CURRENT-RELEASE</version>  <!-- literal — Maven can't resolve a property in <parent> -->
  <relativePath/>
</parent>

<properties>
  <stoatflow.mainClass>com.example.Main</stoatflow.mainClass>
</properties>
```

See the &#x2A;*[Maven reference](https://stoatflow.io/docs/reference/maven-reference)** for the BOM, every property, and the container + native builds.

### Option B — configure it yourself

If you'd rather not use the plugin, set the toolchain and JVM args directly:

::code-tabs{group="build"}
```kotlin [Gradle]
// build.gradle.kts — with the `application` plugin applied
java {
    toolchain { languageVersion = JavaLanguageVersion.of(25) }
}
tasks.withType<JavaCompile> { options.compilerArgs.add("--enable-preview") }

application {
    applicationDefaultJvmArgs = listOf(
        "--enable-preview",
        "--enable-native-access=ALL-UNNAMED",
        "-XX:+UseG1GC",
    )
}
```

```xml [Maven]
<!-- pom.xml -->
<properties>
  <maven.compiler.release>25</maven.compiler.release>
</properties>

<build>
  <plugins>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-compiler-plugin</artifactId>
      <configuration>
        <compilerArgs><arg>--enable-preview</arg></compilerArgs>
      </configuration>
    </plugin>
  </plugins>
</build>

<!-- Run with: java --enable-preview --enable-native-access=ALL-UNNAMED -XX:+UseG1GC -jar your-app.jar -->
```
::

## Next steps

- **[License configuration](https://stoatflow.io/docs/getting-started/license-configuration)** — every license env var / system property / YAML key, for local development and CI/CD.
- **[Your first app](https://stoatflow.io/docs/getting-started/first-app)** — build and run a complete word-count app on the runtime.
- **[AI assistants](https://stoatflow.io/docs/getting-started/ai-assistants)** — install the skills pack so your AI coding assistant writes correct StoatFlow build files instead of hallucinated Kafka Streams.
- **[Architecture](https://stoatflow.io/docs/concepts/architecture)** — the single-instance model, lane dispatcher, and commit barriers.
- Stuck? [Get in touch](https://stoatflow.io/contact) — real people read every email during the alpha.


# License configuration

StoatFlow checks that a license key is **present** at startup (a missing key still fails fast), then runs the **activating validation \~5 minutes after start** on a background thread — and re-checks it on a periodic heartbeat. Until that first validation the license reports `PENDING`; the app serves traffic normally, and on a validation failure it shuts down gracefully. You give it two things: your **license key** and — for production — an **environment** label. This page covers every way to provide them, for local development and CI/CD.

::tldr-panel
- Set your **license key** via `STOATFLOW_LICENSE_KEY`, the `stoatflow.license.key` system property, a `~/.stoatflow/license.key` file, or `application.yaml`.
- **Resolution:** an explicit **key** beats a key **file**; within each, an environment variable beats a system property beats `application.yaml`.
- The **environment** label scopes your license seat (`SHA-256(application-id || environment)`). Production **requires** it explicitly.
- **CI/CD:** on GitHub Actions the environment is auto-derived as `cicd-<run id>`; elsewhere set `STOATFLOW_LICENSE_ENVIRONMENT` yourself.
- **Deferred validation:** the key must be present at start, but the activating validation runs \~5 minutes later — short-lived test instances never contact the license server and never consume a seat.
::

## The license key

Your key is the signed string from your onboarding email — it starts with `key/`. The runtime resolves it by looking for an explicit **key** first, then a key **file** — and within each, an environment variable beats a system property (which is what an `application.yaml` value becomes):

1. **An explicit key** — `STOATFLOW_LICENSE_KEY` (environment variable), else the `stoatflow.license.key` system property (which `application.yaml`'s `stoatflow.license.key` feeds).
2. **A key file** — `STOATFLOW_LICENSE_FILE` (environment variable), else the `stoatflow.license.file` system property (fed by `application.yaml`'s `stoatflow.license.file`), else the default `~/.stoatflow/license.key`.

So an explicit key always wins over a key file, regardless of where each came from.

::callout{icon="i-lucide-info"}
`application.yaml` values are applied **below** environment variables and `-D` system properties — so an env var always wins over YAML. This lets you commit a YAML default and override it per environment without touching the file.
::

Pick whichever fits — by source:

::code-group
```bash [Shell env]
# Highest precedence. Good for containers, CI, and one-off local runs.
export STOATFLOW_LICENSE_KEY="key/...your key from the onboarding email..."
```

```bash [Key file]
# Set once, every local app picks it up — no env var needed.
# Must be chmod 600 (owner-only) on Linux/macOS, or startup is refused.
mkdir -p ~/.stoatflow
printf '%s' 'key/...your key...' > ~/.stoatflow/license.key
chmod 600 ~/.stoatflow/license.key
```

```yaml [application.yaml]
# Runtime module only. Interpolate from an env var so the literal key
# never lands in source control.
stoatflow:
  license:
    key: ${STOATFLOW_LICENSE_KEY}
```
::

## The environment label

The **environment** is a label for the deployment a license seat is bound to — e.g. `prod`, `prod-eu-west`, `staging`, `dev`. StoatFlow derives a machine fingerprint from it:

```text
fingerprint = SHA-256(application-id || environment)
```

So the same `application-id` + the same `environment` is **one seat**; distinct environments are distinct seats. Set it with `STOATFLOW_LICENSE_ENVIRONMENT`, the `stoatflow.license.environment` system property, or `stoatflow.license.environment` in `application.yaml`.

- **Local development** (Trial / Developer tiers): if you don't set it, the environment defaults to `<your-username>-<hostname>` — one seat per developer machine, no configuration needed.
- **Production**: the Production tier **requires an explicit environment** — the engine refuses to run if it has to auto-generate one (the check fires at the deferred validation, \~5 minutes after start). Set it per deployment (`prod`, `prod-eu-west`, …) so each gets its own node-locked seat.

## Local development

The simplest local setup is the **key file** — write it once and every StoatFlow app you run locally is licensed, with no env vars and an auto-derived per-machine environment:

```bash
mkdir -p ~/.stoatflow
printf '%s' 'key/...your developer key...' > ~/.stoatflow/license.key
chmod 600 ~/.stoatflow/license.key
```

Prefer per-run configuration? Export the key (and optionally the environment) in your shell:

```bash
export STOATFLOW_LICENSE_KEY="key/...your developer key..."
# Optional — defaults to <username>-<hostname> if unset:
export STOATFLOW_LICENSE_ENVIRONMENT="dev-$(whoami)"
```

## CI/CD

Use your **CI/CD-tier** key (it carries a higher machine cap suited to short-lived build agents). Store it as an encrypted secret — never in the repo.

On **GitHub Actions**, StoatFlow auto-detects the run and sets the environment to `cicd-<run id>` for you, so each run gets a fresh, ephemeral seat — you only need to provide the key:

```yaml
# .github/workflows/your-workflow.yml
jobs:
  integration-test:
    runs-on: ubuntu-latest
    env:
      STOATFLOW_LICENSE_KEY: ${{ secrets.STOATFLOW_CI_LICENSE }}
      # Optional — auto-derived as cicd-${GITHUB_RUN_ID} on GitHub Actions:
      # STOATFLOW_LICENSE_ENVIRONMENT: cicd-${{ github.run_id }}
    steps:
      - uses: actions/checkout@v4
      # ... build + run StoatFlow tests ...
```

On other CI systems (GitLab CI, Jenkins, …) there's no auto-detection — set the environment explicitly to something unique-per-run so concurrent builds don't contend for one seat:

```bash
export STOATFLOW_LICENSE_KEY="$STOATFLOW_CI_LICENSE"
export STOATFLOW_LICENSE_ENVIRONMENT="cicd-${CI_PIPELINE_ID:-$(date +%s)}"
```

### Deferred validation & CI/testcontainers

The activating license validation — the call that registers a machine against your license — runs &#x2A;*\~5 minutes after `start()`**, not at startup. Startup only checks the key is *present* (offline, no network). The consequence for tests:

- A test instance (testcontainers, integration tests, CI jobs) that starts and stops **within \~5 minutes** makes **zero license-server API calls** and consumes **zero machine activations** — the usual start-app-per-test-class pattern is free, no matter how often CI runs.
- During that window the license status reports `PENDING`: readiness is UP, the app processes normally, and the `stoatflow.license.valid` gauge reads `0` until validation succeeds (see [Metrics](https://stoatflow.io/docs/runtime/metrics)).
- An instance that **outlives** the window validates and consumes **one activation** for its fingerprint. Long-running soak/perf tests therefore each take a seat — the CI/CD tier's higher machine cap covers this; seats from ended runs are reaped automatically by the license server.

## All license settings

Every setting, with its environment variable, system property, and `application.yaml` key (under `stoatflow.license`). Environment variable wins over system property wins over YAML.

| `application.yaml` (`stoatflow.license.*`) | Environment variable               | System property                    | Default                        | Purpose                                   |
| ------------------------------------------ | ---------------------------------- | ---------------------------------- | ------------------------------ | ----------------------------------------- |
| `key`                                      | `STOATFLOW_LICENSE_KEY`            | `stoatflow.license.key`            | —                              | The signed license key (`key/…`).         |
| `file`                                     | `STOATFLOW_LICENSE_FILE`           | `stoatflow.license.file`           | `~/.stoatflow/license.key`     | Path to a file containing the key.        |
| `environment`                              | `STOATFLOW_LICENSE_ENVIRONMENT`    | `stoatflow.license.environment`    | `<username>-<hostname>` (auto) | Seat label; **required** for Production.  |
| `cache-dir`                                | `STOATFLOW_LICENSE_CACHE_DIR`      | `stoatflow.license.cache-dir`      | `~/.stoatflow/license-cache/`  | Where the offline-grace cache is stored.  |
| `verbose-banner`                           | `STOATFLOW_LICENSE_VERBOSE_BANNER` | `stoatflow.license.verbose-banner` | `false`                        | Log a detailed license banner at startup. |

::callout{icon="i-lucide-info"}
**Heartbeat cadence and offline-grace are set by your license tier, not by configuration.** They govern how quickly a license change takes effect and how long the engine tolerates a Keygen outage — so they're tuned per tier rather than left to each deployment. If your environment needs a longer offline-grace window (intermittent connectivity, restricted egress), [talk to us](https://stoatflow.io/contact) and we'll set it on your license.
::

## Troubleshooting

- **Startup fails: license key missing.** The presence check hard-fails fast at `start()`. Check the key is set (precedence above).
- **App shuts down \~5 minutes after start: license invalid / expired / revoked.** The deferred validation rejected the key — the log carries `[LICENSE] deferred validation failed (code=…)`. Check the key is current; renew via your onboarding contact.
- **`license key file has insecure permissions`.** The key file must be owner-only — `chmod 600 ~/.stoatflow/license.key`.
- **Production refuses to run without an environment.** Set `STOATFLOW_LICENSE_ENVIRONMENT` (or `stoatflow.license.environment`) explicitly — Production won't auto-generate one. The refusal fires at the deferred validation (\~5 minutes after start), once the tier is known.
- **Brief Keygen outage.** The runtime keeps serving from its offline cache for the grace window, then fails on the next check. Normal restarts within the window are fine.
- **Lost your key or token?** We re-issue rather than retrieve — reply to your onboarding email and we'll rotate it.

## Next steps

- **[Architecture](https://stoatflow.io/docs/concepts/architecture)** — how the engine runs your topology.
- Questions about your license tier or seats? [Get in touch](https://stoatflow.io/contact).


# Your first app

Let's build a complete StoatFlow app on the `:runtime` module: a **word count** that reads lines of text, splits them into words, and keeps a running total for each word.

This page assumes you've finished [Installation](https://stoatflow.io/docs/getting-started/installation) — the private Maven repository, the `io.stoatflow:stoatflow-runtime` dependency, and the `io.stoatflow` Gradle plugin — and [configured a license](https://stoatflow.io/docs/getting-started/license-configuration).

::tldr-panel
- **What you build:** counts words from a `text-lines` topic into a `word-counts` topic — a stateful aggregation.
- **What you need:** a running Kafka 4.x broker, your license key, and the two topics.
- **Run:** `./gradlew run`, then produce a few lines of text and watch the counts climb.
::

## The topology

`StoatFlowRuntime.fromConfig(...)` loads `application.yaml`, starts the HTTP + metrics server, and runs your topology until terminated. The topology reads `text-lines`, splits each line into lowercase words, groups by word, counts, and writes each word's running total to `word-counts`. Counts are `Long`, so the sink uses a `Long` value serde — everything else is `String`.

Every operator is given an explicit name (`Named.as(...)`, `Grouped.as(...)`, …); StoatFlow uses these stable names for its topology graph, metrics, and state-store identity.

::code-tabs{group="lang"}
```kotlin [Kotlin]
package com.example

import io.stoatflow.core.state.StateStore
import io.stoatflow.core.topology.Consumed
import io.stoatflow.core.topology.Grouped
import io.stoatflow.core.topology.Materialized
import io.stoatflow.core.topology.Named
import io.stoatflow.core.topology.Produced
import io.stoatflow.core.topology.StreamsBuilder
import io.stoatflow.runtime.StoatFlowRuntime
import org.apache.kafka.common.serialization.Serdes

fun main() {
    val runtime = StoatFlowRuntime.fromConfig(
        topologyBuilder = { buildTopology(it) },
        configure = {
            streamsConfigOverrides {
                defaultKeySerde(Serdes.String())
                defaultValueSerde(Serdes.String())
            }
        },
    )
    runtime.start()
    runtime.awaitTermination()
}

private fun buildTopology(builder: StreamsBuilder) {
    val whitespace = "\\s+".toRegex()
    builder
        .stream<String, String>("text-lines", Consumed.`as`("source"))
        .flatMapValues(
            { line -> line.lowercase().split(whitespace).filter { it.isNotBlank() } },
            Named.`as`("split-words"),
        )
        .groupBy({ _, word -> word }, Grouped.`as`("group-by-word"))
        .count(Named.`as`("count"), Materialized.`as`<String, Long, StateStore>("word-counts"))
        .toStream(Named.`as`("to-stream"))
        .to(
            "word-counts",
            Produced.`as`<String, Long>("sink")
                .withKeySerde(Serdes.String())
                .withValueSerde(Serdes.Long()),
        )
}
```

```java [Java]
package com.example;

import io.stoatflow.core.state.StateStore;
import io.stoatflow.core.topology.Consumed;
import io.stoatflow.core.topology.Grouped;
import io.stoatflow.core.topology.KStream;
import io.stoatflow.core.topology.Materialized;
import io.stoatflow.core.topology.Named;
import io.stoatflow.core.topology.Produced;
import io.stoatflow.core.topology.StreamsBuilder;
import io.stoatflow.core.topology.ValueMapper;
import io.stoatflow.runtime.StoatFlowRuntime;
import org.apache.kafka.common.serialization.Serdes;

import java.util.Arrays;
import java.util.stream.Collectors;

public class Main {

    public static void main(String[] args) {
        var runtime = StoatFlowRuntime.fromConfig(
            Main::buildTopology,
            builder -> builder.streamsConfigOverrides(cfg -> {
                cfg.defaultKeySerde(Serdes.String());
                cfg.defaultValueSerde(Serdes.String());
            })
        );
        runtime.start();
        runtime.awaitTermination();
    }

    private static void buildTopology(StreamsBuilder builder) {
        KStream<String, String> lines = builder.stream("text-lines", Consumed.as("source"));
        lines
            .flatMapValues(
                (ValueMapper<String, Iterable<String>>) line ->
                    Arrays.stream(line.toLowerCase().split("\\s+"))
                        .filter(word -> !word.isBlank())
                        .collect(Collectors.toList()),
                Named.as("split-words"))
            .groupBy((key, word) -> word, Grouped.as("group-by-word"))
            .count(Named.as("count"), Materialized.<String, Long, StateStore>as("word-counts"))
            .toStream(Named.as("to-stream"))
            .to(
                "word-counts",
                Produced.<String, Long>as("sink")
                    .withKeySerde(Serdes.String())
                    .withValueSerde(Serdes.Long()));
    }
}
```
::

## Configuration

Add `src/main/resources/application.yaml`. Topic names live in the topology code above; this file configures the engine, your license, and the runtime's HTTP + metrics endpoints:

```yaml
stoatflow:
  application-id: word-count
  bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVERS:-localhost:9092}

  license:
    key: ${STOATFLOW_LICENSE_KEY}

runtime:
  http:
    enabled: true
    port: ${HTTP_PORT:-8080}
  metrics:
    enabled: true
```

::callout{color="info" icon="i-lucide-database"}
Word count is **stateful** — StoatFlow keeps a `word-counts` state store (the running totals) and backs it with a changelog topic so the state survives restarts and recovers quickly. Both are created automatically with sensible defaults; you don't need to configure anything extra to run locally.
::

## Run it

Make sure Kafka is running and the topics exist, export your license key, then start the app:

```bash
export KAFKA_BOOTSTRAP_SERVERS=localhost:9092
export STOATFLOW_LICENSE_KEY="key/...your key from the onboarding email..."

# Create the topics (one-off)
kafka-topics.sh --bootstrap-server localhost:9092 --create --topic text-lines --if-not-exists
kafka-topics.sh --bootstrap-server localhost:9092 --create --topic word-counts --if-not-exists

# Run (the io.stoatflow Gradle plugin provides the `run` task + the JVM flags)
./gradlew run
```

Health and metrics come up on port 8080:

```bash
curl -s localhost:8080/health/ready
```

Now feed it some text:

```bash
printf 'the quick brown fox\nthe lazy dog\nthe quick fox\n' \
  | kafka-console-producer.sh --bootstrap-server localhost:9092 --topic text-lines
```

And watch the counts. The values are `Long`, so tell the consumer to print the key and decode the value with the `LongDeserializer`:

```bash
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic word-counts \
  --from-beginning --property print.key=true \
  --value-deserializer org.apache.kafka.common.serialization.LongDeserializer
```

You'll see each word's total update as the lines are processed — `the` climbing to 3, `quick` and `fox` to 2, the rest at 1:

```text
the 1
quick   1
brown   1
fox 1
the 2
lazy    1
dog 1
the 3
quick   2
fox 2
```

::callout{color="info" icon="i-lucide-info"}
`count()` produces a `KTable`, so the output is a **changelog**: every time a word's total changes, StoatFlow emits that word's new count. That's why you see `the` reported as `1`, then `2`, then `3` rather than only the final tally — each is the running total at that point in the stream.
::

## Next steps

- **[Architecture](https://stoatflow.io/docs/concepts/architecture)** — how the single-instance engine runs your topology (lane dispatcher, commit barriers, state stores).
- **[License configuration](https://stoatflow.io/docs/getting-started/license-configuration)** — the full license reference for local development and CI/CD.
- Building something and stuck? [Get in touch](https://stoatflow.io/contact) — real people read every email during the alpha.


# Project setup

A StoatFlow application has non-trivial build requirements: a JDK 25 toolchain, two mandatory JVM flags at **compile, test, and run** time, a fat JAR with the right `Main-Class`, and — when you opt in — a container-aware heap entrypoint and a GraalVM native image with the correct metadata. You don't hand-maintain any of that. StoatFlow ships first-class build conventions for **both Gradle and Maven**, kept deliberately symmetric, so a production-ready build is a few lines of config.

::tldr-panel
- **Gradle** → the `io.stoatflow` convention plugin: one `plugins { }` line + a `stoatflow { }` block.
- **Maven** → the `io.stoatflow:stoatflow-parent` parent POM (+ the `stoatflow-bom` and `stoatflow-maven-plugin`): a `<parent>` and a few `<properties>`.
- **Both give you:** the JDK 25 toolchain, `--enable-preview` + `--enable-native-access=ALL-UNNAMED` (compile/test/run), `-XX:+UseG1GC`, a runnable fat JAR, a JUnit-Platform `IntegrationTest` tag-split, and opt-in [Docker](https://stoatflow.io/docs/runtime/docker) + [native-image](https://stoatflow.io/docs/runtime/native-image) builds.
- **First:** wire up the repository and credentials in [Installation](https://stoatflow.io/docs/getting-started/installation) — this page picks up from there.
::

## Apply the conventions

Register the StoatFlow repository (and, for Gradle, the plugin-marker repository) as shown in [Installation](https://stoatflow.io/docs/getting-started/installation#_1-register-the-private-maven-repository). Then apply the convention:

::code-tabs{group="build"}
```kotlin [Gradle]
// build.gradle.kts
plugins {
    kotlin("jvm") version "2.4.0"      // omit for a Java-only project
    id("io.stoatflow") version "<stoatflow-version>"
}

stoatflow {
    mainClass.set("com.example.MainKt") // your application entry point
}

dependencies {
    implementation("io.stoatflow:stoatflow-runtime:<stoatflow-version>")
}
```

```xml [Maven]
<!-- pom.xml -->
<parent>
  <groupId>io.stoatflow</groupId>
  <artifactId>stoatflow-parent</artifactId>
  <version>REPLACE-WITH-CURRENT-RELEASE</version>   <!-- literal — Maven can't resolve a property in <parent> -->
  <relativePath/>
</parent>

<properties>
  <stoatflow.mainClass>com.example.Main</stoatflow.mainClass>
</properties>

<dependencies>
  <dependency>
    <groupId>io.stoatflow</groupId>
    <artifactId>stoatflow-runtime</artifactId>
    <!-- version managed by the BOM the parent imports -->
  </dependency>
</dependencies>
```
::

The current published version is :stoatflow-version — substitute it for the placeholders above (the Maven `<parent>` version must be a **literal**; Maven does not interpolate properties in parent coordinates). For a Kotlin `fun main()` in `Main.kt` the entry point is the synthetic `MainKt` class (`com.example.MainKt`); for a Java `public static void main` it's the class itself (`com.example.Main`).

::callout{color="info" icon="i-lucide-info"}
The Maven parent imports `stoatflow-bom` for you, so you **omit the `<version>`** on every `io.stoatflow:*` dependency. If you already have a corporate parent POM, import the BOM directly instead and set the toolchain by hand — see [Maven reference → Using just the BOM](https://stoatflow.io/docs/reference/maven-reference#using-just-the-bom).
::

## What you get

Applying the convention — with no further configuration — already gives you all of this:

| Capability                            | What it does                                                                                                                                                                                                                        |
| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **JDK 25 toolchain**                  | Compiles and runs on JDK 25 — required for the virtual-thread / structured-concurrency preview features StoatFlow is built on.                                                                                                      |
| **Mandatory JVM flags**               | `--enable-preview` and `--enable-native-access=ALL-UNNAMED` at compile, test, **and** run time. Non-negotiable and always added.                                                                                                    |
| **Default GC flag**                   | `-XX:+UseG1GC` — overridable, but on by default for predictable pauses even on a single core or a small container.                                                                                                                  |
| **Runnable fat JAR**                  | A shaded `-all` JAR with the `Main-Class` manifest and merged service files. `java -jar` just works.                                                                                                                                |
| **Test conventions**                  | JUnit Platform with the preview/native-access flags; an `integrationTest` task that runs only `@Tag("IntegrationTest")` tests (the unit run excludes them).                                                                         |
| **Docker image*&#x2A; &#x2A;(opt-in)* | A reproducible JVM container via [Jib](https://github.com/GoogleContainerTools/jib){rel="&#x22;nofollow&#x22;"} — no `Dockerfile`, no Docker daemon to assemble the layers. See [Docker](https://stoatflow.io/docs/runtime/docker). |
| **Native image*&#x2A; &#x2A;(opt-in)* | A GraalVM native image with all of StoatFlow's reflection / JNI / FFM metadata already registered. One line to turn on. See [Native image](https://stoatflow.io/docs/runtime/native-image).                                         |

::callout{color="info" icon="i-lucide-info"}
`-XX:+UseG1GC` is set explicitly because the JVM only auto-selects G1 on "server-class" hardware (≥ 2 CPUs and ≥ \~2 GB memory), falling back to the single-threaded Serial collector below that. Setting it explicitly gives short, predictable GC pauses even on a single core — which matters for stream processing, where a long stop-the-world pause stalls commit barriers. See [Installation](https://stoatflow.io/docs/getting-started/installation#_3-configure-the-jvm-toolchain) for the full reasoning.
::

## Configure the build

`mainClass` is the only property most projects set. The container and native-image builds are opt-in and default to off. Here are the common knobs — the **full** surface (every property, task, and default) lives in the reference pages, linked below.

::code-tabs{group="build"}
```kotlin [Gradle]
// build.gradle.kts
stoatflow {
    mainClass.set("com.example.MainKt")

    docker {
        enabled.set(true)               // opt in to the Jib image
        imageName.set("acme/word-count")
    }

    nativeImage {
        enabled.set(true)               // opt in to the GraalVM native image
        gc.set("G1")                    // Oracle GraalVM; omit for Serial (stateless only)
    }
}
```

```xml [Maven]
<!-- pom.xml — container/native config lives in <properties> -->
<properties>
  <stoatflow.mainClass>com.example.Main</stoatflow.mainClass>
  <stoatflow.docker.imageName>acme/word-count</stoatflow.docker.imageName>
  <stoatflow.nativeImage.gc>G1</stoatflow.nativeImage.gc>
</properties>
```
::

::callout{color="warning" icon="i-lucide-triangle-alert"}
**Maven profiles are activated with `-P`, not `<properties>`.** Setting `<stoatflow.docker.enabled>` does **not** turn the build on — Maven `<activation>` reads system/user properties, not your project's. Activate the build with `mvn -Pstoatflow-docker …` / `mvn -Pstoatflow-native …` (the [build commands](https://stoatflow.io/#build-and-run) below). The `<properties>` above only *configure* the build once a profile is active.
::

- **Full Gradle DSL** — every `stoatflow { }` property, task, and default: [Gradle plugin reference](https://stoatflow.io/docs/reference/gradle-plugin-reference).
- **Full Maven surface** — the three artifacts, every `<properties>` key, and the four goals: [Maven reference](https://stoatflow.io/docs/reference/maven-reference).

## Gradle ⇄ Maven: the same conventions, both tools

Everything one tool does has a counterpart on the other. If you've configured one, this is the map to the other:

| Concept                     | Gradle (`io.stoatflow` plugin)                                      | Maven (`stoatflow-parent` + plugin)                              |
| --------------------------- | ------------------------------------------------------------------- | ---------------------------------------------------------------- |
| Apply the conventions       | `plugins { id("io.stoatflow") }`                                    | `<parent>io.stoatflow:stoatflow-parent</parent>`                 |
| Entry point                 | `stoatflow { mainClass.set("…") }`                                  | `<stoatflow.mainClass>…</stoatflow.mainClass>`                   |
| Fat JAR                     | `./gradlew shadowJar` (`-all.jar`)                                  | `mvn package` (`-all.jar`, via shade)                            |
| Run (forked, preview on)    | `./gradlew run`                                                     | `mvn stoatflow:run`                                              |
| Build a Docker image        | `docker { enabled.set(true) }` → `./gradlew jibDockerBuild`         | `<stoatflow.docker…>` → `mvn -Pstoatflow-docker package`         |
| Build a native image        | `nativeImage { enabled.set(true) }` → `./gradlew nativeDockerBuild` | `<stoatflow.nativeImage…>` → `mvn stoatflow:native-docker-build` |
| Aligned dependency versions | resolved transitively from `stoatflow-runtime`                      | import `stoatflow-bom` (the parent does this for you)            |

::callout{color="info" icon="i-lucide-git-compare"}
**One source of truth under the hood.** Both build tools bundle the *same* native-image argument file, the *same* generic `Dockerfile.native`, and the *same* container entrypoint scripts (shared from one internal module). A change to the native flags or the heap model lands on both at once — they can't drift.
::

## Shared build behaviours

These are identical across Gradle and Maven — worth understanding once.

- **Reproducible images.** By default the image `creationTime` (JVM/Jib) and `SOURCE_DATE_EPOCH` (native) are pinned to your **HEAD commit timestamp**, not wall-clock — so identical source produces an identical digest with a meaningful `Created` date. Outside a git repo the timestamp falls back to the Unix epoch. Turn it off with `reproducibleBuild = false` to restore wall-clock.
- **Container-aware heap sizing.** The bundled entrypoint sizes the JVM (or native) heap from the **container's** memory limit (cgroup v1/v2), not the host's — no `-Xmx` guesswork under Kubernetes limits. The full model lives in [Docker → Container-aware heap sizing](https://stoatflow.io/docs/runtime/docker#container-aware-heap-sizing).
- **RocksDB off-heap auto-detection.** If `org.rocksdb` is on the runtime classpath (it is, via `stoatflow-runtime`/`stoatflow-core`), the image sets `STOATFLOW_ROCKSDB_MB=256` so the entrypoint reserves off-heap memory for the state store before sizing the heap. Override the reservation explicitly, or set it to `0` to skip the reservation for an app that only uses in-memory stores. Automatic on both build tools.

## Build and run

::code-tabs{group="build"}
```bash [Gradle]
./gradlew run             # run locally with the required JVM flags
./gradlew shadowJar       # build the fat JAR  → build/libs/<name>-<version>-all.jar
./gradlew test            # unit tests (IntegrationTest tag excluded)
./gradlew integrationTest # integration tests (only the IntegrationTest tag)
./gradlew jibDockerBuild  # build the container image   (docker.enabled)
./gradlew nativeDockerBuild # build the native image      (nativeImage.enabled)
```

```bash [Maven]
mvn stoatflow:run               # run locally (forked JVM, --enable-preview)
mvn package                     # fat JAR  → target/<name>-<version>-all.jar
mvn test                        # unit tests (IntegrationTest tag excluded)
mvn verify                      # + integration tests (failsafe)
mvn -Pstoatflow-docker package  # build the container image (one command)
mvn stoatflow:native-docker-build # build the native image (run mvn package first)
```
::

::callout{color="info" icon="i-lucide-terminal"}
**`mvn stoatflow:…` short prefix.** To call the plugin's standalone goals by their short prefix, add `io.stoatflow` to `<pluginGroups>` in your `~/.m2/settings.xml`. Without it, use full coordinates — `mvn io.stoatflow:stoatflow-maven-plugin:run`. See the [Maven reference](https://stoatflow.io/docs/reference/maven-reference#set-up).
::

## Worked examples

A complete build for each tool, both languages:

::code-tabs{group="build"}
```kotlin [Gradle]
// build.gradle.kts
plugins {
    kotlin("jvm") version "2.4.0"   // drop this line for a Java-only project
    id("io.stoatflow") version "<stoatflow-version>"
}

dependencies {
    implementation("io.stoatflow:stoatflow-runtime:<stoatflow-version>")
    testImplementation("io.stoatflow:stoatflow-test-utils:<stoatflow-version>")
}

stoatflow {
    mainClass.set("com.example.MainKt")   // Java: "com.example.Main"
    docker {
        enabled.set(true)
        imageName.set("acme/word-count")
    }
}
```

```xml [Maven]
<!-- pom.xml — a StoatFlow app on the parent POM -->
<project>
  <modelVersion>4.0.0</modelVersion>
  <parent>
    <groupId>io.stoatflow</groupId>
    <artifactId>stoatflow-parent</artifactId>
    <version>REPLACE-WITH-CURRENT-RELEASE</version>  <!-- literal — Maven can't resolve a property in <parent> -->
    <relativePath/>
  </parent>

  <groupId>com.example</groupId>
  <artifactId>word-count</artifactId>
  <version>1.0.0</version>

  <properties>
    <stoatflow.mainClass>com.example.MainKt</stoatflow.mainClass>   <!-- Java: com.example.Main -->
    <stoatflow.docker.imageName>acme/word-count</stoatflow.docker.imageName>
  </properties>

  <!-- Kotlin only: point Maven at src/main/kotlin and add the stdlib -->
  <build><sourceDirectory>src/main/kotlin</sourceDirectory></build>

  <dependencies>
    <dependency>
      <groupId>io.stoatflow</groupId>
      <artifactId>stoatflow-runtime</artifactId>
    </dependency>
    <dependency>
      <groupId>org.jetbrains.kotlin</groupId>
      <artifactId>kotlin-stdlib</artifactId>
      <version>${kotlin.version}</version>   <!-- omit for a Java app -->
    </dependency>
  </dependencies>
</project>
```
::

::callout{color="info" icon="i-lucide-info"}
Don't want a build plugin at all? Both tools have a "configure it yourself" path — set the JDK 25 toolchain and the two JVM flags by hand. See [Installation → Option B](https://stoatflow.io/docs/getting-started/installation#option-b-configure-it-yourself).
::

## Next steps

- **[Your first app](https://stoatflow.io/docs/getting-started/first-app)** — build and run a complete word-count app on the runtime.
- **[AI assistants](https://stoatflow.io/docs/getting-started/ai-assistants)** — install the skills pack so your AI coding assistant writes correct StoatFlow build files.
- **[Docker](https://stoatflow.io/docs/runtime/docker)*&#x2A; and &#x2A;*[Native image](https://stoatflow.io/docs/runtime/native-image)** — the full container and GraalVM build guides.
- **[Gradle plugin reference](https://stoatflow.io/docs/reference/gradle-plugin-reference)*&#x2A; · &#x2A;*[Maven reference](https://stoatflow.io/docs/reference/maven-reference)** — every knob, task, and default.
- Stuck on the build? [Get in touch](https://stoatflow.io/contact) — real people read every email during the alpha.


# AI assistants

StoatFlow is source-compatible with Kafka Streams 4.3 via an import swap — which is exactly why AI coding assistants get it *confidently wrong*. Every mainstream model is trained on `org.apache.kafka.streams.*`, so it reaches for the wrong import root, the Kafka Streams `at_least_once` default, `replicas: N` scaling, Maven Central coordinates, and Kafka Streams watermark semantics — all of which are wrong on StoatFlow. The **StoatFlow AI Assistant Skills** pack puts the right answers in front of your assistant.

::tldr-panel
- **What:** a public, versioned pack of skills / rules files — six task-cut skills (build, test, port, configure, set up, operate) + a shared primer.
- **Install (Claude Code):** `/plugin marketplace add stoatflow/skills` → `/plugin install stoatflow@stoatflow`.
- **Install (any agent):** `npx skills add stoatflow/skills`, or copy the `AGENTS.md` / editor rule file into your repo.
- **Pin the version:** the pack ships in lockstep with StoatFlow releases — use the tag matching your version.
- **Nothing to install?** Point your agent at [`stoatflow.io/llms.txt`](https://stoatflow.io/llms.txt){rel=""nofollow""} — these docs, machine-readable.
::

## What it does

The pack overrides the Kafka-Streams priors that produce subtly-wrong StoatFlow code. Its six skills fire on what you're doing:

| Skill                      | Fires when you're…                                                                 |
| -------------------------- | ---------------------------------------------------------------------------------- |
| `stoatflow-build-topology` | writing topology code — DSL, Processor API, serdes, state stores, DLQ              |
| `stoatflow-test`           | writing tests — `TopologyTestDriver`, integration tests                            |
| `stoatflow-port-from-ks`   | porting a Kafka Streams / KSML app — code **and** state                            |
| `stoatflow-configure`      | configuring an app — `application.yaml`, guarantees, lanes, HA                     |
| `stoatflow-project-setup`  | wiring the build — the private Maven repo, license, JDK 25, Docker, native image   |
| `stoatflow-operate`        | deploying and running it — single-instance Kubernetes, HA, probes, metrics, tuning |

Because the pack is drift-checked against the porting guide, the [compatibility matrix](https://stoatflow.io/docs/reference/ks-compatibility-matrix), and the config schema on every release, it stays accurate instead of rotting into authoritative stale answers.

## Install

::callout{color="info" icon="i-lucide-shield"}
The pack is free and open (Apache-2.0). It only *documents* StoatFlow — you still need customer credentials to resolve the library itself from `maven.stoatflow.io`.
::

| Tool                        | Install                                                                            |
| --------------------------- | ---------------------------------------------------------------------------------- |
| **Claude Code**             | `/plugin marketplace add stoatflow/skills` → `/plugin install stoatflow@stoatflow` |
| **Any agent (skills CLI)**  | `npx skills add stoatflow/skills`                                                  |
| **Codex / AGENTS.md tools** | copy `AGENTS.md` into your app repo                                                |
| **Cursor**                  | copy `cursor/rules/stoatflow.mdc` → `.cursor/rules/` (or use `AGENTS.md`)          |
| **GitHub Copilot**          | copy `copilot/stoatflow.instructions.md` → `.github/instructions/`                 |
| **JetBrains AI / Junie**    | copy `jetbrains/guidelines.md` → `.junie/guidelines.md`                            |

All of these live in the public repo [`stoatflow/skills`](https://github.com/stoatflow/skills){rel="&#x22;nofollow&#x22;"}.

## Pin the matching version

The pack version **is*&#x2A; the StoatFlow version it targets. Each artifact carries a &#x2A;"Targets StoatFlow :stoatflow-version "* banner. If your StoatFlow version differs from the pack you installed, switch to the pack's matching tag so the divergence rules and config reference line up with your release.

## The other half: `llms.txt`

The pack is the **instruction** side — files your assistant loads *before* it writes code.
[`stoatflow.io/llms.txt`](https://stoatflow.io/llms.txt){rel="&#x22;nofollow&#x22;"} is the **retrieval** side: a machine-readable
index of this documentation, for agents that fetch a URL when they need an answer.

| File                                                                               | What it is                                                                                                                                                         |
| ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| [`/llms.txt`](https://stoatflow.io/llms.txt){rel="&#x22;nofollow&#x22;"}           | Curated index — every docs page with its title and description, grouped by section                                                                                 |
| [`/llms-full.txt`](https://stoatflow.io/llms-full.txt){rel="&#x22;nofollow&#x22;"} | The same documentation concatenated into one markdown bundle, for direct ingestion                                                                                 |
| `/raw/<page>.md`                                                                   | Any indexed page as raw markdown — e.g. [`/raw/docs/concepts/exactly-once.md`](https://stoatflow.io/raw/docs/concepts/exactly-once.md){rel="&#x22;nofollow&#x22;"} |

They follow the [llms.txt convention](https://llmstxt.org/){rel="&#x22;nofollow&#x22;"}, cost nothing to use, and need no install.
All three are regenerated on every deploy, so they never lag the docs.

## Next steps

- **[Install StoatFlow](https://stoatflow.io/docs/getting-started/installation)*&#x2A; and &#x2A;*[set up your project](https://stoatflow.io/docs/getting-started/project-setup)** — the build wiring the `stoatflow-project-setup` skill also teaches.
- **[Migrating from Kafka Streams](https://stoatflow.io/docs/migration)** — the `stoatflow-port-from-ks` skill drives the same [import codemod](https://stoatflow.io/docs/migration/automated-port) and [state migration](https://stoatflow.io/docs/migration/migration-tool) conversationally.
- Something the pack got wrong? [Get in touch](https://stoatflow.io/contact) — the pack is maintained upstream and improves every release.


# Core concepts

These pages explain the model behind the StoatFlow runtime: how one process runs your whole topology, what guarantees it makes, and where the knobs are. They build on each other — read them top to bottom the first time, then come back to any one on its own.

::tldr-panel
- **Start here:** [Architecture](https://stoatflow.io/docs/concepts/architecture) — the single-instance model everything else rests on.
- **Then:** exactly-once, lanes, state, event time — the four guarantees that follow from it.
- **Then:** the configuration model and the error-handling model — how you tune and how it behaves under failure.
- **If you know Kafka Streams:** finish with [how StoatFlow differs](https://stoatflow.io/docs/concepts/how-stoatflow-differs-from-ks).
::

## Recommended reading order

Each page assumes the ones before it. The architecture page is the anchor — it states the conceptual model, and the rest go deeper on one facet each without re-deriving it.

::path-cards
  :::path-card
  ---
  icon: i-lucide-layers
  title: 1. Architecture
  to: https://stoatflow.io/docs/concepts/architecture
  ---
  The single-instance model, the data path from source to sink, and the operational surface. The foundation for every other concept page.
  :::

  :::path-card
  ---
  icon: i-lucide-shield-check
  title: 2. Exactly-once
  to: https://stoatflow.io/docs/concepts/exactly-once
  ---
  The commit barrier: how state, output, and offsets commit atomically in one Kafka transaction — and what at-least-once mode trades away.
  :::

  :::path-card
  ---
  icon: i-lucide-split
  title: 3. Lanes and parallelism
  to: https://stoatflow.io/docs/concepts/lanes-and-parallelism
  ---
  How key affinity routes records to lanes, why lane count is decoupled from partition count, and why blocking I/O is cheap.
  :::

  :::path-card
  ---
  icon: i-lucide-database
  title: 4. State and thread-safety
  to: https://stoatflow.io/docs/concepts/state-and-thread-safety
  ---
  Global state stores, why concurrent access across lanes is safe, the store types available, and changelog-backed durability.
  :::

  :::path-card
  ---
  icon: i-lucide-clock
  title: 5. Event time and watermarks
  to: https://stoatflow.io/docs/concepts/event-time-and-watermarks
  ---
  Event time versus arrival time, how watermarks advance, when windows close, and the policies for late records.
  :::

  :::path-card
  ---
  icon: i-lucide-sliders-horizontal
  title: 6. Configuration model
  to: https://stoatflow.io/docs/concepts/configuration-model
  ---
  How configuration is layered and resolved — what you set in YAML, what the engine derives, and what stays automatic.
  :::

  :::path-card
  ---
  icon: i-lucide-triangle-alert
  title: 7. Error-handling model
  to: https://stoatflow.io/docs/concepts/error-handling-model
  ---
  The failure policies the runtime applies — skip, fail, or dead-letter — and what each one means for your data and your topology.
  :::

  :::path-card
  ---
  icon: i-lucide-arrow-left-right
  title: 8. How StoatFlow differs from Kafka Streams
  to: https://stoatflow.io/docs/concepts/how-stoatflow-differs-from-ks
  ---
  The conceptual deltas if you already know Kafka Streams: single instance, in-memory repartitioning, lanes instead of tasks.
  :::
::

## Where these pages stop

The concept pages describe **behaviour and design** — the model, the guarantees, and what you can observe at runtime. They do not document the internal algorithms, data structures, or protocols that implement them; the source code is the source of truth for those. The [Architecture](https://stoatflow.io/docs/concepts/architecture) page sets that boundary, and the rest of this section holds to it.

## Where to go next

- [Building topologies](https://stoatflow.io/docs/building) — the DSL and the Processor API, once the model is clear.
- [Your first app](https://stoatflow.io/docs/getting-started/first-app) — a complete word-count app on the runtime, if you haven't run one yet.
- [Features](https://stoatflow.io/product/features) and [Motivation](https://stoatflow.io/product/motivation) — the full capability list and the reasoning behind the design.


# Architecture

This page is the conceptual architecture of StoatFlow — the model that runs your topology, the mechanisms behind the exactly-once and per-key-order guarantees, and what you can observe at runtime. It describes **behaviour and design**, not implementation: the source code is the source of truth for the specific algorithms, data structures, and internal protocols.

![StoatFlow architecture — single-replica engine with key-affinity lanes, global state, and transactional producer](https://stoatflow.io/assets/docs/architecture/StoatFlow_high-level_architecture_detailed_20260517.png)

## The single-instance model

The starting point is unusual for stream processing: a StoatFlow application runs as exactly one JVM process. There is no cluster, no scheduler, no worker pool. The process opens a Kafka consumer group with one member — itself — and that member is assigned every partition of every source topic the topology reads from.

Several properties follow directly from this:

- **No rebalancing** — there is no group to rebalance.
- **Global state** — every state store lives in this one process; any processing context can read or write any key.
- **Deterministic behaviour** — no inter-instance race, no clock skew between replicas, no split-brain between active instances.
- **One coordinator** — exactly-once commits are coordinated within the process, not across nodes.

What this model forbids is equally explicit. You cannot run two **active** replicas of the same StoatFlow application pointed at the same source topics. By default there is no second instance to coordinate with, and Kafka's consumer-group semantics would assign all partitions to one and idle the other. High availability comes from fast restart by default; an opt-in [hot-standby](https://stoatflow.io/docs/operating/high-availability) tier adds one or more *passive* standbys for near-instant failover — still a single *active* instance, never two.

The design reasoning — including the trade-offs you accept by giving up open-ended horizontal scaling — is on [Motivation](https://stoatflow.io/product/motivation).

## Data flow

The diagram above shows the path a record takes from source topic to sink topic.

1. **Kafka consumer.** A single consumer reads from every partition of every source topic. Records arrive in batches.
2. **Record dispatch.** The dispatcher inspects each record's key, decides which processing lane handles it (via consistent hashing — see the next section), and places records onto per-lane queues. The dispatcher also injects commit barriers into the lanes when it's time to commit (see *Exactly-once semantics* below).
3. **Processing lanes.** Each lane runs the topology — your `mapValues`, `filter`, `join`, `aggregate`, custom `Processor` code — for the records assigned to it. Stateful operators read and write state stores.
4. **State stores.** Backed by RocksDB or held in memory. Globally accessible from any lane; see the next-but-one section.
5. **Sink collection.** Records emitted by the topology buffer in a sink collector, ready to publish to Kafka.
6. **Transactional producer.** A single Kafka producer writes the buffered output records, updates the consumer-group offsets, and commits — under exactly-once, all three of those happen atomically on a commit barrier.

That is the full data path. There is no broker round-trip between processing steps inside the topology, no cluster shuffle, no external coordinator. Repartitioning — moving a record from one lane to another because a `selectKey` or `groupBy` changed its key — happens in-memory between lanes; there is no internal repartition topic.

## Processing lanes and key affinity

A **lane** is a unit of concurrent processing inside the JVM. Each lane runs the topology independently of all other lanes, but against the same shared state stores.

Records are routed to lanes by **key affinity**. The dispatcher hashes each record's key and consistently picks one lane — same key, same lane, every time. This guarantees that for any given key, the topology processes events in the order Kafka delivered them. Different keys process in parallel.

Two consequences are worth naming:

**Lane count is decoupled from Kafka partition count.** In the standard Kafka Streams model, processing parallelism is bounded by the partition count of the input topics — one task per partition, with each stream thread running one or more tasks. StoatFlow doesn't have that coupling. The consumer reads all partitions, then the dispatcher distributes work across however many lanes you configure. Lane count scales with cores, not partitions.

**Blocking I/O is cheap.** Lanes run on virtual threads — JDK 21's GA primitive. A lane blocked on a REST call, a database query, or an AI-inference response parks at near-zero cost; the JVM keeps making progress on other lanes. This is what makes in-line external enrichment natural — no `CompletableFuture` chains, no reactive frameworks, no callback wiring required to keep throughput up under blocking calls.

Records that the topology re-keys with `selectKey`, `groupBy`, or a key-changing join get re-hashed and routed to a different lane. That is the in-memory equivalent of Kafka Streams' repartition topic, without the broker round-trip or extra serialization — and, as in Kafka Streams, it happens only where a downstream operator actually needs the new key's affinity (an aggregation, a join, a materialised table, or an explicit `repartition()`), not at every key change. See [Lanes and parallelism](https://stoatflow.io/docs/concepts/lanes-and-parallelism#where-the-handoff-actually-happens).

## State stores and durability

State is **global**. Every state store lives in the JVM that's running the topology, and any lane can read or write any key. There's no partition-scoped isolation, no inter-instance lookup protocol — and because there's a single process holding all state, no replication of the same data across multiple JVMs.

**State stores are safe under concurrent access** across lanes. The correctness story falls out of key affinity (see *Processing lanes* above): records with the same key always route to the same lane, so updates to any given key are processed serially by one lane in arrival order. Different keys update in parallel across different lanes — no contention, no global lock. For custom Processors that need to read-modify-write multiple keys atomically (rare, but real for some patterns), the runtime provides a key-lock utility so the cross-key invariant holds without forcing single-threaded execution.

StoatFlow ships several store types — key-value, window, session, versioned (timestamped lookups), and timer — each available in a RocksDB-backed (persistent, on-disk) variant or an in-memory variant. Stateful DSL operators (`count`, `reduce`, `aggregate`, joins, windowed counts, suppress) choose the appropriate store type automatically. Custom Processors can declare their own.

Durability is provided by **Kafka changelog topics**. Every state write produces a changelog entry. Changelog topics are compacted by key, so the latest value for every key is preserved indefinitely without unbounded storage growth. State updates and the corresponding changelog publish are coupled atomically — when a commit barrier completes, you can be confident the changelog has the same data your in-memory state does.

On restart, the runtime rebuilds local state from the changelog. Restoration runs in parallel across stores so a topology with many state stores recovers concurrently rather than serially. For workloads with large state, the changelog read dominates restart time; see [Benchmarks](https://stoatflow.io/product/benchmarks) for measured cold-start numbers on representative workloads.

## Exactly-once semantics — the commit barrier

Exactly-once is the conceptual centrepiece of the runtime, and the mechanism is the **commit barrier**.

A commit barrier is a marker — not a data record, not user content — that the dispatcher periodically injects into the lanes. As records flow through the topology the barrier flows with them, propagating across each sub-topology boundary — an in-memory repartition or a join — in a staged cascade. When it has reached every lane in every sub-topology, the runtime executes a single Kafka transaction that commits, atomically:

- Every state-store write since the previous barrier (via the changelog topics).
- Every sink output record produced since the previous barrier.
- The Kafka consumer-group offsets for every input partition that contributed records.

Either all three commit together, or none do. If the JVM crashes mid-barrier, the in-flight Kafka transaction aborts. The partial work — uncommitted state changes, uncommitted output records, uncommitted offset advances — is discarded. On restart, processing resumes from the previous successful barrier as if the interrupted epoch had never happened. No duplicate outputs. No lost state. No replayed offsets.

Records that cross a sub-topology boundary carry the epoch they belong to, and the receiving side briefly holds back any that run ahead of the barrier until it arrives there too — so the staged cascade never lets the next epoch's data slip into the current commit, and the cut stays exact across the boundary, not just within a single lane.

The barrier protocol is in the **Chandy-Lamport family** of distributed-snapshot algorithms — the lineage, and what StoatFlow's single-process scope changes about it, are on [Exactly-once semantics](https://stoatflow.io/docs/concepts/exactly-once#the-commit-barrier).

**At-least-once mode** bypasses the barrier entirely. The producer commits its output records and the consumer commits its offsets on independent, faster cadences. You accept that on a crash some records may be processed twice and downstream consumers may see duplicates. The trade-off is a lower commit-cadence floor on end-to-end latency — useful when downstream systems are already idempotent or duplicate-tolerant.

The barrier scheduling cadence, the recovery handshake, the bounded-wait protocol for the transaction itself, and the recovery accounting are implementation concerns and stay in the source.

## Event time and watermarks

Stream processing has to handle time. Records arrive out of order. Network buffers can hold a batch for an unpredictable interval. A topic with many producers carries events generated at very different wall-clock times. The runtime needs a consistent model for *when did this happen* that's independent of *when did this arrive*.

That model is **event time**. Every input record carries a timestamp — the Kafka record timestamp by default, or whatever a custom `TimestampExtractor` returns. Stateful operators that care about time (windowed aggregations, session windows, joins with time bounds) reason in event time.

A **watermark** is a claim made by the runtime: "I do not expect any further records earlier than time *T*." Watermarks are tracked **per source partition**; the runtime combines them into a single global watermark for the application — because there's only one application instance, there's no distributed watermark-coordination protocol. The global watermark advances together with the commit barrier, so windowed-result records are committed alongside the watermark progress that produced them — recovery sees a consistent snapshot of "what the app has seen up to."

When the global watermark passes a window's end — plus any **grace period** configured on the window — the runtime knows the window can close, and no further records will be accepted into it. **Late records** — records whose event time is older than the current watermark — are still folded into their window while it remains within its grace period, updating the result; once the window has closed, a later record that would have belonged to it is dropped. Grace is set on the window specification (for example `TimeWindows.ofSizeAndGrace(...)`).

Custom Processors can register **event-time timers** and **processing-time timers** that fire callbacks when the relevant clock advances past a registered moment, independent of incoming records. See [Features](https://stoatflow.io/product/features) for the watermark strategies and timer API.

## Lifecycle: startup, restart, recovery

**Cold start** runs in three steps:

1. The runtime opens a Kafka consumer in the configured group and gets every partition of every source topic assigned. (Before this, startup checks a license key is *present* — offline, no network; the activating license validation runs \~5 minutes later on a background thread, so it never delays the cold start.)
2. State stores restore from their changelog topics, in parallel. For stores without a local snapshot, this reads the whole changelog; for stores with a local snapshot, only the records since the last commit need to be read.
3. Once every store has caught up, the consumer seeks to the last committed input offsets and processing begins.

On every start, the runtime decides **per store** how to bring local state up to date. There are three outcomes, fastest to slowest:

| Local state                                      | What the runtime does                                                       | Typical time           |
| ------------------------------------------------ | --------------------------------------------------------------------------- | ---------------------- |
| Present and consistent with the committed offset | **Reuse as-is**, no replay                                                  | sub-second             |
| Present but behind the changelog                 | **Delta restore** — replay only the records after the last committed offset | seconds                |
| Missing, corrupt, or first-ever start            | **Full restore** — replay the whole changelog from the beginning            | scales with state size |

The runtime knows which case applies because it stores the committed changelog offset **inside RocksDB itself**, in the same atomic write as the data — there is no separate metadata file to fall out of sync. On start it compares that stored offset against the changelog's end offset in Kafka and picks delta, full, or skip. (In-memory stores have no local state to reuse, so they always full-restore.)

```mermaid
flowchart TD
    S{"Local RocksDB present, valid,<br/>with a recorded offset?"}
    S -->|"no — missing, corrupt,<br/>or first-ever start"| F["Full restore —<br/>replay the whole changelog"]
    S -->|yes| O{"Recorded offset vs<br/>changelog end offset"}
    O -->|consistent| K["Reuse as-is —<br/>no replay"]
    O -->|behind| D["Delta restore —<br/>replay only the records<br/>after the committed offset"]
```

This per-store decision is also what makes a [state-carrying migration](https://stoatflow.io/docs/migration/with-data-migration) possible without any engine support: a first start over pre-seeded changelog topics is simply the "first-ever start" branch — a forced full restore.

**Clean shutdown** is the reverse:

1. The runtime stops accepting new records into the dispatcher.
2. Records already in flight drain through the topology.
3. The dispatcher injects one final commit barrier.
4. When that barrier completes, the runtime commits, closes the consumer and producer, and exits.

**After a crash**, the flow is similar to cold start with one detail:

1. The JVM exits non-cleanly; in-flight work was uncommitted, by design.
2. On restart, the runtime opens the consumer at the last committed offsets — every record after that offset will be re-read.
3. Local state stores may have partial in-memory data still on disk from before the crash; the runtime uses what's there as a head-start and the changelog fills the gap to the last committed barrier.
4. Processing resumes. Under exactly-once, the previous epoch's partial work was aborted at the broker; downstream consumers reading with `read_committed` isolation see no duplicates.

Restart times scale with state size — see [Benchmarks](https://stoatflow.io/product/benchmarks) for measured cold-start numbers on representative workloads.

## Failure modes and observability

Production architecture is partly about what happens when things go right, and partly about what you observe when they don't. The runtime handles common failures with explicit, configurable policies; the admin endpoints expose the state you need to diagnose and respond.

### Common failure modes

**A processor throws an exception.** The configured processing-exception handler decides: log and continue (skip the record), log and fail (stop the topology), or send to a dead-letter queue with the original record and error context. Silent skipping should be a deliberate choice, not an unexamined default. &#x2A;What you see:* `processor-error` metrics, an error-level log entry with stack trace and record metadata, the offending record in the configured DLQ topic.

**A record fails deserialization on input.** Same machinery as a processor exception, with its own configurable handler. Useful for source topics that may contain malformed records — keep processing, route the broken records to a DLQ for offline inspection. &#x2A;What you see:* `deserialization-error` metrics, DLQ records carrying the original key/value bytes and the offending exception.

**A commit transaction times out or fails.** The runtime aborts the in-flight Kafka transaction, treats it as a fatal commit failure, and exits — Kubernetes restarts the process. On restart, the previous epoch's partial work was aborted at the broker (per Kafka's transaction semantics); the new process resumes from the last successful barrier with no duplicates downstream. &#x2A;What you see:* `commit-stall` metrics and the `/debug/barriers` endpoint show the stuck barrier before exit; the restarting instance enters the restoration phase visible via `/state` and `/health/ready`.

**The Kafka broker is unavailable.** Producer and consumer retry per the Kafka client's exponential-backoff defaults. Short outages cause throughput to dip and recover. Sustained outages eventually exceed configured retry budgets and trigger a fatal failure, on the same exit-and-restart pattern as a commit failure. &#x2A;What you see:* Kafka-client error metrics, growing consumer-lag metric, `/health/ready` flipping to 503 once the runtime can no longer make progress.

**State restoration is slow on cold start.** Restoration proceeds store-by-store from the changelog topics; the time scales with state size. The `/health/ready` probe returns 503 until restoration completes — Kubernetes won't route traffic, and load balancers won't think the instance is ready. &#x2A;What you see:* `/health/ready` returns 503 with the `stoatflow` health component reporting the lifecycle state (`VALIDATING_STATE` / `RESTORING`); per-store restoration progress is exposed on `/metrics` via the `stoatflow.restoration.*` meters (stores completed vs total, records and bytes restored per `store_name`); `/state` reports the lifecycle state and its transition history.

### Always-on observability

For steady-state diagnostics and capacity planning, the admin endpoints expose:

- **Health probes** — `/health/live` and `/health/ready` for Kubernetes or any HTTP-probe-aware orchestrator.
- **Prometheus metrics** — `/metrics` exposes JVM, Kafka-client, and StoatFlow internal counters in the standard Prometheus format. Scrape from your existing monitoring stack; no agents to install.
- **Topology introspection** — `/topology` renders the processor DAG; `/state` reports lifecycle state and transition history; `/watermarks` shows per-partition watermark state.
- **Debug endpoints** — `/debug/threads` and `/debug/barriers` give live views of every lane's thread state and the commit-barrier coordinator. Useful when a topology behaves unexpectedly and you need to see exactly what every lane is doing right now.
- **Plugin and lifecycle hooks** — register custom indicators, listeners, or shutdown hooks programmatically. User code can run on `PRE_START`, `POST_START`, `PRE_STOP`, and `POST_STOP`.
- **Logs** — emitted through SLF4J/Logback; per-logger levels are adjustable from the runtime's `logging:` configuration group without touching `logback.xml`, and JSON output plugs in through standard Logback encoder configuration.

## Where to go next

- [Features](https://stoatflow.io/product/features) — every distinguishing capability, by area
- [Motivation](https://stoatflow.io/product/motivation) — why StoatFlow is built this way
- [Comparison matrix](https://stoatflow.io/product/comparison-matrix) — feature-by-feature against Kafka Streams and self-hosted / managed Flink
- [Benchmarks](https://stoatflow.io/product/benchmarks) — measured throughput, latency, resource use, cold-start times
- [Migration](https://stoatflow.io/docs/migration) — porting an existing Kafka Streams topology


# Exactly-once semantics

Exactly-once is StoatFlow's default processing guarantee. This page explains the mechanism that delivers it — the **commit barrier** — what "exactly-once" means here in concrete terms, how recovery works after a crash, and when to switch to the cheaper at-least-once mode instead. It describes **behaviour**, not implementation; the [Architecture](https://stoatflow.io/docs/concepts/architecture) page is the conceptual ceiling, and the source is the source of truth for the protocol details.

::tldr-panel
- **Mechanism:** a commit barrier flows through the topology; when it completes, **one Kafka transaction** atomically commits state, output records, and input offsets — all or nothing.
- **Guarantee:** under `read_committed`, downstream consumers see each input's effect exactly once, even across crashes.
- **Default:** exactly-once is on by default. No multi-property config dance — it's `EXACTLY_ONCE` out of the box.
- **Trade-off:** at-least-once mode drops the transaction for lower commit latency, accepting possible duplicates on recovery.
::

## What "exactly-once" means here

Exactly-once is an end-to-end statement about the effect of each input record, not a claim that any record is physically processed only once. After a crash, StoatFlow re-reads and re-processes records — that's how recovery works. The guarantee is that the **observable result** is as if each input contributed exactly once:

- Every output record a downstream consumer reads (with `read_committed` isolation) reflects each input's effect once — no duplicates, no gaps.
- The state stores backing your aggregations, joins, and windows end up consistent with the committed output and the committed input offsets.
- A crash mid-processing leaves no partial trace: no half-applied state, no orphaned output, no advanced offsets pointing past work that was discarded.

This is the same guarantee Kafka Streams calls `exactly_once_v2` and Flink calls exactly-once checkpointing. What differs is the scope and the operational model — see [Comparison matrix](https://stoatflow.io/product/comparison-matrix) for the feature-by-feature contrast against Kafka Streams and self-hosted / managed Flink.

## The commit barrier

The whole guarantee rests on one mechanism: the **commit barrier**.

A commit barrier is a marker — not a data record, not user content — that the runtime periodically injects into the processing flow. As your records move through the topology (`mapValues`, `filter`, `join`, `aggregate`, custom `Processor` code), the barrier travels with them. When the barrier has reached the end of every processing path — cascading across each sub-topology boundary (an in-memory repartition or a join) in turn — the runtime executes **one Kafka transaction** that commits, atomically:

- **State** — every state-store write since the previous barrier, published to the changelog topics that back those stores.
- **Output** — every sink record the topology produced since the previous barrier.
- **Offsets** — the Kafka consumer-group offsets for every input partition that contributed records during that span.

Either all three commit together, or none of them do. There is no window in which output has been published but offsets haven't, or offsets have advanced but state hasn't been persisted. That atomicity is what makes the three facts above hold simultaneously.

The span between two successful barriers is one **epoch** — the unit of atomic commit. An epoch is bounded by time and by record count — and closes early under memory or state-cache pressure — so it commits on a regular cadence even when input is bursty, processing is slow, or uncommitted state grows large; the exact cadence and how it adapts are implementation concerns and stay in the source.

Because the barrier crosses those in-memory boundaries in stages rather than all at once, the runtime keeps the cut exact with **receiver-side alignment**: every record handed from one sub-topology to the next carries the epoch it belongs to, and the receiving side briefly holds back any record that has run ahead of the barrier until the barrier arrives there too. A commit therefore captures exactly the input prefix each stage had processed — never a partial slice of the following epoch leaking into the transaction.

The whole mechanism on one wall clock — including what the next section's crash undoes:

![A commit barrier cascading downstream through three sub-topologies against a wall-clock axis. The runtime injects barrier N once; it reaches each sub-topology at a different moment, and only when it has reached the end of every path does the transaction commit. One record is handed from sub-topology 1 to sub-topology 2 already tagged epoch N plus 1, and is held there until sub-topology 2 crosses the barrier too, so no part of the next epoch leaks into the commit. The span between two commits is one epoch. Transaction N commits state, output and offsets together — all three or none of them. A crash partway through epoch N plus 1 aborts transaction N plus 1, discarding its state, output and offsets together; that output was never visible to a read\_committed consumer. On restart, state rebuilds from the changelog to the last committed barrier and the consumer resumes from transaction N's offsets, re-processing everything after it into a fresh epoch.](https://stoatflow.io/assets/docs/concepts/commit-barrier-epochs_20260727.svg)

::callout{color="info" icon="i-lucide-git-merge"}
The barrier protocol is in the **Chandy-Lamport family** of distributed-snapshot algorithms — the same lineage Flink's checkpoint barriers descend from. What's different in StoatFlow is the scope: one process, one barrier, one transaction covers the entire topology. There are no per-task transactions to coordinate and no external checkpoint store to configure. The conceptual model is on [Architecture](https://stoatflow.io/docs/concepts/architecture#exactly-once-semantics-the-commit-barrier).
::

### Why the changelog is part of the transaction

State durability in StoatFlow comes from **Kafka changelog topics**: every state write produces a changelog entry, and the changelog is what gets restored on restart. Because the changelog publish is committed inside the same transaction as the output and the offsets, your durable state can never disagree with what you emitted or where you resumed from. When a barrier completes, the changelog holds exactly the same data your in-memory state does. See [State and thread safety](https://stoatflow.io/docs/concepts/state-and-thread-safety) for how the stores themselves work.

## Crash recovery

The barrier is also the recovery anchor. Picture a crash partway through an epoch — output has been produced and state mutated in memory, but the barrier hasn't completed yet.

1. **The in-flight transaction aborts.** When the JVM exits non-cleanly, the open Kafka transaction is never committed; Kafka's broker-side transaction semantics abort it. The uncommitted output records, the uncommitted changelog entries, and the un-advanced offsets are all discarded together.
2. **The process restarts — or a standby takes over.** The last committed barrier is the recovery anchor whether the instance restarts cold (the default) or an opt-in [hot standby](https://stoatflow.io/docs/operating/high-availability) is promoted in its place. The same transactional fencing that makes a commit atomic is what makes a standby handoff split-brain-proof: at most one instance can ever commit. See [Architecture](https://stoatflow.io/docs/concepts/architecture#the-single-instance-model).
3. **State restores to the last barrier.** Local stores rebuild from their changelog topics up to the last committed barrier — never past it, because the changelog was committed transactionally with everything else.
4. **The consumer resumes from the last committed offsets.** Every input record after the last successful barrier is re-read and re-processed into a fresh epoch.

The records from the interrupted epoch are re-processed, but the aborted transaction means none of their earlier effects survived. Downstream consumers reading with `read_committed` isolation never saw the aborted output, so the re-processed epoch produces the result exactly once. No duplicate outputs, no lost state, no replayed offsets.

::callout{color="warning" icon="i-lucide-eye"}
The guarantee reaches downstream consumers only when they read with **`read_committed`** isolation. A consumer reading `read_uncommitted` (the Kafka default for raw consumers) can observe records from a transaction that later aborts — that's a property of how that consumer is configured, not of StoatFlow. Configure downstream consumers (including other StoatFlow apps) for `read_committed` to get the end-to-end guarantee.
::

For a plain Kafka consumer, that's one property:

```properties
# Downstream consumer — never observe records from aborted transactions
isolation.level=read_committed
```

A downstream StoatFlow app sets the same property through its consumer passthrough:

```yaml
stoatflow:
  kafka:
    consumer:
      isolation.level: read_committed
```

There's one more behaviour worth naming, because it shows up at the operational layer rather than in your topology code: &#x2A;*a stalled commit is a fatal event, not a silent hang.** If a commit transaction times out or fails, the runtime aborts the in-flight transaction and exits; the orchestrator restarts the process, and recovery proceeds exactly as above — resume from the last successful barrier, no duplicates downstream. You observe this through `commit-stall` metrics and the `/debug/barriers` endpoint before exit, and the restoration phase via `/state` and `/health/ready` on the way back up. The bounded-wait protocol behind this is an implementation concern and stays in the source.

## At-least-once mode and its trade-off

Exactly-once carries a cost: every commit is a Kafka transaction, and transactions add per-commit overhead (\~8–30ms in practice). For pipelines where downstream systems are already idempotent or tolerate duplicates, **at-least-once** mode removes that overhead.

In at-least-once mode the runtime uses a non-transactional producer and commits consumer offsets directly, on a faster cadence (\~3–5ms per commit). You give up the atomic all-or-nothing boundary. On a crash:

- Records processed but not yet offset-committed are replayed — downstream consumers may see **duplicate output**.
- State stores may be ahead of the committed offsets, so windowed aggregations may **over-count** on recovery. (Plain key-value updates are typically idempotent and converge; aggregations that add per record are the case to watch.)

In exchange you get a lower commit-cadence floor on end-to-end latency and a simpler operational model. It's the right choice when the cost of an occasional duplicate is lower than the cost of the transaction overhead — and the wrong choice when downstream effects aren't idempotent.

The guarantee is selected by a single property, `processing-guarantee`, with two values: `EXACTLY_ONCE` (the default) and `AT_LEAST_ONCE`. It is process-wide — it applies to the whole topology, not per-operator — and switching it does not change your topology code; the same build runs under either guarantee. On the runtime it's one line of `application.yaml`; with `:core` directly it's one builder call:

```yaml
# application.yaml — only needed to switch away from the EXACTLY_ONCE default
stoatflow:
  processing-guarantee: AT_LEAST_ONCE
```

::code-tabs{group="lang"}
```kotlin [Kotlin]
val config = StreamsConfig.builder("order-processor", "localhost:9092")
    .processingGuarantee(ProcessingGuarantee.AT_LEAST_ONCE)
    .build()
```

```java [Java]
var config = StreamsConfig.builder("order-processor", "localhost:9092")
    .processingGuarantee(ProcessingGuarantee.AT_LEAST_ONCE)
    .build();
```
::

## How this compares to Kafka Streams

If you're coming from Kafka Streams, two differences matter most:

- **It's the default, with no property dance.** Kafka Streams ships `at_least_once` as the default and requires `processing.guarantee=exactly_once_v2` plus a transactional broker setup to opt in. StoatFlow is exactly-once out of the box — a single instance coordinating a single transaction, with nothing extra to wire up.
- **One transaction, not per-task.** Kafka Streams runs a transaction per stream task and coordinates them across instances. StoatFlow's single-instance model means one barrier and one transaction cover the entire topology — there are no cross-instance two-phase commits to reason about.

The DSL semantics around exactly-once match Kafka Streams; see [How StoatFlow differs from Kafka Streams](https://stoatflow.io/docs/concepts/how-stoatflow-differs-from-ks) for the broader behavioural map and [Comparison matrix](https://stoatflow.io/product/comparison-matrix) for the side-by-side EOS comparison against Kafka Streams and Flink.

## Where to go next

- [Architecture](https://stoatflow.io/docs/concepts/architecture) — the single-instance model and the commit barrier in its full context
- [State and thread safety](https://stoatflow.io/docs/concepts/state-and-thread-safety) — how global state stays consistent under the commit barrier
- [Event time and watermarks](https://stoatflow.io/docs/concepts/event-time-and-watermarks) — why watermark progress commits alongside the barrier
- [Error handling model](https://stoatflow.io/docs/concepts/error-handling-model) — what the runtime does when a processor or commit fails
- [Comparison matrix](https://stoatflow.io/product/comparison-matrix) — EOS feature-by-feature against Kafka Streams and Flink


# Lanes and parallelism

A **lane** is StoatFlow's unit of concurrent processing inside the single JVM. Records are routed to lanes by their key: the same key always lands on the same lane, so the topology sees that key's events in arrival order; different keys run on different lanes in parallel. This page explains what that buys you and how to think about lane count. For the runtime mechanism behind it, see [Architecture](https://stoatflow.io/docs/concepts/architecture).

::tldr-panel
- **Same key → same lane → in-order.** Per-key ordering is preserved without a global lock.
- **Different keys → different lanes → parallel.** Concurrency comes from key spread, not partition count.
- **Lane count is decoupled from Kafka partitions.** It scales with CPU cores, configured at startup.
- **Lanes run on virtual threads**, so a lane blocked on a REST call, DB query, or AI inference parks at near-zero cost — in-line enrichment is natural.
- **Re-keying happens in-memory** between lanes — no repartition topic, no broker round-trip.
::

## Key affinity: same key, same lane

When a record arrives, the engine inspects its key and consistently routes it to one lane — the same key maps to the same lane every time. Because one lane processes one key's events sequentially, in the order Kafka delivered them, **per-key ordering is guaranteed** even though many lanes run at once.

This is also what makes state safe under concurrency. A stateful operator — `count`, `reduce`, `aggregate`, a join, or a custom `Processor` reading and writing a store — only ever touches a given key from the single lane that owns it. Updates to one key are serialized on that lane; updates to different keys proceed in parallel on other lanes. No global lock, no contention on the common path. The reasoning is spelled out under [State stores and durability](https://stoatflow.io/docs/concepts/architecture#state-stores-and-durability) on the architecture page; the [State and thread safety](https://stoatflow.io/docs/concepts/state-and-thread-safety) concept page goes deeper on the rare cross-key case.

The flip side: parallelism is a function of how your keys spread across lanes. A stream where every record carries the same key runs effectively single-threaded — every record routes to one lane, by design, to preserve that key's order. Throughput on a stateful topology comes from having many distinct, evenly distributed keys, not from adding lanes alone.

## Different keys run in parallel

Distinct keys are spread across lanes, so independent keys process concurrently. In a word-count topology, `the`, `quick`, and `fox` each own their running total on whichever lane their key maps to, and all three advance at the same time. The aggregation you write stays sequential *per key* — exactly what correctness needs — while the engine extracts parallelism *across keys* for free.

You do not wire any of this up. The DSL operators and your custom processors are written as if single-threaded per key; the engine provides the cross-key concurrency underneath. The grouping key you choose (`groupBy`, the join key, the record key on a source) is therefore also your unit of parallelism — pick a key with enough distinct values to keep lanes busy.

## Lane count is decoupled from partition count

In standard Kafka Streams, processing parallelism is capped by the partition count of the input topics: one task per partition, with each stream thread running one or more tasks. To process more in parallel you add partitions — a topic-level change with downstream consequences.

StoatFlow breaks that coupling. The single consumer reads **every** partition of every source topic, and the engine then distributes work across however many lanes you configure. Lane count is independent of partition count — it scales with the cores on the machine, set once at startup.

![A source topic with three partitions feeding one consumer, which passes every record to the engine; the engine hashes each record's key and picks one of six key-affinity lanes, so three partitions in become six lanes out and the two counts are independent. Both ord-17 records hash to lane 2 and are processed there in order even though ord-08 arrived between them in the same partition; ord-08, ord-31, ord-23 and ord-55 each land on a different lane, and no order key lands on lane 3. A groupBy on customerId then re-keys every record: both ord-17 records become cust-9, which owns lane 3, so they leave lane 2 while cust-8 arrives there from lane 5 — orders from two lanes converge onto one customer's lane. The move happens in memory between lanes, with no repartition topic, no broker hop and no extra serialization, which is what Kafka Streams would write a topic for.](https://stoatflow.io/assets/docs/concepts/lanes-routing-rekey_20260727.svg)

Configure it with `numLanes` in `StreamsConfig`, or the `stoatflow.lanes.count` key in `application.yaml`. The default is `max(2, available CPU cores)`.

::code-tabs{group="lang"}
```kotlin [Kotlin]
val config =
    StreamsConfig(
        applicationId = "map-filter-example",
        bootstrapServers = "localhost:9092",
        defaultKeySerde = Serdes.String(),
        defaultValueSerde = Serdes.String(),
        numLanes = 6,
    )
```

```java [Java]
var config =
    StreamsConfig.builder("map-filter-example", "localhost:9092")
        .defaultKeySerde(Serdes.String())
        .defaultValueSerde(Serdes.String())
        .numLanes(6)
        .build();
```
::

On the `:runtime` module the same setting lives in `application.yaml`:

```yaml
stoatflow:
  application-id: map-filter-example
  bootstrap-servers: localhost:9092
  lanes:
    count: 6
```

### How many lanes?

Lane count is a throughput-vs-overhead trade-off, not a correctness knob — ordering and exactly-once hold at any value. Some guidance at the architecture level:

- **Start at the default** (`max(2, CPU cores)`) and tune from measured throughput, not guesswork.
- **More lanes than cores** can help when lanes spend time *blocked* — on external calls, disk, or downstream systems — because parked lanes free their carrier thread for others. It rarely helps a purely CPU-bound topology, where you can't run more concurrent work than you have cores.
- **More lanes cost more.** Each lane carries its own queue and dispatch bookkeeping; past a point you pay overhead without gaining throughput.
- **Your keyspace caps the benefit.** Lanes beyond the number of distinct, well-distributed keys sit idle — they can't manufacture parallelism the data doesn't contain.

Pin numbers come from [Benchmarks](https://stoatflow.io/product/benchmarks); the precise scheduling and dispatch behaviour is an implementation concern and stays in the source.

## Virtual threads make blocking I/O cheap

Lanes run on **virtual threads** — JDK 21's GA primitive. A lane that blocks on a REST call, a database query, or an AI-inference response parks at near-zero cost while the JVM keeps making progress on every other lane. That changes how you write enrichment: a synchronous, blocking call inside a `Processor` is the idiomatic approach — no `CompletableFuture` chains, no reactive frameworks, no callback wiring just to keep throughput up while a call is in flight.

This processor enriches each flight by looking values up from a state store and forwarding the result — straight-line, blocking-style code. A remote lookup against an external service would read the same way; the lane simply parks until it returns.

(`STATE_STORE_AIRPORT_INFO` here is a **global** store, materialized by `builder.globalTable(...)`, so the processor reads it without declaring it and gets it read-only. An ordinary store must be connected to the node — see [Processor API](https://stoatflow.io/docs/building/processor-api#attaching-state-stores).)

::code-tabs{group="lang"}
```kotlin [Kotlin]
class AirportEnrichmentProcessor :
    ContextualProcessor<String, Flight, String, FlightEnriched>() {

    private lateinit var airportInfoStore: ReadOnlyKeyValueStore<String, AirportInfoI18n>

    override fun init(context: ProcessorContext<String, FlightEnriched>) {
        super.init(context)
        airportInfoStore = context.getStateStore(STATE_STORE_AIRPORT_INFO)
    }

    override fun process(record: Record<String, Flight>) {
        val flight = record.value ?: return
        // Blocking-style lookups; a remote enrichment call would read the same way.
        val depInfo = flight.departureAirport?.let { airportInfoStore.get(it) }
        val arrInfo = flight.arrivalAirport?.let { airportInfoStore.get(it) }
        context().forward(record.withValue(toFlightEnriched(flight, depInfo, arrInfo)))
    }
}
```

```java [Java]
public class AirportEnrichmentProcessor
        extends ContextualProcessor<String, Flight, String, FlightEnriched> {

    private ReadOnlyKeyValueStore<String, AirportInfoI18n> airportInfoStore;

    @Override
    public void init(ProcessorContext<String, FlightEnriched> context) {
        super.init(context);
        this.airportInfoStore = context.getStateStore(STATE_STORE_AIRPORT_INFO);
    }

    @Override
    public void process(Record<String, Flight> record) {
        final Flight flight = record.value();
        if (flight == null) {
            return;
        }
        // Blocking-style lookups; a remote enrichment call would read the same way.
        final String dep = flight.getDepartureAirport();
        final String arr = flight.getArrivalAirport();
        final AirportInfoI18n depInfo = dep != null ? airportInfoStore.get(dep) : null;
        final AirportInfoI18n arrInfo = arr != null ? airportInfoStore.get(arr) : null;
        context().forward(record.withValue(toFlightEnriched(flight, depInfo, arrInfo)));
    }
}
```
::

::callout{color="warning" icon="i-lucide-triangle-alert"}
Virtual threads scale blocking I/O; they don't make CPU-bound work faster. A lane crunching numbers occupies a real carrier thread for the whole computation — concurrency there is still bounded by cores. Lanes win when work *waits*.
::

See [State and thread safety](https://stoatflow.io/docs/concepts/state-and-thread-safety) for the rules on what a lane may touch, and [Processor API](https://stoatflow.io/docs/building/processor-api) for the full processor surface.

## In-memory re-keying instead of repartition topics

When the topology changes a record's key — `selectKey`, `groupBy`, or a key-changing join — that record may now belong to a different lane. StoatFlow re-hashes the new key and routes it to the lane that owns it **in-memory**, between lanes. There is no internal repartition topic, no extra serialization round-trip, and no broker hop.

In Kafka Streams the same key change forces a write to a repartition topic and a re-read on the other side. StoatFlow's single-instance model removes that round-trip entirely — re-keying is an in-process handoff. The output is identical; the path is shorter.

The e-commerce daily-summary topology re-keys twice — once on `groupBy` to aggregate per customer, then again coming out of the windowed table — and both are in-memory handoffs:

::code-tabs{group="lang"}
```kotlin [Kotlin]
// Re-key the merged stream by customerId for aggregation.
val windowedStream =
    mergedStreams
        .groupBy(
            { _, v -> v.purchase?.customerId ?: v.webActivity.customerId },
            Grouped.with("grouped-by-customer-id", stringSerde, webActivityOrPurchaseSerde),
        )
        .windowedBy(dailyWindows)

// ...aggregate, then re-key back to customerId on the way out.
val dailyAggregatesStream =
    dailyAggregation
        .mapValues({ wk, v -> /* set date from window start */ }, Named.`as`("daily-aggregates-mapvalues"))
        .toStream({ wk, _ -> wk.key }, Named.`as`("daily-aggregates-to-stream"))
```

```java [Java]
// Re-key the merged stream by customerId for aggregation.
TimeWindowedKStream<String, WebActivityOrPurchase> windowedStream = mergedStreams
    .groupBy(
        (k, v) -> v.purchase() != null ? v.purchase().customerId() : v.webActivity().customerId(),
        Grouped.with("grouped-by-customer-id", stringSerde, webActivityOrPurchaseSerde))
    .windowedBy(dailyWindows);

// ...aggregate, then re-key back to customerId on the way out.
KStream<String, DailyAggregates> dailyAggregatesStream = dailyAggregation
    .mapValues((wk, v) -> /* set date from window start */, Named.as("daily-aggregates-mapvalues"))
    .toStream((wk, v) -> wk.getKey(), Named.as("daily-aggregates-to-stream"));
```
::

After a re-key, the new key gets its own owning lane — the affinity property travels with the record across the handoff, so one key is still never processed by two lanes at once. That is what keeps state safe: aggregations, joins and suppression downstream of a `groupBy` still see a single writer per key.

Ordering is the narrower claim. Records that shared the **old** key stay on one lane throughout and keep their relative order. Records that a re-key newly brings together under one key arrived on *different* lanes, and those lanes run concurrently — so their relative order at the new key is not guaranteed, exactly as in Kafka Streams, where the same records would have been processed by different tasks. If you need a total order per key downstream of a merging re-key, you need it at the source: make the grouping key a function of the source key.

Exactly-once alignment does travel across the handoff: where the topology crosses a sub-topology boundary, a record that runs ahead of the commit barrier is briefly held on the receiving lane until the barrier reaches it there — so the committed cut stays exact across the boundary, not only within a single lane (see [Exactly-once](https://stoatflow.io/docs/concepts/exactly-once)). This is also why your grouping key is your parallelism unit: the lane spread of a `groupBy` aggregation follows the distribution of the *grouped* key, not the source key.

### Where the handoff actually happens

A re-key does not, by itself, trigger the handoff. StoatFlow inserts one where a downstream operator genuinely needs it — the same rule Kafka Streams uses to decide whether to materialise a repartition topic:

- **grouped aggregations** (`groupBy(...).count/reduce/aggregate`, windowed and session included),
- **joins** (stream–stream, stream–table, table–table, foreign-key),
- **`toTable()`** and any operator that materialises a per-key store,
- **an explicit `repartition()`**, which always forces one.

**`process()` / `processValues()` are deliberately not on that list** — Kafka Streams never repartitions before a Processor API node either, even when it has connected stores. StoatFlow did until 1.0.0; `stoatflow.topology.processor-api-key-affinity: presumed` restores it. The trade-off after a **many-to-one** re-key is real and is spelled out in [Key affinity after a re-key](https://stoatflow.io/docs/building/processor-api#key-affinity-after-a-re-key) — including the shape StoatFlow refuses to compile.

Everything else runs where it is: `selectKey → mapValues → to()` re-keys the record and writes it out on the *source*-key lane, with no handoff at all, because nothing downstream cares which lane it is on. That is one fewer set of lanes, queues and barrier stages than a boundary would cost.

::callout{color="warning" icon="i-lucide-triangle-alert"}
**If you relied on a re-key to spread work.** The handoff re-distributes records across lanes as a side effect, and deferring it means that no longer happens automatically. A pipeline shaped like `source(few or skewed keys) → selectKey(high cardinality) → expensive-op` now runs the expensive operator on the *source*-key lanes — possibly a handful, or one for a constant key.
::

The fix is the same idiom you would use in Kafka Streams: insert an explicit &#x2A;*`repartition()`** where you want the re-spread. `stream.selectKey { ... }.repartition().mapValues { expensive(it) }` forces the handoff and fans the work back out.

Genuinely `null` keys are unaffected — they are round-robined across lanes rather than hashed. An *empty but non-null* key is not: it hashes deterministically to one lane.

::callout{color="info" icon="i-lucide-info"}
**Lane *counts* move too, not just the distribution.** `numberOfLanes` set on a `Consumed` / `Grouped` / `Repartitioned` applies to the sub-topology that declaration feeds. A node whose handoff is deferred stays in the sub-topology above it, so it inherits *that* one's lane count rather than starting its own at the global default. Worth re-checking after upgrading if you tuned lanes per sub-topology.
::

To restore the pre-1.0.0 behaviour globally (a handoff at every key change), set `stoatflow.topology.sub-topology-split: eager`. To restore it only at Processor API nodes, set `stoatflow.topology.processor-api-key-affinity: presumed`.

## Where to go next

- [Architecture](https://stoatflow.io/docs/concepts/architecture) — the single-instance engine, lane dispatch, and commit barriers in one place
- [State and thread safety](https://stoatflow.io/docs/concepts/state-and-thread-safety) — what a lane may touch, and the cross-key exception
- [Exactly-once](https://stoatflow.io/docs/concepts/exactly-once) — how per-lane processing commits atomically
- [Processor API](https://stoatflow.io/docs/building/processor-api) — writing custom processors that run on lanes
- [Benchmarks](https://stoatflow.io/product/benchmarks) — measured throughput across lane counts and workloads
- [How StoatFlow differs from KS](https://stoatflow.io/docs/concepts/how-stoatflow-differs-from-ks) — the partition-coupling and repartition-topic contrasts in full


# State and thread-safety

State in StoatFlow is **global**: every state store lives in the single JVM running your topology, and any processing lane can read or write any key. This page explains why that's safe under concurrent access, the one case where you have to coordinate it yourself, and the store types you can choose from.

::tldr-panel
- **One process, all state.** No partition-scoped isolation, no inter-instance lookup — any lane can touch any key.
- **Safe by construction.** Key affinity routes every record for a given key to the same lane, so per-key updates are serial. Different keys run in parallel with no global lock.
- **One sharp edge.** A custom `Processor` that read-modify-writes *multiple keys* atomically needs the key-lock utility (`KeyLockManager`) — the only case where ordering doesn't do the work for you.
- **Five store types**, each RocksDB-backed (persistent) or in-memory: key-value, window, session, versioned, and timer.
::

This is the canonical reference for the thread-safety model. For the engine-level picture of lanes, the commit barrier, and durability, see [Architecture](https://stoatflow.io/docs/concepts/architecture). For the how-to — declaring stores, choosing types, and accessing them from the DSL or a custom `Processor` — see [State stores](https://stoatflow.io/docs/building/state-stores).

## The global state model

In the standard Kafka Streams model, state is partitioned: a task owns a slice of the keyspace, and its state stores hold only the keys for the partitions that task is assigned. Reading a key that lives on another task means an interactive-query round-trip to another instance.

StoatFlow has no partitions to scope state to. Because the application runs as [exactly one instance](https://stoatflow.io/docs/concepts/architecture#the-single-instance-model), every state store holds the entire keyspace, in one process. Three things follow:

- **Any lane can read or write any key.** There is no "this key belongs to that task" boundary — a lane processing one record can look up a value written by a record on a completely different key.
- **No replication of the same data across JVMs.** A single process holds all state; there's no copy of the same store living on another node to keep consistent.
- **No inter-instance lookup protocol.** A value is either in the local store or it doesn't exist yet — there's never a network hop to find it.

Durability is unchanged from the Kafka Streams approach: every write produces a Kafka changelog entry, and on restart the runtime rebuilds local state from the changelog. That side of the story — changelog coupling to the commit barrier and parallel restoration — is on [Architecture](https://stoatflow.io/docs/concepts/architecture#state-stores-and-durability).

## Why concurrent access is safe

Global state plus many lanes running in parallel sounds like it should require locking around every store access. It doesn't, and the reason is **key affinity** (see [Lanes and parallelism](https://stoatflow.io/docs/concepts/lanes-and-parallelism)).

Records are routed to lanes by their key. The same key always lands on the same lane, every time. So for any single key:

- All updates to that key are processed **serially**, by one lane, in the order Kafka delivered them.
- There is never a second lane writing the same key concurrently.

And across keys:

- Different keys live on different lanes and update **in parallel**.
- Because they're different keys, those updates touch different store entries — no contention.

The net effect is per-key serialization *without* a global store lock. You get the ordering guarantee you'd expect from single-threaded processing on each key, and the parallelism of many lanes across the keyspace. Reading and writing a state store from inside an operator — `count`, `aggregate`, a join, or your own `Processor` — needs no synchronization from you for the common single-key case.

Here, a custom `Processor` reads the current value for the record's key, updates it, and writes it back. Because the record's key pins it to one lane, this read-modify-write is already serial for that key — no locking required:

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.processor.ContextualProcessor
import io.stoatflow.core.processor.ProcessorContext
import io.stoatflow.core.processor.Record
import io.stoatflow.core.state.KeyValueStore

class RunningSumProcessor(
    private val storeName: String,
) : ContextualProcessor<String, Long, String, Long>() {
    private lateinit var store: KeyValueStore<String, Long>

    override fun init(context: ProcessorContext<String, Long>) {
        super.init(context)
        // storeName must be connected to this node — declare it from ProcessorSupplier.stores(),
        // or name it at the call site: process(supplier, storeName).
        store = context.getStateStore(storeName)
    }

    override fun process(record: Record<String, Long>) {
        // Single-key read-modify-write: safe with no locking,
        // because this key is always handled by the same lane.
        val current = store.get(record.key()) ?: 0L
        val updated = current + record.value()
        store.put(record.key(), updated)
        context().forward(record.withValue(updated))
    }
}
```

```java [Java]
import io.stoatflow.core.processor.ContextualProcessor;
import io.stoatflow.core.processor.ProcessorContext;
import io.stoatflow.core.processor.Record;
import io.stoatflow.core.state.KeyValueStore;

public class RunningSumProcessor
        extends ContextualProcessor<String, Long, String, Long> {

    private final String storeName;
    private KeyValueStore<String, Long> store;

    public RunningSumProcessor(String storeName) {
        this.storeName = storeName;
    }

    @Override
    public void init(ProcessorContext<String, Long> context) {
        super.init(context);
        // storeName must be connected to this node — declare it from ProcessorSupplier.stores(),
        // or name it at the call site: process(supplier, storeName).
        this.store = context.getStateStore(storeName);
    }

    @Override
    public void process(Record<String, Long> record) {
        // Single-key read-modify-write: safe with no locking,
        // because this key is always handled by the same lane.
        long current = store.get(record.key()) == null ? 0L : store.get(record.key());
        long updated = current + record.value();
        store.put(record.key(), updated);
        context().forward(record.withValue(updated));
    }
}
```
::

## Custom processors: instance fields are per-lane

A custom `Processor` (or `FixedKeyProcessor`) is instantiated **once per lane** — a lane is StoatFlow's analog of a Kafka Streams task, so this mirrors KS's one-instance-per-task model. Your instance fields are therefore **per-lane and single-threaded**: safe by construction for per-key scratch state, with no synchronization needed, exactly as in KS.

Two things follow from per-lane instances:

- A **punctuator** fires once over global state on the punctuation lane — it runs against a *different* instance than your record lanes, so it **cannot see instance fields** set in `process(...)`. For a scheduled write to a key, use a **timer** (`onTimer`), which runs in that key's lane with full read/write access.
- For state that must be **shared across lanes** or **read from a punctuator**, put it in a **state store**, not an instance field — that's exactly what the global state store is for. Punctuators have full read/write store access. For ephemeral scratch that needn't survive a restart, an in-memory store with `withLoggingDisabled()` is the lightweight option. See [Attaching state stores](https://stoatflow.io/docs/building/processor-api#attaching-state-stores).

## The one case that needs coordination: cross-key atomic updates

Key affinity serializes access *per key*. It says nothing about two **different** keys handled by two different lanes at the same time. That's almost always fine — different keys are independent. But there's one pattern it doesn't cover: a custom `Processor` that needs to read-modify-write **more than one key as a single atomic step**, where another lane might be touching one of those same keys concurrently.

Examples: moving a balance from account A to account B (debit one, credit the other, with no moment where the total is wrong), or maintaining an invariant that spans a pair of related keys. Because A and B may hash to different lanes, two records could enter the critical section at the same time, and the interleaving would corrupt the invariant.

For this — and **only** this — StoatFlow provides a key-lock utility, `KeyLockManager` (`io.stoatflow.core.state.KeyLockManager`). You instantiate and own it; the runtime never creates one for you. It guards a short critical section across one or more keys, acquiring locks in a deterministic order so concurrent multi-key sections can't deadlock against each other.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.processor.ContextualProcessor
import io.stoatflow.core.processor.ProcessorContext
import io.stoatflow.core.processor.Record
import io.stoatflow.core.state.KeyLockManager
import io.stoatflow.core.state.KeyValueStore

class TransferProcessor(
    private val storeName: String,
) : ContextualProcessor<String, Transfer, String, Transfer>() {
    private lateinit var balances: KeyValueStore<String, Long>

    // One instance per store; share one across stores for cross-store atomicity.
    private val keyLocks = KeyLockManager()

    override fun init(context: ProcessorContext<String, Transfer>) {
        super.init(context)
        // Connected at the call site: process(supplier, storeName).
        balances = context.getStateStore(storeName)
    }

    override fun process(record: Record<String, Transfer>) {
        val t = record.value()
        // Both keys may live on different lanes — lock both for the atomic move.
        keyLocks.withLocks(t.from, t.to) {
            val fromBalance = balances.get(t.from) ?: 0L
            val toBalance = balances.get(t.to) ?: 0L
            balances.put(t.from, fromBalance - t.amount)
            balances.put(t.to, toBalance + t.amount)
        }
        context().forward(record)
    }
}
```

```java [Java]
import io.stoatflow.core.processor.ContextualProcessor;
import io.stoatflow.core.processor.ProcessorContext;
import io.stoatflow.core.processor.Record;
import io.stoatflow.core.state.KeyLockManager;
import io.stoatflow.core.state.KeyValueStore;

public class TransferProcessor
        extends ContextualProcessor<String, Transfer, String, Transfer> {

    private final String storeName;
    private KeyValueStore<String, Long> balances;

    // One instance per store; share one across stores for cross-store atomicity.
    private final KeyLockManager keyLocks = new KeyLockManager();

    public TransferProcessor(String storeName) {
        this.storeName = storeName;
    }

    @Override
    public void init(ProcessorContext<String, Transfer> context) {
        super.init(context);
        // Connected at the call site: process(supplier, storeName).
        this.balances = context.getStateStore(storeName);
    }

    @Override
    public void process(Record<String, Transfer> record) {
        Transfer t = record.value();
        // Both keys may live on different lanes — lock both for the atomic move.
        keyLocks.withLocks(new Object[]{ t.from, t.to }, () -> {
            long fromBalance = balances.get(t.from) == null ? 0L : balances.get(t.from);
            long toBalance = balances.get(t.to) == null ? 0L : balances.get(t.to);
            balances.put(t.from, fromBalance - t.amount);
            balances.put(t.to, toBalance + t.amount);
            return null;
        });
        context().forward(record);
    }
}
```
::

::callout{color="warning" icon="i-lucide-triangle-alert"}
The block you pass to `withLock` / `withLocks` **must be fast and non-blocking**. Holding a lock across an I/O call, an external lookup, or any blocking operation stalls other lanes and can hold up the commit barrier. Do the blocking work *before* you enter the lock; keep the locked section to the in-memory read-modify-write. `withLocks` accepts at most 8 keys — if you need more, the access pattern likely wants redesigning.
::

If a lock can't be acquired within the configured timeout (default 10 seconds), the call throws `KeyLockTimeoutException` (`io.stoatflow.core.exception`) rather than blocking forever — a loud signal of a deadlock or excessive contention rather than a silent stall.

::callout{color="info" icon="i-lucide-info"}
You almost never need this. Single-key updates — which is what the DSL aggregations, joins, and the overwhelming majority of custom processors do — are already safe through key affinity alone. Reach for `KeyLockManager` only when a genuine cross-key invariant exists.
::

::callout{color="warning" icon="i-lucide-triangle-alert"}
**What a lock cannot do: cross the commit barrier.** `compute()`, `merge()` and `KeyLockManager` are all atomic **within a commit epoch**. Across a barrier rotation none of them is sufficient, and this is structural rather than a gap to be closed: a lane still draining epoch N is forbidden from reading a value another lane wrote into epoch N+1, because that would import uncommitted state into a committed transaction. Both writes then derive from the same base and one is durably lost — with the lock held. A lock makes the safe ordering *deterministic when it already exists*; it cannot create it.

It is unreachable while the key you *store under* is the key the record is *laned by* — which is what the DSL does everywhere, and what a Processor API node does whenever it writes `record.key()`. Two shapes break that correspondence:

- **A many-to-one re-key into a Processor API node.** Records arrive on the lane of their old key, so two records that re-keyed to the same new key run on different lanes. StoatFlow [refuses to compile the provable version of this](https://stoatflow.io/docs/building/processor-api#key-affinity-after-a-re-key) rather than leave you to lock your way around it. The remedies there are `repartition()` or `topology.processor-api-key-affinity: presumed`.
- **Writing a store key that is not the record key** — exactly what the transfer example above does, and what `KeyLockManager` exists for. Many `orderId`s updating one `customerId` aggregate is the everyday version. The compiler cannot see this one: the store key is computed inside your processor, at runtime.

So `withLocks` is doing real work in that example — it serialises the two account keys *within* an epoch, which is where interleaving actually happens on a busy lane. What it does not do is extend across a rotation. If a cross-key update must never lose a write under any interleaving, key the state by the record key and let key affinity do it; reach for a lock when the invariant genuinely spans keys and a rotation-boundary collision is acceptable.
::

## Store types

StoatFlow ships five store types. Stateful DSL operators (`count`, `reduce`, `aggregate`, joins, windowed and session aggregations, suppress) pick the appropriate type automatically; custom `Processor`s declare the stores they need. Each type comes in two backends:

- **RocksDB-backed** — persistent, on-disk. The default. State larger than memory is fine; local data survives process restarts and gives the changelog a head-start on recovery.
- **In-memory** — held entirely in the JVM heap. Lower per-operation overhead; bounded by available memory. Still durable via the changelog — the in-memory store is rebuilt from it on restart.

| Store type    | What it holds                                      | Typical use                                               |
| ------------- | -------------------------------------------------- | --------------------------------------------------------- |
| **Key-value** | One value per key                                  | `count`, `reduce`, `aggregate`, general `Processor` state |
| **Window**    | Values bucketed by time window                     | Windowed aggregations, tumbling/hopping/sliding windows   |
| **Session**   | Values grouped into activity-gap sessions          | Session-window aggregations                               |
| **Versioned** | Historical values queryable by timestamp (KIP-889) | Point-in-time lookups, temporal joins                     |
| **Timer**     | Scheduled callbacks keyed by fire time             | `Processor` event-time / processing-time timers           |

The backend is chosen per store. The DSL uses RocksDB by default; you switch a materialized store to in-memory with `Materialized.withStoreType(StoreType.IN_MEMORY)`, and custom processors choose via the `Stores` factory. The full how-to — declaring stores, the precedence rules, and the factory API — is on [State stores](https://stoatflow.io/docs/building/state-stores).

## Where to go next

- [State stores](https://stoatflow.io/docs/building/state-stores) — declaring stores, choosing types, and accessing them from the DSL or a `Processor`
- [Lanes and parallelism](https://stoatflow.io/docs/concepts/lanes-and-parallelism) — the key-affinity routing that makes concurrent state access safe
- [Architecture](https://stoatflow.io/docs/concepts/architecture) — the single-instance engine, commit barrier, and changelog durability
- [Processor API](https://stoatflow.io/docs/building/processor-api) — building custom processors that own their state


# Event time and watermarks

Stream processing has to answer two different questions about time: *when did this event happen* and *when did the runtime see it*. This page explains how StoatFlow separates the two, how watermarks track event-time progress, and how that drives window closing, late-record handling, and timers. It describes the conceptual model and the public API; the mechanism that tracks and merges watermarks internally stays in the source — see [Architecture](https://stoatflow.io/docs/concepts/architecture) for the boundary.

::tldr-panel
- **Event time** is when the event happened (the record timestamp, or whatever a custom extractor returns). **Processing time** is when the runtime handled it. Windowed operators reason in event time.
- A **watermark** is the runtime's claim "I don't expect any record earlier than time *T*." It's tracked per source partition and merged into one global watermark for the application.
- When the global watermark passes a window's end, the window can **close** — no more in-time records will arrive for it.
- **Late records** (event time older than the watermark) are handled by the window's **grace period**: a record that lands while its window is still inside grace is folded into the aggregate; once the window's grace has elapsed, late records for it are dropped.
- Public watermark strategies: `forBoundedOutOfOrderness`, `forMonotonousTimestamps`, `noWatermarks` — set per source via `Consumed.withWatermarkStrategy(...)`.
::

## Event time vs processing time

**Processing time** is wall-clock time on the machine running the topology — *when the runtime got around to this record*. It's trivially available and monotonic, but it's not reproducible: re-run the same input and the processing-time boundaries land differently, because they depend on broker latency, network buffering, and how busy the runtime was. Two records produced milliseconds apart can be processed seconds apart if a batch sat in a buffer.

**Event time** is *when the event actually happened*, carried on the record itself. It's stable across replays — the same input always produces the same windowed results, regardless of how fast or slow the runtime consumed it. That reproducibility is why every stateful operator that cares about time — windowed aggregations, session windows, time-bounded joins — reasons in event time.

The runtime needs both. Processing time drives wall-clock timers and "do this every N seconds" punctuation. Event time drives windows and late-record decisions.

## The record timestamp and custom extraction

Every input record carries a timestamp. By default that's the **Kafka record timestamp** — set by the producer when it sent the record, or by the broker on append, depending on the topic's `message.timestamp.type`. For many topologies that default is exactly right and you configure nothing.

When the meaningful event time lives *inside the value* — an `eventTime` field on your domain object, a timestamp parsed from a log line — supply a custom assigner. In StoatFlow this is done on the **watermark strategy** (the strategy combines timestamp extraction and watermark generation into one object, the same way Flink's `WatermarkStrategy` does). The KS-style `TimestampExtractor` functional interface is also supported and adapts onto a strategy internally.

The assigner receives the key, the value, and the Kafka record timestamp, and returns the event time in epoch milliseconds:

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.topology.Consumed
import io.stoatflow.core.watermark.WatermarkStrategy
import java.time.Duration

// Event time comes from a field on the value; allow 30s of out-of-orderness.
val strategy = WatermarkStrategy
    .forBoundedOutOfOrderness<String, Order>(Duration.ofSeconds(30))
    .withTimestampAssigner { _, value, _ -> value?.eventTime ?: 0L }

builder.stream(
    "orders",
    Consumed.with(Serdes.String(), orderSerde)
        .withWatermarkStrategy(strategy),
)
```

```java [Java]
import io.stoatflow.core.topology.Consumed;
import io.stoatflow.core.watermark.WatermarkStrategy;
import java.time.Duration;

// Event time comes from a field on the value; allow 30s of out-of-orderness.
WatermarkStrategy<String, Order> strategy = WatermarkStrategy
    .<String, Order>forBoundedOutOfOrderness(Duration.ofSeconds(30))
    .withTimestampAssigner((key, value) -> value.getEventTime());

builder.stream(
    "orders",
    Consumed.with(Serdes.String(), orderSerde)
        .withWatermarkStrategy(strategy));
```
::

::callout{color="info" icon="i-lucide-info"}
The assigner may be called with a `null` value (tombstones) and a `null` key (records with no key). Guard for it — return a sensible fallback rather than dereferencing. The Java `withTimestampAssigner` overload shown above takes a `(key, value)` `BiFunction`; a single-argument `Function<V, Long>` overload is also available when only the value matters (used in the [e-commerce example](https://stoatflow.io/#examples)).
::

## Watermarks: the "no earlier than T" claim

A **watermark*&#x2A; is a claim the runtime makes about event-time progress: &#x2A;"I do not expect any further records with an event time earlier than T."* It's not a record and not user data — it's a moving low-water mark that tells time-sensitive operators when it's safe to act.

Watermarks are derived from the event timestamps the runtime observes. As records flow in with increasing event times, the watermark advances behind them. How far behind depends on the strategy (next section). Crucially, a watermark is a *claim*, not a guarantee: a record older than the current watermark can still physically arrive — that's a **late record**, and whether it still counts toward its window depends on that window's grace period (see below).

Watermarks are tracked **per source partition** — each partition has its own view of how far its event time has progressed. The runtime combines those per-partition views into a **single global watermark** for the whole application. Because a StoatFlow app is exactly one instance (see [Architecture](https://stoatflow.io/docs/concepts/architecture#the-single-instance-model)), there is no distributed watermark-coordination protocol to configure — the global watermark is computed in-process. The global watermark advances together with the commit barrier, so windowed results are committed alongside the watermark progress that produced them; recovery sees a consistent snapshot of "what the app has seen up to."

The current per-partition watermark state is observable at runtime on the `/watermarks` admin endpoint.

### Idleness

If one partition stops receiving records — a quiet partition, an off-hours source — its event time stops advancing, which would hold the global watermark back and stall windows that depend on the busier partitions. A strategy configured `.withIdleness(timeout)` marks a partition idle after it goes quiet for that long, excluding it from the global watermark so the rest of the application keeps making event-time progress.

::code-tabs{group="lang"}
```kotlin [Kotlin]
val strategy = WatermarkStrategy
    .forBoundedOutOfOrderness<String, Order>(Duration.ofSeconds(30))
    .withTimestampAssigner { _, value, _ -> value?.eventTime ?: 0L }
    .withIdleness(Duration.ofMinutes(2))
```

```java [Java]
WatermarkStrategy<String, Order> strategy = WatermarkStrategy
    .<String, Order>forBoundedOutOfOrderness(Duration.ofSeconds(30))
    .withTimestampAssigner((key, value) -> value.getEventTime())
    .withIdleness(Duration.ofMinutes(2));
```
::

The idleness timeout must be positive — a zero or negative duration is rejected.

## The watermark strategies

StoatFlow ships three watermark strategies, all created from `WatermarkStrategy` factory methods and attached per source topic via `Consumed.withWatermarkStrategy(...)`. They're modelled on Flink's strategies.

### `forBoundedOutOfOrderness(maxOutOfOrderness)`

The common case. Events may arrive out of order by up to `maxOutOfOrderness` before being considered late. The watermark is `maxObservedTimestamp - maxOutOfOrderness` — the larger the tolerance you set, the longer windows stay open for stragglers, at the cost of holding results back longer. `maxOutOfOrderness` must not be negative.

::code-tabs{group="lang"}
```kotlin [Kotlin]
Consumed.with(Serdes.String(), orderSerde)
    .withWatermarkStrategy(
        WatermarkStrategy.forBoundedOutOfOrderness(Duration.ofSeconds(30)),
    )
```

```java [Java]
Consumed.with(Serdes.String(), orderSerde)
    .withWatermarkStrategy(
        WatermarkStrategy.forBoundedOutOfOrderness(Duration.ofSeconds(30)));
```
::

### `forMonotonousTimestamps()`

Use when events are known to arrive in strict timestamp order — a single-partition topic, log-append-time topics where the broker assigns timestamps, or events pre-sorted before production. It's equivalent to `forBoundedOutOfOrderness(Duration.ZERO)` with slightly less overhead. Any record whose timestamp goes backwards is treated as late.

::code-tabs{group="lang"}
```kotlin [Kotlin]
Consumed.with(Serdes.String(), eventSerde)
    .withWatermarkStrategy(WatermarkStrategy.forMonotonousTimestamps())
```

```java [Java]
Consumed.with(Serdes.String(), eventSerde)
    .withWatermarkStrategy(WatermarkStrategy.forMonotonousTimestamps());
```
::

### `noWatermarks()`

Produces no watermarks at all — the watermark stays at `Long.MIN_VALUE` forever. Use for processing-time-only topologies, or sources without meaningful event timestamps. The trade-off is explicit: with no watermarks, **event-time windows never close** and **event-time timers never fire**. Processing-time semantics still work.

::code-tabs{group="lang"}
```kotlin [Kotlin]
Consumed.with(Serdes.String(), eventSerde)
    .withWatermarkStrategy(WatermarkStrategy.noWatermarks())
```

```java [Java]
Consumed.with(Serdes.String(), eventSerde)
    .withWatermarkStrategy(WatermarkStrategy.noWatermarks());
```
::

All three accept `.withTimestampAssigner(...)` and `.withIdleness(...)`, plus Flink-style `.withWatermarkAlignment(group, maxDrift)` for multi-source topologies where one source's event time must not run too far ahead of the others in the same alignment group. When you set neither a strategy nor an assigner, the source uses the Kafka record timestamp as event time.

## Window closing

A windowed operator — a tumbling/hopping `TimeWindows` aggregation, a session window, a sliding window — groups records into windows by their event time. The runtime can't emit a window's final result until it's confident no more in-time records will land in it. That confidence comes from the watermark.

When the **global watermark passes a window's end**, the runtime knows the window can close: any record that would belong inside it would have an event time the watermark has already moved past. At that point the window's result is finalised. Until then, the window stays open and accumulates.

This is why watermark choice has a direct, visible effect on latency: a 30-second `maxOutOfOrderness` means a window's result is held back at least 30 seconds past its end before it can close. Tighter tolerance closes windows sooner but treats more stragglers as late.

Window specifications can carry a **grace period** of their own (for example `TimeWindows.ofSizeAndGrace(size, grace)`), which keeps a window open for late records for an extra interval *after* its end before it's permanently closed — see the next section.

The whole cycle on one timeline. The axis is event time; the circled numbers are the order the records *arrived* in, which is what actually drives the watermark:

![One window from 10:00 to 10:05 with a 1 minute grace period, under a 30 second out-of-orderness watermark, and eight records numbered by arrival order. The watermark sits at the highest event time seen so far minus 30 seconds and only advances when a record arrives with a higher event time. Records 1 to 4 land in the window in time — record 4 arrives out of order at 10:03:45 after record 3 at 10:04:00, but is still ahead of the watermark, so it is not late. Record 5 at 10:05:20 belongs to a later window and drags the watermark to 10:04:50. Record 6 at 10:04:30 is now behind the watermark and therefore late, but the window has not closed yet, so it is folded in and the result corrected. Record 7 at 10:06:40 pushes the watermark to 10:06:10, past the close point of 10:06, and the window closes. Record 8 at 10:04:45 would have belonged to the window but arrives after the close, so it is dropped. Emitting on every update produces one result per record that lands in the window; emitting on close produces a single final result.](https://stoatflow.io/assets/docs/concepts/watermark-window-grace_20260727.svg)

## Late records and grace

A **late record** is one whose event time is older than the current watermark — it arrived after the runtime had already claimed not to expect anything that old. How a windowed operator responds is governed by the **grace period** on the window the record belongs to:

- **Within grace** — if the late record's window is still inside its configured grace window, the record is folded into the aggregate and an updated result is emitted. Grace is set on the window specification (for example `TimeWindows.ofSizeAndGrace(...)`), and bounds how long after a window's end late records are still accepted.
- **Past grace** — once a window's end plus its grace has elapsed, the window is closed and finalised; any later record that would have belonged to it is **dropped** before it can change the result.

Grace is the knob that trades completeness against finality: a longer grace catches more stragglers and corrects more aggregates, but holds the window's state longer and delays the point at which the result is truly final. A window with no grace closes as soon as the watermark passes its end, dropping every straggler.

A past-grace drop is **not** separately metered: a record discarded because its window has already closed produces no dedicated metric or callback. The runtime does count late *arrivals* — records whose event time is already behind the watermark — via a debug-level counter, but that measures arrivals, not grace-exceeded drops: many counted arrivals are still folded into open windows within grace, and a window that closes with no further records produces no drop signal at all. If knowing how often stragglers are discarded matters to you, account for it in your own logic rather than relying on a built-in drop metric.

## Event-time and processing-time timers

The DSL's windowed operators cover the common time-driven patterns declaratively. For logic that has to fire on a schedule independent of incoming records — flush a buffer at a deadline, expire an entry, emit a heartbeat — the [Processor API](https://stoatflow.io/docs/building/processor-api) lets a custom `Processor` register **timers**:

- **Event-time timers** fire when the **watermark** advances past a registered moment. They're driven by event-time progress, so they respect the same out-of-order tolerance as windows — and, like windows, they never fire under `noWatermarks()`.
- **Processing-time timers** fire when **wall-clock** time passes a registered moment, regardless of whether any records are flowing.

Both deliver a callback with full state-store access in the same epoch-aligned context as record processing, so anything a timer writes commits atomically with the surrounding work (see [Exactly-once semantics](https://stoatflow.io/docs/concepts/exactly-once)). The registration API and callback signatures are covered in [Processor API](https://stoatflow.io/docs/building/processor-api).

For a *topology-level* periodic source — generating records on an interval or a cron schedule without consuming from Kafka — see [Scheduled sources](https://stoatflow.io/docs/building/scheduled-sources), which can be driven by either `STREAM_TIME` (event time) or `WALL_CLOCK_TIME` (processing time).

## Examples

The e-commerce daily-customer-behaviour example (under the project's `examples/ecommerce-daily-customer-behaviour` module) wires custom event-time extraction into a windowed aggregation. Each source attaches a strategy that pulls event time from the record's own timestamp field, with a daily window that carries a grace period for late arrivals:

```java
// Custom timestamp extraction off the value (single-arg Function overload)
WatermarkStrategy<String, WebActivityEvent> webActivityWatermark = WatermarkStrategy
    .<String, WebActivityEvent>forBoundedOutOfOrderness(Duration.ZERO)
    .withTimestampAssigner(v -> toUtcFromLocal(v.timestamp(), LOCAL_ZONE_ID).toEpochMilli());

KStream<String, WebActivityEvent> webActivity = builder.stream(
    WEB_ACTIVITY_SOURCE_TOPIC,
    Consumed.with(stringSerde, webActivityEventSerde)
        .withWatermarkStrategy(webActivityWatermark)
        .withName("web-activity-source"));

// Daily windows with a 15-minute grace period for late events
TimeWindows dailyWindows = TimeWindows.ofSizeAndGrace(Duration.ofDays(1), Duration.ofMinutes(15));
```

The full source includes the windowed aggregation and a downstream join enriching each daily summary against a customer-profile `KTable`.

## Where to go next

- [Architecture](https://stoatflow.io/docs/concepts/architecture#event-time-and-watermarks) — where watermark tracking sits in the engine, and the single-instance reason there's no distributed coordination
- [Windowing](https://stoatflow.io/docs/building/windowing) — tumbling, hopping, session, and sliding windows; grace and emit strategies
- [Aggregations](https://stoatflow.io/docs/building/aggregations) — `count`, `reduce`, `aggregate` over grouped and windowed streams
- [Processor API](https://stoatflow.io/docs/building/processor-api) — registering event-time and processing-time timers
- [Scheduled sources](https://stoatflow.io/docs/building/scheduled-sources) — topology-level periodic sources on stream time or wall-clock time
- [Exactly-once semantics](https://stoatflow.io/docs/concepts/exactly-once) — how windowed results and watermark progress commit atomically


# The configuration model

This page explains how a StoatFlow application is configured: the sources you can set values from, the order they win in, and the line between what you configure and what the engine decides for itself. It describes the model, not every key — for the exact key list and defaults, the configuration classes in the source are authoritative.

::tldr-panel
- **One file to start:** `application.yaml` on the classpath, with three top-level groups — `stoatflow:` (the engine), `runtime:` (the HTTP/metrics/health wrapper), and `logging:` (per-logger levels).
- **Layered:** YAML defaults can be overlaid by external files, environment variables, and a programmatic override block — each layer wins over the one below it.
- **Opinionated defaults:** almost everything has a sensible default, so a minimal `application.yaml` is two lines plus a license. You override only what your deployment needs.
- **Some behaviour is adaptive:** the commit cadence, for example, self-tunes at runtime rather than reading a fixed interval. You set bounds and intent; the engine picks the value inside them.
::

## Two configuration surfaces

A StoatFlow application is configured at two layers, and it helps to keep them straight.

The **engine** — the `:core` library — is configured by a `StreamsConfig` object. If you build directly against `:core`, you construct that object yourself, in code.

Porting a Kafka Streams application onto `:core` doesn't mean rewriting that config from scratch. A KS-keyed `Properties` or `Map` adapts onto the same typed `StreamsConfig` — `StreamsConfig.fromProperties(props)` / `fromMap(map)`, and `new StreamsConfig(props)` works directly (the KS-faithful constructor shape). The adapter covers the Kafka-Streams-4.x overlap surface — typed keys, class-name pluggables (serdes, exception handlers), and the `consumer.` / `producer.` / `admin.` client prefixes — and treats a key StoatFlow doesn't recognise leniently: a warning, not a failure, unless you opt into `stoatflow.config.strict`. `StreamsConfig.toProperties()` emits the resolved config back out for inspection. The [KS-compatibility matrix](https://stoatflow.io/docs/reference/ks-compatibility-matrix#configuration) lists the exact key coverage.

The **runtime** — the `:runtime` module, the batteries-included wrapper most applications use — adds YAML loading on top. It reads an `application.yaml`, maps it onto the engine's `StreamsConfig`, and configures its own infrastructure (the HTTP server, metrics, health checks) from the same file. This page is mostly about that runtime layer, because that's where the layering and precedence live. (Why a runtime wrapper exists at all is covered on [Architecture](https://stoatflow.io/docs/concepts/architecture) and [Features](https://stoatflow.io/product/features).)

The YAML splits into three top-level groups:

- **`stoatflow:`** — the engine. Kafka connection, lane parallelism, commit barriers, state stores, changelog topics, watermarks, the license, and the rest of the processing behaviour.
- **`runtime:`** — the infrastructure wrapper. The HTTP server, Prometheus metrics, endpoint visibility, and health checks.
- **`logging:`** — per-logger levels, applied programmatically over whatever `logback.xml` defines.

A minimal file is short, because the defaults carry the rest:

```yaml
stoatflow:
  application-id: word-count
  bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVERS:-localhost:9092}

  license:
    key: ${STOATFLOW_LICENSE_KEY}

runtime:
  http:
    enabled: true
    port: ${HTTP_PORT:-8080}
  metrics:
    enabled: true
```

The `${VAR:-default}` syntax is environment-variable interpolation with a fallback — resolve the variable, or use the value after `:-` when it's unset. Use it to keep deployment-specific values (broker addresses, ports, secrets) out of the committed file.

## The layers, and how they stack

Configuration is resolved by overlaying several sources. A value set in a higher-priority layer overrides the same value in every layer below it. Anything no layer sets falls through to the built-in default.

From highest priority to lowest:

1. **Programmatic overrides** — a code block passed when you construct the runtime. Applied last, so they win over everything from YAML and the environment. This is also the *only* layer that can carry a live object — a serde built with constructor arguments, an exception handler, a callback. YAML gets partway: it can name a class that has a no-arg constructor and let the runtime instantiate it. Anything past that exists only in code.
2. **Environment variables (`__` separator)** — the recommended form for overriding nested keys from the environment.
3. **External overlay files** — extra YAML files named via the `STOATFLOW_CONFIG_FILES` environment variable.
4. **`application.yml`** on the classpath.
5. **`application.yaml`** on the classpath.
6. **Built-in defaults** — the values baked into the configuration classes.

![A precedence stack of the six configuration layers, highest first: programmatic overrides, environment variables in the double-underscore form, external overlay files, application.yml on the classpath, application.yaml on the classpath, and the built-in defaults. One key, stoatflow.lanes.count, is traced down the stack: the programmatic layer does not set it, the environment variable STOATFLOW\_\_LANES\_\_COUNT=32 does and wins, and the overlay file's 24, the application.yaml's 12 and the derived default of 8 are all struck through as read but discarded. A separate mark on each band shows what that layer can carry: layers two to five carry text, only the programmatic layer carries a live object such as a serde instance, and a panel below explains that YAML can still name a class with a no-arg constructor but cannot construct one with arguments or pass a lambda.](https://stoatflow.io/assets/docs/concepts/config-layers_20260727.svg)

The sections below cover each layer that needs explaining.

### The classpath YAML

`application.yaml` (or `application.yml`) on the classpath is the base layer. For most applications it's the only configuration file. It lives at `src/main/resources/application.yaml` in a typical project.

This is where you express the *intended* shape of the deployment — the values that are the same across every environment, with `${VAR:-default}` interpolation for the handful that differ. The example below tunes lanes, the commit barrier, state, and the changelog; everything it omits keeps its default.

```yaml
stoatflow:
  application-id: stock-tick-filter
  bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVERS:-localhost:9092}

  lanes:
    count: 12
    queue-capacity: 1000

  commit-barrier:
    interval-ms: 200
    timeout-ms: 20000

  state:
    dir: ${java.io.tmpdir}/stoatflow/stock-tick-filter
    restoration-enabled: true

  changelog:
    enabled: true
    replication-factor: 1
    num-partitions: 6

runtime:
  http:
    enabled: true
    port: ${HTTP_PORT:-8080}
  metrics:
    enabled: true
    common-tags:
      env: development
```

### External overlay files

Set `STOATFLOW_CONFIG_FILES` to a comma-separated list of file paths and the runtime merges each one *over* the bundled classpath YAML, at the property level — a deep merge, not a wholesale replace. A file that sets only `stoatflow.bootstrap-servers` overrides that one value and leaves everything else from the base file intact.

```bash
# Single overlay
STOATFLOW_CONFIG_FILES=/etc/myapp/production.yaml

# Multiple overlays — later files win over earlier ones
STOATFLOW_CONFIG_FILES=/etc/myapp/base.yaml,/etc/myapp/overrides.yaml
```

This is the natural fit for environment-specific layering: a base `application.yaml` baked into the artifact, and a per-environment overlay mounted at deploy time (a Kubernetes ConfigMap or Secret, for example). The files listed in `STOATFLOW_CONFIG_FILES` are required — if one is named but missing, startup fails rather than silently skipping it.

### Environment variables

Any property can be set from the environment, which is what makes the YAML overridable without rebuilding the artifact. There is one naming form.

It uses `__` (double underscore) as the path separator between nested keys, and a single `_` for word boundaries inside a key name:

```bash
STOATFLOW__LANES__COUNT=32                          # → stoatflow.lanes.count
STOATFLOW__COMMIT_BARRIER__INTERVAL_MS=500          # → stoatflow.commit-barrier.interval-ms
RUNTIME__HTTP__PORT=9090                            # → runtime.http.port
```

Separating the two roles is what lets a variable name address a multi-word key like `commit-barrier` at all: with `_` doing both jobs there is no way to tell a path boundary from a word boundary. Names are case-insensitive, and only keys under the known top-level groups (`stoatflow`, `runtime`, `logging`) are read.

### Programmatic overrides

The highest-priority layer is a code block you pass when constructing the runtime. It's applied *after* YAML and environment variables are resolved, so anything it sets wins. Two reasons to use it:

- **Non-serializable values.** Serdes, exception handlers, and callbacks are Java/Kotlin objects, not strings. YAML can name one by fully qualified class name provided it has a no-arg constructor — see [class-name configuration](https://stoatflow.io/docs/configuration#class-name-configuration-in-yaml). Anything that needs a constructor argument, or is a lambda, has to be set here.
- **Programmatic control.** Anything you'd rather compute in code than spell out in YAML.

This is the `streamsConfigOverrides` block from [Your first app](https://stoatflow.io/docs/getting-started/first-app) — here setting the default serdes, which have no YAML representation:

::code-tabs{group="lang"}
```kotlin [Kotlin]
val runtime = StoatFlowRuntime.fromConfig(
    topologyBuilder = { buildTopology(it) },
    configure = {
        streamsConfigOverrides {
            defaultKeySerde(Serdes.String())
            defaultValueSerde(Serdes.String())
        }
    },
)
runtime.start()
runtime.awaitTermination()
```

```java [Java]
var runtime = StoatFlowRuntime.fromConfig(
    Main::buildTopology,
    builder -> builder.streamsConfigOverrides(cfg -> {
        cfg.defaultKeySerde(Serdes.String());
        cfg.defaultValueSerde(Serdes.String());
    })
);
runtime.start();
runtime.awaitTermination();
```
::

`StoatFlowRuntime.fromConfig(...)` loads `application.yaml`, applies the environment-variable layers, then runs your `streamsConfigOverrides` block last — so a value you set here overrides the same key from YAML or the environment.

::callout{color="info" icon="i-lucide-info"}
If you build against `:core` directly rather than `:runtime`, there is no YAML layer — you construct the engine's `StreamsConfig` in code, which is the same surface the override block writes to. The runtime's whole job here is to load YAML and the environment, then hand control to that same builder.
::

## Why the defaults are opinionated

Most configuration keys have a default, and the defaults are chosen to make a typical application run well without tuning. Lane count defaults to the available CPU count; the commit barrier, state store sizes, changelog topic creation, and watermark behaviour all start from values that suit a common workload. The minimal two-line `application.yaml` works because of this — you override only the keys your deployment actually needs to change.

The intent is that configuration is *subtractive*: you start from a working default and adjust the few knobs your workload demands (broker address, lane count, commit-interval bounds, state directory), rather than assembling a working configuration from scratch.

Some defaults are also **derived rather than fixed** — they're computed from the environment at startup. Lane count, for instance, defaults to a function of the available processor count, so the same `application.yaml` adapts to the machine it runs on instead of pinning a number that's wrong on half your hardware. These derived defaults still sit at the bottom of the precedence stack: set the key explicitly and your value wins.

## What stays automatic

Not everything that affects behaviour is a configuration key. Some of the engine's behaviour is **adaptive** — decided at runtime from what the workload is actually doing, rather than read from a fixed value.

The clearest example is the **commit cadence**. Rather than committing on a fixed timer, the runtime adjusts how often it commits based on observed runtime conditions, so the cadence tracks the workload instead of forcing you to guess a number that's right for both a quiet topic and a burst. What you configure is the *envelope* — bounds and intent, like the minimum and maximum interval between barriers — and the engine chooses a value inside it. The [Architecture](https://stoatflow.io/docs/concepts/architecture) page frames this distinction: the barrier scheduling cadence and related runtime decisions are implementation concerns, and the algorithms that drive them stay in the source.

The practical consequence for configuring an application: you don't set the commit interval to a single number and hope it fits every load. You set the bounds that must hold — the slowest acceptable cadence, the fastest — and let the engine pick within them. The same philosophy shows up elsewhere in the engine, where a mode like `AUTO` lets the runtime detect the right setting from the topology rather than asking you to specify it. Where a knob is exposed, it's there because a deployment can reasonably need to set it; where behaviour is left automatic, it's because the engine has better information at runtime than you have at configuration time.

## Inspecting the resolved configuration

Because configuration is layered, the value an application ends up running with isn't always obvious from any single file. The runtime exposes the **merged, effective configuration** at the `/config` HTTP endpoint, with sensitive values (passwords, secrets, tokens) masked. When you need to confirm what actually won across all the layers, that's the source of truth for a running instance — see the operational surface on [Architecture](https://stoatflow.io/docs/concepts/architecture).

## Where to go next

- [Architecture](https://stoatflow.io/docs/concepts/architecture) — the model the configuration tunes, and the runtime's full operational surface.
- [Your first app](https://stoatflow.io/docs/getting-started/first-app) — a complete `application.yaml` and `streamsConfigOverrides` block in context.
- [Error-handling model](https://stoatflow.io/docs/concepts/error-handling-model) — the failure policies you configure and what each one does.
- [Features](https://stoatflow.io/product/features) — the runtime capabilities the `runtime:` group switches on.


# The error-handling model

This page is the conceptual model for how a StoatFlow application behaves when something goes wrong with a record or a commit. It describes **which failures are distinct**, **the policies you can apply to each**, and **what the runtime does that you don't get to configure** — the commit failure that ends in a process restart. The concrete handler classes and the configuration keys that select them live in [Error handling and DLQ](https://stoatflow.io/docs/building/error-handling-dlq); this page stays at the level of behaviour.

The starting point is the same single-process model the rest of these pages build on. Because there's one instance and one transaction per commit (see [Architecture](https://stoatflow.io/docs/concepts/architecture) and [Exactly-once](https://stoatflow.io/docs/concepts/exactly-once)), every failure either gets handled *inside* an epoch — without disturbing the exactly-once guarantee — or it's fatal and the epoch is thrown away. There's no middle ground where a node limps along in a degraded state.

::tldr-panel
- **Three failure classes, each with its own policy:** a record fails to **deserialize** on input, a **processor** throws while handling a record, or **production** (serialize + send) fails on output.
- **Three policies per class:** *log and continue* (skip the record), *log and fail* (stop the topology), or *dead-letter* (route the record and its error context to a DLQ topic).
- **DLQ records are part of the transaction:** they commit on the same barrier as everything else, so a dead-lettered record is never lost and never duplicated.
- **Producer errors are classified** retriable vs fatal: transient send errors retry with backoff; fatal ones end the epoch.
- **A failed commit is not configurable:** the in-flight transaction aborts, the process exits, and the orchestrator restarts it from the last successful barrier.
::

## Three classes of failure

The runtime distinguishes failures by *where in the record's lifecycle* they happen. Each class is independent — you can log-and-continue on bad input while still failing hard on a processor bug, or vice versa.

### Deserialization (on the way in)

A record arrives from Kafka as raw bytes. Before the topology can touch it, the key and value bytes must deserialize through the configured serdes. If that fails — malformed JSON, a schema the deserializer can't read, a corrupt payload — it's a **deserialization failure**. The topology never ran for this record; there's nothing to roll back. This is the failure class for source topics that may carry records your app can't read (a producer upstream changed format, a poison record slipped in).

### Processing (inside the topology)

The record deserialized fine and entered the topology, and then a processor threw — a `NullPointerException` in a `mapValues`, a validation error in a custom `Processor`, an unchecked exception from an enrichment call. This is a **processing failure**: it happens on a processing lane, mid-DAG, after the record was already accepted. Whatever state the record touched before the throw is part of the current epoch and is governed by the same commit barrier as everything else — a skipped or dead-lettered record doesn't leak half-applied state into the committed snapshot.

### Production (on the way out)

The topology produced an output record and the runtime tried to publish it. Two things can go wrong here, and the runtime treats them as one class with sub-cases:

- **Serialization** — the output key or value can't be turned into bytes (a serde rejects the value, a Schema Registry call fails). This is deterministic: the same record will fail the same way every time, so it's never retried.
- **Send** — serialization succeeded but the producer couldn't deliver (broker unreachable, request timed out, an authorization error). Some send failures are transient and worth retrying; others are permanent.

This is a **production failure**. It's the only class where *retry* is a meaningful policy, because it's the only class where the failure might be transient — see *Producer error classification* below.

## The handler model: continue, fail, or dead-letter

Each failure class is governed by an **exception handler** — a policy the runtime consults when that class of failure occurs. There are three behaviours a handler can express, and the built-in handlers give you one each:

| Policy                | What it does                                                                                   | When it fits                                                                                                                               |
| --------------------- | ---------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| **Log and continue**  | Logs the error and **skips** the offending record. Processing carries on with the next record. | Lossy-tolerant streams where one bad record shouldn't stop the pipeline and you don't need to recover the record later.                    |
| **Log and fail**      | Logs the error and **stops** the topology. The process shuts down.                             | Correctness-critical streams where a bad record means something is wrong upstream that a human needs to look at. The conservative default. |
| **Dead-letter (DLQ)** | Routes the record and its error context to a configured **dead-letter topic**, then continues. | You want to keep processing *and* keep the bad records for offline inspection or replay.                                                   |

![A record travelling left to right past three gates — deserialize before the topology, process mid-DAG on a lane, and produce, which splits into serialize (never retried) and send (may be transient). Each gate offers the same log-and-continue, log-and-fail and dead-letter verdicts and carries its own default; produce adds a fourth, retry. Every dead-letter verdict feeds one Kafka transaction that commits the output records, the DLQ records and the state plus source offsets together on the same barrier.](https://stoatflow.io/assets/docs/concepts/failure-gates_20260727.svg)

The defaults are deliberately strict. Deserialization and processing both default to **log and fail** — silently skipping records should be a choice you make on purpose, not a behaviour you inherit. Production defaults to a handler that retries transient send errors and fails on the rest. You opt into continue-or-dead-letter behaviour per class; the building guide shows how the handlers are named and wired.

::callout{color="warning" icon="i-lucide-shield"}
**Skipping is data loss with a log line.** *Log and continue* drops the record and moves on — the only trace is a log entry and an error-metric tick on `/metrics`. If you might ever need the record back, choose the DLQ policy instead, where the bytes are preserved on a topic you control.
::

These handler classes follow the shape of Kafka Streams' KIP-1033 (processing exception handler, Kafka 3.9.0) and KIP-1034 (dead letter queue, Kafka 4.2.0) error-handling design, so the mental model carries over directly if you know Kafka Streams. The behavioural deltas are catalogued in the [Kafka Streams compatibility matrix](https://stoatflow.io/docs/reference/ks-compatibility-matrix).

Credit where it is due: both KIPs were coauthored by Damien Gasparina, [Loïc Greffier](https://www.linkedin.com/in/loicgreffier/){rel="&#x22;nofollow&#x22;"} and [Sébastien Viale](https://www.linkedin.com/in/s%C3%A9bastien-viale-86a245138/){rel="&#x22;nofollow&#x22;"} at Michelin, and grew out of Michelin's [Kstreamplify](https://github.com/michelin/kstreamplify){rel="&#x22;nofollow&#x22;"}. Their Current London 2025 talk [Processing Exception Handling and Dead Letter Queue in Kafka Streams](https://current.confluent.io/post-conference-videos-2025/processing-exception-handling-and-dead-letter-queue-in-kafka-streams-lnd25){rel="&#x22;nofollow&#x22;"}, and Michelin's write-ups on [processing error handling](https://blogit.michelin.io/processing-error-handling-in-kafka-streams/){rel="&#x22;nofollow&#x22;"} and [the DLQ in Spring Kafka 4.1](https://blogit.michelin.io/kafka-streams-dead-letter-queue-in-spring-kafka-4-1/){rel="&#x22;nofollow&#x22;"}, are the best introductions to the design this page inherits.

One escape hatch sits above the per-class policies. A *failed* record error (the **log and fail** outcome) surfaces as a fatal stream error, and by default it stops the instance. An application can register the KS-compatible `StreamsUncaughtExceptionHandler`; returning `REPLACE_THREAD` makes the runtime recover with an **in-place engine restart** — the processing engine is torn down and rebuilt inside the same process, resuming from the last committed barrier with the interrupted epoch aborted, exactly as a crash recovery would. It's the single-instance analogue of Kafka Streams replacing a failed stream thread, and it's bounded: a restart budget shuts the application down if faults keep recurring.

## Dead-letter queue semantics

A DLQ handler doesn't publish to the dead-letter topic out of band. It hands the failed record back to the runtime, and that record is sent through **the same transactional producer, on the same commit barrier** as the epoch's normal output. For the deserialization, processing, and synchronous-production failure classes, this is the property that makes the DLQ trustworthy:

- **No loss.** The dead-lettered record commits atomically with the rest of the epoch. If the epoch commits, the DLQ record is on the topic; if the epoch aborts (a crash mid-barrier), the DLQ record is discarded along with everything else and the source record will be re-read and re-handled after restart.
- **No duplicates.** Because it rides the exactly-once transaction, a dead-lettered record appears on the DLQ topic exactly once for downstream consumers reading with `read_committed` isolation — same guarantee as your real output.

**The asynchronous (broker-side) production failure path reaches the same guarantee by a different route.** When the broker rejects an already-sent record (the classic case: a record exceeding the topic's `max.message.bytes`), the rejection poisons the in-flight transaction — nothing in it can commit, so the epoch's outputs, changelog writes and state are all aborted. Rather than advancing past them, the runtime **holds the source offsets** and **replays the epoch**: it commits a secondary transaction carrying only the poison's DLQ record, quarantines the poison's source coordinates, restarts the engine in place, and reprocesses the epoch with exactly that record skipped. Every other record in it is processed again and commits normally.

| Verdict    | Source offsets | Dead letter                                  | Innocent records                                       | Application                            |
| ---------- | -------------- | -------------------------------------------- | ------------------------------------------------------ | -------------------------------------- |
| *continue* | not advanced   | written (value omitted for a size rejection) | preserved — replayed with the poison skipped           | keeps running, up to the replay budget |
| *retry*    | not advanced   | written                                      | preserved — replayed, and the failed send re-attempted | keeps running, up to the replay budget |
| *fail*     | not advanced   | written                                      | preserved — replayed on operator restart               | stops; no in-place restart             |

::callout{color="warning" icon="i-lucide-alert-triangle"}
**Loss is bounded to the poison record's own outputs *and its state contribution* — it is not zero.** Quarantining a source record skips *all* of its outputs, including sinks whose output was fine, so a record that fans out to several topics loses the good sends along with the bad one. It is skipped before deserialization, so it never reaches a processor either: a `count()` over its key stays permanently one short, and every aggregate derived from that key is wrong from then on. And a poison derived from **accumulated state** — an aggregate that outgrew `max.message.bytes` — does not converge: skipping the triggering record does not shrink the accumulator, so the next record on that key reproduces it, each round quarantining one more offset (and drifting the state by one more record) until the replay budget runs out.

An over-limit producer *batch* also fails as a unit: Kafka fails such a batch outright only when it holds a single record, otherwise it expires and *every* record in it fails. Your handler is told those innocent records failed, and *continue* quarantines them. The bound is therefore "the records whose sends actually failed", which under batching can include batch-mates.

That budget (`stoatflow.commit-barrier.max-poison-replays`, default 10 per minute) is **in-memory**. Exhausting it ends the process; Kubernetes restarts the pod and the budget returns clean. So *continue* is bounded **per process**, never end to end — a non-converging poison will crash-loop the pod with no framework-level bound. Alert on **both** `stoatflow.dlq.poison.replays.total` and `stoatflow.dlq.poison.replay.budget.exhausted.total`: the first counts replays, so it goes quiet precisely when the budget runs out and the crash loop starts, which is what the second one catches. Prevent the case by sizing the target topic's `max.message.bytes` for your largest output — and size the **DLQ** topic at least as large as every topic that feeds it, or the dead letter for an oversized record can be refused too.
::

![A wall-clock timeline of one epoch. Source records are read while innocent output records and state writes accumulate, and one output record is sent that the broker rejects asynchronously. At the commit barrier the whole transaction aborts, but the source offsets are held rather than advanced: a secondary transaction commits only the poison's dead-letter record, the engine restarts in place, and the epoch is replayed with the poison record skipped so the innocent records commit normally.](https://stoatflow.io/assets/docs/concepts/poison-epoch-abort_20260813.svg)

Each replay costs a full engine restart, so it is loud rather than silent: `stoatflow.dlq.poison.quarantined.total` counts the records being skipped, `stoatflow.dlq.poison.replays.total` counts the replays, and `stoatflow.dlq.poison.quarantine.size` shows how many offsets a replay is currently skipping (0 in steady state). The pod is briefly unready while the engine rebuilds.

Two cases cannot be replayed and stop the instance immediately, with the source offsets held so an operator restart replays the epoch:

- A **`fail` verdict**, which now means what it says — stop before further harm, rather than causing the harm and then stopping.
- A &#x2A;*`continue`** on a poison with **no attributable source record** — an output emitted by a punctuator, a watermark-driven window close, a suppression flush, a timer or a scheduled source. `continue&#x60; asserts the emission itself is poison, and those planes fire on their own schedule, so they would re-poison every attempt: the runtime names the plane and shuts down rather than burning the budget discovering that. A &#x2A;*`retry`** on one of those planes is a different claim — the *send* failed, not the emission — so it replays like any other `retry`, and the send is re-attempted.

**Kafka Streams shares the failure class, and answers it differently.** Verified against 4.3.1: it honours *continue* locally, and then its offsets ride `sendOffsetsToTransaction`, which throws once the producer is in `ABORTABLE_ERROR`, so the source position never advances — the [open bug](https://issues.apache.org/jira/browse/KAFKA-15259){rel="&#x22;nofollow&#x22;"} is precisely this scenario. Its own KIP-1034 dead-letter record is appended to the transaction the rejection already doomed, so it is aborted with everything else and never becomes visible to a `read_committed` consumer. The commit then throws, the stream thread dies, and the default `SHUTDOWN_CLIENT` stops the client — under a supervisor, a restart loop onto the same record, with no evidence written anywhere. StoatFlow also stops advancing, deliberately; the difference is that the dead letter survives, the innocent records survive, and the instance either recovers or stops with the poison's coordinates named.

What each DLQ record carries depends on the failure class:

- **Deserialization DLQ records** preserve the **original raw key/value bytes** — nothing was successfully deserialized, so the untouched payload is exactly what you need to diagnose or replay it.
- **Processing and production DLQ records** carry the record as it was at the point of failure, plus error-context metadata.

Every DLQ record is annotated with **error-context headers** under a StoatFlow-specific `__stoatflow.errors.*` namespace (KIP-1034-shaped): the failure type, the exception class and message, an optional stack trace, the originating source topic / partition / offset, and which component failed. DLQ tooling that filters out the conventional `__`-prefixed internal headers must opt in to surface these. The exact header set is documented in [Error handling and DLQ](https://stoatflow.io/docs/building/error-handling-dlq).

::callout{color="info" icon="i-lucide-info"}
A dead-letter topic is an ordinary Kafka topic that you create and own. The runtime only produces to it — it doesn't read it back. Replaying dead-lettered records into the source (after you've fixed the upstream cause) is your call to make, with your tooling.
::

## Producer error classification: retriable vs fatal

Production is the one failure class where the runtime makes a finer distinction, because a send failure might be transient. When the producer reports an error, the runtime classifies it before deciding what to do:

- **Retriable.** Transient conditions — a request timeout, a momentarily unreachable broker — that may succeed if tried again. The default production handler returns *retry*. What that buys you depends on which send failed. On a **synchronous** send the runtime retries in place with exponential backoff for a bounded number of attempts, and writes any attached dead letter only once those are exhausted — so a retry that succeeds leaves no trace. On an **asynchronous** broker rejection the producer's own retries are already spent by the time the callback fires, so *retry* instead aborts and replays the epoch: identical to *continue* except that the record is **not** quarantined, because the verdict says the record is fine and the send wasn't. Under `AT_LEAST_ONCE` there is no transaction to abort and no replay to offer, so *retry* on that path is treated as *fail*.
- **Fatal.** Conditions that won't improve by retrying — a serialization error (deterministic, the same record fails the same way), an authentication or authorization failure, a broker-incompatibility error, or a producer-fencing condition. These can't be handled away inside the epoch, so they end it.

The serialization-vs-send split matters here: serialization failures are *always* classified non-retriable (retrying can't change a deterministic outcome), while send failures are classified case by case. A fatal production error that has a DLQ configured will still emit the DLQ record before the epoch ends, so even a fatal failure leaves a forensic trail.

The classification rules are stricter than Kafka Streams' in one specific way: StoatFlow has a single instance and no task to migrate, so the failure conditions Kafka Streams would resolve by moving a task elsewhere have nowhere to go and are treated as fatal. That's a direct consequence of the [single-instance model](https://stoatflow.io/docs/concepts/architecture#the-single-instance-model), not a separate design choice.

## When the commit itself fails

Everything above happens *inside* an epoch and leaves the exactly-once guarantee intact. There is one failure the runtime does **not** hand to a configurable policy: the **commit barrier itself failing**.

When the per-epoch Kafka transaction can't complete — it times out, the broker rejects it, the producer gets fenced — there is no safe way to continue. The runtime aborts the in-flight transaction and the process exits. As described on the [architecture page](https://stoatflow.io/docs/concepts/architecture#exactly-once-semantics-the-commit-barrier), the aborted transaction's partial work — uncommitted state writes, uncommitted output, uncommitted offset advances — is discarded at the broker. Your orchestrator (Kubernetes, typically) restarts the process, which resumes from the **last successful barrier** and re-reads every record after it. Downstream consumers reading with `read_committed` isolation never see the aborted epoch's output.

This is intentional and not tunable: a single-instance, single-transaction design has no notion of "commit failed but keep going." A stalled or failed commit is treated as a fatal condition, the epoch is thrown away whole, and recovery is a clean restart from a known-good point. The same restart-on-fatal pattern covers a sustained broker outage that exhausts the retry budget — the runtime stops making progress, the process exits, and the orchestrator brings it back. The internal protocol that detects a stalled commit and converts it into this abort-and-exit behaviour stays in the source; what you observe is the restart.

::callout{color="info" icon="i-lucide-heart-pulse"}
A restart is the recovery path, not an outage you have to engineer around. High availability under this model comes from **fast restart** — or, with an opt-in [hot standby](https://stoatflow.io/docs/operating/high-availability), failover to a warm passive peer — see the [lifecycle and recovery](https://stoatflow.io/docs/concepts/architecture#lifecycle-startup-restart-recovery) section. The readiness probe stays down while the restarting instance restores state, so traffic and load balancers wait until it's caught up.
::

## What you observe

Each failure class surfaces the same way you'd expect from the architecture page's operational framing — a metric signal, a log line, and (for DLQ policies) the records themselves. The column below names the *signal*, not the exact Prometheus metric ID; the precise names are what you'll see exposed on `/metrics`.

| Failure                | Signal                        | Log                                              | Record disposition                                 |
| ---------------------- | ----------------------------- | ------------------------------------------------ | -------------------------------------------------- |
| Deserialization        | deserialization-error counter | error/warn with topic-partition-offset           | skipped, failed, or on the DLQ with original bytes |
| Processing             | processing-error counter      | error/warn with processor name + record metadata | skipped, failed, or on the DLQ                     |
| Production (retriable) | Kafka-client error metrics    | warn ("will retry")                              | retried with backoff                               |
| Production (fatal)     | Kafka-client error metrics    | error                                            | DLQ if configured, then the epoch ends             |
| Commit failure         | commit-stall counter          | error + thread dump                              | epoch aborted at broker; process restarts          |

Steady-state failure diagnosis uses the always-on admin surface — `/metrics`, `/health/ready`, and the debug endpoints — covered in the [architecture page's observability section](https://stoatflow.io/docs/concepts/architecture#failure-modes-and-observability).

## Where to go next

- **[Error handling and DLQ](https://stoatflow.io/docs/building/error-handling-dlq)** — the concrete handler classes, the configuration keys that select them per failure class, the full DLQ header set, and worked examples in Kotlin and Java.
- **[Exactly-once](https://stoatflow.io/docs/concepts/exactly-once)** — why DLQ records and skipped-record offsets commit atomically with the rest of the epoch.
- **[Architecture](https://stoatflow.io/docs/concepts/architecture#failure-modes-and-observability)** — the full failure-mode and observability catalogue at the system level.
- **[Kafka Streams compatibility matrix](https://stoatflow.io/docs/reference/ks-compatibility-matrix)** — how StoatFlow's error-handling surface lines up against Kafka Streams.


# How StoatFlow differs from Kafka Streams

If you know Kafka Streams, the fastest way to understand StoatFlow is by the differences. The DSL you write is the same — the same `StreamsBuilder`, the same `KStream` / `KTable`, the same operators. What changes is the runtime model underneath: how many processes run, where state lives, how parallelism is expressed, how re-keying happens, and how exactly-once is committed. This page lays out those deltas. For the model in its own right, start with [Architecture](https://stoatflow.io/docs/concepts/architecture); for porting an existing topology, see [Migration](https://stoatflow.io/docs/migration).

::tldr-panel
- **One instance, no rebalancing** — a StoatFlow app runs as exactly one JVM, not a cluster of stream threads that join and leave a group.
- **In-memory re-keying** — `selectKey` / `groupBy` / key-changing joins hand records between lanes in-process; there is no internal repartition topic.
- **Barrier-based exactly-once** — one Kafka transaction per commit barrier covers the whole topology, rather than per-task transactional writes.
- **Global state** — every store lives in the one process and any key is reachable from anywhere; no partition-scoped isolation, no co-partitioning requirement.
- **Lanes, not tasks** — parallelism comes from key-affinity lanes that scale with cores, not from one task per input partition.
- **The DSL is the same.** Your topology code reads the same; the engine that runs it is different.
::

## What carries over unchanged: the DSL

The thing you spend most of your time on — the topology — is the part that does **not** change. StoatFlow implements the Kafka Streams DSL: `StreamsBuilder`, `KStream`, `KTable`, `KGroupedStream`, the windowed and session variants, the joins, the Processor API, the `Consumed` / `Produced` / `Materialized` / `Grouped` config objects, and the KS-compatible functional interfaces (`ValueMapper`, `KeyValueMapper`, `ValueJoiner`, `Reducer`, `Aggregator`, …). The full method-by-method status is in the [Kafka Streams compatibility matrix](https://stoatflow.io/docs/reference/ks-compatibility-matrix).

Concretely, this map-filter topology is written the same way against StoatFlow as against Kafka Streams:

::code-tabs{group="lang"}
```kotlin [Kotlin]
val intermediate =
    stream1
        .selectKey { _, _ -> "lala" }
        .map { key, value -> KeyValue(value.substring(0, 3), "$key:$value") }
        .filter { _, value -> value.length > 5 }
        .mapValues { value -> value.uppercase() }

intermediate.to(
    "output-topic",
    Produced.`as`<String, String>("sink1")
        .withKeySerde(Serdes.String())
        .withValueSerde(Serdes.String()),
)
```

```java [Java]
KStream<String, String> intermediate =
    stream1
        .selectKey((k, v) -> "lala")
        .map((key, value) -> KeyValue.pair(value.substring(0, 3), key + ":" + value))
        .filter((k, value) -> value.length() > 5)
        .mapValues(value -> value.toUpperCase());

intermediate.to(
    "output-topic",
    Produced.<String, String>as("sink1")
        .withKeySerde(Serdes.String())
        .withValueSerde(Serdes.String()));
```
::

The imports differ — StoatFlow's DSL lives under `io.stoatflow.core.topology.*` (and the runtime under `io.stoatflow.runtime.*`) rather than `org.apache.kafka.streams.*` — and the entry point differs (covered below). The operators in between are the same shape, which is what makes a port mechanical rather than a rewrite.

There are deliberate API-surface differences where the single-instance model makes a Kafka Streams concept unnecessary or replaces it with something Flink-shaped — for example, event-time extraction is expressed as a Flink-style `WatermarkStrategy` on `Consumed` (the KS `TimestampExtractor` is still accepted and adapts onto one), and the partitioner settings on `TableJoined` are accepted as no-ops because there are no partition-bound tasks to route between. Those are catalogued in the [Kafka Streams compatibility matrix](https://stoatflow.io/docs/reference/ks-compatibility-matrix); everything below is the conceptual *why*.

## Single instance vs. multi-instance and rebalancing

This is the root difference; the rest follow from it.

**Kafka Streams** is a cluster. You run N instances, each with some number of stream threads. The instances form a Kafka consumer group, the group coordinator assigns tasks (each bound to an input partition) across the live members, and that assignment changes — *rebalances* — whenever an instance joins, leaves, or fails. Rebalancing is the mechanism that gives Kafka Streams its horizontal scaling and its failover, and it's also the source of much of its operational complexity: state has to migrate or restore on the new owner, processing pauses during the handoff, and you tune around it (standby replicas, static membership, cooperative rebalancing).

**StoatFlow** runs as exactly one active JVM process, assigned every partition of every source topic, with no group to rebalance. High availability comes from fast restart by default, with an opt-in [hot-standby](https://stoatflow.io/docs/operating/high-availability) cluster (one active + warm standbys) for near-instant failover — still a single *active* instance, never a second active taking over a partition (ADR-001).

The trade-off is explicit: you give up open-ended horizontal scale-out and instead scale a single instance vertically with cores and memory. You cannot run two *active* replicas of the same StoatFlow application against the same source topics; the supported way to run more than one instance is the *passive* [hot-standby](https://stoatflow.io/docs/operating/high-availability) cluster, which coordinates a single active explicitly. The reasoning behind accepting that trade is on [Motivation](https://stoatflow.io/product/motivation); the consequences for operations are on [Architecture](https://stoatflow.io/docs/concepts/architecture#the-single-instance-model).

## Lane parallelism vs. partition-bound tasks

Because there's no cluster, parallelism is expressed differently.

In **Kafka Streams**, processing parallelism is capped by the partition count of the input topics: one task per partition, one stream thread runs one or more tasks. To process more in parallel you add partitions — a topic-level change with downstream consequences for every consumer of that topic. The unit of concurrency is the partition, and it's fixed by your topic layout.

In **StoatFlow**, the single consumer reads **all** partitions, and the engine then distributes work across **lanes** — key-affinity units of concurrent processing inside the JVM. The same key always routes to the same lane (so per-key order is preserved); different keys run on different lanes in parallel. Lane count is decoupled from partition count: you set it at startup, and it scales with CPU cores, not with how many partitions the source topic happens to have. Lanes run on virtual threads, so a lane blocked on a REST call or a database query parks at near-zero cost while other lanes make progress — which makes in-line blocking enrichment natural in a way the partition-bound model isn't.

The full treatment — choosing a lane count, why your keyspace caps the benefit, and the blocking-I/O story — is on [Lanes and parallelism](https://stoatflow.io/docs/concepts/lanes-and-parallelism).

## In-memory re-keying vs. repartition topics

When a topology changes a record's key — `selectKey`, `groupBy`, or a key-changing join — the record may need to move to a different unit of parallelism.

**Kafka Streams** handles this by writing the re-keyed record to an internal **repartition topic** and re-reading it on the other side, so that the new key lands on the correct partition (and therefore the correct task). That's a broker round-trip plus an extra serialize/deserialize per re-keyed record. You can see it in a Kafka Streams topology as an explicit or implicit `repartition()` boundary:

```kotlin
// Kafka Streams — re-keying forces a repartition topic round-trip
stream1
    .selectKey { _, _ -> "lala" }
    .map { key, value -> KeyValue(value.substring(0, 3), "$key:$value") }
    .repartition()          // ← writes to an internal topic, re-reads on the other side
    .filter { _, value -> value.length > 5 }
```

**StoatFlow** re-hashes the new key and hands the record to the lane that owns it **in-memory**, between lanes in the same process. There is no internal repartition topic, no broker hop, and no extra serialization round-trip — the same `selectKey` / `map` chain needs no `repartition()` call at all:

::code-tabs{group="lang"}
```kotlin [Kotlin]
// StoatFlow — re-keying is an in-process handoff; no repartition() needed
stream1
    .selectKey { _, _ -> "lala" }
    .map { key, value -> KeyValue(value.substring(0, 3), "$key:$value") }
    .filter { _, value -> value.length > 5 }
```

```java [Java]
// StoatFlow — re-keying is an in-process handoff; no repartition() needed
stream1
    .selectKey((k, v) -> "lala")
    .map((key, value) -> KeyValue.pair(value.substring(0, 3), key + ":" + value))
    .filter((k, value) -> value.length() > 5);
```
::

The output is identical to what Kafka Streams produces; the path is shorter (ADR-010). `repartition()` still exists in StoatFlow's DSL for source compatibility, but it's an in-memory operation rather than a topic round-trip. After a re-key, the new key gets its own owning lane — the affinity property travels with the record across the handoff, so one key is never processed by two lanes at once. Ordering carries over for records that shared the old key; records newly brought together under one key have no guaranteed relative order, the same as in Kafka Streams, where they would have been processed by different tasks. &#x2A;*Where the handoff happens matches Kafka Streams.** StoatFlow inserts one exactly where Kafka Streams would materialise a repartition topic — before a grouped aggregation, a join, `toTable()`, or an explicit `repartition()` — not at every key change. So `describe()` reports the same sub-topology structure as the KS original, which matters when you are comparing a port against the app it replaced. That includes `process()` / `processValues()`, where Kafka Streams never repartitions either — StoatFlow did until 1.0.0, and stopping was the last piece of sub-topology-shape parity (`topology.processor-api-key-affinity: presumed` restores the old boundary; see [Key affinity after a re-key](https://stoatflow.io/docs/building/processor-api#key-affinity-after-a-re-key) for the trade-off after a many-to-one re-key). The mechanism is detailed on [Lanes and parallelism](https://stoatflow.io/docs/concepts/lanes-and-parallelism#in-memory-re-keying-instead-of-repartition-topics), including the one case where the difference is visible: the lazy rule does not implicitly re-spread work across lanes after a re-key.

## Barrier-based exactly-once vs. per-task transactions

Both systems deliver exactly-once over Kafka transactions; the **scope** of the transaction is what differs.

**Kafka Streams** coordinates exactly-once across the cluster. Each task commits its own work, and the framework manages the transactional producers and offset commits across all the tasks and instances participating — a distributed coordination problem, and historically a fiddly one to configure correctly.

**StoatFlow** uses a single **commit barrier** that flows through the entire topology, cascading across each in-memory sub-topology boundary in turn. When the barrier completes, the runtime executes one Kafka transaction that atomically commits every state-store write (via changelog topics), every sink output record, and the consumer-group offsets for every contributing input partition — all three together, or none. Because there's one process, there's one barrier and one transaction covering the whole topology: no per-task transactions to coordinate, no cross-instance two-phase commit, no external checkpoint store. Where the topology crosses a sub-topology boundary, epoch alignment rides along with the records — the receiving side briefly holds back anything that runs ahead of the barrier — so the one transaction still commits a consistent cut across the boundary, not a partial slice of the next epoch. Exactly-once is the default rather than something you opt into and tune (ADR-004). The protocol is in the Chandy-Lamport family of distributed-snapshot algorithms — see [Exactly-once](https://stoatflow.io/docs/concepts/exactly-once#the-commit-barrier).

The full mechanism, what "exactly-once" means in concrete terms here, crash recovery, and the at-least-once trade-off are on [Exactly-once semantics](https://stoatflow.io/docs/concepts/exactly-once). The barrier scheduling cadence and the transaction protocol itself are implementation concerns and stay in the source.

## Global state vs. partition-scoped state

State placement follows directly from the single-instance model.

In **Kafka Streams**, state is **partition-scoped**: each task owns the state for its partitions, in a local store, and a key is only reachable from the task that owns it. This is why `KTable`-`KTable` joins require **co-partitioning** — both tables must be partitioned the same way so that matching keys land in the same task — and why interactive queries across a cluster need partition routing or RPC to reach the instance that holds a given key.

In **StoatFlow**, state is **global**: every store lives in the one JVM, and any processing lane can read or write any key. There's no partition-scoped isolation, no inter-instance lookup protocol, and no replication of the same data across JVMs. Two consequences fall out:

- **No co-partitioning requirement.** `KTable`-`KTable` joins work without aligning the tables' partitioning, because there are no partition-bound tasks to align (ADR-007). Foreign-key joins likewise don't need a co-partitioned subscription topic.
- **No partition routing for queries.** Interactive Queries reach any store directly — there's no `withPartition(...)` and no cross-instance RPC, because all state is locally accessible (ADR-025).

Correctness under concurrency is preserved by key affinity, not by partition isolation: each key is only ever touched by the single lane that owns it, so updates to one key are serialized while different keys update in parallel. The depth — why that's safe, the one cross-key case where you coordinate yourself, and the store types available — is on [State and thread safety](https://stoatflow.io/docs/concepts/state-and-thread-safety).

## The entry point and the deployment unit

Two practical differences you hit immediately when porting.

**Entry point.** Where Kafka Streams takes a `Topology` plus a `Properties` and a `KafkaStreams` object you `start()`, StoatFlow has two front doors:

- `StoatFlow.fromBuilder(config, builder)` on the `:core` module — the DSL-and-engine entry point, configured with a typed `StreamsConfig`, with `start()` / `awaitTermination()` / `close()`.
- `StoatFlowRuntime.fromConfig(...)` on the `:runtime` module — the batteries-included wrapper that loads `application.yaml`, starts the HTTP admin + metrics server and health endpoints, and runs your topology until terminated. This is what [Your first app](https://stoatflow.io/docs/getting-started/first-app) uses.

**Deployment unit.** A Kafka Streams deployment is a fleet you size by instance count and partition count. A StoatFlow deployment is a single pod you size by cores and memory — one active replica, scaled vertically. High availability is a fast, clean restart by default, or an opt-in [hot-standby](https://stoatflow.io/docs/operating/high-availability) cluster where a warm passive standby takes over in seconds. The operational surface for that — health probes, metrics, the debug endpoints, and the failure policies — is described on [Architecture](https://stoatflow.io/docs/concepts/architecture#failure-modes-and-observability).

## The deltas at a glance

| Aspect              | Kafka Streams                                     | StoatFlow                                                         |
| ------------------- | ------------------------------------------------- | ----------------------------------------------------------------- |
| Deployment          | Multiple instances, rebalancing consumer group    | Single instance, no group to rebalance                            |
| Scaling             | Horizontal — add instances / partitions           | Vertical — add cores / memory                                     |
| Parallelism unit    | Task per input partition                          | Key-affinity lane, decoupled from partitions                      |
| Re-keying           | Internal repartition topic (broker round-trip)    | In-memory handoff between lanes                                   |
| Exactly-once        | Per-task transactions, coordinated across cluster | One commit barrier → one transaction, whole topology              |
| Default guarantee   | Opt-in and tuned                                  | Exactly-once by default                                           |
| State model         | Partition-scoped, local to a task                 | Global, reachable from any lane                                   |
| Table joins         | Require co-partitioning                           | No co-partitioning needed                                         |
| Interactive Queries | Partition routing / RPC across instances          | Direct — all state is local                                       |
| Blocking I/O        | Blocks a stream thread                            | Parks a virtual thread cheaply                                    |
| Entry point         | `KafkaStreams(topology, props)`                   | `StoatFlow.fromBuilder(...)` / `StoatFlowRuntime.fromConfig(...)` |
| DSL                 | Kafka Streams DSL                                 | The same DSL                                                      |

The full method-by-method parity table — including the handful of KS APIs that StoatFlow deliberately drops or replaces — is the [Kafka Streams compatibility matrix](https://stoatflow.io/docs/reference/ks-compatibility-matrix).

## Where to go next

- [Architecture](https://stoatflow.io/docs/concepts/architecture) — the single-instance model the deltas above all follow from
- [Lanes and parallelism](https://stoatflow.io/docs/concepts/lanes-and-parallelism) — lanes, key affinity, and in-memory re-keying in full
- [Exactly-once](https://stoatflow.io/docs/concepts/exactly-once) — the commit barrier and crash recovery
- [State and thread safety](https://stoatflow.io/docs/concepts/state-and-thread-safety) — the global state model and co-partitioning-free joins
- [Comparison matrix](https://stoatflow.io/product/comparison-matrix) — feature-by-feature against Kafka Streams and Flink
- [Migration](https://stoatflow.io/docs/migration) — porting an existing Kafka Streams topology


# Building topologies

A **topology** is the processing graph your application runs: source topics in, operators in the middle, sink topics out. You describe it once with `StreamsBuilder`, hand the builder to the runtime, and the engine executes it. The DSL is Kafka Streams compatible — if you know `stream()`, `mapValues()`, `groupBy()`, and `to()`, you already know how to write one.

This page is the orientation for the section: the entry point, the stream abstractions at a glance, and the one rule that trips up people coming from Kafka Streams — the fan-out rule. Each operator family has its own page; links are at the bottom.

::tldr-panel
- **Entry point:** `StreamsBuilder` — call `.stream(...)` / `.table(...)` to read, chain operators, call `.to(...)` to write.
- **Abstractions:** `KStream` (event stream), `KTable` (changelog table), and the grouped/windowed types you reach via `groupBy()` / `windowedBy()`.
- **Fan-out rule:** to branch a stream multiple ways, **reuse the same `KStream` reference**. Calling `builder.stream()` twice on the same topic is rejected at build time.
::

## StreamsBuilder — the entry point

Everything starts with a `StreamsBuilder`. Its source methods (`stream`, `table`, `globalTable`, `scheduled`) return DSL objects you chain operators onto; terminal operations (`to`, `forEach`, `print`) end a branch and return nothing. You never construct `KStream` or `KTable` directly — the builder hands them to you.

A source method takes a topic name and an optional `Consumed` for serdes, watermark strategy, and offset-reset policy. Every operator takes an optional `Named` so its node gets a stable identity in the topology graph, metrics, and state-store names. The convention across the examples is to name each operator explicitly.

Here is a minimal end-to-end topology — read, transform, filter, write:

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.topology.Consumed
import io.stoatflow.core.topology.Named
import io.stoatflow.core.topology.Produced
import io.stoatflow.core.topology.StreamsBuilder
import org.apache.kafka.common.serialization.Serdes

fun buildTopology(builder: StreamsBuilder) {
    builder
        .stream<String, String>("input-topic", Consumed.`as`("source"))
        .mapValues({ value -> value.uppercase() }, Named.`as`("upper"))
        .filter({ _, value -> value.length > 5 }, Named.`as`("longer-than-5"))
        .to(
            "output-topic",
            Produced.`as`<String, String>("sink")
                .withKeySerde(Serdes.String())
                .withValueSerde(Serdes.String()),
        )
}
```

```java [Java]
import io.stoatflow.core.topology.Consumed;
import io.stoatflow.core.topology.Named;
import io.stoatflow.core.topology.Produced;
import io.stoatflow.core.topology.StreamsBuilder;
import org.apache.kafka.common.serialization.Serdes;

void buildTopology(StreamsBuilder builder) {
    builder
        .<String, String>stream("input-topic", Consumed.as("source"))
        .mapValues(value -> value.toUpperCase(), Named.as("upper"))
        .filter((key, value) -> value.length() > 5, Named.as("longer-than-5"))
        .to(
            "output-topic",
            Produced.<String, String>as("sink")
                .withKeySerde(Serdes.String())
                .withValueSerde(Serdes.String()));
}
```
::

::callout{color="info" icon="i-lucide-info"}
`as` is a soft keyword in Kotlin, so the factory methods are written ``Consumed.`as`(...)`` with backticks. In Java it's plain `Consumed.as(...)`. Imports come from `io.stoatflow.core.*` (DSL) and `io.stoatflow.runtime.*` (runtime wrapper). See [Your first app](https://stoatflow.io/docs/getting-started/first-app) for the full runnable shell that calls `buildTopology`.
::

When you're done describing the graph, the builder is what you pass to the runtime — `StoatFlowRuntime.fromConfig(topologyBuilder = { buildTopology(it) }, ...)`. The runtime calls `build()` for you and validates the topology before it starts. (If you're embedding the core engine directly rather than using the runtime wrapper, you call `builder.build()` yourself.)

## The stream abstractions at a glance

The DSL has two foundational types and several specialised ones you reach by grouping or windowing:

| Type                           | What it is                                                                                                               | You get it from                                                               |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------- |
| `KStream<K, V>`                | An unbounded stream of independent events. Each record is a fact.                                                        | `builder.stream(...)`, `KTable.toStream()`, `builder.scheduled(...)`          |
| `KTable<K, V>`                 | A changelog table — one current value per key. A new record for a key replaces the previous one.                         | `builder.table(...)`, `KGroupedStream.count()/reduce()`, `KStream.toTable()`  |
| `KGroupedStream<K, V>`         | A stream re-grouped by key, ready to aggregate.                                                                          | `KStream.groupBy(...)` / `groupByKey()`                                       |
| `KGroupedTable<K, V>`          | A table re-grouped by key, with add/subtract changelog semantics.                                                        | `KTable.groupBy(...)`                                                         |
| `TimeWindowedKStream<K, V>`    | A grouped stream bucketed into fixed-size time windows (tumbling/hopping) **or** event-driven sliding windows (KIP-450). | `KGroupedStream.windowedBy(TimeWindows...)` / `windowedBy(SlidingWindows...)` |
| `SessionWindowedKStream<K, V>` | A grouped stream bucketed by activity gaps.                                                                              | `KGroupedStream.windowedBy(SessionWindows...)`                                |
| `CogroupedKStream<K, V>`       | Multiple streams aggregated jointly into one result.                                                                     | `KGroupedStream.cogroup(...)`                                                 |
| `BranchedKStream<K, V>`        | The result of splitting one stream into named, predicate-routed branches.                                                | `KStream.split(...)`                                                          |

`StoatFlow` also adds **scheduled sources** — a source that emits records on an interval or cron schedule instead of consuming a topic. It returns a `KStream` and flows through the topology like any other source. See [Scheduled sources](https://stoatflow.io/docs/building/scheduled-sources).

The mental model is the same as Kafka Streams: a `KStream` is a stream of events; a `KTable` is the latest-value-per-key view of a changelog. `KStream.toStream()` doesn't exist (it's already a stream), but `KTable.toStream()` turns a table back into its change events, and `KStream.toTable()` materialises a stream as a table. For the full operator-by-operator parity status against Kafka Streams, see the [Kafka Streams compatibility matrix](https://stoatflow.io/docs/reference/ks-compatibility-matrix).

## The fan-out rule

This is the one rule worth internalising before you write anything non-trivial.

To send one input stream down **multiple** processing paths — say, one branch to topic A and another to topic B — you **reuse the same `KStream` reference** for each branch. You do **not** call `builder.stream()` twice on the same topic. Subscribing the same topic from two source nodes is **rejected when the topology is built**, with a `TopologyValidationException`.

::callout{color="error" icon="i-lucide-circle-x"}
**Don't do this** — two source nodes on one topic:

```kotlin
builder.stream<String, String>("orders").filter(...).to("a")   // source node 1
builder.stream<String, String>("orders").mapValues(...).to("b") // source node 2 — REJECTED at build()
```

`build()` throws: *"Multiple source nodes consuming the same topic are not supported."*
::

Instead, hold the source `KStream` in a variable and branch off it. Each operator chain that starts from the shared reference is an independent path through the topology; they all share the single source subscription:

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.topology.Consumed
import io.stoatflow.core.topology.KStream
import io.stoatflow.core.topology.Named
import io.stoatflow.core.topology.Produced
import io.stoatflow.core.topology.StreamsBuilder

fun buildTopology(builder: StreamsBuilder) {
    // Read the topic ONCE, keep the reference.
    val orders: KStream<String, String> =
        builder.stream("orders", Consumed.`as`("source"))

    // Branch 1: filter, write to "large-orders".
    orders
        .filter({ _, v -> v.length > 100 }, Named.`as`("only-large"))
        .to("large-orders", Produced.`as`("large-sink"))

    // Branch 2: transform the SAME source, write to "orders-upper".
    orders
        .mapValues({ v -> v.uppercase() }, Named.`as`("upper"))
        .to("orders-upper", Produced.`as`("upper-sink"))
}
```

```java [Java]
import io.stoatflow.core.topology.Consumed;
import io.stoatflow.core.topology.KStream;
import io.stoatflow.core.topology.Named;
import io.stoatflow.core.topology.Produced;
import io.stoatflow.core.topology.StreamsBuilder;

void buildTopology(StreamsBuilder builder) {
    // Read the topic ONCE, keep the reference.
    KStream<String, String> orders =
        builder.stream("orders", Consumed.as("source"));

    // Branch 1: filter, write to "large-orders".
    orders
        .filter((key, v) -> v.length() > 100, Named.as("only-large"))
        .to("large-orders", Produced.as("large-sink"));

    // Branch 2: transform the SAME source, write to "orders-upper".
    orders
        .mapValues(v -> v.toUpperCase(), Named.as("upper"))
        .to("orders-upper", Produced.as("upper-sink"));
}
```
::

The same rule applies to every topic-backed source: reuse the returned reference from `table()` and `globalTable()` rather than calling them twice for the same topic. `scheduled()` is **not** topic-backed — each call defines its own independent source (there is no shared Kafka topic to conflict over), so two `scheduled()` calls are never rejected. To branch a scheduled source's output, reuse its returned `KStream` reference the same way you would any other source.

::callout{color="info" icon="i-lucide-git-branch"}
For **first-match** routing — where each record should go to exactly one of several mutually exclusive branches based on a predicate — use `KStream.split()` instead of manual fan-out. It returns a `BranchedKStream` and routes each record to the first matching branch (with an optional default). Manual fan-out (above) sends *every* record down *every* branch; `split()` sends each record down *one*.
::

## Where to go next

Each operator family has a dedicated how-to:

- **[StreamsBuilder](https://stoatflow.io/docs/building/streams-builder)** — sources, sinks, global/read-only stores, and `build()` in depth.
- **[KStream and KTable](https://stoatflow.io/docs/building/kstream-ktable)** — `map`, `filter`, `flatMap`, `selectKey`, `merge`, `split`, table operations, and stream↔table conversion.
- **[Aggregations](https://stoatflow.io/docs/building/aggregations)** — `count`, `reduce`, `aggregate`, grouping, and cogroup.
- **[Windowing](https://stoatflow.io/docs/building/windowing)** — tumbling, hopping, sliding, and session windows.
- **[Joins](https://stoatflow.io/docs/building/joins)** — stream-stream, stream-table, and table-table (incl. foreign-key) joins.
- **[Processor API](https://stoatflow.io/docs/building/processor-api)** — custom `Processor` / `FixedKeyProcessor`, state-store access, timers, and punctuators.
- **[Scheduled sources](https://stoatflow.io/docs/building/scheduled-sources)** — interval- and cron-driven sources (a StoatFlow extension).
- **[Serdes](https://stoatflow.io/docs/building/serdes)** — configuring serialization for keys and values.
- **[State stores](https://stoatflow.io/docs/building/state-stores)** — store types, materialization, and the `Materialized` config.
- **[Error handling & DLQ](https://stoatflow.io/docs/building/error-handling-dlq)** — deserialization and processing exception handlers.
- **[Testing](https://stoatflow.io/docs/building/testing)** — the in-memory `TopologyTestDriver`, no broker required.

For the engine that runs the topology you build here — lanes, commit barriers, state durability — see [Architecture](https://stoatflow.io/docs/concepts/architecture).


# Error handling and DLQ

StoatFlow has three independent exception handlers, one per failure stage: a record that fails to **deserialize** on the way in, a **processor** that throws while handling a record, and a **production** failure when serializing or sending an output record. Each handler decides whether to skip the record and continue, stop the application, or route the record to a dead-letter queue (DLQ) — and each is wired the same way, in code via `streamsConfigOverrides` or in `application.yaml` by class name.

For the design rationale — why these three stages, when "continue" is safe, and how DLQ records ride the same commit barrier as your output — see [Error-handling model](https://stoatflow.io/docs/concepts/error-handling-model).

The surface follows Kafka Streams' KIP-1033 (processing exception handler, Kafka 3.9.0) and KIP-1034 (dead letter queue, Kafka 4.2.0), so it should look familiar if you have used those. Both were coauthored by Damien Gasparina, [Loïc Greffier](https://www.linkedin.com/in/loicgreffier/){rel="&#x22;nofollow&#x22;"} and [Sébastien Viale](https://www.linkedin.com/in/s%C3%A9bastien-viale-86a245138/){rel="&#x22;nofollow&#x22;"} at Michelin, and grew out of Michelin's [Kstreamplify](https://github.com/michelin/kstreamplify){rel="&#x22;nofollow&#x22;"}; their Current London 2025 talk [Processing Exception Handling and Dead Letter Queue in Kafka Streams](https://current.confluent.io/post-conference-videos-2025/processing-exception-handling-and-dead-letter-queue-in-kafka-streams-lnd25){rel="&#x22;nofollow&#x22;"} and Michelin's write-ups on [processing error handling](https://blogit.michelin.io/processing-error-handling-in-kafka-streams/){rel="&#x22;nofollow&#x22;"} and [the DLQ in Spring Kafka 4.1](https://blogit.michelin.io/kafka-streams-dead-letter-queue-in-spring-kafka-4-1/){rel="&#x22;nofollow&#x22;"} are the best background.

::tldr-panel
- **Three handlers:** `deserializationExceptionHandler`, `processingExceptionHandler`, `productionExceptionHandler`.
- **Built-in behaviours:** log-and-fail (default for deser + processing), log-and-continue, and dead-letter-queue.
- **DLQ records** carry the original bytes plus error metadata in `__stoatflow.errors.*` headers, and are committed transactionally alongside your normal output.
- **Wire it:** in code via `streamsConfigOverrides { ... }`, or in YAML by fully qualified class name (no-arg handlers only).
::

## The three handler stages

| Stage           | Config property                   | Fires when                                            | Default                                     |
| --------------- | --------------------------------- | ----------------------------------------------------- | ------------------------------------------- |
| Deserialization | `deserializationExceptionHandler` | A source record's key or value cannot be deserialized | `LogAndFailDeserializationExceptionHandler` |
| Processing      | `processingExceptionHandler`      | A processor throws during `process()`                 | `LogAndFailProcessingExceptionHandler`      |
| Production      | `productionExceptionHandler`      | An output record fails to serialize or send           | `DefaultProductionExceptionHandler`         |

Each handler returns a decision:

- **`CONTINUE`** — skip the failed record and keep processing. Optionally emit one or more DLQ records. On the **asynchronous** production path below it skips the record across an epoch *replay* rather than in place, but the effect is the same: the failed record is skipped, everything else is processed.
- **`FAIL`** — stop the application. The in-flight epoch's work is discarded and the process exits; see [Architecture](https://stoatflow.io/docs/concepts/architecture) for the recovery behaviour on restart. Optionally emit DLQ records before failing. The source offsets are **not** advanced on any path, including the asynchronous production one, so the epoch replays on restart — `FAIL` means stop before further harm.
- **`RETRY`** — production handlers only. On a **synchronous** send the runtime retries in place with exponential backoff, and writes any attached DLQ record only once those attempts are exhausted. On the **asynchronous** production path below the producer's own retries are already spent, so `RETRY` aborts and replays the epoch instead: identical to `CONTINUE` except the record is **not** quarantined, because the verdict says the record is fine and the send wasn't. Under `AT_LEAST_ONCE` there is no transaction to abort and no replay to offer, so `RETRY` on that path is treated as `FAIL`.

::callout{color="warning" icon="i-lucide-alert-triangle"}
The deserialization and processing defaults are **fail-fast** (`LogAndFail*`). A malformed record stops the application until you choose a continue-or-DLQ policy — StoatFlow does not silently drop data unless you opt in.
::

::callout{color="info" icon="i-lucide-info"}
**On Kubernetes, a `FAIL` shutdown *is* a restart — and it will fail again.** The pod exits, the kubelet starts it, the offending record is still at the same offset, and the same handler returns `FAIL`. That is the design: what you get is not recovery but a loud `CrashLoopBackOff` and an alert. Choose it deliberately — it is the right answer when a record you cannot read means something upstream is broken and processing *should* halt until a human looks. If you would rather keep running, handle the poison record: `CONTINUE`, or dead-letter it.

This now describes the **asynchronous production** path too. It used to be the exception — the offsets had already advanced by the time `FAIL` was applied, so the restart resumed *past* the record and stopped once, after the loss. `FAIL` now holds the offsets on every path, so the loop above is what you get there as well: loud, and with the poison still on the topic where you can inspect it.
::

## Built-in handlers

All built-in handlers live in `io.stoatflow.core.exception`.

| Handler class                                    | Stage           | Behaviour                                                                                                                               |
| ------------------------------------------------ | --------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| `LogAndFailDeserializationExceptionHandler`      | Deserialization | Log at error, then `FAIL` (default)                                                                                                     |
| `LogAndContinueDeserializationExceptionHandler`  | Deserialization | Log at warn, then `CONTINUE` (record skipped)                                                                                           |
| `DeadLetterQueueDeserializationExceptionHandler` | Deserialization | Log at warn, emit DLQ record, then `CONTINUE`                                                                                           |
| `LogAndFailProcessingExceptionHandler`           | Processing      | Log at error, then `FAIL` (default)                                                                                                     |
| `LogAndContinueProcessingExceptionHandler`       | Processing      | Log at warn, then `CONTINUE` (record skipped)                                                                                           |
| `DeadLetterQueueProcessingExceptionHandler`      | Processing      | Log at warn, emit DLQ record, then `CONTINUE`                                                                                           |
| `DefaultProductionExceptionHandler`              | Production      | Retry retriable send errors; `FAIL` on serialization and non-retriable send errors; route to DLQ first if a `dlqTopic` is set (default) |

`DefaultProductionExceptionHandler` is the only built-in production handler; it covers retry-vs-fail classification and optional DLQ routing in one place, so you configure behaviour by whether you pass it a `dlqTopic` rather than by swapping the class.

## Configuring a handler in code

Set handlers through `streamsConfigOverrides` on the `:runtime` builder. The same builder methods exist on `StreamsConfig.builder(...)` if you run on `:core` directly.

This example routes deserialization failures to a DLQ topic (skip + capture), keeps the fail-fast processing default, and adds a DLQ topic for production failures:

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.exception.DeadLetterQueueDeserializationExceptionHandler
import io.stoatflow.core.exception.DefaultProductionExceptionHandler
import io.stoatflow.runtime.StoatFlowRuntime
import org.apache.kafka.common.serialization.Serdes

val runtime = StoatFlowRuntime.fromConfig(
    topologyBuilder = { buildTopology(it) },
    configure = {
        streamsConfigOverrides {
            defaultKeySerde(Serdes.String())
            defaultValueSerde(Serdes.String())

            deserializationExceptionHandler(
                DeadLetterQueueDeserializationExceptionHandler(dlqTopic = "my-app.deserialization-errors.dlq"),
            )
            productionExceptionHandler(
                DefaultProductionExceptionHandler(dlqTopic = "my-app.production-errors.dlq"),
            )
        }
    },
)
runtime.start()
runtime.awaitTermination()
```

```java [Java]
import io.stoatflow.core.exception.DeadLetterQueueDeserializationExceptionHandler;
import io.stoatflow.core.exception.DefaultProductionExceptionHandler;
import io.stoatflow.runtime.StoatFlowRuntime;
import org.apache.kafka.common.serialization.Serdes;

var runtime = StoatFlowRuntime.fromConfig(
    Main::buildTopology,
    builder -> builder.streamsConfigOverrides(cfg -> {
        cfg.defaultKeySerde(Serdes.String());
        cfg.defaultValueSerde(Serdes.String());

        cfg.deserializationExceptionHandler(
            new DeadLetterQueueDeserializationExceptionHandler("my-app.deserialization-errors.dlq"));
        cfg.productionExceptionHandler(
            new DefaultProductionExceptionHandler("my-app.production-errors.dlq"));
    })
);
runtime.start();
runtime.awaitTermination();
```
::

To skip bad records without a DLQ, swap in the log-and-continue handler instead:

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.exception.LogAndContinueDeserializationExceptionHandler

streamsConfigOverrides {
    deserializationExceptionHandler(LogAndContinueDeserializationExceptionHandler())
}
```

```java [Java]
import io.stoatflow.core.exception.LogAndContinueDeserializationExceptionHandler;

builder.streamsConfigOverrides(cfg ->
    cfg.deserializationExceptionHandler(new LogAndContinueDeserializationExceptionHandler()));
```
::

The DLQ handlers take optional flags: `includeStackTrace` (default `true`) controls whether the full stack trace is written to a header, and the deserialization handler additionally takes `includeOriginalHeaders` (default `true`) to copy the source record's headers onto the DLQ record under a prefix.

## Configuring a handler in YAML

The `:runtime` module can resolve handlers by **fully qualified class name** under `stoatflow.*`. Builder overrides (above) take precedence over YAML; YAML takes precedence over the defaults.

```yaml
stoatflow:
  application-id: my-app
  bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVERS:-localhost:9092}

  deserialization-exception-handler: io.stoatflow.core.exception.LogAndContinueDeserializationExceptionHandler
  processing-exception-handler: io.stoatflow.core.exception.LogAndFailProcessingExceptionHandler
  production-exception-handler: io.stoatflow.core.exception.DefaultProductionExceptionHandler
```

::callout{color="warning" icon="i-lucide-info"}
**YAML can only configure handlers with a public no-arg constructor.** The DLQ handlers require a `dlqTopic` constructor argument, so they cannot be set from YAML directly — configure them in code via `streamsConfigOverrides` (above), or write a thin no-arg subclass that hard-codes the topic and reference *that* class from YAML. A class with no matching no-arg constructor fails at startup with a clear instantiation error.
::

## DLQ records and headers

When a DLQ handler fires, it builds a `ProducerRecord` targeting your configured `dlqTopic` and returns it to the engine. That record is **sent through the same transactional producer as your normal output**, committed atomically on the next commit barrier — so a DLQ record exists if and only if the rest of that epoch committed. No separate producer, no lost-or-duplicated error records.

::callout{color="warning" icon="i-lucide-alert-triangle"}
**One case takes a different route: an asynchronous production failure aborts and replays the epoch.** When the broker rejects a record that was already sent — classically, one exceeding the topic's `max.message.bytes` — the rejection poisons the in-flight transaction and nothing in it can commit. Rather than advancing past the epoch, the runtime **holds the source offsets**: it commits a secondary transaction carrying only the poison's DLQ record, quarantines the poison's source coordinates, restarts the engine in place, and reprocesses the epoch with exactly that record skipped. The other records in the epoch are processed again and commit normally.

**Loss is bounded to the poison record's own outputs *and its state contribution* — not zero.** Skipping a source record skips *all* of its outputs, so a record that fans out to several sinks loses the good sends with the bad one; it is skipped before deserialization, so it never updates state either, and a `count()` over its key stays permanently one short. And a poison derived from **accumulated state** (an aggregate that outgrew `max.message.bytes`) does not converge: skipping the triggering record does not shrink the accumulator, so the next record on that key reproduces it. Each replay costs a full engine restart and consumes `stoatflow.commit-barrier.max-poison-replays` (default 10 per minute); that budget is in-memory, so exhausting it ends the process and the pod restarts with a clean budget — `CONTINUE` is bounded **per process**, not end to end.

**Two cases stop immediately instead**, with the offsets held so an operator restart replays the epoch: a `FAIL` verdict, and a `CONTINUE` on a poison with no attributable source record (emitted by a punctuator, a window close, a suppression flush, a timer or a scheduled source — those planes fire on their own schedule and would re-poison every attempt, so the runtime names the plane and stops). A `RETRY` on one of those planes replays as usual: it says the send failed, not the emission.

Alert on `stoatflow.dlq.poison.replays.total` **and** `stoatflow.dlq.poison.replay.budget.exhausted.total` — the first stops incrementing exactly when the budget runs out, which is what the second catches; `stoatflow.dlq.poison.quarantined.total` and `stoatflow.dlq.poison.quarantine.size` show what is being skipped. Prevent the case entirely by sizing the target topic's `max.message.bytes` for your largest output, and size the DLQ topic at least as large as every topic that feeds it. Full rationale, including how Kafka Streams behaves on the same scenario: [Error-handling model](https://stoatflow.io/docs/concepts/error-handling-model#dead-letter-queue-semantics).
::

The original key and value bytes are preserved where available:

- **Deserialization** failures carry the **raw source bytes** verbatim (key and value), since deserialization never succeeded.
- **Processing** failures serialize the in-flight key and value with `toString().toByteArray()` — the raw bytes are no longer available after the record was deserialized for processing.
- **Production** failures carry the (already-serialized) record bytes that failed to send — **except a size rejection**, which carries no value at all.

### Size rejections carry no value

When the broker refuses an output record with `RecordTooLargeException`, the dead letter is emitted with a **null value**, three `__stoatflow.errors.value.*` markers, and a capped stack trace. Carrying the refused value would make the dead letter *strictly larger* than the record the broker just rejected — the original headers plus ten-odd metadata headers go on top of it — so a DLQ topic sized like the output topic would reject it too, and that second rejection surfaces only as a later commit-time failure with no dead letter written at all.

The payload is still recoverable: the dead letter records `(__stoatflow.errors.topic, .partition, .offset)`, so seek to those coordinates on the **source** topic — while that topic's retention lasts. A short `retention.ms` on a busy source topic turns this recovery path into a dead link, so size the retention against your mean time-to-response.

Size the DLQ topic for the metadata too. The `__stoatflow.errors.*` header set is roughly 600 bytes before the stack trace, so a DLQ topic wants a few KiB of `max.message.bytes` no matter how small your values are.

### Header reference

All DLQ headers use the `__stoatflow.errors.*` namespace. The double-underscore prefix follows the Kafka convention for system headers, so DLQ tooling that filters `__`-prefixed headers must opt in to surface these.

| Header key                                | Present on                          | Value                                                                                                      |
| ----------------------------------------- | ----------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| `__stoatflow.errors.type`                 | all                                 | `DESERIALIZATION`, `PROCESSING`, or `PRODUCTION`                                                           |
| `__stoatflow.errors.exception`            | all                                 | Exception class name                                                                                       |
| `__stoatflow.errors.message`              | all                                 | Exception message                                                                                          |
| `__stoatflow.errors.stacktrace`           | all (if `includeStackTrace`)        | Stack trace — full, except capped on the size-rejection path                                               |
| `__stoatflow.errors.topic`                | all                                 | Original source topic                                                                                      |
| `__stoatflow.errors.partition`            | all                                 | Original source partition                                                                                  |
| `__stoatflow.errors.offset`               | all                                 | Original source offset                                                                                     |
| `__stoatflow.errors.component`            | deser, production                   | `KEY` / `VALUE` (deser); `KEY_SERIALIZATION` / `VALUE_SERIALIZATION` / `SEND` / `PARTITIONER` (production) |
| `__stoatflow.errors.processor`            | processing, production              | Failing processor name; sink-node id on multi-sink topologies                                              |
| `__stoatflow.errors.timestamp`            | processing                          | Record timestamp                                                                                           |
| `__stoatflow.errors.target.topic`         | production                          | The topic the record was being sent to                                                                     |
| `__stoatflow.errors.value.omitted`        | production, size rejections         | `true` — the value was dropped rather than carried                                                         |
| `__stoatflow.errors.value.size`           | production, size rejections         | Serialized size in bytes of the omitted value                                                              |
| `__stoatflow.errors.value.omitted.reason` | production, size rejections         | Why it was dropped (`record-too-large`)                                                                    |
| `__stoatflow.errors.original.<key>`       | deser (if `includeOriginalHeaders`) | Each original source header, copied under this prefix                                                      |

The deserialization handler is the only one that prefixes the failed record's own headers, and it appends them **last**. The processing and production handlers copy them **verbatim and first**, so `lastHeader()` on a `__stoatflow.errors.*` key always resolves to the error metadata rather than to a colliding original.

Inspect a DLQ record with the console consumer:

```bash
kafka-console-consumer.sh --bootstrap-server localhost:9092 \
  --topic my-app.deserialization-errors.dlq --from-beginning \
  --property print.headers=true --property print.key=true
```

## Production-error classification

The production stage distinguishes errors that are worth retrying from those that are not. `DefaultProductionExceptionHandler` treats Kafka client `RetriableException`s and `TimeoutException` (including when they appear as the `cause`) as retriable and returns `RETRY`; serialization errors and non-retriable send errors return `FAIL` (routing to the DLQ first if a topic is configured).

Underneath, StoatFlow classifies producer exceptions into three categories that determine engine-level behaviour:

| Category     | Meaning                                                                                                                                                                         |
| ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `FATAL`      | The engine cannot recover — triggers shutdown regardless of handler. Includes producer-fencing, authentication/authorization, serialization, and broker-incompatibility errors. |
| `ABORTABLE`  | Routed to your `productionExceptionHandler.handle(...)`; the handler's decision is respected.                                                                                   |
| `SUPPRESSED` | Trace-logged only; a follow-up after a prior abort (e.g. `TransactionAbortedException`).                                                                                        |

Because StoatFlow runs as a [single instance](https://stoatflow.io/docs/concepts/architecture), there is no task to migrate a fenced producer to — so producer-fencing and related errors that Kafka Streams would treat as recoverable task migrations are `FATAL` here. The application exits and a fresh process resumes from the last committed barrier.

## Custom handlers

Implement the handler interface for the stage you want to control. The `handle` methods use the Kafka Streams-compatible parameter order `(context, record, exception)`: the context is the KS-shaped `ErrorHandlerContext`, and the record is the in-flight `Record` (processing), raw `ConsumerRecord<ByteArray, ByteArray>` (deserialization), or `ProducerRecord<ByteArray?, ByteArray?>` (production). A handler returns a response object combining the `CONTINUE` / `FAIL` decision with an optional list of DLQ `ProducerRecord`s — the engine sends those transactionally for you.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.exception.ErrorHandlerContext
import io.stoatflow.core.exception.ProcessingExceptionHandler
import io.stoatflow.core.exception.ProcessingHandlerResponse
import io.stoatflow.core.processor.Record

class SkipNullPointerHandler : ProcessingExceptionHandler {
    override fun handle(
        context: ErrorHandlerContext,
        record: Record<*, *>,
        exception: Exception,
    ): ProcessingHandlerResponse =
        if (exception is NullPointerException) {
            ProcessingHandlerResponse.continueProcessing()
        } else {
            ProcessingHandlerResponse.fail()
        }
}
```

```java [Java]
import io.stoatflow.core.exception.ErrorHandlerContext;
import io.stoatflow.core.exception.ProcessingExceptionHandler;
import io.stoatflow.core.exception.ProcessingHandlerResponse;
import io.stoatflow.core.processor.Record;

public class SkipNullPointerHandler implements ProcessingExceptionHandler {
    @Override
    public ProcessingHandlerResponse handle(
            ErrorHandlerContext context, Record<?, ?> record, Exception exception) {
        if (exception instanceof NullPointerException) {
            return ProcessingHandlerResponse.continueProcessing();
        }
        return ProcessingHandlerResponse.fail();
    }
}
```
::

The `ErrorHandlerContext` carries the source topic, partition, offset, timestamp, headers, failing processor node id, and the raw source key/value bytes, so your handler can log or tag DLQ records precisely. To emit a DLQ record from a custom handler, build a `ProducerRecord<ByteArray?, ByteArray?>` and pass it to `continueProcessing(listOf(...))` or `fail(listOf(...))`.

A custom handler referenced from YAML must have a public no-arg constructor; one configured in code can take any constructor arguments you like.

## Application-level DLQ routing

The exception handlers above catch **failures** — records that throw or fail to (de)serialize. For **business-rule rejections** (a record that deserializes fine but fails your validation), route it explicitly inside the topology with an ordinary sink. This is just normal DSL: branch the invalid records and `.to(...)` a DLQ topic.

::code-tabs{group="lang"}
```kotlin [Kotlin]
val validated = builder.stream<String, Order>("orders")

validated
    .filterNot({ _, order -> order.isValid() }, Named.`as`("invalid"))
    .to("orders.invalid.dlq", Produced.`as`("dlq-sink"))

validated
    .filter({ _, order -> order.isValid() }, Named.`as`("valid"))
    .mapValues({ order -> order.normalize() })
    .to("orders.normalized")
```

```java [Java]
KStream<String, Order> validated = builder.stream("orders");

validated
    .filterNot((k, order) -> order.isValid(), Named.as("invalid"))
    .to("orders.invalid.dlq", Produced.as("dlq-sink"));

validated
    .filter((k, order) -> order.isValid(), Named.as("valid"))
    .mapValues(Order::normalize)
    .to("orders.normalized");
```
::

This pattern gives you full control over the DLQ record shape (you choose the serde and payload) and is the right tool when "this record is bad" is a domain decision rather than an exception. The `stock-tick-filter` example app uses exactly this approach to send validation failures to a DLQ topic in Avro form.

## Related

- [Error-handling model](https://stoatflow.io/docs/concepts/error-handling-model) — the why behind the three stages, CONTINUE-vs-FAIL trade-offs, transactional DLQ delivery, and the one path where *continue* costs an epoch of output.
- [Metrics reference](https://stoatflow.io/docs/reference/metrics-reference) — the `stoatflow.dlq.poison.*` meters and the `stoatflow.dropped.records.total` reason vocabulary.
- [Serdes](https://stoatflow.io/docs/building/serdes) — where deserialization failures originate, and how to make malformed data fail predictably.
- [Event time and watermarks](https://stoatflow.io/docs/concepts/event-time-and-watermarks) — late-record policies, which are configured separately from these exception handlers.
- [Architecture](https://stoatflow.io/docs/concepts/architecture) — what a `FAIL` decision does to the running process and how recovery works on restart.


# State stores

State stores hold the data your stateful operators accumulate — running counts, window aggregates, join buffers, custom per-key state. This page covers the two factories you use to configure them (`Stores` and `Materialized`), how to choose between RocksDB-backed and in-memory variants, how to attach a store to a custom `Processor`, and how to read a store back from a running application.

For the conceptual model — global state, thread-safety under concurrent lanes, and changelog-backed durability — see [State and thread safety](https://stoatflow.io/docs/concepts/state-and-thread-safety).

::tldr-panel
- **`Stores`** is the factory for store *suppliers*: `persistentKeyValueStore(...)`, `inMemoryWindowStore(...)`, etc. Persistent = RocksDB (survives restarts); in-memory = lost on restart.
- **`Materialized`** configures the store behind a DSL operator (`count`, `reduce`, `aggregate`, joins): store name, store type (`withStoreType`), caching, changelog logging.
- **Default is RocksDB.** Most stateful DSL operators pick the right store type for you — you only reach for `Stores` / `withStoreType` to override.
- **Read back** a store from a running engine with `store(StoreQueryParameters.fromNameAndType(...))`, or from tests with `TopologyTestDriver.getKeyValueStore(...)`.
::

## RocksDB vs. in-memory

Every store type comes in two variants, and the choice is the same each time:

|                  | Persistent (RocksDB)                       | In-memory                                      |
| ---------------- | ------------------------------------------ | ---------------------------------------------- |
| Survives restart | Yes — restored from local disk + changelog | No — rebuilt from changelog on every start     |
| Memory footprint | Bounded; data spills to disk               | Whole store lives on the heap                  |
| Exactly-once     | Yes                                        | Yes (changelog still committed on the barrier) |
| Best for         | Production, large state, fast restart      | Tests, small bounded state, prototyping        |

::callout{color="info" icon="i-lucide-info"}
RocksDB is the **default and the recommended production type**. In-memory stores keep their data on the JVM heap, so they're bounded by available memory and have to read the entire changelog at startup — fine for tests and small state, but RocksDB recovers faster and scales past heap size in production. Note that "in-memory store" is *not* "no durability": both variants write a changelog and commit it atomically on the [commit barrier](https://stoatflow.io/docs/concepts/exactly-once). The difference is local persistence and cold-start cost.
::

## The `Stores` factory

`Stores` is a static factory (`io.stoatflow.core.state.Stores`) that returns a *supplier* — a deferred description of a store. Suppliers are handed to `Materialized` (for DSL operators) or to `StreamsBuilder.addStateStore(...)` (for custom Processors). The factory methods, grounded in `Stores.kt`:

| Store type                   | Persistent (RocksDB)                                            | In-memory                                                     |
| ---------------------------- | --------------------------------------------------------------- | ------------------------------------------------------------- |
| Key-value                    | `persistentKeyValueStore(name)`                                 | `inMemoryKeyValueStore(name)`                                 |
| LRU cache                    | —                                                               | `lruMap(name, maxCacheSize)`                                  |
| Window                       | `persistentWindowStore(name, retention, windowSize)`            | `inMemoryWindowStore(name, retention, windowSize)`            |
| Session                      | `persistentSessionStore(name, retention, inactivityGap)`        | `inMemorySessionStore(name, retention, inactivityGap)`        |
| Timestamped KV (KIP-258)     | `persistentTimestampedKeyValueStore(name)`                      | `inMemoryTimestampedKeyValueStore(name)`                      |
| Timestamped window (KIP-258) | `persistentTimestampedWindowStore(name, retention, windowSize)` | `inMemoryTimestampedWindowStore(name, retention, windowSize)` |
| Versioned KV (KIP-889)       | `persistentVersionedKeyValueStore(name, historyRetention)`      | `inMemoryVersionedKeyValueStore(name, historyRetention)`      |

All duration arguments are `java.time.Duration`. Window and session retention must be at least the window size (the factory enforces this and throws `IllegalArgumentException` otherwise). Store names must be non-empty and contain only `[a-zA-Z0-9._-]`.

::callout{color="info" icon="i-lucide-clock"}
**Versioned stores** (`persistentVersionedKeyValueStore` / `inMemoryVersionedKeyValueStore`) keep multiple versions of each value over time for point-in-time lookups and temporal joins. The single `historyRetention` argument is dual-purpose: it's both how long old versions are kept *and* the grace period — out-of-order records older than that window are rejected. Unlike Kafka Streams, which only ships a RocksDB-backed versioned store, StoatFlow offers an in-memory variant too.
::

## `Materialized` — stores behind DSL operators

Stateful DSL operators (`count`, `reduce`, `aggregate`, windowed counts, joins) take a `Materialized` to describe their store. The most common forms (grounded in `Materialized.kt`):

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.state.StateStore
import io.stoatflow.core.state.StoreType
import io.stoatflow.core.topology.Materialized
import org.apache.kafka.common.serialization.Serdes

// Named store, default (RocksDB) type, with serdes
val counts =
    Materialized.`as`<String, Long, StateStore>("word-counts")
        .withKeySerde(Serdes.String())
        .withValueSerde(Serdes.Long())

// Pick the store type explicitly (clears any explicit supplier)
val inMem =
    Materialized.`as`<String, Long, StateStore>("counts")
        .withStoreType(StoreType.IN_MEMORY)

// Auto-generated store name, store type only
val autoNamed = Materialized.`as`<String, Long, StateStore>(StoreType.ROCKS_DB)
```

```java [Java]
import io.stoatflow.core.state.StateStore;
import io.stoatflow.core.state.StoreType;
import io.stoatflow.core.topology.Materialized;
import org.apache.kafka.common.serialization.Serdes;

// Named store, default (RocksDB) type, with serdes
var counts =
    Materialized.<String, Long, StateStore>as("word-counts")
        .withKeySerde(Serdes.String())
        .withValueSerde(Serdes.Long());

// Pick the store type explicitly (clears any explicit supplier)
var inMem =
    Materialized.<String, Long, StateStore>as("counts")
        .withStoreType(StoreType.IN_MEMORY);

// Auto-generated store name, store type only
var autoNamed = Materialized.<String, Long, StateStore>as(StoreType.ROCKS_DB);
```
::

You can also hand `Materialized.as(...)` a supplier directly when you need parameters the DSL doesn't expose (window retention overrides, an explicit window/session/versioned supplier):

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.state.Stores
import io.stoatflow.core.topology.Materialized
import java.time.Duration

// Explicit persistent window store supplier
val windowed =
    Materialized.`as`<String, Long, StateStore>(
        Stores.persistentWindowStore("hourly", Duration.ofHours(2), Duration.ofHours(1)),
    )

// Explicit in-memory key-value supplier
val cache =
    Materialized.`as`<String, Long, StateStore>(Stores.inMemoryKeyValueStore("cache"))
```

```java [Java]
import io.stoatflow.core.state.Stores;
import io.stoatflow.core.topology.Materialized;
import java.time.Duration;

// Explicit persistent window store supplier
var windowed =
    Materialized.<String, Long, StateStore>as(
        Stores.persistentWindowStore("hourly", Duration.ofHours(2), Duration.ofHours(1)));

// Explicit in-memory key-value supplier
var cache =
    Materialized.<String, Long, StateStore>as(Stores.inMemoryKeyValueStore("cache"));
```
::

### Store type precedence

When more than one store source is set on a `Materialized`, resolution follows a fixed order (ADR-029):

1. `withStoreType(...)` (the `DslStoreSuppliers` / `StoreType`) — wins absolutely, and clears any explicit supplier.
2. An explicit supplier passed to `Materialized.as(supplier)`.
3. Default — RocksDB (matching Kafka Streams).

### `Materialized` options

| Method                                                     | Effect                                                                                                                                                                                                                                                           |
| ---------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `withKeySerde(serde)` / `withValueSerde(serde)`            | Serdes for the store (and its changelog). Falls back to the configured defaults if unset.                                                                                                                                                                        |
| `withStoreType(StoreType.IN_MEMORY | ROCKS_DB)`            | Selects store type; takes precedence over any explicit supplier.                                                                                                                                                                                                 |
| `withRetention(duration)`                                  | Overrides retention for windowed stores. Must be at least window size + grace.                                                                                                                                                                                   |
| `withCachingEnabled()` / `withCachingEnabled(config)`      | Compacts multiple writes to the same key within a barrier interval; only the final value per key is emitted downstream via `toStream()`.                                                                                                                         |
| `withCachingDisabled()`                                    | Emits every intermediate update downstream.                                                                                                                                                                                                                      |
| `withLoggingEnabled()` / `withLoggingEnabled(topicConfig)` | Forces the changelog topic on, optionally with custom topic config (e.g. `retention.ms`).                                                                                                                                                                        |
| `withLoggingDisabled()`                                    | Disables the changelog for this store — state is then only locally persisted (RocksDB) and **cannot be recovered from Kafka** after a crash. Use only for state you can recompute.                                                                               |
| `withRecordHeaders()` / `withoutRecordHeaders()`           | Persists the processing record's `Headers` alongside the value (KIP-1271/KIP-1285). See [record headers in state stores](https://stoatflow.io/#record-headers-in-state-stores). Explicit opt-in or opt-out always beats the global `stoatflow.dsl.store-format`. |

::callout{color="info" icon="i-lucide-info"}
**Caching only affects what flows *downstream*.** Whether or not caching is enabled, changelog writes are always compacted to the final value per key per barrier interval. `withCachingDisabled()` is what you want when a downstream consumer needs to observe every intermediate update (as the [word-count walkthrough](https://stoatflow.io/docs/getting-started/first-app) shows — counts climbing 1, 2, 3 rather than only the final tally). When unset, caching follows the global default.
::

### Worked example: a counting KTable

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.state.StateStore
import io.stoatflow.core.topology.Grouped
import io.stoatflow.core.topology.Materialized
import io.stoatflow.core.topology.Named
import io.stoatflow.core.topology.StreamsBuilder
import org.apache.kafka.common.serialization.Serdes

fun buildTopology(builder: StreamsBuilder) {
    builder
        .stream<String, String>("words")
        .groupBy({ _, word -> word }, Grouped.`as`("group-by-word"))
        .count(
            Named.`as`("count"),
            Materialized.`as`<String, Long, StateStore>("word-counts")
                .withKeySerde(Serdes.String())
                .withValueSerde(Serdes.Long()),
        )
}
```

```java [Java]
import io.stoatflow.core.state.StateStore;
import io.stoatflow.core.topology.Grouped;
import io.stoatflow.core.topology.Materialized;
import io.stoatflow.core.topology.Named;
import io.stoatflow.core.topology.StreamsBuilder;
import org.apache.kafka.common.serialization.Serdes;

static void buildTopology(StreamsBuilder builder) {
    builder
        .<String, String>stream("words")
        .groupBy((key, word) -> word, Grouped.as("group-by-word"))
        .count(
            Named.as("count"),
            Materialized.<String, Long, StateStore>as("word-counts")
                .withKeySerde(Serdes.String())
                .withValueSerde(Serdes.Long()));
}
```
::

## Attaching a store to a custom Processor

For the Processor API you create a store builder yourself, register it on the `StreamsBuilder` with `addStateStore(...)`, and **connect it to the processor** by naming it in the `process(...)` call — registering alone is not enough. Inside the processor you then look it up with `context.getStateStore(name)`. The builder comes from `Stores.keyValueStoreBuilder(supplier, keySerde, valueSerde)` (and the `windowStoreBuilder` / `sessionStoreBuilder` / `versionedKeyValueStoreBuilder` equivalents).

The alternative is to declare the store from `ProcessorSupplier.stores()`, which connects it automatically — see [Processor API](https://stoatflow.io/docs/building/processor-api#attaching-state-stores).

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.processor.Processor
import io.stoatflow.core.processor.ProcessorContext
import io.stoatflow.core.processor.Record
import io.stoatflow.core.state.KeyValueStore
import io.stoatflow.core.state.Stores
import io.stoatflow.core.topology.StreamsBuilder
import org.apache.kafka.common.serialization.Serdes

fun buildTopology(builder: StreamsBuilder) {
    builder.addStateStore(
        Stores.keyValueStoreBuilder(
            Stores.persistentKeyValueStore("seen-counts"),
            Serdes.String(),
            Serdes.Long(),
        ),
    )

    builder
        .stream<String, String>("events")
        .process({ CountingProcessor("seen-counts") }, "seen-counts")
}

class CountingProcessor(
    private val storeName: String,
) : Processor<String, String, String, Long> {
    private lateinit var store: KeyValueStore<String, Long>

    override fun init(context: ProcessorContext<String, Long>) {
        store = context.getStateStore(storeName)
    }

    override fun process(record: Record<String, String>) {
        val next = (store.get(record.key) ?: 0L) + 1
        store.put(record.key, next)
    }
}
```

```java [Java]
import io.stoatflow.core.processor.Processor;
import io.stoatflow.core.processor.ProcessorContext;
import io.stoatflow.core.processor.Record;
import io.stoatflow.core.state.KeyValueStore;
import io.stoatflow.core.state.Stores;
import io.stoatflow.core.topology.StreamsBuilder;
import org.apache.kafka.common.serialization.Serdes;

static void buildTopology(StreamsBuilder builder) {
    builder.addStateStore(
        Stores.keyValueStoreBuilder(
            Stores.persistentKeyValueStore("seen-counts"),
            Serdes.String(),
            Serdes.Long()));

    builder
        .<String, String>stream("events")
        .process(() -> new CountingProcessor("seen-counts"), "seen-counts");
}

class CountingProcessor implements Processor<String, String, String, Long> {
    private final String storeName;
    private KeyValueStore<String, Long> store;

    CountingProcessor(String storeName) {
        this.storeName = storeName;
    }

    @Override
    public void init(ProcessorContext<String, Long> context) {
        store = context.getStateStore(storeName);
    }

    @Override
    public void process(Record<String, String> record) {
        long next = (store.get(record.key()) == null ? 0L : store.get(record.key())) + 1;
        store.put(record.key(), next);
    }
}
```
::

::callout{color="warning" icon="i-lucide-lock"}
**A processor may only reach a store it declared.** `getStateStore(name)` throws `StoreNotConnectedException` if `name` is not connected to that node — the Kafka Streams rule, and a fatal error rather than something `processing.exception.handler` can skip. Two exemptions need no declaration: **global stores** (`addGlobalStore(...)` and `globalTable(...)`) and **read-only stores** (`addReadOnlyStateStore(...)`).
::

A global store reached **without** declaring it is handed to the processor **read-only** — `put`/`delete` throw `UnsupportedOperationException`, as in Kafka Streams. Its own state-update processor writes to it normally. This covers every store family: plain key-value, window and session stores, and equally the timestamped, versioned and headers-aware ones. The store keeps its declared type, so `getStateStore<TimestampedKeyValueStore<K, V>>(name)` still returns a `TimestampedKeyValueStore`; only its writes throw.

To write to a global store from your own processor, **declare it** — `process(supplier, "my-global")` or `ProcessorSupplier.stores()` — which opts into write access. That is a StoatFlow extension (Kafka Streams rejects the declaration itself): on a single instance all state is already global, so a "global" store is an ordinary store, and the read-only default exists for Kafka Streams portability rather than safety.

::callout{color="info" icon="i-lucide-shield-check"}
Within a processor, state access is safe without any locking of your own: records with the same key always run on the same processing context in arrival order, so per-key read-modify-write is serial. The store also exposes atomic `compute(key, fn)` and `merge(key, value, fn)` for read-modify-write in a single call — atomic **within a commit epoch**. See [State and thread safety](https://stoatflow.io/docs/concepts/state-and-thread-safety) for the full model, including the cross-key key-lock utility and the two shapes where key affinity does not hold: [a many-to-one re-key into a Processor API node](https://stoatflow.io/docs/building/processor-api#key-affinity-after-a-re-key), which StoatFlow refuses to compile rather than let you lock your way around, and storing under a key that is not the record key, which is computed at runtime and so cannot be caught at build time.
::

## Record headers in state stores

A store can persist the processing record's `Headers` alongside the value — a schema id, a
`traceparent`, tenant or routing metadata, lineage — so it survives aggregation and comes back through
interactive queries and on the changelog. This is Kafka's KIP-1271 (storage) and KIP-1285 (DSL opt-in),
and StoatFlow implements both.

Opt in per store, or globally:

::code-tabs{group="lang"}
```kotlin [Kotlin]
builder
    .stream<String, String>("orders")
    .groupByKey()
    .count(
        Materialized.`as`<String, Long, StateStore>("order-counts")
            .withKeySerde(Serdes.String())
            .withValueSerde(Serdes.Long())
            .withRecordHeaders(),
    )
```

```java [Java]
builder
    .stream("orders", Consumed.with(Serdes.String(), Serdes.String()))
    .groupByKey()
    .count(
        Materialized.<String, Long, StateStore>as("order-counts")
            .withKeySerde(Serdes.String())
            .withValueSerde(Serdes.Long())
            .withRecordHeaders());
```
::

```yaml
stoatflow:
  dsl:
    store-format: HEADERS   # every DSL aggregation persists headers by default
```

Headers-aware stores are **timestamped-only** (as in Kafka Streams), covering timestamped key-value,
versioned, timestamped window and session stores. Join buffers are out of scope in both engines. Query
them with the `QueryableStoreTypes.*WithHeaders()` views.

### Turning record headers on

Flipping the flag on is a **lazy in-place upgrade** — no migration pause, no I/O spike. The store opens
with a second RocksDB keyspace: reads check the new one and fall back to the old, converting an old-format
hit on the way out (old entries read back with empty headers, which is correct — they predate the
feature); writes land in the new keyspace and clear the old copy, so each key upgrades itself the first
time it is touched. Windowed and session stores do this per time segment.

### Turning record headers off again

Turning them **off** is not the mirror image, and it is worth knowing before you flip the flag.

The new keyspace is created eagerly the first time the store opens in headers mode, so it exists on disk
even if nothing was ever written to it — and RocksDB refuses to open a database without naming every
keyspace present. StoatFlow therefore classifies the mismatch at startup, before the store opens, and
treats it as a **declared format change, not corruption**:

| Situation                                                   | What happens                                                                                                                                                                                                                   |
| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Nothing was written in headers mode                         | The empty keyspace is **dropped in place**. Free: no acknowledgment, no restore, not a byte of data touched.                                                                                                                   |
| Data has migrated                                           | Startup **refuses** with an actionable message. Nothing on disk is touched, so an accidental flag removal costs a restart, never data.                                                                                         |
| Data has migrated, and you acknowledge                      | Set `stoatflow.state.format-downgrade: wipe-and-restore`. That store's local state is deleted and rebuilt from its changelog — one full restore, bounded by changelog size.                                                    |
| Data has migrated, and the store has **no recovery source** | Refused **unconditionally**. The acknowledgment does not apply: with the changelog disabled (per store via `withLoggingDisabled()`, or globally) there is nothing to rebuild from, so wiping would be unrecoverable data loss. |

::callout{color="warning" icon="i-lucide-alert-triangle"}
**A downgrade sheds persisted headers, and turning the flag back on does not bring them back.** The old
format cannot hold headers, so they are dropped from the local store; and re-enabling headers upgrades
lazily, converting legacy values with *empty* headers. Only a forced full restore repacks them. Kafka
Streams behaves the same way — it refuses the downgrade outright and leaves you to delete local state by
hand; the acknowledgment key is StoatFlow's addition, so the operation is one config line rather than a
manual reset.
::

The outcome is metered as `stoatflow.store.format.downgrade.total{outcome=dropped-empty|wiped|refused}`.
If you take the rebuild, the [restoration tuning guide](https://stoatflow.io/docs/operating/tuning) covers making it fast.

## Reading a store back

A store registered in your topology is queryable by name. The read-only views are typed via `QueryableStoreTypes` and selected with `StoreQueryParameters.fromNameAndType(...)`:

| Store kind         | `QueryableStoreTypes` factory | Read-only view                                   |
| ------------------ | ----------------------------- | ------------------------------------------------ |
| Key-value          | `keyValueStore()`             | `ReadOnlyKeyValueStore<K, V>`                    |
| Window             | `windowStore()`               | `ReadOnlyWindowStore<K, V>`                      |
| Session            | `sessionStore()`              | `ReadOnlySessionStore<K, AGG>`                   |
| Timestamped KV     | `timestampedKeyValueStore()`  | `ReadOnlyKeyValueStore<K, ValueAndTimestamp<V>>` |
| Timestamped window | `timestampedWindowStore()`    | `ReadOnlyWindowStore<K, ValueAndTimestamp<V>>`   |

`ReadOnlyKeyValueStore` gives you `get(key)`, `containsKey(key)`, `all()`, `reverseAll()`, `range(from, to)`, `reverseRange(from, to)`, `prefixScan(prefix, prefixKeySerializer)`, and `approximateNumEntries()`. Iterators must be closed after use.

Range bounds follow Kafka Streams semantics (KIP-763): a `null` bound is open-ended (both `null` ≡ `all()`), `reverseRange` takes its bounds in the same ascending `(low, high)` order as `range`, and an inverted range (`from > to` on the serialized key bytes) returns an empty iterator with a WARN. The same null-bound rules apply to `ReadOnlyWindowStore.fetch(keyFrom, keyTo, …)` and the `ReadOnlySessionStore` key-range queries.

::callout{color="info" icon="i-lucide-info"}
StoatFlow's single-instance model means **all state is globally accessible** — there is no partition routing and no cross-instance RPC. Consequently `StoreQueryParameters` has no `withPartition(...)`; a query against any key resolves locally. Use `enableStaleStores()` to allow reads while the application is still restoring.
::

### Interactive queries against the running engine

The query entry point is `store(...)` on the core `StoatFlow` engine, which returns the requested read-only view:

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.state.QueryableStoreTypes
import io.stoatflow.core.state.ReadOnlyKeyValueStore
import io.stoatflow.core.state.StoreQueryParameters

val store: ReadOnlyKeyValueStore<String, Long> =
    stoatflow.store(
        StoreQueryParameters.fromNameAndType(
            "word-counts",
            QueryableStoreTypes.keyValueStore(),
        ),
    )

val count: Long? = store.get("hello")
```

```java [Java]
import io.stoatflow.core.state.QueryableStoreTypes;
import io.stoatflow.core.state.ReadOnlyKeyValueStore;
import io.stoatflow.core.state.StoreQueryParameters;

ReadOnlyKeyValueStore<String, Long> store =
    stoatflow.store(
        StoreQueryParameters.fromNameAndType(
            "word-counts",
            QueryableStoreTypes.keyValueStore()));

Long count = store.get("hello");
```
::

`store(...)` throws `IllegalStateException` unless the application is `RUNNING` (or `RESTORING` / `VALIDATING_STATE` when `enableStaleStores()` was set), `IllegalArgumentException` if the named store does not exist, and `IllegalArgumentException` if the store's type does not match the requested `QueryableStoreType`. `storeNames()` lists every registered store name for discovery.

`store(...)` and `storeNames()` are defined on the core `StoatFlow` engine, so programmatic interactive queries need a handle to that engine. A `:core` application holds it directly. The `:runtime` wrapper (`StoatFlowRuntime.fromConfig`) currently keeps its engine instance private and exposes no public query accessor, so typed in-process interactive queries are not available from a `:runtime` deployment today — use the `:core` API directly if you need them.

### Reading a store in a test

In `TopologyTestDriver` (the in-memory [test driver](https://stoatflow.io/docs/building/testing)), read stores back directly with `getKeyValueStore(name)` (and `getWindowStore` / `getSessionStore`). Call `commitBarrier()` first if the store has caching enabled, so cached emissions are flushed:

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.testutils.TopologyTestDriver

val driver = TopologyTestDriver.fromBuilder(builder)
// ... pipe input ...
driver.commitBarrier() // flush cached writes before reading

val store = driver.getKeyValueStore<String, Long>("word-counts")
assertThat(store.get("the")).isEqualTo(3L)
```

```java [Java]
import io.stoatflow.testutils.TopologyTestDriver;
import io.stoatflow.core.state.KeyValueStore;

var driver = TopologyTestDriver.fromBuilder(builder);
// ... pipe input ...
driver.commitBarrier(); // flush cached writes before reading

KeyValueStore<String, Long> store = driver.getKeyValueStore("word-counts");
assertThat(store.get("the")).isEqualTo(3L);
```
::

## Related

- [State and thread safety](https://stoatflow.io/docs/concepts/state-and-thread-safety) — global state, concurrent-access correctness, durability
- [Aggregations](https://stoatflow.io/docs/building/aggregations) and [Windowing](https://stoatflow.io/docs/building/windowing) — operators that materialize stores
- [Serdes](https://stoatflow.io/docs/building/serdes) — the serialization the store uses for keys, values, and changelog
- [Testing](https://stoatflow.io/docs/building/testing) — reading stores back from the test driver
- [Kafka Streams compatibility matrix](https://stoatflow.io/docs/reference/ks-compatibility-matrix) — store-type parity against Kafka Streams


# Testing topologies

`stoatflow-test-utils` runs your topology synchronously, in-process, with no Kafka broker. You pipe records in, read records out, advance time and watermarks by hand, and read state stores directly for assertions. Because processing is synchronous and single-threaded, tests are deterministic and easy to debug.

::tldr-panel
- **Driver:** `TopologyTestDriver.fromBuilder(builder)` — captures your topology, processors, and state stores.
- **In / out:** `createInputTopic(...).pipeInput(...)` and `createOutputTopic(...).readRecord()`.
- **Time:** `advanceWallClockTime(Duration)` and `advanceWatermark(timestampMs)` drive punctuators, timers, window closes, and scheduled sources.
- **State:** `getKeyValueStore(name)` / `getWindowStore` / `getSessionStore` for direct assertions.
- **Config-driven:** `StoatFlowTestDriver.fromConfig(...)` builds a driver from `application.yaml` on the test classpath — the test-time mirror of `StoatFlowRuntime.fromConfig(...)`.
::

## Add the dependency

`stoatflow-test-utils` is a `testImplementation` dependency. If you load configuration from `application.yaml` in tests (see [Config-driven tests](https://stoatflow.io/#config-driven-tests-stoatflowtestdriver)), also add `stoatflow-test-runtime`.

::code-tabs{group="build"}
```kotlin [Gradle]
// build.gradle.kts
dependencies {
    testImplementation("io.stoatflow:stoatflow-test-utils:<stoatflow-version>")
    // Config-driven tests (loads application.yaml):
    testImplementation("io.stoatflow:stoatflow-test-runtime:<stoatflow-version>")
}
```

```xml [Maven]
<!-- pom.xml -->
<dependency>
  <groupId>io.stoatflow</groupId>
  <artifactId>stoatflow-test-utils</artifactId>
  <version>${stoatflow.version}</version>
  <scope>test</scope>
</dependency>
<!-- Config-driven tests (loads application.yaml): -->
<dependency>
  <groupId>io.stoatflow</groupId>
  <artifactId>stoatflow-test-runtime</artifactId>
  <version>${stoatflow.version}</version>
  <scope>test</scope>
</dependency>
```
::

The current version is :stoatflow-version. See [Installation](https://stoatflow.io/docs/getting-started/installation) for the repository and version setup.

## A first test

`TopologyTestDriver.fromBuilder(builder)` compiles the topology from a `StreamsBuilder` and wires up the processor and state-store registries automatically — it's the recommended entry point. Then `createInputTopic` / `createOutputTopic` give you typed handles for piping records and reading results.

The serdes you pass to `createInputTopic` / `createOutputTopic` are **applied**, exactly as the engine would: piped keys and values are serialized to wire bytes and re-deserialized through the source-node serde before the first processor sees them, and sink output is deserialized through the serdes you pass at read time. (KS-shaped `createInputTopic(topic, Serializer, Serializer)` / `createOutputTopic(topic, Deserializer, Deserializer)` overloads exist too.) Records flow through the topology **synchronously** on `pipeInput` — by the time `pipeInput` returns, the output is already on the sink queue.

The driver also performs the engine's **boundary key serialization**: wherever a record (or a window-close / suppress / punctuator emission) crosses a sub-topology boundary — where a re-keyed record reaches an aggregation, a join, a materialised table, or an explicit `repartition()` — its key is serialized with the same [boundary key serde](https://stoatflow.io/docs/building/serdes#boundary-key-serdes-lane-assignment) the engine resolves, and an unresolvable boundary raises the same `LaneKeySerializationException` your application would hit in production. A topology that would die on its first record can no longer run green in a unit test.

The driver compiles the topology with the same `topology.sub-topology-split` and `topology.processor-api-key-affinity` settings as the engine, so the boundaries it exercises are the ones your application will actually have — and **topology-validation rules set to `error` fail the driver too**, with no broker. A re-key feeding a store-connected Processor API node is refused in your unit tests rather than in production ([Key affinity after a re-key](https://stoatflow.io/docs/building/processor-api#key-affinity-after-a-re-key)).

One thing the driver cannot show you: it is **single-lane**, so any concurrency property — including the lane-affinity trade-off above — is invisible to it by construction. Those need a real broker.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.topology.StreamsBuilder
import io.stoatflow.testutils.TopologyTestDriver
import org.apache.kafka.common.serialization.Serdes
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.AfterEach
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Test

class UppercaseTopologyTest {
    private lateinit var driver: TopologyTestDriver

    @BeforeEach
    fun setup() {
        val builder = StreamsBuilder()
        builder
            .stream<String, String>("input-topic")
            .mapValues { value -> value.uppercase() }
            .filter { _, value -> value.length > 3 }
            .to("output-topic")

        driver = TopologyTestDriver.fromBuilder(builder, TopologyTestDriver.STRING_CONFIG)
    }

    @AfterEach
    fun teardown() = driver.close()

    @Test
    fun `maps and filters values`() {
        val input = driver.createInputTopic("input-topic", Serdes.String(), Serdes.String())
        val output = driver.createOutputTopic<String, String>("output-topic", Serdes.String(), Serdes.String())

        input.pipeInput("k1", "hello")
        input.pipeInput("k2", "no")   // filtered out (length <= 3 after uppercasing)

        assertThat(output.readRecord()?.value).isEqualTo("HELLO")
        assertThat(output.isEmpty()).isTrue()
    }
}
```

```java [Java]
import io.stoatflow.core.topology.StreamsBuilder;
import io.stoatflow.testutils.TestInputTopic;
import io.stoatflow.testutils.TestOutputTopic;
import io.stoatflow.testutils.TopologyTestDriver;
import org.apache.kafka.common.serialization.Serdes;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;

import static org.assertj.core.api.Assertions.assertThat;

class UppercaseTopologyTest {

    private TopologyTestDriver driver;

    @BeforeEach
    void setup() {
        StreamsBuilder builder = new StreamsBuilder();
        builder.<String, String>stream("input-topic")
            .mapValues(value -> value.toUpperCase())
            .filter((key, value) -> value.length() > 3)
            .to("output-topic");

        driver = TopologyTestDriver.fromBuilder(builder, TopologyTestDriver.STRING_CONFIG);
    }

    @AfterEach
    void teardown() {
        driver.close();
    }

    @Test
    void mapsAndFiltersValues() {
        TestInputTopic<String, String> input =
            driver.createInputTopic("input-topic", Serdes.String(), Serdes.String());
        TestOutputTopic<String, String> output =
            driver.createOutputTopic("output-topic", Serdes.String(), Serdes.String());

        input.pipeInput("k1", "hello");
        input.pipeInput("k2", "no");   // filtered out

        assertThat(output.readRecord().getValue()).isEqualTo("HELLO");
        assertThat(output.isEmpty()).isTrue();
    }
}
```
::

::callout{color="warning" icon="i-lucide-triangle-alert"}
`TopologyTestDriver` is **not thread-safe** — it is designed for single-threaded test execution. Create a fresh instance per test case (`@BeforeEach`) and `close()` it in `@AfterEach` when running parallel test runners.
::

### Picking a config

`fromBuilder(builder)` defaults to `ByteArray` key/value serdes (matching the production `StreamsConfig` defaults). For String-keyed topologies, pass the bundled `TopologyTestDriver.STRING_CONFIG`, which sets `String` default serdes. For anything else, build a `StreamsConfig` explicitly, or use `StoatFlowTestDriver.fromConfig(...)` to load your real `application.yaml` (see below).

`fromBuilder` is overloaded with an optional `initialWallClockTime` (an `Instant`) — useful when a test asserts on absolute wall-clock-based fire times of scheduled sources or processing-time timers.

## Piping input

`TestInputTopic` pipes records into a source topic. The simplest form takes a key and value; overloads add an explicit timestamp and headers, and a `TestRecord` form.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.state.KeyValue

val input = driver.createInputTopic("input-topic", Serdes.String(), Serdes.String())

input.pipeInput("k1", "v1")                       // uses the topic's current timestamp
input.pipeInput("k1", "v2", timestamp = 1_000L)   // explicit event time (ms)
input.pipeInput("k1", null)                        // tombstone (null value)

// Bulk helpers (KeyValue, matching Kafka Streams)
input.pipeKeyValueList(listOf(KeyValue("k1", "a"), KeyValue("k2", "b")))
input.pipeValueList(listOf("x", "y"))              // null keys

// Auto-advancing timestamps: each record steps 1s forward
input.withTimestamp(0L).withAutoAdvance(1_000L)
input.pipeInput("k1", "t0")   // ts = 0
input.pipeInput("k1", "t1")   // ts = 1000
```

```java [Java]
import io.stoatflow.core.state.KeyValue;

TestInputTopic<String, String> input =
    driver.createInputTopic("input-topic", Serdes.String(), Serdes.String());

input.pipeInput("k1", "v1");                  // uses the topic's current timestamp
input.pipeInput("k1", "v2", 1_000L);          // explicit event time (ms)
input.pipeInput("k1", null);                  // tombstone (null value)

// Bulk helpers (KeyValue, matching Kafka Streams)
input.pipeKeyValueList(List.of(KeyValue.pair("k1", "a"), KeyValue.pair("k2", "b")));
input.pipeValueList(List.of("x", "y"));       // null keys

// Auto-advancing timestamps: each record steps 1s forward
input.withTimestamp(0L).withAutoAdvance(1_000L);
input.pipeInput("k1", "t0");   // ts = 0
input.pipeInput("k1", "t1");   // ts = 1000
```
::

::callout{color="info" icon="i-lucide-info"}
`withTimestamp` and `withAutoAdvance` mutate state on the `TestInputTopic` instance that persists across calls. Call `reset()` to restore defaults (auto-advance disabled, timestamp back to the driver's current wall-clock time) if you reuse one input topic across test cases.
::

The event timestamp you pass to `pipeInput` is the record's **event time** — it drives windowing, joins, and watermark-based logic. If the source was configured with a `WatermarkStrategy` that carries a timestamp assigner, the driver applies it the same way the runtime does, overriding the passed timestamp. See [Event time and watermarks](https://stoatflow.io/docs/concepts/event-time-and-watermarks).

## Reading output

`TestOutputTopic` drains records the topology produced to a sink topic. `readRecord()` returns a `TestRecord` (or `null` when empty); there are convenience readers for keys, values, and bulk drains.

::code-tabs{group="lang"}
```kotlin [Kotlin]
val output = driver.createOutputTopic<String, Long>("output-topic", Serdes.String(), Serdes.Long())

val record = output.readRecord()        // TestRecord<String, Long>? — null if empty
record?.key                              // "k1"
record?.value                            // 3L
record?.timestamp                        // event time of the emission

output.readValue()                       // next value only
output.readKeyValue()                    // next KeyValue (or null if empty)
output.readRecordsToList()               // drain all remaining as a list of TestRecord
output.readKeyValuesToList()             // drain all remaining as a list of KeyValues
output.readKeyValuesToMap()              // drain all; last value per key wins
output.getQueueSize()                    // records currently waiting
output.isEmpty()                         // true when drained
```

```java [Java]
TestOutputTopic<String, Long> output =
    driver.createOutputTopic("output-topic", Serdes.String(), Serdes.Long());

TestRecord<String, Long> record = output.readRecord();   // null if empty
record.getKey();        // "k1"
record.getValue();      // 3L
record.getTimestamp();  // event time of the emission

output.readValue();              // next value only
output.readKeyValue();           // next KeyValue (or null if empty)
output.readRecordsToList();      // drain all remaining as a list of TestRecord
output.readKeyValuesToList();    // drain all remaining as a list of KeyValues
output.readKeyValuesToMap();     // drain all; last value per key wins
output.getQueueSize();           // records currently waiting
output.isEmpty();                // true when drained
```
::

::callout{color="info" icon="i-lucide-info"}
Type parameters on `TestOutputTopic` are erased at runtime — they are not validated. If the topology emits a different type than declared, you get a `ClassCastException` only when you read a property. Match the declared `K`/`V` to the actual sink output types.
::

A common assertion for a `KTable`-backed aggregation is the changelog stream: `count()` / `aggregate()` emit the running result per key, so `readKeyValuesToMap()` (last-value-per-key) is the idiomatic way to assert the final state.

## Advancing time

Stateful time-based logic — windowed aggregations with `OnWindowClose`, suppress, punctuators, timers, scheduled sources — fires when the relevant clock advances. By default the **event-time** clock advances automatically to each piped record's timestamp (matching Kafka Streams' `TopologyTestDriver`), so piping a record *past* a window's close already flushes it. You can also drive either clock manually:

- **`advanceWatermark(timestampMs)`** advances **event time**. This fires event-time timers and `STREAM_TIME` punctuators, closes windows past their end + grace, flushes suppressed results, and fires `STREAM_TIME` scheduled sources. The watermark is monotonic — by default a backward request is ignored (a silent no-op), since each `pipeInput` already drives event time forward; an explicit call is only needed to push event time *beyond* the last piped record.
- **`advanceWallClockTime(Duration)`** advances **processing time**. This fires processing-time timers and `WALL_CLOCK_TIME` punctuators, fires `WALL_CLOCK_TIME` (and cron) scheduled sources, and triggers a barrier commit (which flushes any caching-store emissions — see [commitBarrier()](https://stoatflow.io/#flushing-caches-with-commitbarrier)).

::callout{icon="i-lucide-info"}
Set `test.driver.commit-per-pipe=false` in your config `Properties` to opt out of the per-pipe parity: pipes then neither commit the cache nor advance event time, and you control both explicitly via `commitBarrier()` / `advanceWatermark()` (where a backward request throws, the legacy manual-control mode).
::

A windowed aggregation that only emits on window close needs event time pushed past the window's end plus its grace period — either by piping a later-timestamped record, or with an explicit `advanceWatermark`:

::code-tabs{group="lang"}
```kotlin [Kotlin]
// Window [0, 5min), grace 30s — emits only on close
val builder = StreamsBuilder()
builder
    .stream<String, Int>("input")
    .groupByKey()
    .windowedBy(TimeWindows.ofSizeAndGrace(Duration.ofMinutes(5), Duration.ofSeconds(30)))
    .emitStrategy(EmitStrategy.onWindowClose())
    .count()
    .toStream()
    .map { windowedKey, count -> KeyValue(windowedKey.key, count) }
    .to("output")

val driver = TopologyTestDriver.fromBuilder(builder, TopologyTestDriver.STRING_CONFIG)
val input = driver.createInputTopic("input", Serdes.String(), Serdes.Integer())
val output = driver.createOutputTopic<String, Long>("output", Serdes.String(), Serdes.Long())

input.pipeInput("k1", 1, timestamp = 1_000L)
input.pipeInput("k1", 2, timestamp = 2_000L)

assertThat(output.isEmpty()).isTrue()           // window not closed yet

// Advance past window end (5min) + grace (30s) = 330s
driver.advanceWatermark(330_001L)

assertThat(output.readRecord()?.value).isEqualTo(2L)   // two records counted
```

```java [Java]
// Window [0, 5min), grace 30s — emits only on close
StreamsBuilder builder = new StreamsBuilder();
builder.<String, Integer>stream("input")
    .groupByKey()
    .windowedBy(TimeWindows.ofSizeAndGrace(Duration.ofMinutes(5), Duration.ofSeconds(30)))
    .emitStrategy(EmitStrategy.onWindowClose())
    .count()
    .toStream()
    .map((windowedKey, count) -> new KeyValue<>(windowedKey.getKey(), count))
    .to("output");

TopologyTestDriver driver = TopologyTestDriver.fromBuilder(builder, TopologyTestDriver.STRING_CONFIG);
TestInputTopic<String, Integer> input =
    driver.createInputTopic("input", Serdes.String(), Serdes.Integer());
TestOutputTopic<String, Long> output =
    driver.createOutputTopic("output", Serdes.String(), Serdes.Long());

input.pipeInput("k1", 1, 1_000L);
input.pipeInput("k1", 2, 2_000L);

assertThat(output.isEmpty()).isTrue();           // window not closed yet

// Advance past window end (5min) + grace (30s) = 330s
driver.advanceWatermark(330_001L);

assertThat(output.readRecord().getValue()).isEqualTo(2L);   // two records counted
```
::

For session windows and suppress, the runtime pattern is the same one the example apps use: push the watermark past the inactivity gap (plus grace) to close the session and let suppress emit its final result. `getCurrentWatermark()` and `getWallClockTime()` return the current clocks for assertions.

## Reading state stores

For stateful topologies you can read the state store directly instead of (or in addition to) asserting on output records. The accessor matches the store type the DSL operator created.

| Accessor                            | Returns                          | For                                                   |
| ----------------------------------- | -------------------------------- | ----------------------------------------------------- |
| `getKeyValueStore(name)`            | `KeyValueStore<K, V>`            | `count`, `reduce`, `aggregate`, materialized `KTable` |
| `getWindowStore(name)`              | `WindowStore<K, V>`              | time / sliding windowed aggregations                  |
| `getSessionStore(name)`             | `SessionStore<K, V>`             | session windowed aggregations                         |
| `getTimestampedKeyValueStore(name)` | `TimestampedKeyValueStore<K, V>` | when you also want the value's event timestamp        |
| `getTimestampedWindowStore(name)`   | `TimestampedWindowStore<K, V>`   | windowed, with value timestamps                       |
| `getVersionedKeyValueStore(name)`   | `VersionedKeyValueStore<K, V>`   | versioned (point-in-time) stores                      |
| `getStateStore(name)`               | `S : StateStore`                 | custom Processor stores                               |

`getKeyValueStore` transparently unwraps DSL-created timestamped stores, so it returns the raw value `V` (matching Kafka Streams behaviour). The state-store name is the one you set via `Materialized.as("...")` or the store supplier.

::code-tabs{group="lang"}
```kotlin [Kotlin]
val builder = StreamsBuilder()
builder
    .stream<String, String>("input-topic")
    .groupByKey()
    .count(Materialized.`as`<String, Long, StateStore>("count-store"))
    .toStream()
    .to("output-topic")

val driver = TopologyTestDriver.fromBuilder(builder, TopologyTestDriver.STRING_CONFIG)
val input = driver.createInputTopic("input-topic", Serdes.String(), Serdes.String())

input.pipeInput("a", "x")
input.pipeInput("a", "y")
input.pipeInput("b", "z")

val store = driver.getKeyValueStore<String, Long>("count-store")
assertThat(store.get("a")).isEqualTo(2L)
assertThat(store.get("b")).isEqualTo(1L)
assertThat(store.get("missing")).isNull()
```

```java [Java]
StreamsBuilder builder = new StreamsBuilder();
builder.<String, String>stream("input-topic")
    .groupByKey()
    .count(Materialized.<String, Long, StateStore>as("count-store"))
    .toStream()
    .to("output-topic");

TopologyTestDriver driver = TopologyTestDriver.fromBuilder(builder, TopologyTestDriver.STRING_CONFIG);
TestInputTopic<String, String> input =
    driver.createInputTopic("input-topic", Serdes.String(), Serdes.String());

input.pipeInput("a", "x");
input.pipeInput("a", "y");
input.pipeInput("b", "z");

KeyValueStore<String, Long> store = driver.getKeyValueStore("count-store");
assertThat(store.get("a")).isEqualTo(2L);
assertThat(store.get("b")).isEqualTo(1L);
assertThat(store.get("missing")).isNull();
```
::

::callout{color="info" icon="i-lucide-info"}
The test driver does **not** write changelog topics — it runs without a broker, so state writes use a no-op changelog. Tests cover processing and state logic, not state backup / restore. To exercise changelog and recovery behaviour, use a broker-backed integration test (the example apps pair each `...AppTest` with a `...IntegrationTest`).
::

## Flushing caches with commitBarrier()

State stores with caching enabled (via `Materialized.withCachingEnabled()`) suppress intermediate updates and emit only the final value per key on commit. In the runtime that happens on a commit barrier; in the test driver you trigger it explicitly with `commitBarrier()`. `advanceWallClockTime(...)` also triggers one. If a caching-enabled topology emits nothing after `pipeInput`, call `commitBarrier()` (or advance wall-clock time) to flush.

::code-tabs{group="lang"}
```kotlin [Kotlin]
input.pipeInput("a", "x")
input.pipeInput("a", "y")

// Nothing emitted yet — caching holds intermediate updates
assertThat(output.isEmpty()).isTrue()

driver.commitBarrier()                 // flush cached emissions

assertThat(output.readKeyValuesToMap()).containsEntry("a", 2L)
```

```java [Java]
input.pipeInput("a", "x");
input.pipeInput("a", "y");

// Nothing emitted yet — caching holds intermediate updates
assertThat(output.isEmpty()).isTrue();

driver.commitBarrier();                // flush cached emissions

assertThat(output.readKeyValuesToMap()).containsEntry("a", 2L);
```
::

## Triggering scheduled sources

A [scheduled source](https://stoatflow.io/docs/building/scheduled-sources) generates records on a clock rather than from Kafka. You drive it two ways:

- **`triggerScheduledSource(name)`** fires the emitter once, immediately, without advancing time — handy for asserting the emitter logic in isolation.
- **`advanceWallClockTime(...)`** (for `WALL_CLOCK_TIME&#x60; / cron sources) or &#x2A;*`advanceWatermark(...)`** (for `STREAM_TIME` sources) fires every scheduled execution due up to the new time.

The default name of the first `builder.scheduled(...)` source is `scheduled-source-0`; pass a `Named` to the operator to set your own. In Kotlin the key serde is optional (defaults to `null`); in Java, pass it explicitly since Java has no default arguments.

::code-tabs{group="lang"}
```kotlin [Kotlin]
val builder = StreamsBuilder()
builder
    .scheduled<String, String>(
        interval = Duration.ofSeconds(10),
        type = PunctuationType.WALL_CLOCK_TIME,
        emitter = { ctx -> ctx.forward("hb", "heartbeat-${ctx.currentWallClockTime()}") },
    )
    .to("output", Produced.with(Serdes.String(), Serdes.String()))

val driver = TopologyTestDriver.fromBuilder(builder)
val output = driver.createOutputTopic<String, String>("output", Serdes.String(), Serdes.String())

// Fire once, no time advance
driver.triggerScheduledSource("scheduled-source-0")
assertThat(output.readRecord()?.key).isEqualTo("hb")

// Or fire by advancing the wall clock past the interval
driver.advanceWallClockTime(Duration.ofSeconds(10))
assertThat(output.isEmpty()).isFalse()
```

```java [Java]
StreamsBuilder builder = new StreamsBuilder();
builder.<String, String>scheduled(
        Duration.ofSeconds(10),
        PunctuationType.WALL_CLOCK_TIME,
        Serdes.String(),   // key serde (Java has no default args)
        ctx -> ctx.forward("hb", "heartbeat-" + ctx.currentWallClockTime()))
    .to("output", Produced.with(Serdes.String(), Serdes.String()));

TopologyTestDriver driver = TopologyTestDriver.fromBuilder(builder);
TestOutputTopic<String, String> output =
    driver.createOutputTopic("output", Serdes.String(), Serdes.String());

// Fire once, no time advance
driver.triggerScheduledSource("scheduled-source-0");
assertThat(output.readRecord().getKey()).isEqualTo("hb");

// Or fire by advancing the wall clock past the interval
driver.advanceWallClockTime(Duration.ofSeconds(10));
assertThat(output.isEmpty()).isFalse();
```
::

## Config-driven tests (StoatFlowTestDriver)

`StoatFlowTestDriver.fromConfig(...)` is the test-time mirror of the production `StoatFlowRuntime.fromConfig(...)` (see [Your first app](https://stoatflow.io/docs/getting-started/first-app)). It loads `application.yaml` from the classpath, maps it into a `StreamsConfig`, builds your topology with the configured default serdes, and returns a ready `TopologyTestDriver`. No broker, no HTTP server — just the topology under the same configuration your app runs with.

Configuration is resolved from the classpath, so a `src/test/resources/application.yaml` shadows the main one — letting tests pin a deterministic config. Environment variables with the `STOATFLOW_` prefix take highest priority; the optional `configOverrides` lambda overrides programmatically above everything else.

The topology-builder callback receives both the `StreamsBuilder` and the loaded `StreamsConfig`, so a topology that needs config values (e.g. `config.schemaRegistryUrl`) can read them.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.testruntime.StoatFlowTestDriver
import io.stoatflow.testutils.TopologyTestDriver

private fun buildDriver(): TopologyTestDriver =
    StoatFlowTestDriver.fromConfig(
        topologyBuilder = { builder, config -> MyApp.buildTopology(builder, config) },
        // optional: configOverrides = { /* StreamsConfig.Builder receiver */ },
    )
```

```java [Java]
import io.stoatflow.testruntime.StoatFlowTestDriver;
import io.stoatflow.testutils.TopologyTestDriver;

static TopologyTestDriver buildDriver() {
    return StoatFlowTestDriver.fromConfig(
        (builder, config) -> MyApp.buildTopology(builder),
        null   // optional Consumer<StreamsConfig.Builder> for overrides
    );
}
```
::

From there it is the same `TopologyTestDriver` API — `createInputTopic`, `createOutputTopic`, `advanceWatermark`, `getKeyValueStore`, and so on. This is the pattern the example apps use; see `examples/stock-tick-filter` (Kotlin) and `examples/ecommerce-daily-customer-behaviour` / `examples/news-feed-subscription-processor` (Java) for full end-to-end topology tests, each paired with a broker-backed integration test.

## Porting Kafka Streams tests

If you're migrating a Kafka Streams test suite, you don't have to rewrite it around `fromBuilder`. The driver also carries the KS-shaped surface, so an existing KS test ports with the same import swap as the topology it exercises:

- **KS constructors** — `TopologyTestDriver(topology)`, `(topology, Properties)`, `(topology, Instant)`, and `(topology, Properties, Instant)`. The `Properties` overloads accept KS-keyed config (`application.id`, default serde class names, …) through the same adapter the engine uses. Typed `(topology, StreamsConfig[, Instant])` forms exist too.
- **`fromTopology(topology[, config][, initialWallClockTime])`** — for a `Topology` assembled with the Processor API (`addSource` / `addProcessor` / `addSink`) rather than a `StreamsBuilder`.
- **`getAllStateStores()`** — every state store keyed by name, matching KS.
- **`producedTopicNames()`** — the set of topics the driver has produced records to, matching KS.

::code-tabs{group="lang"}
```kotlin [Kotlin]
val props = Properties()
props[StreamsConfig.APPLICATION_ID_CONFIG] = "uppercase-test"
props[StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG] = Serdes.String()::class.java
props[StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG] = Serdes.String()::class.java

// The KS constructor idiom, unchanged
val driver = TopologyTestDriver(builder.build(), props)

input.pipeInput("k1", "hello")

driver.producedTopicNames()   // setOf("output-topic")
driver.getAllStateStores()    // all stores by name
```

```java [Java]
var props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "uppercase-test");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());

// The KS constructor idiom, unchanged
var driver = new TopologyTestDriver(builder.build(), props);

input.pipeInput("k1", "hello");

driver.producedTopicNames();   // Set.of("output-topic")
driver.getAllStateStores();    // all stores by name
```
::

For new StoatFlow tests, prefer `fromBuilder` (above) — it skips the `builder.build()` step and the `Properties` indirection. The KS-shaped surface exists so ported suites keep passing as-is; the [KS compatibility matrix](https://stoatflow.io/docs/reference/ks-compatibility-matrix#test-utilities) tracks the full test-utils parity.

## Next steps

- **[Streams builder](https://stoatflow.io/docs/building/streams-builder)** — building the topology you're testing.
- **[State stores](https://stoatflow.io/docs/building/state-stores)** — the store types the accessors above return.
- **[Windowing](https://stoatflow.io/docs/building/windowing)*&#x2A; and &#x2A;*[Event time and watermarks](https://stoatflow.io/docs/concepts/event-time-and-watermarks)** — the time model behind `advanceWatermark`.
- **[Scheduled sources](https://stoatflow.io/docs/building/scheduled-sources)** — clock-driven sources and how to test them.


# Sources and sinks

Every topology starts at a source and ends at a sink. `StreamsBuilder` opens source topics — `stream` for an event stream, `table` / `globalTable` for a changelog table — and `KStream.to` writes records back to Kafka. `Consumed` configures the read side (serdes, offset reset, watermarks); `Produced` configures the write side (serdes, partitioner, dynamic topic routing).

This page covers the source and sink methods only. For the operators in between, see [KStream and KTable](https://stoatflow.io/docs/building/kstream-ktable). For the engine that runs the topology, see [Architecture](https://stoatflow.io/docs/concepts/architecture).

::tldr-panel
- **Read a stream:** `builder.stream<K, V>("topic", consumed)` returns a `KStream<K, V>`.
- **Read a table:** `builder.table(...)` / `builder.globalTable(...)` return a changelog-backed `KTable` / `GlobalKTable` — always materialized into a state store.
- **Configure the read:** `Consumed.with(keySerde, valueSerde)`, then chain `.withOffsetResetPolicy(...)`, `.withWatermarkStrategy(...)`, `.withName(...)`.
- **Write a sink:** `stream.to("topic", Produced.with(keySerde, valueSerde))`, or `stream.to(topicNameExtractor, produced)` for per-record routing.
- **Fan-out:** reuse the same `KStream` reference for multiple branches — never call `builder.stream(...)` twice on the same topic.
::

## Reading a stream

`builder.stream` subscribes to one or more topics and returns a `KStream<K, V>`. Pass a `Consumed` to configure deserialization and source behaviour; omit it to fall back to the default serdes from your config.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.topology.Consumed
import io.stoatflow.core.topology.StreamsBuilder
import org.apache.kafka.common.serialization.Serdes

fun buildTopology(builder: StreamsBuilder) {
    val orders =
        builder.stream(
            "orders",
            Consumed.with(Serdes.String(), Serdes.String()),
        )
    // orders: KStream<String, String>
}
```

```java [Java]
import io.stoatflow.core.topology.Consumed;
import io.stoatflow.core.topology.KStream;
import io.stoatflow.core.topology.StreamsBuilder;
import org.apache.kafka.common.serialization.Serdes;

void buildTopology(StreamsBuilder builder) {
    KStream<String, String> orders =
        builder.stream(
            "orders",
            Consumed.with(Serdes.String(), Serdes.String()));
}
```
::

### Multiple topics and patterns

`stream` also accepts a collection of topic names — all merged into one stream — or a `java.util.regex.Pattern` for dynamic subscription, where new topics matching the pattern are picked up at runtime.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import java.util.regex.Pattern

// Several named topics, merged into one stream
builder.stream(listOf("orders-eu", "orders-us"), Consumed.with(keySerde, valueSerde))

// Pattern subscription — matching topics are added dynamically
builder.stream(Pattern.compile("orders-.*"), Consumed.with(keySerde, valueSerde))
```

```java [Java]
import java.util.List;
import java.util.regex.Pattern;

// Several named topics, merged into one stream
builder.stream(List.of("orders-eu", "orders-us"), Consumed.with(keySerde, valueSerde));

// Pattern subscription — matching topics are added dynamically
builder.stream(Pattern.compile("orders-.*"), Consumed.with(keySerde, valueSerde));
```
::

::callout{color="warning" icon="i-lucide-triangle-alert"}
A topic may only be registered as a source **once** per topology. Subscribing to it from two separate `builder.stream(...)` calls is rejected when the topology is built. To process the same input two ways, reuse one `KStream` reference — see [The fan-out rule](https://stoatflow.io/#the-fan-out-rule) below.
::

## Reading a table

`builder.table` interprets a topic as a **changelog** — the latest value per key is the current state — and returns a `KTable<K, V>` backed by a state store. Unlike a plain stream, a table is **always materialized**: when you don't pass a `Materialized`, StoatFlow auto-creates a store named `{topic}-store`. (See [State stores](https://stoatflow.io/docs/building/state-stores) for store configuration.)

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.state.StateStore
import io.stoatflow.core.topology.Consumed
import io.stoatflow.core.topology.Materialized

// Auto-materialized into "customers-store"
val customers = builder.table("customers", Consumed.with(Serdes.String(), Serdes.String()))

// Explicit store name + serdes
val customersNamed =
    builder.table(
        "customers",
        Consumed.`as`("customers-source"),
        Materialized.`as`<String, String, StateStore>("customers")
            .withKeySerde(Serdes.String())
            .withValueSerde(Serdes.String()),
    )
```

```java [Java]
import io.stoatflow.core.state.StateStore;
import io.stoatflow.core.topology.Consumed;
import io.stoatflow.core.topology.KTable;
import io.stoatflow.core.topology.Materialized;

// Auto-materialized into "customers-store"
KTable<String, String> customers =
    builder.table("customers", Consumed.with(Serdes.String(), Serdes.String()));

// Explicit store name + serdes
KTable<String, String> customersNamed =
    builder.table(
        "customers",
        Consumed.as("customers-source"),
        Materialized.<String, String, StateStore>as("customers")
            .withKeySerde(Serdes.String())
            .withValueSerde(Serdes.String()));
```
::

`builder.globalTable(...)` has the same signatures and returns a `GlobalKTable`. In StoatFlow's single-instance model all state is already global (see [Architecture](https://stoatflow.io/docs/concepts/architecture)), so `globalTable` is functionally identical to `table` — it exists for Kafka Streams API compatibility, and new StoatFlow code should prefer `table(...)`.

### Source-topic reuse for compacted tables

When the source topic is log-compacted, StoatFlow can restore the table's state directly from the source topic instead of creating a separate changelog topic — avoiding duplicated data. This is automatic by default (controlled globally) and can be forced per-table with `Consumed.withMaterializeFromSourceTopic(...)`:

```kotlin
// Force source-topic reuse — no separate changelog created
builder.table("users", Consumed.with(Serdes.String(), userSerde).withMaterializeFromSourceTopic(true))

// Force a dedicated changelog topic even if the source is compacted
builder.table("events", Consumed.with(Serdes.String(), eventSerde).withMaterializeFromSourceTopic(false))
```

`true` forces source-topic restoration (no changelog, no compaction check); `false` forces a dedicated changelog topic; `null` (the default) uses the global setting plus automatic compaction detection.

## Configuring the read — `Consumed`

`Consumed<K, V>` carries everything about how a source is read. Create one with a static factory, then chain `with*` methods — each returns a new immutable instance.

| Factory                                                   | Produces                     |
| --------------------------------------------------------- | ---------------------------- |
| `Consumed.with(keySerde, valueSerde)`                     | both serdes set              |
| `Consumed.keySerde(serde)` / `Consumed.valueSerde(serde)` | one serde set                |
| `Consumed.with(watermarkStrategy)`                        | watermark strategy set       |
| `Consumed.as("name")`                                     | named source, default serdes |
| `Consumed.offsetResetPolicy(policy)`                      | offset-reset policy set      |

| Builder method                                    | Effect                                               |
| ------------------------------------------------- | ---------------------------------------------------- |
| `.withKeySerde(serde)` / `.withValueSerde(serde)` | override the key / value deserializer                |
| `.withName(name)`                                 | give the source node a stable name (same as `Named`) |
| `.withOffsetResetPolicy(AutoOffsetReset.…)`       | per-source `auto.offset.reset` override              |
| `.withWatermarkStrategy(strategy)`                | per-source event-time + watermark strategy           |
| `.withMaterializeFromSourceTopic(bool?)`          | KTable changelog reuse (tables only)                 |

When a serde is not set on `Consumed`, the source falls back to the default key/value serde configured for the application.

::callout{color="info" icon="i-lucide-info"}
`Consumed.as(...)` and `.withName(...)` give the source a stable name that flows into the topology graph, metrics, and state-store identity. Naming your sources (and operators) makes `/topology` and your dashboards readable — see how the [first app](https://stoatflow.io/docs/getting-started/first-app) names every node.
::

### Offset reset policy

`AutoOffsetReset` controls where the consumer starts when there is no committed offset for a partition. The per-source override beats the global `auto.offset.reset` config. It mirrors the KIP-1106 shape of Kafka's `AutoOffsetReset`: a sealed type with static factories — in Kotlin you can also reference the objects (`AutoOffsetReset.Earliest`) directly.

| Factory                         | Behaviour                                                 |
| ------------------------------- | --------------------------------------------------------- |
| `AutoOffsetReset.earliest()`    | start from the beginning of the partition                 |
| `AutoOffsetReset.latest()`      | start from the end — only records produced after start    |
| `AutoOffsetReset.none()`        | throw if no committed offset exists                       |
| `AutoOffsetReset.byDuration(d)` | start from the first offset at/after `now − d` (KIP-1106) |

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.topology.AutoOffsetReset

// Historical/reference data — read everything from the start
builder.stream(
    "events",
    Consumed.with(Serdes.String(), eventSerde).withOffsetResetPolicy(AutoOffsetReset.Earliest),
)

// Real-time commands — only new records
builder.stream(
    "commands",
    Consumed.with(Serdes.String(), commandSerde).withOffsetResetPolicy(AutoOffsetReset.Latest),
)
```

```java [Java]
import io.stoatflow.core.topology.AutoOffsetReset;

// Historical/reference data — read everything from the start
builder.stream(
    "events",
    Consumed.with(Serdes.String(), eventSerde).withOffsetResetPolicy(AutoOffsetReset.earliest()));

// Real-time commands — only new records
builder.stream(
    "commands",
    Consumed.with(Serdes.String(), commandSerde).withOffsetResetPolicy(AutoOffsetReset.latest()));
```
::

All sources reading the same topic must agree on the offset-reset policy; conflicting policies are rejected when the topology is built.

### Timestamp extraction and watermarks

Event time and watermarks are configured **per source** through a `WatermarkStrategy`, set with `Consumed.withWatermarkStrategy(...)`. The strategy does two jobs: it extracts the event timestamp from each record (via `withTimestampAssigner`), and it generates watermarks. With no strategy set, the source uses the application's global watermark strategy.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.watermark.WatermarkStrategy
import java.time.Duration

val strategy =
    WatermarkStrategy
        .forBoundedOutOfOrderness<String, Order>(Duration.ofSeconds(30))
        .withTimestampAssigner { _, value, _ -> value?.eventTime ?: 0L }
        .withIdleness(Duration.ofMinutes(2))

builder.stream(
    "orders",
    Consumed.with(Serdes.String(), orderSerde).withWatermarkStrategy(strategy),
)
```

```java [Java]
import io.stoatflow.core.watermark.WatermarkStrategy;
import java.time.Duration;

WatermarkStrategy<String, Order> strategy =
    WatermarkStrategy
        .<String, Order>forBoundedOutOfOrderness(Duration.ofSeconds(30))
        .withTimestampAssigner((key, value) -> value != null ? value.eventTime() : 0L)
        .withIdleness(Duration.ofMinutes(2));

builder.stream(
    "orders",
    Consumed.with(Serdes.String(), orderSerde).withWatermarkStrategy(strategy));
```
::

The factory methods are `WatermarkStrategy.forBoundedOutOfOrderness(maxOutOfOrderness)`, `WatermarkStrategy.forMonotonousTimestamps()`, and `WatermarkStrategy.noWatermarks()`. In Java, `withTimestampAssigner` also accepts a value-only `Function<V, Long>` or a key-and-value `BiFunction<K, V, Long>`, as shown above. For the full event-time model — watermarks, idleness, late-record handling — see [Event time and watermarks](https://stoatflow.io/docs/concepts/event-time-and-watermarks).

## Writing a sink — `to()` and `Produced`

`KStream.to(topic, produced)` writes the stream to a Kafka topic. It's a terminal operation — it returns nothing. `Produced<K, V>` configures the write side; when a serde is omitted, the sink falls back to the default serde.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.topology.Produced

stream.to(
    "output",
    Produced.with(Serdes.String(), Serdes.Long()),
)

// Named sink with serdes — readable in the topology graph
stream.to(
    "word-counts",
    Produced.`as`<String, Long>("sink")
        .withKeySerde(Serdes.String())
        .withValueSerde(Serdes.Long()),
)
```

```java [Java]
import io.stoatflow.core.topology.Produced;

stream.to(
    "output",
    Produced.with(Serdes.String(), Serdes.Long()));

// Named sink with serdes — readable in the topology graph
stream.to(
    "word-counts",
    Produced.<String, Long>as("sink")
        .withKeySerde(Serdes.String())
        .withValueSerde(Serdes.Long()));
```
::

`Produced` factories mirror `Consumed`: `Produced.with(keySerde, valueSerde)`, `Produced.keySerde(serde)`, `Produced.valueSerde(serde)`, `Produced.streamPartitioner(partitioner)`, and `Produced.as("name")`. The chainable builders are `.withKeySerde`, `.withValueSerde`, `.withStreamPartitioner`, and `.withName`.

### Custom partitioning

By default the producer partitions by key. Supply a `StreamPartitioner<K, V>` to control the target partition yourself — it receives the topic, key, value, and partition count, and returns an `Optional<Set<Integer>>` of 0-indexed partitions (KIP-837): `Optional.empty()` falls back to the default partitioner, a single-element set routes to that partition, an empty set drops the record, and multiple elements multicast (sink path only).

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.topology.StreamPartitioner
import java.util.Optional

stream.to(
    "output",
    Produced.with(Serdes.String(), valueSerde)
        .withStreamPartitioner(
            StreamPartitioner { _, key, _, numPartitions ->
                Optional.of(setOf(Math.floorMod(key.hashCode(), numPartitions)))
            },
        ),
)
```

```java [Java]
import io.stoatflow.core.topology.StreamPartitioner;
import java.util.Optional;
import java.util.Set;

stream.to(
    "output",
    Produced.with(Serdes.String(), valueSerde)
        .withStreamPartitioner(
            (StreamPartitioner<String, Value>) (topic, key, value, numPartitions) ->
                Optional.of(Set.of(Math.floorMod(key.hashCode(), numPartitions)))));
```
::

### Dynamic topic routing

To pick the destination topic per record, pass a `TopicNameExtractor<K, V>` instead of a topic name. The extractor receives the key, value, and a `RecordContext` (source topic, partition, offset, timestamp, headers) and returns the topic name.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.topology.TopicNameExtractor

stream.to(
    TopicNameExtractor { _, value, _ -> "orders-${value.region}" },
    Produced.with(Serdes.String(), orderSerde),
)
```

```java [Java]
import io.stoatflow.core.topology.TopicNameExtractor;

stream.to(
    (TopicNameExtractor<String, Order>) (key, value, ctx) -> "orders-" + value.region(),
    Produced.with(Serdes.String(), orderSerde));
```
::

::callout{color="info" icon="i-lucide-info"}
For records that originate inside the topology (for example, results emitted as a window closes), the `RecordContext` source fields may be unknown: `topic` is `null`, and `partition` / `offset` are `-1` (`RecordContext.UNKNOWN_PARTITION` / `UNKNOWN_OFFSET`). The `timestamp` is always available. Handle those cases when your routing depends on source metadata. Dynamic-sink topics are not part of the topology's static sink-topic set.
::

## The fan-out rule

To process one source two (or more) ways, **reuse the `KStream` reference** — do not call `builder.stream(...)` twice on the same topic. A topic may only back one source node; a second subscription is rejected when the topology is built.

::code-tabs{group="lang"}
```kotlin [Kotlin]
val orders = builder.stream("orders", Consumed.with(Serdes.String(), orderSerde))

// Branch 1
orders.filter { _, v -> v.isHighValue }.to("high-value-orders")

// Branch 2 — same reference, no second subscription
orders.mapValues { v -> v.summary() }.to("order-summaries")
```

```java [Java]
KStream<String, Order> orders = builder.stream("orders", Consumed.with(Serdes.String(), orderSerde));

// Branch 1
orders.filter((k, v) -> v.isHighValue()).to("high-value-orders");

// Branch 2 — same reference, no second subscription
orders.mapValues(Order::summary).to("order-summaries");
```
::

The same rule applies to `table`, `globalTable`, and any other source. To merge two streams into one, use `KStream.merge(...)` (see the [stock-tick-filter example](https://stoatflow.io/docs/building/kstream-ktable)); to split one stream into named branches by predicate, use `split` / `branch`. Fan-out and merge happen in-memory between processing lanes — no broker round-trip, no repartition topic. See [Lanes and parallelism](https://stoatflow.io/docs/concepts/lanes-and-parallelism) for the routing model.

## Worked example

The runtime [word count](https://stoatflow.io/docs/getting-started/first-app) reads `text-lines`, counts by word, and writes `word-counts` — a `String` source and a `Long` sink:

```kotlin
builder
    .stream<String, String>("text-lines", Consumed.`as`("source"))
    .flatMapValues { line -> line.lowercase().split(Regex("\\s+")).filter { it.isNotBlank() } }
    .groupBy({ _, word -> word })
    .count(Materialized.`as`<String, Long, StateStore>("word-counts"))
    .toStream()
    .to(
        "word-counts",
        Produced.`as`<String, Long>("sink")
            .withKeySerde(Serdes.String())
            .withValueSerde(Serdes.Long()),
    )
```

## Next steps

- **[KStream and KTable](https://stoatflow.io/docs/building/kstream-ktable)** — the operators between source and sink: `map`, `filter`, `merge`, `split`, table conversions.
- **[Serdes](https://stoatflow.io/docs/building/serdes)** — choosing and configuring serializers for keys and values.
- **[State stores](https://stoatflow.io/docs/building/state-stores)** — `Materialized` and the store types behind `table` and the aggregations.
- **[Scheduled sources](https://stoatflow.io/docs/building/scheduled-sources)** — emit records on an interval or cron schedule without reading from Kafka.
- **[Event time and watermarks](https://stoatflow.io/docs/concepts/event-time-and-watermarks)** — the full event-time model behind `withWatermarkStrategy`.


# KStream and KTable operations

`KStream<K, V>` is an unbounded stream of records; `KTable<K, V>` is a changelog table holding one value per key. This page covers the stateless `KStream` transforms and the basic `KTable` operations. Aggregations, joins, windowing, and the Processor API have their own pages.

Both types are created through the [`StreamsBuilder`](https://stoatflow.io/docs/building/streams-builder), never constructed directly. The operators are Kafka Streams compatible — Java code written against the KS DSL compiles against StoatFlow with no changes (see the [Kafka Streams compatibility matrix](https://stoatflow.io/docs/reference/ks-compatibility-matrix)).

::tldr-panel
- **Stateless `KStream` transforms:** `mapValues`, `map`, `filter`, `filterNot`, `flatMapValues`, `flatMap`, `selectKey`, `peek`, `merge`, `split`.
- **`KTable` basics:** `mapValues`, `filter`, `filterNot`, `toStream`, plus `KStream.toTable`.
- **Re-keying:** `map`, `flatMap`, `selectKey`, and `groupBy` change the key — records are re-routed to the lane that owns the new key. `mapValues`, `filter`, and `peek` preserve the key and stay on the same lane.
- **Every operator** takes an optional trailing `Named` to give the node a stable name.
::

## Two kinds of operator: value-preserving vs key-changing

The single most important distinction in this API is whether an operator changes the **key**. StoatFlow routes every record to a processing lane by key affinity — same key, same lane, in order. An operator that changes the key forces the record to be re-routed to whichever lane owns the new key. (This is StoatFlow's in-memory equivalent of a Kafka Streams repartition topic — no broker round-trip. See [Lanes and parallelism](https://stoatflow.io/docs/concepts/lanes-and-parallelism) and [Architecture](https://stoatflow.io/docs/concepts/architecture).)

| Operator               | Changes key? | Effect                                   |
| ---------------------- | ------------ | ---------------------------------------- |
| `mapValues`            | No           | Stays on the same lane                   |
| `filter` / `filterNot` | No           | Stays on the same lane                   |
| `flatMapValues`        | No           | All output records keep the input key    |
| `peek`                 | No           | Pure side effect                         |
| `merge`                | No           | Records keep their original key affinity |
| `split`                | No           | Routing only — keys unchanged            |
| `map`                  | **Yes**      | Re-routed by new key                     |
| `flatMap`              | **Yes**      | Each output re-routed by its key         |
| `selectKey`            | **Yes**      | Re-routed by new key                     |
| `groupBy`              | **Yes**      | Re-keys, then groups                     |

Prefer the value-only operators (`mapValues`, `flatMapValues`) when you don't need to change the key — they avoid re-routing entirely.

## Naming operators

Every operator accepts an optional trailing `Named` parameter. StoatFlow uses these stable names for the topology graph, metrics, and (for stateful operators) state-store identity. In Kotlin `as` is a soft keyword, so escape it with backticks: ``Named.`as`("uppercase")``. In Java it's a plain method: `Named.as("uppercase")`.

```kotlin
import io.stoatflow.core.topology.Named
```

Naming is optional but recommended — unnamed nodes get auto-generated names that change as you edit the topology.

## Value transforms — `mapValues`

`mapValues` transforms each value and keeps the key. There's a value-only form and a key-aware form (the key is read-only — it is not changed).

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.topology.Named

// value-only
val upper = stream.mapValues({ v -> v.uppercase() }, Named.`as`("uppercase"))

// key-aware (key is read-only)
val tagged = stream.mapValues({ k, v -> "$k:$v" }, Named.`as`("tag-with-key"))
```

```java [Java]
import io.stoatflow.core.topology.Named;
import io.stoatflow.core.topology.ValueMapper;
import io.stoatflow.core.topology.ValueMapperWithKey;

// value-only
KStream<String, String> upper =
    stream.mapValues((ValueMapper<String, String>) v -> v.toUpperCase(), Named.as("uppercase"));

// key-aware (key is read-only)
KStream<String, String> tagged =
    stream.mapValues((ValueMapperWithKey<String, String, String>) (k, v) -> k + ":" + v,
        Named.as("tag-with-key"));
```
::

::callout{color="warning" icon="i-lucide-triangle-alert"}
Values may be `null` (a tombstone). Your mapper, predicate, and action functions must handle `null` values — StoatFlow does not filter them out for you.
::

## Key + value transform — `map`

`map` returns a `KeyValue<KR, VR>`, replacing both key and value. **This is a key-changing operation** — the output is re-routed to the lane that owns the new key.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.state.KeyValue
import io.stoatflow.core.topology.Named

val rekeyed = stream.map(
    { key, value -> KeyValue(value.substring(0, 3), "$key:$value") },
    Named.`as`("rekey-by-prefix"),
)
```

```java [Java]
import io.stoatflow.core.state.KeyValue;
import io.stoatflow.core.topology.KeyValueMapper;
import io.stoatflow.core.topology.Named;

KStream<String, String> rekeyed = stream.map(
    (KeyValueMapper<Long, String, KeyValue<String, String>>) (key, value) ->
        new KeyValue<>(value.substring(0, 3), key + ":" + value),
    Named.as("rekey-by-prefix"));
```
::

## Filtering — `filter` and `filterNot`

`filter` keeps records that match the predicate; `filterNot` keeps records that don't. Neither changes the key.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.topology.Named

val longEnough = stream.filter({ _, v -> v.length > 5 }, Named.`as`("keep-long"))
val notEmpty = stream.filterNot({ _, v -> v.isEmpty() }, Named.`as`("drop-empty"))
```

```java [Java]
import io.stoatflow.core.topology.Named;
import io.stoatflow.core.topology.Predicate;

KStream<Long, String> longEnough =
    stream.filter((Predicate<Long, String>) (k, v) -> v.length() > 5, Named.as("keep-long"));
KStream<Long, String> notEmpty =
    stream.filterNot((Predicate<Long, String>) (k, v) -> v.isEmpty(), Named.as("drop-empty"));
```
::

## One-to-many — `flatMapValues` and `flatMap`

`flatMapValues` turns each value into zero or more values, all keeping the input key — value-only, so no re-routing. There's also a key-aware form that reads (but does not change) the key.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.topology.Named

val whitespace = "\\s+".toRegex()
val words = lines.flatMapValues(
    { line -> line.lowercase().split(whitespace).filter { it.isNotBlank() } },
    Named.`as`("split-words"),
)
```

```java [Java]
import io.stoatflow.core.topology.Named;
import io.stoatflow.core.topology.ValueMapper;

import java.util.Arrays;
import java.util.stream.Collectors;

KStream<String, String> words = lines.flatMapValues(
    (ValueMapper<String, Iterable<String>>) line ->
        Arrays.stream(line.toLowerCase().split("\\s+"))
            .filter(w -> !w.isBlank())
            .collect(Collectors.toList()),
    Named.as("split-words"));
```
::

`flatMap` returns an `Iterable<KeyValue<KR, VR>>` — each output record carries its own key, so **this is a key-changing operation** and each output is re-routed independently.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.state.KeyValue
import io.stoatflow.core.topology.Named

val perTag = stream.flatMap(
    { _, value -> value.tags.map { tag -> KeyValue(tag, value.id) } },
    Named.`as`("explode-tags"),
)
```

```java [Java]
import io.stoatflow.core.state.KeyValue;
import io.stoatflow.core.topology.KeyValueMapper;
import io.stoatflow.core.topology.Named;

import java.util.stream.Collectors;

KStream<String, String> perTag = stream.flatMap(
    (KeyValueMapper<String, Doc, Iterable<KeyValue<String, String>>>) (k, value) ->
        value.tags().stream()
            .map(tag -> new KeyValue<>(tag, value.id()))
            .collect(Collectors.toList()),
    Named.as("explode-tags"));
```
::

## Re-keying — `selectKey`

`selectKey` replaces only the key, leaving the value alone. **It is a key-changing operation** — use it before `groupByKey` or a join when you need to key the stream on a value field.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.topology.Named

val byCustomer = orders.selectKey({ _, order -> order.customerId }, Named.`as`("key-by-customer"))
```

```java [Java]
import io.stoatflow.core.topology.KeyValueMapper;
import io.stoatflow.core.topology.Named;

KStream<String, Order> byCustomer =
    orders.selectKey((KeyValueMapper<String, Order, String>) (k, order) -> order.customerId(),
        Named.as("key-by-customer"));
```
::

## Side effects — `peek`

`peek` runs an action for each record (logging, counters, debugging) and forwards the record unchanged. The key is never modified. (`forEach` is the terminal variant — it runs the action but returns nothing, ending the chain.)

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.topology.Named

val observed = stream.peek({ k, v -> println("$k => $v") }, Named.`as`("log"))
```

```java [Java]
import io.stoatflow.core.topology.ForeachAction;
import io.stoatflow.core.topology.Named;

KStream<Long, String> observed =
    stream.peek((ForeachAction<Long, String>) (k, v) -> System.out.println(k + " => " + v),
        Named.as("log"));
```
::

The Java overloads take a `ForeachAction`, so void lambdas work directly — no `return Unit.INSTANCE`.

## Combining streams — `merge`

`merge` combines two streams of the same key/value types into one. It is **not** a key-changing operation: records keep their original key affinity. Both streams must come from the same `StreamsBuilder`.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.topology.Named

val combined = streamA.merge(streamB, Named.`as`("merge-a-b"))
```

```java [Java]
import io.stoatflow.core.topology.Named;

KStream<String, String> combined = streamA.merge(streamB, Named.as("merge-a-b"));
```
::

## Branching — `split`

`split` routes each record to **exactly one** branch using first-match semantics: predicates are evaluated in order, and the first match wins. `split()` returns a `BranchedKStream`; the terminal `defaultBranch()` / `noDefaultBranch()` call returns a map of branch name to `KStream`. Branch names are the split name plus the `Branched` suffix — e.g. split name `router` plus suffix `-high` gives key `router-high`. `split` does not change the key.

Finish the chain with `defaultBranch()` (unmatched records go to a default branch), `defaultBranch(Branched.as("-suffix"))` (named default), or `noDefaultBranch()` (unmatched records are dropped).

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.topology.Branched
import io.stoatflow.core.topology.Named

val branches: Map<String, KStream<Long, String>> = stream
    .split(Named.`as`("router"))
    .branch({ _, v -> v.length > 20 }, Branched.`as`("-high"))
    .branch({ _, v -> v.length > 10 }, Branched.`as`("-medium"))
    .defaultBranch(Branched.`as`("-low"))

branches["router-high"]!!.to("high-topic")
branches["router-medium"]!!.to("medium-topic")
branches["router-low"]!!.to("low-topic")
```

```java [Java]
import io.stoatflow.core.topology.Branched;
import io.stoatflow.core.topology.Named;
import io.stoatflow.core.topology.Predicate;

import java.util.Map;

Map<String, KStream<Long, String>> branches = stream
    .split(Named.as("router"))
    .branch((Predicate<Long, String>) (k, v) -> v.length() > 20, Branched.as("-high"))
    .branch((Predicate<Long, String>) (k, v) -> v.length() > 10, Branched.as("-medium"))
    .defaultBranch(Branched.as("-low"));

branches.get("router-high").to("high-topic");
branches.get("router-medium").to("medium-topic");
branches.get("router-low").to("low-topic");
```
::

`Branched` can also inline a branch transform or side effect instead of returning the stream in the map: `Branched.withFunction(s -> s.mapValues(...), "-medium")` applies a transform and stores the result; `Branched.withConsumer(s -> s.to("topic"), "-low")` applies a side effect and stores the original branch stream.

## KTable basics

A `KTable` represents the latest value per key as a changelog. You get one from `StreamsBuilder.table(...)`, from a stream-to-table conversion (`KStream.toTable`), or as the output of an aggregation. The stateless KTable operators mirror the stream ones but operate on the table's changelog.

### `mapValues` and `filter`

`KTable.mapValues` transforms values (value-only and key-aware forms). `KTable.filter` / `filterNot` keep or drop entries by predicate. Without a `Materialized` argument these are derived, on-the-fly views over the source table's state and keep no store of their own.

::callout{color="info" icon="i-lucide-info"}
A materialized `KTable.filter` differs semantically from the non-materialized form: when an entry stops matching, a materialized filter writes a **tombstone** (null) to its store and forwards it downstream, whereas the non-materialized filter simply doesn't forward the record. Pass a `Materialized` argument to opt into the store. See [State stores](https://stoatflow.io/docs/building/state-stores).
::

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.topology.Named

val upper = table
    .mapValues({ v -> v.uppercase() }, Named.`as`("table-upper"))
    .filter({ _, v -> v.isNotEmpty() }, Named.`as`("table-non-empty"))
```

```java [Java]
import io.stoatflow.core.topology.Named;
import io.stoatflow.core.topology.Predicate;
import io.stoatflow.core.topology.ValueMapper;

KTable<String, String> upper = table
    .mapValues((ValueMapper<String, String>) v -> v.toUpperCase(), Named.as("table-upper"))
    .filter((Predicate<String, String>) (k, v) -> !v.isEmpty(), Named.as("table-non-empty"));
```
::

### `toStream` — table back to stream

`KTable.toStream` converts the table's changelog into a `KStream`: every update to a key becomes a record. This is how you write a table's results to a topic. (`count`, `reduce`, and `aggregate` produce a `KTable`, so `toStream` is the usual bridge to a sink — see [Aggregations](https://stoatflow.io/docs/building/aggregations).)

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.topology.Named

table
    .mapValues({ v -> v.uppercase() }, Named.`as`("table-upper"))
    .toStream(Named.`as`("to-stream"))
    .to("output-topic")
```

```java [Java]
import io.stoatflow.core.topology.Named;
import io.stoatflow.core.topology.ValueMapper;

table
    .mapValues((ValueMapper<String, String>) v -> v.toUpperCase(), Named.as("table-upper"))
    .toStream(Named.as("to-stream"))
    .to("output-topic");
```
::

There's a key-changing `toStream(keyMapper, named)` overload that derives a new key from each entry — like `selectKey`, it re-routes the resulting records to the lane that owns the new key.

### `toTable` — stream to table

`KStream.toTable` interprets a stream as a changelog: each record updates the key's value in the resulting table, and a null value is a tombstone that deletes the key. It preserves the key (no re-routing). Pass a `Materialized` to control the backing store and its name; without one StoatFlow auto-generates a store name.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.state.StateStore
import io.stoatflow.core.topology.Materialized
import org.apache.kafka.common.serialization.Serdes

val latest = stream.toTable(
    Materialized.`as`<String, String, StateStore>("latest-by-key")
        .withKeySerde(Serdes.String())
        .withValueSerde(Serdes.String()),
)
```

```java [Java]
import io.stoatflow.core.state.StateStore;
import io.stoatflow.core.topology.Materialized;
import org.apache.kafka.common.serialization.Serdes;

KTable<String, String> latest = stream.toTable(
    Materialized.<String, String, StateStore>as("latest-by-key")
        .withKeySerde(Serdes.String())
        .withValueSerde(Serdes.String()));
```
::

## Putting it together

A complete stateless pipeline: read a topic, re-key, filter, uppercase, and write the result. (Adapted from the `map-filter` example.)

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.state.KeyValue
import io.stoatflow.core.topology.Consumed
import io.stoatflow.core.topology.Named
import io.stoatflow.core.topology.Produced
import io.stoatflow.core.topology.StreamsBuilder
import org.apache.kafka.common.serialization.Serdes

fun buildTopology(builder: StreamsBuilder) {
    builder
        .stream("input-topic", Consumed.`as`<Long, String>("source").withKeySerde(Serdes.Long()))
        .map({ key, value -> KeyValue(value.substring(0, 3), "$key:$value") }, Named.`as`("rekey"))
        .filter({ _, value -> value.length > 5 }, Named.`as`("keep-long"))
        .mapValues({ value -> value.uppercase() }, Named.`as`("uppercase"))
        .to(
            "output-topic",
            Produced.`as`<String, String>("sink")
                .withKeySerde(Serdes.String())
                .withValueSerde(Serdes.String()),
        )
}
```

```java [Java]
import io.stoatflow.core.state.KeyValue;
import io.stoatflow.core.topology.Consumed;
import io.stoatflow.core.topology.KStream;
import io.stoatflow.core.topology.KeyValueMapper;
import io.stoatflow.core.topology.Named;
import io.stoatflow.core.topology.Predicate;
import io.stoatflow.core.topology.Produced;
import io.stoatflow.core.topology.StreamsBuilder;
import io.stoatflow.core.topology.ValueMapper;
import org.apache.kafka.common.serialization.Serdes;

void buildTopology(StreamsBuilder builder) {
    KStream<Long, String> input =
        builder.stream("input-topic", Consumed.<Long, String>as("source").withKeySerde(Serdes.Long()));
    input
        .map((KeyValueMapper<Long, String, KeyValue<String, String>>) (key, value) ->
                new KeyValue<>(value.substring(0, 3), key + ":" + value),
            Named.as("rekey"))
        .filter((Predicate<String, String>) (k, value) -> value.length() > 5, Named.as("keep-long"))
        .mapValues((ValueMapper<String, String>) String::toUpperCase, Named.as("uppercase"))
        .to(
            "output-topic",
            Produced.<String, String>as("sink")
                .withKeySerde(Serdes.String())
                .withValueSerde(Serdes.String()));
}
```
::

## Next steps

- **[Aggregations](https://stoatflow.io/docs/building/aggregations)** — `groupByKey`, `groupBy`, `count`, `reduce`, `aggregate`.
- **[Joins](https://stoatflow.io/docs/building/joins)** — stream-table, stream-stream, and table-table joins.
- **[Serdes](https://stoatflow.io/docs/building/serdes)** — `Consumed`, `Produced`, and serde resolution.
- **[Lanes and parallelism](https://stoatflow.io/docs/concepts/lanes-and-parallelism)** — how key affinity and re-routing work.
- **[Testing](https://stoatflow.io/docs/building/testing)** — verify these operators with the in-memory test driver.


# Aggregations

Aggregation in StoatFlow follows the Kafka Streams shape: you first **group** records by key, then fold them into a running result with `count`, `reduce`, or `aggregate`. The result is a `KTable<K, R>` — a changelog, one value per key, backed by a state store. This page covers grouping a `KStream` (`groupByKey` / `groupBy` → `KGroupedStream`) and grouping a `KTable` (`groupBy` → `KGroupedTable`, which adds the adder/subtractor changelog form).

Windowed aggregations (`groupByKey().windowedBy(...)`) and co-grouping (`cogroup`) build on the same grouped types but have their own pages — see [Windowing](https://stoatflow.io/docs/building/windowing). Everything here is non-windowed.

::tldr-panel
- **Group first:** `groupByKey()` keeps the key; `groupBy(selector)` re-keys. Either way the aggregation itself opens the sub-topology boundary, because it needs the grouped key's lane affinity.
- **Then fold:** `count()` → `KTable<K, Long>`; `reduce(reducer)` → `KTable<K, V>`; `aggregate(initializer, aggregator)` → `KTable<K, VR>`.
- **Result is a state store.** Name and configure it with `Materialized`; supply key/value serdes via `Grouped` or `Materialized`.
- **`KGroupedTable`** (from `KTable.groupBy`) takes an **adder + subtractor** so a re-keyed value is removed from its old group and added to its new one.
::

## Grouping a stream

Aggregation always starts from a grouped stream. Two ways to get one:

- **`groupByKey()`** — group by the record's current key. No re-keying; the boundary still opens at the aggregation if something upstream changed the key.
- **`groupBy(selector)`** — derive a new grouping key from each record. This is a key-changing operation: records are re-routed by the new key (StoatFlow does this in-memory between lanes — see [Architecture](https://stoatflow.io/docs/concepts/architecture#processing-lanes-and-key-affinity)).

Both return a `KGroupedStream<K, V>`, the entry point to `count` / `reduce` / `aggregate`.

Serdes for the grouping key (and value, where it survives the fold) come from `Grouped`. Provide them when the engine can't infer them from the source — most importantly after `groupBy` changes the key type, or when the value serde differs from the stream's default.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.topology.Grouped
import org.apache.kafka.common.serialization.Serdes

// Group by current key
val byKey = stream.groupByKey(Grouped.with(Serdes.String(), Serdes.String()))

// Re-key, then group (key type changes to the selector's return type)
val byCategory =
    stream.groupBy(
        { _, product -> product.category },
        Grouped.with("by-category", Serdes.String(), productSerde),
    )
```

```java [Java]
import io.stoatflow.core.topology.Grouped;
import org.apache.kafka.common.serialization.Serdes;

// Group by current key
var byKey = stream.groupByKey(Grouped.with(Serdes.String(), Serdes.String()));

// Re-key, then group (key type changes to the selector's return type)
var byCategory =
    stream.groupBy(
        (key, product) -> product.category(),
        Grouped.with("by-category", Serdes.String(), productSerde));
```
::

::callout{color="info" icon="i-lucide-info"}
`Grouped` factories: `Grouped.with(keySerde, valueSerde)`, `Grouped.with(name, keySerde, valueSerde)`, `Grouped.as(name)`, `Grouped.keySerde(...)`, `Grouped.valueSerde(...)`. The name (when given) labels the grouping node in the topology graph and metrics; serdes set here are the default for the downstream aggregation unless `Materialized` overrides them.
::

## count

`count()` returns a `KTable<K, Long>` holding the number of records seen per key. The value serde is `Long` automatically; you only need a key serde (from `Grouped`, `Materialized`, or the topology default). Null values (tombstones) are ignored and do not increment the count.

This is the word-count fold from [Your first app](https://stoatflow.io/docs/getting-started/first-app), in isolation:

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.state.StateStore
import io.stoatflow.core.topology.Grouped
import io.stoatflow.core.topology.Materialized
import io.stoatflow.core.topology.Named

val counts: KTable<String, Long> =
    words
        .groupBy({ _, word -> word }, Grouped.`as`("group-by-word"))
        .count(
            Named.`as`("count"),
            Materialized.`as`<String, Long, StateStore>("word-counts"),
        )
```

```java [Java]
import io.stoatflow.core.state.StateStore;
import io.stoatflow.core.topology.Grouped;
import io.stoatflow.core.topology.KTable;
import io.stoatflow.core.topology.Materialized;
import io.stoatflow.core.topology.Named;

KTable<String, Long> counts =
    words
        .groupBy((key, word) -> word, Grouped.as("group-by-word"))
        .count(
            Named.as("count"),
            Materialized.<String, Long, StateStore>as("word-counts"));
```
::

`count()` has overloads for `(Named)`, `(Materialized)`, `(Named, Materialized)`, and the no-arg form (auto-named store) — pass only what you need.

## reduce

`reduce(reducer)` folds values of the **same type** into one per key. The first value for a key is stored as-is; each subsequent value is combined with the stored aggregate via the `Reducer<V>` — `(aggregate, value) -> aggregate`. The result is a `KTable<K, V>`. Null values are ignored.

Use `reduce` when the running result has the same type as the input (sum, max, latest-wins, string concatenation). The value serde carries through, so it comes from `Grouped`/`Materialized`/the default.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.state.StateStore
import io.stoatflow.core.topology.Materialized
import io.stoatflow.core.topology.Named

// Running total of order amounts per customer
val totals: KTable<String, Long> =
    orderAmounts
        .groupByKey()
        .reduce(
            { runningTotal, amount -> runningTotal + amount },
            Named.`as`("sum-amounts"),
            Materialized.`as`<String, Long, StateStore>("amount-totals"),
        )
```

```java [Java]
import io.stoatflow.core.state.StateStore;
import io.stoatflow.core.topology.KTable;
import io.stoatflow.core.topology.Materialized;
import io.stoatflow.core.topology.Named;

// Running total of order amounts per customer
KTable<String, Long> totals =
    orderAmounts
        .groupByKey()
        .reduce(
            (runningTotal, amount) -> runningTotal + amount,
            Named.as("sum-amounts"),
            Materialized.<String, Long, StateStore>as("amount-totals"));
```
::

::callout{color="info" icon="i-lucide-info"}
In Kotlin, `reduce` takes the reducer lambda first, then optional `Named` / `Materialized`. In Java, the reducer is a `Reducer<V>` — the SAM type for `(V, V) -> V` — so a lambda passed inline converts to it directly. Overloads accept `(Reducer)`, `(Reducer, Named)`, `(Reducer, Materialized)`, and `(Reducer, Named, Materialized)`.
::

## aggregate

`aggregate` is the general fold: the result type `VR` is independent of the input value type. You supply two functions:

- **`Initializer<VR>`** — `() -> VR`, the starting aggregate for a key not seen before.
- **`Aggregator<K, V, VR>`** — `(key, value, aggregate) -> aggregate`, applied for each record.

The result is a `KTable<K, VR>`. Because the aggregate type differs from the value type, give it a value serde via `Materialized.withValueSerde(...)` (or `Materialized.with(keySerde, valueSerde)`); the key serde still falls back to `Grouped`/the default.

This mirrors the per-customer daily aggregation in the e-commerce example (`examples/ecommerce-daily-customer-behaviour`), simplified to a non-windowed running aggregate:

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.state.StateStore
import io.stoatflow.core.topology.Grouped
import io.stoatflow.core.topology.Materialized
import io.stoatflow.core.topology.Named
import org.apache.kafka.common.serialization.Serdes

data class DailyAggregates(
    val productsViewed: Int,
    val purchases: Int,
    val amountSpent: Double,
)

val perCustomer: KTable<String, DailyAggregates> =
    events
        .groupBy(
            { _, e -> e.customerId },
            Grouped.with("by-customer", Serdes.String(), eventSerde),
        )
        .aggregate(
            { DailyAggregates(0, 0, 0.0) },
            { _, event, agg ->
                DailyAggregates(
                    productsViewed = agg.productsViewed + if (event.isView) 1 else 0,
                    purchases = agg.purchases + if (event.isPurchase) 1 else 0,
                    amountSpent = agg.amountSpent + event.amount,
                )
            },
            Named.`as`("daily-aggregation"),
            Materialized.`as`<String, DailyAggregates, StateStore>("daily-aggregation-store")
                .withValueSerde(dailyAggregatesSerde),
        )
```

```java [Java]
import io.stoatflow.core.state.StateStore;
import io.stoatflow.core.topology.Grouped;
import io.stoatflow.core.topology.KTable;
import io.stoatflow.core.topology.Materialized;
import io.stoatflow.core.topology.Named;
import org.apache.kafka.common.serialization.Serdes;

KTable<String, DailyAggregates> perCustomer =
    events
        .groupBy(
            (key, e) -> e.customerId(),
            Grouped.with("by-customer", Serdes.String(), eventSerde))
        .aggregate(
            () -> new DailyAggregates(0, 0, 0.0),
            (key, event, agg) -> new DailyAggregates(
                agg.productsViewed() + (event.isView() ? 1 : 0),
                agg.purchases() + (event.isPurchase() ? 1 : 0),
                agg.amountSpent() + event.amount()),
            Named.as("daily-aggregation"),
            Materialized.<String, DailyAggregates, StateStore>as("daily-aggregation-store")
                .withValueSerde(dailyAggregatesSerde));
```
::

::callout{color="warning" icon="i-lucide-triangle-alert"}
The aggregator must be a **pure transformation** of `(key, value, aggregate)`. Don't mutate the incoming aggregate in place and return it — build and return a new value (the Kotlin example uses a copy; the Java example a new record). The runtime treats the returned object as the new state; mutating shared state defeats the changelog and recovery semantics.
::

In Kotlin, `aggregate` also offers a config-first overload — `aggregate(named, materialized, initializerFn = { ... }) { k, v, agg -> ... }`. Only the **final** parameter (the aggregator) becomes a trailing lambda block; the initializer is still passed as a regular argument (named here as `initializerFn`). The Java-friendly overloads take `Initializer<VR>` and `Aggregator<K, V, VR>` explicitly.

## Aggregating a table: KGroupedTable

`KTable.groupBy(selector)` produces a `KGroupedTable<KR, VR>`, not a `KGroupedStream`. The difference is **changelog semantics**: a table is a stream of updates, so when a record's grouping key changes (a customer moves department, a product is re-categorised), the aggregate for the *old* group must be undone and the aggregate for the *new* group applied. That's why every `KGroupedTable` fold takes a **subtractor** alongside the adder.

::callout{color="info" icon="i-lucide-info"}
The source `KTable` must be **materialized** (have a state store) for `groupBy` — the engine reads the old value to know what to subtract. `KTable.groupBy` will throw at build time otherwise. The selector returns a `KeyValue<KR, VR>`, so it can change both the grouping key and the value. See [KStream & KTable](https://stoatflow.io/docs/building/kstream-ktable) for materialising a table.
::

### count (table)

`count()` on a `KGroupedTable` returns `KTable<K, Long>` and applies the three changelog rules automatically: insert increments, a key change decrements the old key and increments the new, a delete (tombstone) decrements. When a count reaches zero the key is removed from the store.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.state.KeyValue

// Count users per department, correct under department changes
val usersPerDept: KTable<String, Long> =
    usersTable
        .groupBy { _, user -> KeyValue(user.department, user) }
        .count()
```

```java [Java]
import io.stoatflow.core.state.KeyValue;
import io.stoatflow.core.topology.KTable;

// Count users per department, correct under department changes
KTable<String, Long> usersPerDept =
    usersTable
        .groupBy((key, user) -> KeyValue.pair(user.department(), user))
        .count();
```
::

### reduce (table)

`reduce(adder, subtractor)` folds same-typed values. Both are `Reducer<V>` — `(aggregate, value) -> aggregate`. On each update the subtractor removes the old value first, then the adder applies the new one.

::code-tabs{group="lang"}
```kotlin [Kotlin]
// Sum scores by department
val scoreByDept: KTable<String, Int> =
    usersTable
        .groupBy { _, user -> KeyValue(user.department, user.score) }
        .reduce(
            adder = { agg, score -> agg + score },
            subtractor = { agg, score -> agg - score },
        )
```

```java [Java]
import io.stoatflow.core.topology.Reducer;

// Sum scores by department
Reducer<Integer> adder = (agg, score) -> agg + score;
Reducer<Integer> subtractor = (agg, score) -> agg - score;

KTable<String, Integer> scoreByDept =
    usersTable
        .groupBy((key, user) -> KeyValue.pair(user.department(), user.score()))
        .reduce(adder, subtractor);
```
::

### aggregate (table)

`aggregate(initializer, adder, subtractor)` is the general table fold. The adder and subtractor are **both** `Aggregator<K, V, VR>` — `(key, value, aggregate) -> aggregate` (Kafka Streams uses `Aggregator` for both roles; there is no separate `Subtractor` type). Subtractor runs first on an update, then adder.

::code-tabs{group="lang"}
```kotlin [Kotlin]
// Track the set of user IDs per department
val membersByDept: KTable<String, Set<String>> =
    usersTable
        .groupBy { _, user -> KeyValue(user.department, user) }
        .aggregate(
            initializer = { mutableSetOf<String>() },
            adder = { _, user, set -> set.also { it.add(user.id) } },
            subtractor = { _, user, set -> set.also { it.remove(user.id) } },
        )
```

```java [Java]
import io.stoatflow.core.topology.Aggregator;
import io.stoatflow.core.topology.Initializer;

import java.util.HashSet;
import java.util.Set;

// Track the set of user IDs per department
Initializer<Set<String>> initializer = HashSet::new;
Aggregator<String, User, Set<String>> adder = (dept, user, set) -> {
    set.add(user.id());
    return set;
};
Aggregator<String, User, Set<String>> subtractor = (dept, user, set) -> {
    set.remove(user.id());
    return set;
};

KTable<String, Set<String>> membersByDept =
    usersTable
        .groupBy((key, user) -> KeyValue.pair(user.department(), user))
        .aggregate(initializer, adder, subtractor);
```
::

::callout{color="info" icon="i-lucide-info"}
When the source table is backed by a **versioned** store, out-of-order records (timestamp older than the last seen for that key) are ignored and do not update the aggregate — this prevents late records from corrupting the result. See [State stores](https://stoatflow.io/docs/building/state-stores).
::

## The functional interfaces

These are the Java SAM types used across all aggregation methods (Kotlin callers pass lambdas directly). They are import-compatible in name and shape with their Kafka Streams equivalents.

| Interface              | Signature                              | Used by                                                                                                    |
| ---------------------- | -------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| `Initializer<VA>`      | `() -> VA`                             | every `aggregate` (initial value)                                                                          |
| `Aggregator<K, V, VA>` | `(key, value, aggregate) -> aggregate` | every `aggregate` — adder **and** the `KGroupedTable.aggregate` subtractor (KS uses `Aggregator` for both) |
| `Reducer<V>`           | `(aggregate, value) -> aggregate`      | every `reduce` (and table adder/subtractor)                                                                |

All live in `io.stoatflow.core.topology`. For the full DSL parity status against Kafka Streams, see the [Kafka Streams compatibility matrix](https://stoatflow.io/docs/reference/ks-compatibility-matrix).

## Materializing the result

Every aggregation writes to a state store. You control it with `Materialized`:

- **`Materialized.as("store-name")`** — name the store explicitly (recommended; the name is the store's stable identity for changelog topics and recovery).
- **`.withValueSerde(serde)`** — required when the aggregate type differs from the input value type (i.e. for `aggregate`).
- **`Materialized.with(keySerde, valueSerde)`** — set both serdes at once.
- **`Materialized.valueSerde(serde)`** — value serde only, auto-named store.

If you omit `Materialized`, the store gets an auto-generated name and falls back to `Grouped` / the topology default serdes. Stores are durable by default — every aggregation update is committed to a Kafka changelog topic on the commit barrier, so the running result survives restarts. The full store configuration surface (RocksDB vs in-memory, logging, caching, retention) is on [State stores](https://stoatflow.io/docs/building/state-stores).

## Consuming the result

`count` / `reduce` / `aggregate` all return a `KTable`. To emit the running result to a topic, turn it back into a changelog stream with `toStream()` and `to(...)`:

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.topology.Named
import io.stoatflow.core.topology.Produced
import org.apache.kafka.common.serialization.Serdes

counts
    .toStream(Named.`as`("to-stream"))
    .to(
        "word-counts",
        Produced.`as`<String, Long>("sink")
            .withKeySerde(Serdes.String())
            .withValueSerde(Serdes.Long()),
    )
```

```java [Java]
import io.stoatflow.core.topology.Named;
import io.stoatflow.core.topology.Produced;
import org.apache.kafka.common.serialization.Serdes;

counts
    .toStream(Named.as("to-stream"))
    .to(
        "word-counts",
        Produced.<String, Long>as("sink")
            .withKeySerde(Serdes.String())
            .withValueSerde(Serdes.Long()));
```
::

Because the output is a changelog, downstream consumers see every intermediate update of a key's aggregate, not just a final value — that's the expected `KTable` semantics, the same as Kafka Streams.

## Next steps

- **[Windowing](https://stoatflow.io/docs/building/windowing)** — time, sliding, and session windows on grouped streams, plus `cogroup`.
- **[KStream & KTable](https://stoatflow.io/docs/building/kstream-ktable)** — materialising a table so `KTable.groupBy` can track old values.
- **[State stores](https://stoatflow.io/docs/building/state-stores)** — store types, changelog, in-memory vs RocksDB, versioned stores.
- **[Testing](https://stoatflow.io/docs/building/testing)** — assert on aggregation results with the in-memory test driver.


# Windowing

Windowing groups records by key **and** by time, so an aggregation produces one result per key per window instead of a single running total. You reach a windowed aggregation by calling `.windowedBy(...)` on a `KGroupedStream` — the same grouped type you get from [`groupByKey()` / `groupBy()`](https://stoatflow.io/docs/building/aggregations). The result is a `KTable` whose key is a `Windowed<K>`: the original key plus the window bounds.

Windows are reasoned about in **event time**, not wall-clock time — when a window closes is driven by the watermark, not by the clock on the machine. Read [Event time and watermarks](https://stoatflow.io/docs/concepts/event-time-and-watermarks) for the model that underpins everything on this page.

::tldr-panel
- **Entry points:** `.windowedBy(TimeWindows…)`, `.windowedBy(SlidingWindows…)`, `.windowedBy(SessionWindows…)` on a `KGroupedStream`.
- **Window kinds:** tumbling (`ofSizeWithNoGrace`), hopping (`.advanceBy(...)`), sliding (`ofTimeDifference…`), session (`ofInactivityGap…`).
- **Result:** `KTable<Windowed<K>, V>` — key carries `windowStart()` / `windowEnd()`. Serialize it for a sink with `WindowedSerdes`.
- **When to emit:** `EmitStrategy.onWindowUpdate()` (default, every update) vs `EmitStrategy.onWindowClose()` (one final result). Or `KTable.suppress(...)` for buffering control.
::

## Window kinds at a glance

| Kind         | Factory                                                  | Overlap                                                         | Result key window |
| ------------ | -------------------------------------------------------- | --------------------------------------------------------------- | ----------------- |
| **Tumbling** | `TimeWindows.ofSizeWithNoGrace(size)`                    | None — each record in exactly one window                        | `TimeWindow`      |
| **Hopping**  | `TimeWindows.ofSizeWithNoGrace(size).advanceBy(advance)` | Yes — `advance < size` means a record lands in multiple windows | `TimeWindow`      |
| **Sliding**  | `SlidingWindows.ofTimeDifferenceWithNoGrace(diff)`       | Event-driven; two events share a window if `|t1 - t2| <= diff`  | `TimeWindow`      |
| **Session**  | `SessionWindows.ofInactivityGapWithNoGrace(gap)`         | Dynamic — sessions grow and merge across activity               | `SessionWindow`   |

Every factory has a `…AndGrace(…, grace)` variant that accepts late records for `grace` after the window's end before the window closes. The factories are static methods, so they call identically from Kotlin and Java.

The same six records, under all four kinds — trace any record straight down to compare:

![Four window kinds applied to one key and the same six records at 10:02, 10:04, 10:08, 10:12, 10:16 and 10:18. Tumbling (size 5 min) puts each record in exactly one of four non-overlapping windows. Hopping (size 5 min, advance 1 min) overlaps, and the 10:08 record lands in five windows — size divided by advance. Sliding (difference 5 min) anchors each window on the record that closes it, spanning t minus 5 minutes to t. Session (inactivity gap 5 min) shows two sessions separated by 8 minutes of idle time, which merge into one 10:02–10:18 session once the late 10:12 record arrives within 5 minutes of both.](https://stoatflow.io/assets/docs/building/window-kinds_20260727.svg)

## Tumbling and hopping windows

`TimeWindows` covers both fixed-size cases. A **tumbling** window has no overlap — its advance equals its size. Calling `.advanceBy(...)` with an interval smaller than the size turns it into a **hopping** window, where each record falls into several overlapping windows.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.topology.Grouped
import io.stoatflow.core.topology.TimeWindows
import io.stoatflow.core.topology.Windowed
import java.time.Duration

// Tumbling: count page views per user in fixed 5-minute windows
val tumbling: KTable<Windowed<String>, Long> =
    pageViews
        .groupByKey(Grouped.with(Serdes.String(), pageViewSerde))
        .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5)))
        .count()

// Hopping: 5-minute windows that advance every 1 minute (each view lands in 5 windows)
val hopping: KTable<Windowed<String>, Long> =
    pageViews
        .groupByKey(Grouped.with(Serdes.String(), pageViewSerde))
        .windowedBy(
            TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5))
                .advanceBy(Duration.ofMinutes(1)),
        )
        .count()
```

```java [Java]
import io.stoatflow.core.topology.Grouped;
import io.stoatflow.core.topology.TimeWindows;
import io.stoatflow.core.topology.Windowed;
import java.time.Duration;

// Tumbling: count page views per user in fixed 5-minute windows
KTable<Windowed<String>, Long> tumbling =
    pageViews
        .groupByKey(Grouped.with(Serdes.String(), pageViewSerde))
        .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5)))
        .count();

// Hopping: 5-minute windows that advance every 1 minute (each view lands in 5 windows)
KTable<Windowed<String>, Long> hopping =
    pageViews
        .groupByKey(Grouped.with(Serdes.String(), pageViewSerde))
        .windowedBy(
            TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5))
                .advanceBy(Duration.ofMinutes(1)))
        .count();
```
::

`.advanceBy(...)` must be positive and no larger than the window size; an advance equal to the size is just a tumbling window again. To accept late records, swap the factory for `TimeWindows.ofSizeAndGrace(size, grace)` — the window then stays open for `grace` past its end before closing.

`count()`, `reduce(...)`, and `aggregate(...)` are all available on the windowed stream, with the same Kotlin-lambda / Java-functional-interface overloads as the non-windowed aggregations (see [Aggregations](https://stoatflow.io/docs/building/aggregations)). `count()` returns `KTable<Windowed<K>, Long>`; `reduce` and `aggregate` return your value type.

## Sliding windows

A sliding window is defined by a **time difference** rather than a fixed grid position. Two events belong to the same window when their event-time difference is within the configured difference; windows are positioned relative to each event. This is the KIP-450 event-driven model — roughly two to three windows per event, not one window per millisecond of advance.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.topology.SlidingWindows
import java.time.Duration

val slidingCounts: KTable<Windowed<String>, Long> =
    events
        .groupByKey(Grouped.with(Serdes.String(), eventSerde))
        .windowedBy(SlidingWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(10)))
        .count()
```

```java [Java]
import io.stoatflow.core.topology.SlidingWindows;
import java.time.Duration;

KTable<Windowed<String>, Long> slidingCounts =
    events
        .groupByKey(Grouped.with(Serdes.String(), eventSerde))
        .windowedBy(SlidingWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(10)))
        .count();
```
::

For a 10-minute difference, events at `t=0` and `t=5min` share a window (5 ≤ 10), while events at `t=0` and `t=15min` do not (15 > 10). Use `SlidingWindows.ofTimeDifferenceAndGrace(diff, grace)` to admit late records.

## Session windows

Session windows are dynamic. A session grows as long as records keep arriving within the **inactivity gap**; once activity pauses longer than the gap, the session closes and the next record starts a fresh one. When a record bridges two existing sessions, they merge.

Because sessions merge, `aggregate(...)` on a session window **requires a session merger** — a function that combines the aggregates of two sessions being joined. `count()` and `reduce(...)` supply this implicitly (counts sum; the reducer doubles as the merger), but `aggregate` makes you pass it explicitly.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.topology.SessionWindows
import java.time.Duration

// Count user actions per session (30-minute inactivity gap)
val sessionCounts: KTable<Windowed<String>, Long> =
    userActions
        .groupByKey(Grouped.with(Serdes.String(), actionSerde))
        .windowedBy(SessionWindows.ofInactivityGapWithNoGrace(Duration.ofMinutes(30)))
        .count()

// Aggregate with an explicit session merger
val sessionActions: KTable<Windowed<String>, List<Action>> =
    userActions
        .groupByKey(Grouped.with(Serdes.String(), actionSerde))
        .windowedBy(SessionWindows.ofInactivityGapWithNoGrace(Duration.ofMinutes(30)))
        .aggregate(
            { emptyList() },                                   // initializer
            { _, value, agg -> agg + value },                 // aggregator
            { _, agg1, agg2 -> agg1 + agg2 },                 // session merger (required)
        )
```

```java [Java]
import io.stoatflow.core.topology.SessionWindows;
import java.util.ArrayList;
import java.util.List;
import java.time.Duration;

// Count user actions per session (30-minute inactivity gap)
KTable<Windowed<String>, Long> sessionCounts =
    userActions
        .groupByKey(Grouped.with(Serdes.String(), actionSerde))
        .windowedBy(SessionWindows.ofInactivityGapWithNoGrace(Duration.ofMinutes(30)))
        .count();

// Aggregate with an explicit session merger
KTable<Windowed<String>, List<Action>> sessionActions =
    userActions
        .groupByKey(Grouped.with(Serdes.String(), actionSerde))
        .windowedBy(SessionWindows.ofInactivityGapWithNoGrace(Duration.ofMinutes(30)))
        .aggregate(
            ArrayList::new,                                   // initializer
            (key, value, agg) -> { agg.add(value); return agg; },  // aggregator
            (key, agg1, agg2) -> { agg1.addAll(agg2); return agg1; });  // session merger (required)
```
::

`SessionWindows.ofInactivityGapAndGrace(gap, grace)` adds a grace period on top of the inactivity gap before a session is considered closed.

## Windowed keys

Every windowed aggregation keys its output by `Windowed<K>` — the original key plus a `Window` (a `TimeWindow` for time/sliding aggregations, a `SessionWindow` for sessions). The window is a half-open interval `[start, end)`: it includes `start` and excludes `end`.

::code-tabs{group="lang"}
```kotlin [Kotlin]
windowedCounts.toStream().forEach { windowedKey, count ->
    println(
        "key=${windowedKey.key} " +
            "window=${windowedKey.windowStartTime()}..${windowedKey.windowEndTime()} " +
            "count=$count",
    )
}
```

```java [Java]
windowedCounts.toStream().forEach((windowedKey, count) ->
    System.out.println(
        "key=" + windowedKey.getKey()
            + " window=" + windowedKey.windowStartTime() + ".." + windowedKey.windowEndTime()
            + " count=" + count));
```
::

`Windowed<K>` exposes `windowStart()` / `windowEnd()` (epoch millis) and `windowStartTime()` / `windowEndTime()` (`Instant`). In Kotlin the underlying key and window are properties (`windowedKey.key`, `windowedKey.window`); from Java they are the generated accessors `getKey()` and `getWindow()`.

::callout{color="info" icon="i-lucide-info"}
A common pattern is to fold the window bounds into the value before re-keying back to the plain key. The e-commerce example does exactly this — its `mapValues` reads `wk.getWindow().startTime()` to stamp the aggregate with its date, then `toStream((wk, v) -> wk.getKey(), …)` re-keys by the original customer id.
::

## When results are emitted

By default a windowed aggregation emits a result on **every update** — each record that changes a window's aggregate produces a new downstream record (continuous refinement). For high-volume windows that is a lot of intermediate output. `EmitStrategy` lets you switch to emitting only the **final** result when the window closes.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.topology.EmitStrategy
import io.stoatflow.core.topology.TimeWindows
import java.time.Duration

val finalCounts: KTable<Windowed<String>, Long> =
    pageViews
        .groupByKey(Grouped.with(Serdes.String(), pageViewSerde))
        .windowedBy(TimeWindows.ofSizeAndGrace(Duration.ofMinutes(5), Duration.ofSeconds(30)))
        .emitStrategy(EmitStrategy.onWindowClose())
        .count()
```

```java [Java]
import io.stoatflow.core.topology.EmitStrategy;
import io.stoatflow.core.topology.TimeWindows;
import java.time.Duration;

KTable<Windowed<String>, Long> finalCounts =
    pageViews
        .groupByKey(Grouped.with(Serdes.String(), pageViewSerde))
        .windowedBy(TimeWindows.ofSizeAndGrace(Duration.ofMinutes(5), Duration.ofSeconds(30)))
        .emitStrategy(EmitStrategy.onWindowClose())
        .count();
```
::

`EmitStrategy.onWindowClose()` produces exactly one output per window per key, at the cost of latency equal to the grace period — the window can't close until the watermark passes `windowEnd + grace`. It is available on the windowed stream types (`TimeWindowedKStream` — which covers both fixed-size and sliding windows — and `SessionWindowedKStream`) and requires watermark propagation to detect closure. The default is `EmitStrategy.onWindowUpdate()`.

## Suppression

`EmitStrategy` is the declarative way to ask for final-only output. `KTable.suppress(...)` is the lower-level control: it holds updates in a buffer and you choose both the release condition and how the buffer behaves when it fills.

For windowed tables, `Suppressed.untilWindowCloses(bufferConfig)` buffers every update until the window closes — equivalent in effect to `onWindowClose()`, but with explicit buffer-overflow control:

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.topology.Suppressed
import io.stoatflow.core.topology.Suppressed.BufferConfig
import io.stoatflow.core.topology.TimeWindows
import java.time.Duration

val suppressed: KTable<Windowed<String>, Long> =
    pageViews
        .groupByKey(Grouped.with(Serdes.String(), pageViewSerde))
        .windowedBy(TimeWindows.ofSizeAndGrace(Duration.ofMinutes(5), Duration.ofSeconds(30)))
        .count()
        .suppress(Suppressed.untilWindowCloses(BufferConfig.unbounded()))
```

```java [Java]
import io.stoatflow.core.topology.Suppressed;
import io.stoatflow.core.topology.Suppressed.BufferConfig;
import io.stoatflow.core.topology.TimeWindows;
import java.time.Duration;

KTable<Windowed<String>, Long> suppressed =
    pageViews
        .groupByKey(Grouped.with(Serdes.String(), pageViewSerde))
        .windowedBy(TimeWindows.ofSizeAndGrace(Duration.ofMinutes(5), Duration.ofSeconds(30)))
        .count()
        .suppress(Suppressed.untilWindowCloses(BufferConfig.unbounded()));
```
::

For non-windowed tables, `Suppressed.untilTimeLimit(duration, bufferConfig)` rate-limits per key: at most one update per key per interval, keeping only the most recent value within each interval.

### Buffer configuration

`suppress` needs a `BufferConfig` — a type nested inside `Suppressed` (`Suppressed.BufferConfig`, matching Kafka Streams) — describing how much to hold and what to do on overflow:

| Builder                      | Behaviour                                                                                              |
| ---------------------------- | ------------------------------------------------------------------------------------------------------ |
| `BufferConfig.unbounded()`   | No size limit (a `StrictBufferConfig`). Use with care — memory grows with buffered records.            |
| `BufferConfig.maxRecords(n)` | Bound the buffer to `n` records.                                                                       |
| `.emitEarlyWhenFull()`       | On overflow, emit the oldest records to make room (no data loss, but non-final values may be emitted). |
| `.shutDownWhenFull()`        | On overflow, shut the application down rather than emit early or lose data.                            |
| `.withLoggingDisabled()`     | Don't back the buffer with a changelog. Faster, but suppressed records are lost on restart.            |

::callout{color="warning" icon="i-lucide-triangle-alert"}
`BufferConfig.maxBytes(...)` from Kafka Streams is **not supported** — StoatFlow buffers in-memory objects, not serialized bytes, so a byte limit can't be enforced accurately. Use `maxRecords(...)` instead; calling `maxBytes(...)` throws `UnsupportedOperationException`.
::

By default suppress buffers are changelog-backed for fault tolerance. `suppress(...)` with logging enabled requires the upstream `KTable` to carry serdes (provide them via `Materialized` on the aggregation, or disable logging with `BufferConfig.withLoggingDisabled()`).

::callout{color="info" icon="i-lucide-database-backup"}
**Migrating a suppress-heavy topology from Kafka Streams?** The KS suppress buffer's changelog envelope is not byte-translatable, so the [state-carrying migration](https://stoatflow.io/docs/migration/with-data-migration#semantics-caveats-at-the-cutover-boundary) requires draining the buffer at quiesce — cut over at a windows-closed point so KS emits its pending finals before the stop.
::

## Serializing windowed keys to a sink

To write a windowed table to a topic, the `Windowed<K>` key needs a serde. `WindowedSerdes` wraps the inner key serde with the window bounds:

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.topology.Named
import io.stoatflow.core.topology.Produced
import io.stoatflow.core.topology.Windowed
import io.stoatflow.core.topology.WindowedSerdes
import org.apache.kafka.common.serialization.Serdes

val windowedKeySerde = WindowedSerdes.timeWindowedSerdeFrom(Serdes.String())

windowedCounts
    .toStream(Named.`as`("to-stream"))
    .to(
        "windowed-counts",
        Produced.`as`<Windowed<String>, Long>("sink")
            .withKeySerde(windowedKeySerde)
            .withValueSerde(Serdes.Long()),
    )
```

```java [Java]
import io.stoatflow.core.topology.Named;
import io.stoatflow.core.topology.Produced;
import io.stoatflow.core.topology.Windowed;
import io.stoatflow.core.topology.WindowedSerdes;
import org.apache.kafka.common.serialization.Serdes;

var windowedKeySerde = WindowedSerdes.timeWindowedSerdeFrom(Serdes.String());

windowedCounts
    .toStream(Named.as("to-stream"))
    .to(
        "windowed-counts",
        Produced.<Windowed<String>, Long>as("sink")
            .withKeySerde(windowedKeySerde)
            .withValueSerde(Serdes.Long()));
```
::

Use `WindowedSerdes.sessionWindowedSerdeFrom(inner)` for session-windowed keys so deserialization reconstructs `SessionWindow` instances. Both factories take a `Serde<T>` for the inner key (a StoatFlow deviation from Kafka Streams, which takes a `Class<T>`); the window size is omitted because the bounds are encoded in the serialized bytes.

## A complete windowed aggregation

The e-commerce example aggregates per-customer activity into a daily window, then re-keys back to the customer id and joins a profile table. Its core is a tumbling daily window with a 15-minute grace period:

```java
// Daily tumbling windows, 15-minute grace for late events
TimeWindows dailyWindows =
    TimeWindows.ofSizeAndGrace(Duration.ofDays(1), Duration.ofMinutes(15));

TimeWindowedKStream<String, WebActivityOrPurchase> windowedStream = mergedStreams
    .groupBy(
        (k, v) -> v.purchase() != null ? v.purchase().customerId() : v.webActivity().customerId(),
        Grouped.with("grouped-by-customer-id", stringSerde, webActivityOrPurchaseSerde))
    .windowedBy(dailyWindows);

KTable<Windowed<String>, DailyAggregates> dailyAggregation = windowedStream
    .aggregate(
        () -> new DailyAggregates(/* empty */),
        (key, value, agg) -> /* fold value into agg */ agg,
        Named.as("daily-aggregation"),
        Materialized.<String, DailyAggregates, StateStore>as("daily-aggregation-store")
            .withValueSerde(dailyAggregatesSerde));
```

The full source — including how it reads `wk.getWindow().startTime()` to stamp each aggregate with its date and `toStream((wk, v) -> wk.getKey(), …)` to re-key — is in `examples/ecommerce-daily-customer-behaviour`.

## Where to go next

- [Aggregations](https://stoatflow.io/docs/building/aggregations) — `count`, `reduce`, `aggregate` and the grouped types that windowing builds on
- [Event time and watermarks](https://stoatflow.io/docs/concepts/event-time-and-watermarks) — how StoatFlow decides when a window closes
- [Serdes](https://stoatflow.io/docs/building/serdes) — serializers for keys and values, including custom value types
- [State stores](https://stoatflow.io/docs/building/state-stores) — the window/session stores backing these aggregations
- [Testing](https://stoatflow.io/docs/building/testing) — advancing the watermark to drive window closes in tests


# Joins

A join combines records from two sources that share a key. StoatFlow's join DSL is Kafka Streams compatible: the same five join families — stream-stream, stream-table, table-table, foreign-key, and co-grouping — with the same `ValueJoiner`, `JoinWindows`, `Joined`, `StreamJoined`, and `TableJoined` configuration types. Java code written against the KS join API compiles against StoatFlow unchanged (see the [Kafka Streams compatibility matrix](https://stoatflow.io/docs/reference/ks-compatibility-matrix)).

One structural difference is worth stating up front: because StoatFlow holds [global state in a single instance](https://stoatflow.io/docs/concepts/architecture#state-stores-and-durability), the two sides of a join **do not have to be co-partitioned**. There is no requirement that both topics share a partition count. Re-keying for a foreign-key join happens in-memory between lanes, not through a repartition topic.

::tldr-panel
- **Stream-stream:** `left.join(right, joiner, JoinWindows.ofTimeDifferenceWithNoGrace(d))` — time-windowed, both sides buffered in window stores.
- **Stream-table:** `stream.join(table, joiner)` — point-in-time lookup; table updates do **not** re-trigger the join.
- **Table-table:** `left.join(right, joiner)` — key-equal join; updates on either side re-emit.
- **Foreign-key:** `left.join(right, fkExtractor, joiner, TableJoined.as(...))` — join on a key extracted from the left value (inner + left only).
- **Joiner shapes:** `ValueJoiner<V1, V2, VR>` (values only) or `ValueJoinerWithKey<K, V1, V2, VR>` (key-aware).
- **`inner` / `left` / `outer`** map to `join` / `leftJoin` / `outerJoin` (FK joins have no outer).
::

## The joiner

Every join takes a **joiner** — the function that combines a left value and a right value into the result. Two interfaces:

| Interface                           | Signature                      | When                                                      |
| ----------------------------------- | ------------------------------ | --------------------------------------------------------- |
| `ValueJoiner<V1, V2, VR>`           | `(left, right) -> result`      | The result depends only on the two values.                |
| `ValueJoinerWithKey<K, V1, V2, VR>` | `(key, left, right) -> result` | The result also needs the join key. The key is read-only. |

For `leftJoin` the right value may be `null` (no match on the right). For `outerJoin` either side may be `null`, but never both. Write the joiner to tolerate the nulls the join type allows.

## Stream-stream joins

A stream-stream join matches records from two streams that arrive **close together in event time**. Both sides are buffered in window stores, and a record from one stream joins any record from the other whose timestamp falls inside the configured `JoinWindows`. The join is symmetric — a late-arriving record on either side can produce a match.

`JoinWindows` defines the time bounds. `ofTimeDifferenceWithNoGrace(d)` is a symmetric ±`d` window; `before(d)` / `after(d)` make it asymmetric; `ofTimeDifferenceAndGrace(d, grace)` adds a grace period for late records (see [Event time and watermarks](https://stoatflow.io/docs/concepts/event-time-and-watermarks)).

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.topology.JoinWindows
import io.stoatflow.core.topology.StreamJoined
import org.apache.kafka.common.serialization.Serdes
import java.time.Duration

// Inner: emit only when both an order and a payment land within 5 minutes
val matched =
    orders.join(
        payments,
        { order, payment -> OrderPayment(order, payment) },
        JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(5)),
    )

// With serdes + a named, materialized join (recommended in production)
val matchedConfigured =
    orders.join(
        payments,
        { order, payment -> OrderPayment(order, payment) },
        JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(5)),
        StreamJoined.with(Serdes.String(), orderSerde, paymentSerde)
            .withStoreName("order-payment-join"),
    )
```

```java [Java]
import io.stoatflow.core.topology.JoinWindows;
import io.stoatflow.core.topology.StreamJoined;
import org.apache.kafka.common.serialization.Serdes;
import java.time.Duration;

// Inner: emit only when both an order and a payment land within 5 minutes
var matched =
    orders.join(
        payments,
        (order, payment) -> new OrderPayment(order, payment),
        JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(5)));

// With serdes + a named, materialized join (recommended in production)
var matchedConfigured =
    orders.join(
        payments,
        (order, payment) -> new OrderPayment(order, payment),
        JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(5)),
        StreamJoined.with(Serdes.String(), orderSerde, paymentSerde)
            .withStoreName("order-payment-join"));
```
::

`join`, `leftJoin`, and `outerJoin` are all available, each in value-only and key-aware forms:

- **`join`** (inner) — emit only when both sides match within the window.
- **`leftJoin`** — emit for every left record; the right value is `null` when nothing matched by the time the window closes.
- **`outerJoin`** — emit for every record on either side; the unmatched side is `null` when the window closes.

::callout{color="info" icon="i-lucide-clock"}
Unmatched `leftJoin` / `outerJoin` results are emitted when the **window closes**, which is driven by the watermark — not on the bare arrival of the left record. A join window with no grace closes as soon as the watermark passes the window end.
::

![A five minute symmetric join window with two minutes of grace, drawn around two left-stream records on an event-time axis running from 10:00 to 10:35. The left record at 10:06 carries a window from 10:01 to 10:11 plus grace to 10:13, and no right record falls inside it. The right record at 10:16 sits past that window and before the next one, so it matches nothing at all. The second left record at 10:26 carries a window from 10:21 to 10:31: the right record at 10:28 arrives ahead of the watermark and joins, and the right record at 10:24 arrives last, behind the watermark and therefore late, but its window is still inside grace so it joins as well. The watermark, tracking the highest event time seen so far minus one minute, reaches 10:15 once the record at 10:16 arrives, passing 10:13 and closing the first left record's window. An inner join emits only the two matched pairs. A leftJoin emits those same two plus a third result pairing the left value at 10:06 with null, emitted at the close rather than on the left record's arrival.](https://stoatflow.io/assets/docs/building/join-window-timeline_20260727.svg)

### Configuring the join with StreamJoined

`StreamJoined` configures the two window stores that back a stream-stream join, plus naming and serdes:

| Method                                                         | Effect                                                                                           |
| -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
| `StreamJoined.as(name)`                                        | Name the join processor.                                                                         |
| `StreamJoined.with(keySerde, leftValueSerde, rightValueSerde)` | Set the serdes for both sides.                                                                   |
| `.withStoreName(base)`                                         | Derive store names `{base}-left-store` and `{base}-right-store`.                                 |
| `.withThisStoreSupplier(...)` / `.withOtherStoreSupplier(...)` | Supply explicit window stores ("this" = the stream `join` is called on; "other" = the argument). |
| `.withDslStoreSuppliers(StoreType.IN_MEMORY)`                  | Choose in-memory or RocksDB for both stores (takes precedence over explicit suppliers).          |
| `.withLoggingEnabled(...)` / `.withLoggingDisabled()`          | Force changelog logging on/off for the join stores.                                              |

A key-aware joiner uses the same calls — pass a `ValueJoinerWithKey` instead:

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.topology.ValueJoinerWithKey

val labelled =
    orders.join(
        payments,
        ValueJoinerWithKey<String, Order, Payment, String> { customerId, order, payment ->
            "$customerId: ${order.total + payment.amount}"
        },
        JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(5)),
    )
```

```java [Java]
import io.stoatflow.core.topology.ValueJoinerWithKey;

var labelled =
    orders.join(
        payments,
        (ValueJoinerWithKey<String, Order, Payment, String>)
            (customerId, order, payment) ->
                customerId + ": " + (order.total() + payment.amount()),
        JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(5)));
```
::

## Stream-table joins

A stream-table join is an **enrichment lookup**: each stream record is joined against the current value for its key in a `KTable`. This is a point-in-time lookup — unlike a table-table join, **updates to the table do not re-trigger the join**. Only stream records produce output. The table must be materialized (created via `builder.table(...)` or `stream.toTable(...)` with a store).

The stream key is preserved, so the join stays on the same lane — no re-keying.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.topology.Joined

// Inner: enrich each click with the user record, drop clicks with no user
val enriched =
    clicks.join(users) { click, user -> EnrichedClick(click, user) }

// Left: keep every click; user is null when the table has no entry for the key
val enrichedAll =
    clicks.leftJoin(
        users,
        Joined.`as`("click-user-join"),
    ) { click, user -> EnrichedClick(click, user) }
```

```java [Java]
import io.stoatflow.core.topology.Joined;

// Inner: enrich each click with the user record, drop clicks with no user
var enriched =
    clicks.join(users, (click, user) -> new EnrichedClick(click, user));

// Left: keep every click; user is null when the table has no entry for the key
var enrichedAll =
    clicks.leftJoin(
        users,
        (click, user) -> new EnrichedClick(click, user),
        Joined.as("click-user-join"));
```
::

`Joined.as(name)` names the join processor — that's its only *effective* setting. For Kafka Streams source compatibility `Joined` also accepts serdes (`withKeySerde` / `withValueSerde` / `withOtherValueSerde`) and `withGracePeriod(...)`, but these are **advisory** in StoatFlow: they're stored so KS code compiles, while serde resolution actually comes from `Consumed`/`Grouped` upstream (a documented divergence — see the [compatibility matrix](https://stoatflow.io/docs/reference/ks-compatibility-matrix)). Key-aware joiners (`ValueJoinerWithKey`) are available for both `join` and `leftJoin`. There is no `outerJoin` for stream-table joins — an outer join has no stream record to anchor an unmatched table entry to.

::callout{color="info" icon="i-lucide-info"}
**Worked example.** The `news-feed-subscription-processor` example enriches a windowed result stream with subscriber and user detail held in a `KTable`, using exactly this shape — `subscriptionResults.join(subscriptionAndUserTable, joiner, Joined.as("..."))` — to build the final notification before writing to the sink.
::

## Table-table joins

A table-table join matches the two tables **by key**: for each key present in both, it emits the joined value, and it re-emits whenever **either** side updates. The result is itself a `KTable`, so it carries changelog (tombstone) semantics — a deletion on the relevant side deletes the join result.

Both tables must be materialized. The three join types follow KS semantics exactly:

| Method         | Emits                                                 | Result deleted when               |
| -------------- | ----------------------------------------------------- | --------------------------------- |
| `join` (inner) | only when both keys are present                       | either side becomes a tombstone   |
| `leftJoin`     | for every left update; right may be `null`            | the left side becomes a tombstone |
| `outerJoin`    | for every update to either side; either may be `null` | both sides become tombstones      |

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.state.StateStore
import io.stoatflow.core.topology.Materialized
import io.stoatflow.core.topology.Named

// Inner join on equal keys
val joined =
    accounts.join(balances) { account, balance -> AccountBalance(account, balance) }

// Outer join, named and materialized into its own store
val outer =
    accounts.outerJoin(
        balances,
        { account, balance -> AccountBalance(account, balance) },
        Named.`as`("account-balance-join"),
        Materialized.`as`<String, AccountBalance, StateStore>("account-balance"),
    )
```

```java [Java]
import io.stoatflow.core.state.StateStore;
import io.stoatflow.core.topology.Materialized;
import io.stoatflow.core.topology.Named;

// Inner join on equal keys
var joined =
    accounts.join(balances, (account, balance) -> new AccountBalance(account, balance));

// Outer join, named and materialized into its own store
var outer =
    accounts.outerJoin(
        balances,
        (account, balance) -> new AccountBalance(account, balance),
        Named.as("account-balance-join"),
        Materialized.<String, AccountBalance, StateStore>as("account-balance"));
```
::

Each of `join` / `leftJoin` / `outerJoin` has overloads taking an optional `Named` and an optional `Materialized` — name the node, materialize the result, or both.

## Foreign-key joins

A foreign-key (FK) join joins two tables on a key **extracted from the left table's value** rather than on the record key. This is the table-table equivalent of a relational `JOIN ... ON left.fk = right.id`. The right table is keyed by the foreign key; the result keeps the **left** table's key.

The signature is `left.join(right, foreignKeyExtractor, joiner, tableJoined, ...)`:

- **`foreignKeyExtractor`** derives the right table's key from the left value — `(V) -> KO?` (value only) or `(K, V) -> KO?` (key + value). A `null` foreign key means no join is possible and the record is ignored.
- **`joiner`** combines the left value with the matched right value.
- **`TableJoined.as(name)`** names the join processors (its only setting).

Both **inner** (`join`) and **left** (`leftJoin`) FK joins are supported; **outer is not** — there is no left key to anchor an unmatched right row to. The FK join re-emits when the left value changes (re-extract the FK, look up the right table) and when the right value changes (re-evaluate every left row that points at it).

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.state.StateStore
import io.stoatflow.core.topology.Materialized
import io.stoatflow.core.topology.TableJoined

// orders keyed by orderId, customers keyed by customerId.
// Extract customerId from each order, look it up in customers.
val enrichedOrders =
    orders.join(
        customers,
        { order -> order.customerId },           // foreign-key extractor (left value -> right key)
        { order, customer -> EnrichedOrder(order, customer) },
        TableJoined.`as`("order-customer-join"),
        Materialized.`as`<String, EnrichedOrder, StateStore>("enriched-orders")
            .withKeySerde(Serdes.String())
            .withValueSerde(enrichedOrderSerde),
    )
```

```java [Java]
import io.stoatflow.core.state.StateStore;
import io.stoatflow.core.topology.Materialized;
import io.stoatflow.core.topology.TableJoined;

// orders keyed by orderId, customers keyed by customerId.
// Extract customerId from each order, look it up in customers.
var enrichedOrders =
    orders.join(
        customers,
        order -> order.customerId(),             // foreign-key extractor (left value -> right key)
        (order, customer) -> new EnrichedOrder(order, customer),
        TableJoined.as("order-customer-join"),
        Materialized.<String, EnrichedOrder, StateStore>as("enriched-orders")
            .withKeySerde(Serdes.String())
            .withValueSerde(enrichedOrderSerde));
```
::

The Java foreign-key overloads take `java.util.function.Function<V, KO>` (or `BiFunction<K, V, KO>` for the key-aware form) for the extractor — the same shape as the current Kafka Streams FK-join API — and `ValueJoiner<V, VO, VR>` for the joiner, so a method reference like `Order::customerId` works as the extractor and a constructor reference like `EnrichedOrder::new` works as the joiner.

::callout{color="info" icon="i-lucide-database-backup"}
**Migrating join state from Kafka Streams?** FK-join subscription stores and the LEFT/OUTER stream-stream outer store both carry across with the [migration tool](https://stoatflow.io/docs/migration/migration-tool) — migrated foreign-key rows keep re-joining on later table updates, and boundary-unmatched join records still emit their null-side results after the cutover. See [classifying your stores](https://stoatflow.io/docs/migration/with-data-migration#classifying-your-stores).
::

## Co-grouping

Co-grouping aggregates **multiple grouped streams of different value types** into one result, each stream contributing through its own aggregator. It's the join-like answer to "fold orders and payments for the same customer into one summary" — without windowing the two sides against each other.

Start a cogroup from a `KGroupedStream`, chain `.cogroup(other) { ... }` for each additional stream, then `.aggregate(initializer)`:

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.topology.Materialized

val orders = orderStream.groupByKey()       // KGroupedStream<String, Order>
val payments = paymentStream.groupByKey()   // KGroupedStream<String, Payment>

val summary: KTable<String, CustomerSummary> =
    orders
        .cogroup { _, order, agg -> agg.addOrder(order) }
        .cogroup(payments) { _, payment, agg -> agg.addPayment(payment) }
        .aggregate(
            { CustomerSummary() },
            Materialized.`as`("customer-summary"),
        )
```

```java [Java]
import io.stoatflow.core.topology.Materialized;

var orders = orderStream.groupByKey();       // KGroupedStream<String, Order>
var payments = paymentStream.groupByKey();   // KGroupedStream<String, Payment>

KTable<String, CustomerSummary> summary =
    orders
        .cogroup((key, order, agg) -> agg.addOrder(order))
        .cogroup(payments, (key, payment, agg) -> agg.addPayment(payment))
        .aggregate(
            CustomerSummary::new,
            Materialized.as("customer-summary"));
```
::

`aggregate` has overloads taking an optional `Named` and an optional `Materialized`. Co-grouping can also be windowed — call `.windowedBy(TimeWindows...)` or `.windowedBy(SessionWindows...)` before `aggregate`; see [Windowing](https://stoatflow.io/docs/building/windowing). For the non-co-grouped single-stream folds, see [Aggregations](https://stoatflow.io/docs/building/aggregations).

## Choosing a join

| You have                                 | You want                                     | Use                               |
| ---------------------------------------- | -------------------------------------------- | --------------------------------- |
| Two event streams, matches close in time | Correlate within a time window               | **stream-stream** (`JoinWindows`) |
| An event stream + a lookup table         | Enrich each event with current table state   | **stream-table**                  |
| Two changelog tables, same key           | A joined table that updates with either side | **table-table**                   |
| Two tables joined on a value field       | Relational-style FK lookup                   | **foreign-key**                   |
| Several grouped streams, one key         | Fold all of them into one aggregate          | **co-grouping**                   |

## Where to go next

- [Aggregations](https://stoatflow.io/docs/building/aggregations) — group and fold a single stream or table
- [Windowing](https://stoatflow.io/docs/building/windowing) — windowed aggregations and windowed co-grouping
- [Serdes](https://stoatflow.io/docs/building/serdes) — the serializers join stores need for keys and both value sides
- [State stores](https://stoatflow.io/docs/building/state-stores) — store types behind join state
- [Event time and watermarks](https://stoatflow.io/docs/concepts/event-time-and-watermarks) — what drives stream-stream window closing


# The Processor API

When the declarative DSL operators don't cover what you need, drop down to the **Processor API**: a record-by-record interface where you control forwarding, state access, periodic work, and timers directly. It mirrors the Kafka Streams Processor API, so code written for KS ports with only an import change. For the engine model behind state access and timer execution, see [Architecture](https://stoatflow.io/docs/concepts/architecture) and [Lanes and parallelism](https://stoatflow.io/docs/concepts/lanes-and-parallelism).

::tldr-panel
- **`Processor<KIn, VIn, KOut, VOut>`** can change keys (`KStream.process`); **`FixedKeyProcessor<KIn, VIn, VOut>`** preserves the key (`KStream.processValues`).
- **Forward** via `ProcessorContext` / `FixedKeyProcessorContext`; **read/write state** via `getStateStore(name)`.
- **Punctuators** (`context.schedule(...)`) run periodically on stream time or wall-clock time.
- **Timers** (`context.timerService()`) fire per key at an event-time or processing-time instant, calling `onTimer(...)`.
- Attach stores by declaring them in the supplier's `stores()` set, or register with `builder.addStateStore(...)`.
::

## Processor vs FixedKeyProcessor

Two interfaces, chosen by whether your logic changes the record key:

| Interface                           | Attach with                  | Can change key? | Forward         |
| ----------------------------------- | ---------------------------- | --------------- | --------------- |
| `Processor<KIn, VIn, KOut, VOut>`   | `KStream.process(...)`       | Yes             | new key + value |
| `FixedKeyProcessor<KIn, VIn, VOut>` | `KStream.processValues(...)` | No (key fixed)  | value only      |

Use `FixedKeyProcessor` whenever you only transform values — a fixed key keeps the record on the lane that already owns it, so per-key state stays serialised without any re-keying at all. Reach for `Processor` when you need to emit records under a different key, and read [Key affinity after a re-key](https://stoatflow.io/#key-affinity-after-a-re-key) before you keep per-key state in one.

Both interfaces are in `io.stoatflow.core.processor`. The convenience base classes `ContextualProcessor` and `ContextualFixedKeyProcessor` store the context for you so you don't have to keep a field; `context()` returns it after `init`.

### A value-only processor

`FixedKeyProcessor` transforms the value and forwards it under the original key. `process(record)` runs once per input record; forward zero records to filter the record out, or more than one to fan out.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.processor.ContextualFixedKeyProcessor
import io.stoatflow.core.processor.FixedKeyRecord

class UppercaseProcessor : ContextualFixedKeyProcessor<String, String, String>() {
    override fun process(record: FixedKeyRecord<String, String>) {
        val value = record.value
        if (value.isNotBlank()) {
            // key and timestamp are preserved automatically
            context().forward(value.uppercase())
        }
    }
}
```

```java [Java]
import io.stoatflow.core.processor.ContextualFixedKeyProcessor;
import io.stoatflow.core.processor.FixedKeyRecord;

public class UppercaseProcessor
        extends ContextualFixedKeyProcessor<String, String, String> {

    @Override
    public void process(FixedKeyRecord<String, String> record) {
        String value = record.value();
        if (value != null && !value.isBlank()) {
            // key and timestamp are preserved automatically
            context().forward(value.toUpperCase());
        }
    }
}
```
::

### A key-changing processor

`Processor` can emit records under a different key. `KStream.process(...)` is a key-changing operation, so downstream stateful operators see the new key.

::callout{color="info" icon="i-lucide-info"}
`process()` and `processValues()` do **not** open a sub-topology boundary of their own, even after a re-key and even with connected stores — matching Kafka Streams exactly. Before 1.0.0 StoatFlow was stricter here and always opened one; that was the single remaining place a ported topology's `describe()` showed an extra sub-topology, and it is gone. Set `topology.processor-api-key-affinity: presumed` to restore the old boundary. What this costs you after a **many-to-one** re-key is [below](https://stoatflow.io/#key-affinity-after-a-re-key). See also [Lanes and parallelism](https://stoatflow.io/docs/concepts/lanes-and-parallelism#where-the-handoff-actually-happens).
::

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.processor.ContextualProcessor
import io.stoatflow.core.processor.Record

class RouteByTypeProcessor : ContextualProcessor<String, Event, String, Event>() {
    override fun process(record: Record<String, Event>) {
        val event = record.value
        // re-key on the event type, keeping the original timestamp
        context().forward(Record(event.type, event, record.timestamp))
    }
}
```

```java [Java]
import io.stoatflow.core.processor.ContextualProcessor;
import io.stoatflow.core.processor.Record;

public class RouteByTypeProcessor
        extends ContextualProcessor<String, Event, String, Event> {

    @Override
    public void process(Record<String, Event> record) {
        Event event = record.value();
        // re-key on the event type, keeping the original timestamp
        context().forward(new Record<>(event.type(), event, record.timestamp()));
    }
}
```
::

## Suppliers

`process` and `processValues` take a **supplier**, not a processor instance — the engine creates **one instance per key-affinity lane**, calling `get()` for each:

- `ProcessorSupplier<KIn, VIn, KOut, VOut>` — `get()` returns a `Processor`.
- `FixedKeyProcessorSupplier<KIn, VIn, VOut>` — `get()` returns a `FixedKeyProcessor`.

`get()` **must return a fresh instance every time** (Kafka Streams' contract; a supplier that hands back the same reference twice fails loudly). Both are functional interfaces, so a lambda or method reference (`::new`) works. They also expose a `stores()` method (default: empty) for declaring state stores inline with the processor — see [Attaching state stores](https://stoatflow.io/#attaching-state-stores).

::callout{color="info" icon="i-lucide-info"}
**One instance per lane — like a Kafka Streams task.** A key-affinity lane is StoatFlow's analog of a KS task/stream-thread, and each gets its own single-threaded processor instance. So your **instance fields are never raced** — no synchronization needed, exactly as in KS. Three consequences: (1) a punctuator, which fires once over global state, cannot see another lane's instance fields; (2) after a **many-to-one re-key**, one key's records can arrive on several lanes, so per-key state in an instance field is **spread across those lanes' instances** — again exactly as in KS, where it is spread across tasks (see [Key affinity after a re-key](https://stoatflow.io/#key-affinity-after-a-re-key)); (3) for state that must be **shared across lanes or read by a punctuator**, use a **state store** — StoatFlow's global-state channel — not a plain instance field. See [State and thread safety](https://stoatflow.io/docs/concepts/state-and-thread-safety) and [Attaching state stores](https://stoatflow.io/#attaching-state-stores).
::

::callout{color="info" icon="i-lucide-info"}
**Low-level `Topology.addProcessor` uses one extra instance.** A processor added via the low-level `Topology.addProcessor` API gets a **dedicated punctuator-owner instance** in addition to the per-lane record instances — its `init()` is what registers `context.schedule(...)` punctuators, so a punctuator-only processor with no input records still fires. Expect `init()`/`close()` once **per instance** (twice at one lane), records only on lane instances, and punctuators only on the owner. The DSL `process()`/`processValues()` paths instead designate the first lane instance as owner (one instance at one lane). Deliberate divergence from KS's one-instance-per-task model — see the compatibility matrix.
::

::code-tabs{group="lang"}
```kotlin [Kotlin]
val routed: KStream<String, Event> =
    events.process({ RouteByTypeProcessor() }, Named.`as`("route-by-type"))

val shouted: KStream<String, String> =
    lines.processValues({ UppercaseProcessor() }, Named.`as`("uppercase"))
```

```java [Java]
KStream<String, Event> routed =
    events.process(RouteByTypeProcessor::new, Named.as("route-by-type"));

KStream<String, String> shouted =
    lines.processValues(UppercaseProcessor::new, Named.as("uppercase"));
```
::

::callout{color="info" icon="i-lucide-info"}
The `Named` argument is optional but recommended — StoatFlow uses these stable names for the topology graph, metrics, and state-store identity. The same convention as the DSL operators in [Your first app](https://stoatflow.io/docs/getting-started/first-app).
::

## ProcessorContext

`init(context)` is called once before any records. Store the context (or extend a `Contextual*` base class and use `context()`). The context gives you forwarding, record metadata, time access, state stores, punctuator scheduling, and the timer service.

### Forwarding

`ProcessorContext` (key-changing) forwards a full `Record` or a key/value pair:

| Method                                                          | Behaviour                                                                                                                                                       |
| --------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `forward(record)`                                               | Forward a `Record<KOut, VOut>` — the record's timestamp **and headers** ride to the sink (set them via `record.withTimestamp(...)` / `record.withHeaders(...)`) |
| `forward(key, value)`                                           | Forward using the current record's timestamp and headers                                                                                                        |
| `forward(key, value, timestamp)`                                | Forward with an explicit timestamp                                                                                                                              |
| `forward(record, childName)` / `forward(key, value, childName)` | Forward to a specific named child node                                                                                                                          |

`FixedKeyProcessorContext` forwards values only — the key is fixed: `forward(value)`, `forward(value, timestamp)`, and the `FixedKeyRecord` / `childName` variants. Call `key()` on the context to read the (immutable) key.

### Record metadata and time

Both contexts expose the metadata of the record currently being processed and the engine's time notions:

| Method                                 | Returns                                                                                                                                                  |
| -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `timestamp()`                          | Timestamp of the current record                                                                                                                          |
| `headers()`                            | The **source record's** headers (`org.apache.kafka.common.header.Headers`); `record.headers()` carries the same. Empty in timer and punctuator callbacks |
| `topic()` / `partition()` / `offset()` | Source coordinates of the current record                                                                                                                 |
| `applicationId()`                      | The configured application ID                                                                                                                            |
| `currentWatermarkMs()`                 | Global event-time progress (min watermark across non-idle partitions); may be `Long.MIN_VALUE` before any events                                         |
| `currentStreamTimeMs()`                | KS-compatible alias for `currentWatermarkMs()`                                                                                                           |
| `currentSystemTimeMs()`                | Wall-clock time (`System.currentTimeMillis()`)                                                                                                           |
| `getStateStore(name)`                  | A state store by name                                                                                                                                    |

`ProcessorContext` additionally has `isLate()` — true when the current record's timestamp is below the watermark. It's only meaningful during `process(...)`; in timer and punctuator callbacks it returns false. See [Event time and watermarks](https://stoatflow.io/docs/concepts/event-time-and-watermarks).

::callout{color="info" icon="i-lucide-database"}
State stores are epoch-scoped and accessed only from within `process(...)`, `onTimer(...)`, and punctuators — not from `init` background threads of your own making. Reads see your own uncommitted writes; the engine flushes and makes them durable at commit barriers. See [State and thread safety](https://stoatflow.io/docs/concepts/state-and-thread-safety).
::

## Attaching state stores

A processor reads and writes a store by name via `context.getStateStore(name)`. **The store must be connected to that processor** — asking for one that isn't throws `StoreNotConnectedException`, exactly as Kafka Streams does. There are two ways to connect one, and they union:

### Option A — declare stores in the supplier

Override `stores()` on the supplier to return the `StateStoreBuilder`s the processor needs. The engine creates and connects them automatically. Build store builders with the `Stores` factory in `io.stoatflow.core.state`.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.processor.ProcessorSupplier
import io.stoatflow.core.state.StateStoreBuilder
import io.stoatflow.core.state.Stores
import org.apache.kafka.common.serialization.Serdes

val supplier =
    object : ProcessorSupplier<String, Event, String, Event> {
        override fun get() = SessionProcessor("sessions")

        override fun stores(): Set<StateStoreBuilder<*>> =
            setOf(
                Stores.keyValueStoreBuilder(
                    Stores.persistentKeyValueStore("sessions"),
                    Serdes.String(),
                    sessionSerde,
                ),
            )
    }

val out = events.process(supplier, Named.`as`("sessionize"))
```

```java [Java]
import io.stoatflow.core.processor.Processor;
import io.stoatflow.core.processor.ProcessorSupplier;
import io.stoatflow.core.state.StateStoreBuilder;
import io.stoatflow.core.state.Stores;
import org.apache.kafka.common.serialization.Serdes;

import java.util.Set;

ProcessorSupplier<String, Event, String, Event> supplier =
    new ProcessorSupplier<>() {
        @Override
        public Processor<String, Event, String, Event> get() {
            return new SessionProcessor("sessions");
        }

        @Override
        public Set<StateStoreBuilder<?>> stores() {
            return Set.of(
                Stores.keyValueStoreBuilder(
                    Stores.persistentKeyValueStore("sessions"),
                    Serdes.String(),
                    sessionSerde));
        }
    };

KStream<String, Event> out = events.process(supplier, Named.as("sessionize"));
```
::

### Option B — register on the builder

Register the store on the `StreamsBuilder` with `addStateStore(...)`, then **name it in the `process(...)` call** — registering alone does not connect it to any processor. `persistentKeyValueStore` is the recommended store type for production (RocksDB, changelog-backed); `inMemoryKeyValueStore` is available for non-durable state. See [State stores](https://stoatflow.io/docs/building/state-stores).

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.state.Stores
import org.apache.kafka.common.serialization.Serdes

builder.addStateStore(
    Stores.keyValueStoreBuilder(
        Stores.persistentKeyValueStore("sessions"),
        Serdes.String(),
        sessionSerde,
    ),
)

// The trailing store name is what connects "sessions" to this processor.
val out = events.process({ SessionProcessor("sessions") }, Named.`as`("sessionize"), "sessions")
```

```java [Java]
import io.stoatflow.core.state.Stores;
import org.apache.kafka.common.serialization.Serdes;

builder.addStateStore(Stores.keyValueStoreBuilder(
    Stores.persistentKeyValueStore("sessions"),
    Serdes.String(),
    sessionSerde));

// The trailing store name is what connects "sessions" to this processor.
KStream<String, Event> out =
    events.process(() -> new SessionProcessor("sessions"), Named.as("sessionize"), "sessions");
```
::

::callout{color="warning" icon="i-lucide-lock"}
**You can only read a store you declared.** `getStateStore(name)` throws `StoreNotConnectedException` — a fatal, non-recoverable error — if `name` is not connected to this processor node. The message names the processor, the store, and what the node actually declares. Two categories are exempt and need no declaration: **global stores** (`addGlobalStore(...)` and `globalTable(...)`), which are handed to an undeclaring processor **read-only** — declare one anyway and you get write access, a StoatFlow extension Kafka Streams rejects outright; and **read-only stores** (`addReadOnlyStateStore(...)`).
::

Note the failure surfaces on the **first record** rather than at startup, because StoatFlow creates one processor instance per lane on demand — including the `init` where the lookup usually lives. Kafka Streams reports the same mistake at task startup. It is deliberately not routable to `processing.exception.handler`: a `CONTINUE` or DLQ handler cannot swallow a topology-configuration error.

::callout{color="info" icon="i-lucide-share-2"}
**Shared or punctuator-visible state goes in a store, not an instance field.** Because each lane has its own processor instance, state that must be visible across lanes — or read/written from a punctuator (punctuators fire once over global state) — belongs in a state store, StoatFlow's global-state channel. It's the same mechanism the built-in windowed operators use (lanes write in `process`; the punctuation lane reads to emit). Punctuators have **full read/write** store access. For ephemeral scratch that doesn't need to survive a restart, use an **in-memory, logging-disabled** store:

```kotlin
builder.addStateStore(
    Stores
        .keyValueStoreBuilder(Stores.inMemoryKeyValueStore("buffer"), Serdes.String(), valueSerde)
        .withLoggingDisabled(), // ephemeral — no changelog topic
)
```

This is the drop-in replacement for a Kafka Streams processor that kept cross-record state in a plain instance field and read it from a punctuator.
::

### Reading and writing the store

Look up the store in `init`, then use it in `process` (and `onTimer`). `getStateStore` is generic over the store type; in Kotlin pass the type argument, in Java assign to a typed variable. The same connection rule applies in every callback — `process`, `onTimer` and punctuators all see the node's declared set.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.processor.ContextualProcessor
import io.stoatflow.core.processor.ProcessorContext
import io.stoatflow.core.processor.Record
import io.stoatflow.core.state.KeyValueStore

class CountingProcessor : ContextualProcessor<String, String, String, Long>() {
    private lateinit var counts: KeyValueStore<String, Long>

    override fun init(context: ProcessorContext<String, Long>) {
        super.init(context)
        counts = context.getStateStore("counts")
    }

    override fun process(record: Record<String, String>) {
        val next = (counts.get(record.key) ?: 0L) + 1
        counts.put(record.key, next)
        context().forward(record.key, next)
    }
}
```

```java [Java]
import io.stoatflow.core.processor.ContextualProcessor;
import io.stoatflow.core.processor.ProcessorContext;
import io.stoatflow.core.processor.Record;
import io.stoatflow.core.state.KeyValueStore;

public class CountingProcessor
        extends ContextualProcessor<String, String, String, Long> {

    private KeyValueStore<String, Long> counts;

    @Override
    public void init(ProcessorContext<String, Long> context) {
        super.init(context);
        this.counts = context.getStateStore("counts");
    }

    @Override
    public void process(Record<String, String> record) {
        long current = counts.get(record.key()) == null ? 0L : counts.get(record.key());
        long next = current + 1;
        counts.put(record.key(), next);
        context().forward(record.key(), next);
    }
}
```
::

## Key affinity after a re-key

Normally a key's records all run on one lane, so per-key read-modify-write is serial and you need no locking. **A many-to-one re-key immediately upstream of a Processor API node breaks that** — and it is the one shape StoatFlow can detect at build time. (The other way to break it is to write a store under a key that is not the record key; see [state and thread safety](https://stoatflow.io/docs/concepts/state-and-thread-safety#the-one-case-that-needs-coordination-cross-key-atomic-updates). That one is computed inside your processor at runtime, so no compiler can see it.)

Since 1.0.0 a Processor API node does not, by itself, require key affinity (`topology.processor-api-key-affinity: off`) — matching Kafka Streams, which never materialises a repartition before `process()` / `processValues()` even with connected stores. StoatFlow lanes records by the key they arrived with, so:

| Shape                                                    | What happens                                                                                                          |
| -------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
| No re-key upstream                                       | Unchanged — one lane per key.                                                                                         |
| **Injective** re-key (`k → "user-$k"`, 1:1)              | Unchanged — distinct new keys still map to distinct lanes. Safe for free.                                             |
| Node declares a timer (`declaredTimerTypes()` non-empty) | Unchanged — the boundary is kept, because the timer service hashes the *new* key while records lane by the *old* one. |
| **Many-to-one** re-key (several old keys → one new key)  | Records for that new key arrive on **several** lanes and are processed concurrently.                                  |

In the last row, per-key state is **fragmented across lanes**: instance fields stay race-free (one instance per lane), but a given key's state now lives in several of them — which is exactly what happens across *tasks* in Kafka Streams after the same re-key without a `repartition()`. State in a **store** is worse than fragmented, and it is worth being precise about why.

::callout{color="warning" icon="i-lucide-triangle-alert"}
**Neither `compute()` nor a lock closes this.** `compute(key, fn)` and `merge(...)` are atomic **within a commit epoch**. Across a barrier rotation they are not, and no user-level lock can make them be: a lane still draining epoch N is structurally forbidden from reading a value another lane wrote into epoch N+1 — allowing it would import uncommitted state into a committed transaction. So both writes can derive from the same base and one is **durably lost**, with the lock held. A lock makes the safe ordering deterministic when it already exists; it cannot create it.
::

### StoatFlow refuses to compile the provable case

When a re-key feeds a **store-connected** Processor API node with no boundary between them, the topology fails to build — in your own test suite, with no broker, because the check runs at topology-compile time:

```text
Topology validation failed: 1 violation(s) configured as 'error'.

  - [papi-key-affinity-presumed] Processor API node 'papi' connects 'counts' and follows the
    re-key at 'rekey' with no sub-topology boundary between them, so records reach it on the
    lane of their OLD key. ...
```

Three remedies, in order of preference:

1. **Insert `repartition()` before the node.** In StoatFlow this is an **in-memory lane hand-off, not a Kafka topic** — no broker round-trip, no extra latency budget, no topic to provision. If you are arriving from Kafka Streams, this is much cheaper than it looks there.
2. **Set `topology.processor-api-key-affinity: presumed`** to restore the boundary for every Processor API node in the application.
3. **Stop keying the store by the post-re-key key** — keep it keyed by something that is still lane-stable.

If the re-key really is injective, downgrade the rule with `topology.validation.papi-key-affinity-presumed: warn`. Read-only stores (KIP-813) need no action — they cannot be written, so they are excluded from the check.

### When the re-key is only *presumed*

`process()` and `Topology.addProcessor()` declare a key change unconditionally, because StoatFlow cannot see inside your `Processor`. So in a chain like `process(parse).process(count, "store")` — the shape a KSML-generated topology is made of — the "re-key" may not exist at all, and Kafka Streams compiles and runs it without a repartition.

Failing that build would be a refusal on evidence that does not exist, so it gets its own rule, &#x2A;*`papi-key-affinity-presumed-chain`, defaulting to `warn`**:

```text
Topology validation [papi-key-affinity-presumed-chain]: Processor API node 'count' connects
'store' and follows the Processor API node 'parse' with no sub-topology boundary between them.
StoatFlow cannot see whether 'parse' changes the key ... but IF it does, and many-to-one, then ...
```

Only you can answer it. If `parse` does not change the key, nothing is wrong — set the rule to `off`. If it does, the remedies above apply. Making the re-key explicit upstream (a `selectKey`) also moves the finding to the `error`-default rule, where it belongs.

### What the build-time check cannot see

Two things.

**Per-lane instance fields.** No declaration can express them, so nothing can prove the shape. After a many-to-one re-key with the setting off, per-key state held in a processor's own fields is fragmented across lanes — exactly as it is across tasks in Kafka Streams. State held in a **store** is the case StoatFlow refuses to build, because that is the only place StoatFlow would be *worse* than Kafka Streams (KS fragments the value across N stores; StoatFlow can lose an update) rather than merely equal to it.

**A store key that is not the record key.** `store.put(record.value().customerId, …)` has the same cross-barrier window with no re-key anywhere, because the key you store under is not the key you are laned by. It is computed at runtime, so the compiler never sees it. [State and thread safety](https://stoatflow.io/docs/concepts/state-and-thread-safety#the-one-case-that-needs-coordination-cross-key-atomic-updates) covers what a lock does and does not buy you there.

Boundaries elided under `off` are logged once at startup, at `INFO`, naming the nodes — so an elided boundary is visible somewhere other than this page.

::callout{color="info" icon="i-lucide-info"}
Eliding the boundary also puts the node in the **upstream** sub-topology, so it inherits that sub-topology's lane count — including a `Consumed.withNumberOfLanes(...)` set on the source. Under `presumed` it would have started its own. If you tuned lanes per sub-topology, re-check them after the flip.
::

## Punctuators

A **punctuator** is a callback that runs periodically — useful for heartbeats, metrics, or flushing buffered state. Schedule one in `init` via `context.schedule(...)`. It returns a `Cancellable` you can use to stop it (for example in `close()`).

`schedule` takes an interval, a `TimeNotion`, an optional `PunctuatorMode`, and the callback. The callback receives the trigger timestamp.

### Time notion

`TimeNotion` is the same type used for timers and `StreamsBuilder.scheduled(...)`. It carries both StoatFlow/Kafka-Streams and Flink naming for the same two concepts:

| Constant                     | Alias             | Fires when                                    | Timestamp passed        |
| ---------------------------- | ----------------- | --------------------------------------------- | ----------------------- |
| `TimeNotion.STREAM_TIME`     | `EVENT_TIME`      | the watermark advances past the next interval | the current watermark   |
| `TimeNotion.WALL_CLOCK_TIME` | `PROCESSING_TIME` | system time reaches the next interval         | the current system time |

From Java, the `PunctuationType` object exposes `STREAM_TIME` / `WALL_CLOCK_TIME` and `TimeDomain` exposes `EVENT_TIME` / `PROCESSING_TIME` — both resolve to the same `TimeNotion` constants.

### Punctuator mode

`PunctuatorMode` controls how a punctuator interacts with commit barriers:

| Mode                 | Behaviour                                                                                                                                                                                                                  |
| -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `BLOCKING` (default) | The next commit waits for the punctuator to finish; records it forwards commit atomically with that barrier. Strong consistency, higher latency.                                                                           |
| `NON_BLOCKING`       | The punctuator runs asynchronously; records it forwards join whichever barrier is active when they're enqueued. Lower latency, no atomicity guarantee across the run. Best for fire-and-forget work (monitoring, metrics). |

::callout{color="warning" icon="i-lucide-triangle-alert"}
Punctuators execute on a dedicated **punctuation lane** per sub-topology (not on the per-key record lanes), with full state-store access in an epoch-aligned context. Because they run concurrently with the record lanes, a punctuator's read-modify-write on a key is **not** serialized against that key's record processing. For per-key mutations on a time trigger, prefer a **timer** (next section), whose `onTimer` callback runs in the same per-key serialized context as `process(...)`. Each punctuator also has locked-run semantics: if a run is still in progress when the next interval fires, that trigger is skipped.
::

### Example

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.processor.Cancellable
import io.stoatflow.core.processor.ContextualProcessor
import io.stoatflow.core.processor.ProcessorContext
import io.stoatflow.core.processor.PunctuatorMode
import io.stoatflow.core.processor.Record
import io.stoatflow.core.processor.TimeNotion
import java.time.Duration

class HeartbeatProcessor : ContextualProcessor<String, Int, String, Int>() {
    private var schedule: Cancellable? = null

    override fun init(context: ProcessorContext<String, Int>) {
        super.init(context)
        // wall-clock heartbeat every 10 seconds
        schedule =
            context.schedule(Duration.ofSeconds(10), TimeNotion.WALL_CLOCK_TIME) { ts ->
                context.forward("heartbeat", ts.toInt())
            }
    }

    override fun process(record: Record<String, Int>) {
        context().forward(record.key, record.value)
    }

    override fun close() {
        schedule?.cancel()
    }
}
```

```java [Java]
import io.stoatflow.core.processor.Cancellable;
import io.stoatflow.core.processor.ContextualProcessor;
import io.stoatflow.core.processor.ProcessorContext;
import io.stoatflow.core.processor.PunctuationType;
import io.stoatflow.core.processor.Record;
import java.time.Duration;

public class HeartbeatProcessor
        extends ContextualProcessor<String, Integer, String, Integer> {

    private Cancellable schedule;

    @Override
    public void init(ProcessorContext<String, Integer> context) {
        super.init(context);
        // wall-clock heartbeat every 10 seconds
        this.schedule = context.schedule(
            Duration.ofSeconds(10),
            PunctuationType.WALL_CLOCK_TIME,
            ts -> context.forward("heartbeat", (int) ts));
    }

    @Override
    public void process(Record<String, Integer> record) {
        context().forward(record.key(), record.value());
    }

    @Override
    public void close() {
        if (schedule != null) {
            schedule.cancel();
        }
    }
}
```
::

::callout{color="info" icon="i-lucide-info"}
`context.schedule(...)` also has an **anchored** overload that takes a `startTime: Instant`, snapping fire times to a fixed grid aligned to that anchor (KIP-1146) for deterministic, cron-like cadence. For topology-level periodic record generation that isn't tied to a processor, see [Scheduled sources](https://stoatflow.io/docs/building/scheduled-sources).
::

## Timers

A **timer** fires once for a specific key at a specific instant, in event time or processing time. Unlike punctuators, timer callbacks run in the same per-key context as records — so `onTimer(...)` has **full read/write state access** and forwarding, with the same serialized execution as `process(...)` for that key. This follows Flink's timer semantics and is the right tool for per-key timeouts, debounce windows, and scheduled emissions.

### Declaring timer types

A processor that uses timers must override `declaredTimerTypes()` to return the `TimeNotion`s it will register. The engine uses this both to validate at runtime (registering an undeclared type throws `IllegalStateException`) and to optimise the hot path (it can skip watermark tracking when no processor declares event-time timers). The default is the empty set.

### Registering and handling timers

Get the timer service in `init` via `context.timerService()`. Register timers in `process` (or `onTimer`), and handle them by overriding `onTimer(timestamp, key, context)`.

| `TimerService<K>` method                         | Effect                                            |
| ------------------------------------------------ | ------------------------------------------------- |
| `registerEventTimeTimer(key, timestamp)`         | Fire when the watermark advances past `timestamp` |
| `registerProcessingTimeTimer(key, timestamp)`    | Fire when wall-clock time reaches `timestamp`     |
| `deleteEventTimeTimer(key, timestamp)`           | Cancel a pending event-time timer                 |
| `deleteProcessingTimeTimer(key, timestamp)`      | Cancel a pending processing-time timer            |
| `currentWatermark()` / `currentProcessingTime()` | Read the current stream time / wall-clock time    |

Timers are deduplicated by `(key, timestamp)` — registering the same pair twice has no extra effect, and the timer fires exactly once.

The `onTimer` callback receives a `TimerContext`, which provides `forward(...)`, `getStateStore(name)` (read/write), `timerService()` to register follow-up timers, the scheduled `timestamp()`, and `timeDomain()` to distinguish event-time from processing-time timers.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.processor.ContextualProcessor
import io.stoatflow.core.processor.ProcessorContext
import io.stoatflow.core.processor.Record
import io.stoatflow.core.processor.TimeNotion
import io.stoatflow.core.processor.TimerContext
import io.stoatflow.core.processor.TimerService
import io.stoatflow.core.state.KeyValueStore

private const val SESSION_TIMEOUT_MS = 30_000L

class SessionProcessor :
    ContextualProcessor<String, Event, String, Session>() {

    private lateinit var sessions: KeyValueStore<String, Session>
    private lateinit var timers: TimerService<String>

    override fun declaredTimerTypes(): Set<TimeNotion> = setOf(TimeNotion.EVENT_TIME)

    override fun init(context: ProcessorContext<String, Session>) {
        super.init(context)
        sessions = context.getStateStore("sessions")
        timers = context.timerService()
    }

    override fun process(record: Record<String, Event>) {
        val session = sessions.get(record.key)?.add(record.value)
            ?: Session.start(record.value)
        sessions.put(record.key, session)
        // (re)arm the session-timeout timer
        timers.registerEventTimeTimer(record.key, record.timestamp + SESSION_TIMEOUT_MS)
    }

    override fun onTimer(
        timestamp: Long,
        key: String,
        context: TimerContext<String, Session>,
    ) {
        val session = context.getStateStore<KeyValueStore<String, Session>>("sessions").get(key)
        if (session != null && session.lastActivity + SESSION_TIMEOUT_MS <= timestamp) {
            context.forward(key, session.complete())
            context.getStateStore<KeyValueStore<String, Session>>("sessions").delete(key)
        }
    }
}
```

```java [Java]
import io.stoatflow.core.processor.ContextualProcessor;
import io.stoatflow.core.processor.ProcessorContext;
import io.stoatflow.core.processor.Record;
import io.stoatflow.core.processor.TimeNotion;
import io.stoatflow.core.processor.TimerContext;
import io.stoatflow.core.processor.TimerService;
import io.stoatflow.core.state.KeyValueStore;
import org.jspecify.annotations.NonNull;

import java.util.Set;

public class SessionProcessor
        extends ContextualProcessor<String, Event, String, Session> {

    private static final long SESSION_TIMEOUT_MS = 30_000L;

    private KeyValueStore<String, Session> sessions;
    private TimerService<String> timers;

    @Override
    public @NonNull Set<TimeNotion> declaredTimerTypes() {
        return Set.of(TimeNotion.EVENT_TIME);
    }

    @Override
    public void init(ProcessorContext<String, Session> context) {
        super.init(context);
        this.sessions = context.getStateStore("sessions");
        this.timers = context.timerService();
    }

    @Override
    public void process(Record<String, Event> record) {
        Session existing = sessions.get(record.key());
        Session session = existing == null
            ? Session.start(record.value())
            : existing.add(record.value());
        sessions.put(record.key(), session);
        // (re)arm the session-timeout timer
        timers.registerEventTimeTimer(record.key(), record.timestamp() + SESSION_TIMEOUT_MS);
    }

    @Override
    public void onTimer(long timestamp, String key, @NonNull TimerContext<String, Session> context) {
        KeyValueStore<String, Session> store = context.getStateStore("sessions");
        Session session = store.get(key);
        if (session != null && session.lastActivity() + SESSION_TIMEOUT_MS <= timestamp) {
            context.forward(key, session.complete());
            store.delete(key);
        }
    }
}
```
::

::callout{color="info" icon="i-lucide-clock"}
Pick **event-time timers** for data-driven timeouts (sessionization, watermark-aligned flushing) — they advance with your data and are deterministic on replay. Pick **processing-time timers** for wall-clock deadlines that must fire regardless of data flow (the news-portal "publish at scheduled time" pattern). A persistent (RocksDB) timer backend survives restarts; with an in-memory backend, pending timers are rebuilt from your own state on recovery.
::

## A complete processor app

Putting it together: register the store, attach the processor with `process(...)`, and run it on the runtime. The processor reads its store by name, forwards results, and the runtime wires up Kafka, the HTTP/metrics server, and graceful shutdown — see [Your first app](https://stoatflow.io/docs/getting-started/first-app) for the runtime entry point.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.state.Stores
import io.stoatflow.core.topology.Consumed
import io.stoatflow.core.topology.Named
import io.stoatflow.core.topology.Produced
import io.stoatflow.core.topology.StreamsBuilder
import io.stoatflow.runtime.StoatFlowRuntime
import org.apache.kafka.common.serialization.Serdes

fun main() {
    val runtime = StoatFlowRuntime.fromConfig(
        topologyBuilder = { buildTopology(it) },
        configure = {
            streamsConfigOverrides {
                defaultKeySerde(Serdes.String())
                defaultValueSerde(Serdes.String())
            }
        },
    )
    runtime.start()
    runtime.awaitTermination()
}

private fun buildTopology(builder: StreamsBuilder) {
    builder.addStateStore(
        Stores.keyValueStoreBuilder(
            Stores.persistentKeyValueStore("counts"),
            Serdes.String(),
            Serdes.Long(),
        ),
    )

    builder
        .stream<String, String>("input", Consumed.`as`("source"))
        .process({ CountingProcessor() }, Named.`as`("count"), "counts")
        .to("counts-out", Produced.`as`<String, Long>("sink").withValueSerde(Serdes.Long()))
}
```

```java [Java]
import io.stoatflow.core.state.Stores;
import io.stoatflow.core.topology.Consumed;
import io.stoatflow.core.topology.KStream;
import io.stoatflow.core.topology.Named;
import io.stoatflow.core.topology.Produced;
import io.stoatflow.core.topology.StreamsBuilder;
import io.stoatflow.runtime.StoatFlowRuntime;
import org.apache.kafka.common.serialization.Serdes;

public class Main {

    public static void main(String[] args) {
        var runtime = StoatFlowRuntime.fromConfig(
            Main::buildTopology,
            builder -> builder.streamsConfigOverrides(cfg -> {
                cfg.defaultKeySerde(Serdes.String());
                cfg.defaultValueSerde(Serdes.String());
            })
        );
        runtime.start();
        runtime.awaitTermination();
    }

    private static void buildTopology(StreamsBuilder builder) {
        builder.addStateStore(Stores.keyValueStoreBuilder(
            Stores.persistentKeyValueStore("counts"),
            Serdes.String(),
            Serdes.Long()));

        KStream<String, String> input = builder.stream("input", Consumed.as("source"));
        input
            .process(CountingProcessor::new, Named.as("count"), "counts")
            .to("counts-out", Produced.<String, Long>as("sink").withValueSerde(Serdes.Long()));
    }
}
```
::

## Processor wrapping (KIP-1112)

Sometimes you want *one* piece of behaviour applied to **every** processor in a topology — structured logging,
per-node metrics, a tracing span, uniform error handling, or swapping in a mock while testing. Rather than
editing every operator, configure a single `ProcessorWrapper` and StoatFlow applies it to every node — DSL
operators and Processor-API nodes alike — when the topology is compiled.

```java
public final class LoggingWrapper implements ProcessorWrapper {

    @Override
    public <KIn, VIn, KOut, VOut> WrappedProcessorSupplier<KIn, VIn, KOut, VOut> wrapProcessorSupplier(
            String processorName, ProcessorSupplier<KIn, VIn, KOut, VOut> supplier) {
        return ProcessorWrapper.asWrapped(() -> {
            Processor<KIn, VIn, KOut, VOut> inner = supplier.get();
            return new Processor<>() {
                @Override public void init(ProcessorContext<KOut, VOut> ctx) { inner.init(ctx); }
                @Override public void process(Record<KIn, VIn> record) {
                    log.debug("{} <- {}", processorName, record.key());
                    inner.process(record);
                }
                @Override public void close() { inner.close(); }
            };
        });
    }

    @Override
    public <KIn, VIn, VOut> WrappedFixedKeyProcessorSupplier<KIn, VIn, VOut> wrapFixedKeyProcessorSupplier(
            String processorName, FixedKeyProcessorSupplier<KIn, VIn, VOut> supplier) {
        return ProcessorWrapper.asWrappedFixedKey(supplier);   // pass through
    }
}
```

Point `processor.wrapper.class` at it:

```properties
processor.wrapper.class=com.acme.LoggingWrapper
```

The Kafka Streams route — passing a `TopologyConfig` to the builder — works and is the portable choice:

```java
StreamsConfig config = StreamsConfig.fromProperties(props).build();
StreamsBuilder builder = new StreamsBuilder(new TopologyConfig(config));
```

Unlike Kafka Streams, StoatFlow **also honours a wrapper set on the runtime config alone** — it compiles the
topology inside `start()`, so KS's "too late, silently ignored" case doesn't exist here. If both are set to
different classes, the `TopologyConfig` one wins and a warning names the shadowed value.

If your wrapper needs settings, implement `configure(Map)` — it is called once at instantiation with the config
map, exactly as in Kafka Streams.

::note
Return `ProcessorWrapper.asWrapped(supplier)` / `asWrappedFixedKey(supplier)` to pass a node through unchanged.
The default is `NoOpProcessorWrapper`, which does exactly that; with no wrapper configured the runtime record
path is untouched, so there is no cost to leaving the key unset.
::

### What a wrapper can and cannot do

|                                                                              |                                                                                                                                                                                                                                                                                                                                                                             |
| ---------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Your Processor-API nodes** (`process` / `processValues` / `addProcessor`)  | Full control — decorate, **replace** the processor outright, or add state stores via the wrapped supplier's `stores()`                                                                                                                                                                                                                                                      |
| **DSL operator nodes** (`mapValues`, `filter`, aggregations, joins, windows) | The **record path**: observe, decorate, or short-circuit. A decorator may rewrite the record before delegating — `record.withTimestamp(...)` / `withHeaders(...)` flow through to the operator, downstream nodes, and the sink. Replacing a DSL operator with a different processor is rejected at build time — they are type-erased internally                             |
| **`init` / watermark / suppress-flush callbacks on DSL nodes**               | Not routed through the wrapper (they aren't on the `Processor` surface) — in practice: window/session close emissions, the barrier suppress flush, and FK-join/suppress restore. A DSL-node decorator's context also cannot `schedule()` or use `timerService()`. **Your timers are unaffected**: keyed timers only exist on Processor-API nodes, which are wrapped in full |
| **Global / read-only store update processors**                               | Not wrapped — Kafka Streams doesn't wrap them either                                                                                                                                                                                                                                                                                                                        |

Two behavioural notes worth knowing before you write a stateful wrapper:

- **`get()` runs once per lane**, not once per task. Each decorator instance is pinned to one lane and single-
  threaded, so per-instance fields are race-free — but they only see that lane's records. Put cross-lane
  counters on the wrapper object itself (and make it thread-safe), or push to an external sink.
- **Node names are StoatFlow's** — `mapValues-0`, `process-0`, or whatever you passed to `Named.as(...)` — not
  Kafka Streams' `KSTREAM-MAPVALUES-0000000001`. A ported wrapper that switches on node-name patterns compiles
  cleanly and then matches nothing.

## Next steps

- **[State stores](https://stoatflow.io/docs/building/state-stores)** — store types, serdes, changelog backing, and querying.
- **[Scheduled sources](https://stoatflow.io/docs/building/scheduled-sources)** — topology-level periodic record generation (`StreamsBuilder.scheduled()`).
- **[Testing](https://stoatflow.io/docs/building/testing)** — drive processors, advance time, and trigger punctuators with the in-memory test driver.
- **[State and thread safety](https://stoatflow.io/docs/concepts/state-and-thread-safety)** — the execution model behind per-key serialized processing and timer callbacks.
- **[How StoatFlow differs from Kafka Streams](https://stoatflow.io/docs/concepts/how-stoatflow-differs-from-ks)** — Processor API parity and the StoatFlow-specific extensions (timers, time notions).


# Scheduled sources

`StreamsBuilder.scheduled()` is a StoatFlow extension — it has no Kafka Streams equivalent. It creates a topology source that periodically generates records and feeds them downstream, without consuming from any Kafka topic. Use it for heartbeats, periodic state-store scans, external polling, or any time-driven event you want to flow through the same DAG as your Kafka-sourced records.

::tldr-panel
- **What it is:** a source node that emits records on a schedule instead of reading a topic. Returns a `KStream<K, V>` you process like any other.
- **Two schedules:** fixed interval (`STREAM_TIME` or `WALL_CLOCK_TIME`) or a cron expression (`CronExpression.unix/quartz/spring`, always wall-clock).
- **The emitter:** a `ScheduledEmitter<K, V>` callback. Its context gives you `forward(...)` and **read-only** state-store access.
- **StoatFlow-only:** not part of the Kafka Streams DSL — see the [Kafka Streams compatibility matrix](https://stoatflow.io/docs/reference/ks-compatibility-matrix) for the parity picture.
::

## Interval-based sources

The interval overload fires every `Duration` according to a [`TimeNotion`](https://stoatflow.io/docs/concepts/event-time-and-watermarks):

- **`WALL_CLOCK_TIME`** — fires on the system clock, every interval, regardless of data flow. Use for heartbeats, metrics, or polling that must happen on real-world time.
- **`STREAM_TIME`** — fires when the event-time watermark advances past the interval. Data-driven: with no records flowing, the watermark doesn't move and the source doesn't fire.

The interval must be positive (`scheduled()` throws `IllegalArgumentException` otherwise).

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.processor.PunctuationType
import io.stoatflow.core.topology.Produced
import io.stoatflow.core.topology.StreamsBuilder
import org.apache.kafka.common.serialization.Serdes
import java.time.Duration

val builder = StreamsBuilder()

// Emit a heartbeat every 30 seconds on wall-clock time
builder
    .scheduled<String, String>(
        interval = Duration.ofSeconds(30),
        type = PunctuationType.WALL_CLOCK_TIME,
        emitter = { context ->
            context.forward("heartbeat", "alive-${context.currentWallClockTime()}")
        },
    )
    .to("heartbeats", Produced.with(Serdes.String(), Serdes.String()))
```

```java [Java]
import io.stoatflow.core.processor.PunctuationType;
import io.stoatflow.core.topology.Produced;
import io.stoatflow.core.topology.StreamsBuilder;
import org.apache.kafka.common.serialization.Serdes;

import java.time.Duration;

StreamsBuilder builder = new StreamsBuilder();

// Emit a heartbeat every 30 seconds on wall-clock time
builder
    .<String, String>scheduled(
        null,                                 // named (optional)
        Duration.ofSeconds(30),
        PunctuationType.WALL_CLOCK_TIME,
        null,                                 // keySerde (optional)
        context ->
            context.forward("heartbeat", "alive-" + context.currentWallClockTime()))
    .to("heartbeats", Produced.with(Serdes.String(), Serdes.String()));
```
::

::callout{color="info" icon="i-lucide-info"}
For Java, the emitter is delivered as a `java.util.function.Consumer<ScheduledEmitterContext<K, V>>`, so a plain lambda works. The Kotlin lambda overload is hidden from Java (`@JvmSynthetic`) to avoid ambiguity. The `named` and `keySerde` parameters have Kotlin defaults; from Java, pass `null` to skip them.
::

### Naming the source

Pass a [`Named`](https://stoatflow.io/docs/building/streams-builder) to give the source a stable name in the topology graph and metrics. Without it, sources are auto-named `scheduled-source-0`, `scheduled-source-1`, … in declaration order — which is the name you'll reference when [testing](https://stoatflow.io/#testing-scheduled-sources).

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.processor.PunctuationType
import io.stoatflow.core.topology.Named
import java.time.Duration

builder.scheduled<String, String>(
    named = Named.`as`("heartbeat-source"),
    interval = Duration.ofSeconds(30),
    type = PunctuationType.WALL_CLOCK_TIME,
    emitter = { context -> context.forward("heartbeat", "alive") },
)
```

```java [Java]
import io.stoatflow.core.processor.PunctuationType;
import io.stoatflow.core.topology.Named;

import java.time.Duration;

builder.<String, String>scheduled(
    Named.as("heartbeat-source"),
    Duration.ofSeconds(30),
    PunctuationType.WALL_CLOCK_TIME,
    null,                                     // keySerde (optional)
    context -> context.forward("heartbeat", "alive"));
```
::

## Cron-based sources

The cron overload fires according to a [`CronExpression`](https://stoatflow.io/#cron-expressions). Cron scheduling **always uses wall-clock time** — there is no `TimeNotion` parameter, and cron sources do not fire on watermark advance.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.processor.CronExpression
import io.stoatflow.core.topology.Named
import io.stoatflow.core.topology.Produced
import org.apache.kafka.common.serialization.Serdes

// Fire every day at midnight (Unix 5-field cron)
builder
    .scheduled<String, String>(
        named = Named.`as`("daily-report-source"),
        cron = CronExpression.unix("0 0 * * *"),
        emitter = { context ->
            context.forward("report", "daily-${context.currentWallClockTime()}")
        },
    )
    .to("daily-reports", Produced.with(Serdes.String(), Serdes.String()))
```

```java [Java]
import io.stoatflow.core.processor.CronExpression;
import io.stoatflow.core.topology.Named;
import io.stoatflow.core.topology.Produced;
import org.apache.kafka.common.serialization.Serdes;

// Fire every day at midnight (Unix 5-field cron)
builder
    .<String, String>scheduled(
        Named.as("daily-report-source"),
        CronExpression.unix("0 0 * * *"),
        null,                                 // keySerde (optional)
        context ->
            context.forward("report", "daily-" + context.currentWallClockTime()))
    .to("daily-reports", Produced.with(Serdes.String(), Serdes.String()));
```
::

### Cron expressions

`CronExpression` wraps three cron dialects. Pick the factory that matches the dialect of your expression — the field counts differ, so an expression valid in one dialect is usually invalid in another. An unparseable expression throws `IllegalArgumentException`.

| Factory                      | Fields | Format                                               | Example         | Meaning               |
| ---------------------------- | ------ | ---------------------------------------------------- | --------------- | --------------------- |
| `CronExpression.unix(...)`   | 5      | `min hour day-of-month month day-of-week`            | `"0 * * * *"`   | top of every hour     |
| `CronExpression.quartz(...)` | 6–7    | `sec min hour day-of-month month day-of-week [year]` | `"0 0 0 * * ?"` | every day at midnight |
| `CronExpression.spring(...)` | 6      | `sec min hour day-of-month month day-of-week`        | `"0 0 0 * * *"` | every day at midnight |

All three factories are `@JvmStatic`, so they're called identically from Kotlin and Java. Day-of-week numbering differs between dialects (Unix `0–6` with `0 = Sunday`; Quartz `1–7` with `1 = Sunday`), so consult the dialect when porting expressions.

::callout{color="info" icon="i-lucide-clock"}
`CronExpression` resolves the next fire time in the **system default time zone** by default. The next-execution helpers (`nextExecutionTime`, `timeUntilNextExecution`) accept an explicit `ZoneId` if you need a fixed zone.
::

## The emitter and its context

The emitter is a `ScheduledEmitter<K, V>` — a single-method functional interface. Each invocation receives a `ScheduledEmitterContext<K, V>` for emitting records and reading state.

### Forwarding records

The context offers three `forward` overloads:

| Method                           | Timestamp               | Headers           |
| -------------------------------- | ----------------------- | ----------------- |
| `forward(key, value)`            | current wall-clock time | none              |
| `forward(key, value, timestamp)` | explicit                | none              |
| `forward(record: Record<K, V>)`  | from the `Record`       | from the `Record` |

A single emitter invocation may forward zero, one, or many records. The key drives downstream routing, exactly as for Kafka-sourced records, so per-key ordering downstream behaves the same way — see [Lanes & parallelism](https://stoatflow.io/docs/concepts/lanes-and-parallelism).

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.processor.Record
import org.apache.kafka.common.header.internals.RecordHeaders

builder.scheduled<String, String>(
    interval = Duration.ofSeconds(10),
    type = PunctuationType.WALL_CLOCK_TIME,
    emitter = { context ->
        // simplest: key + value, timestamped now
        context.forward("k1", "v1")

        // explicit event-time timestamp
        context.forward("k2", "v2", context.currentWallClockTime())

        // full control via Record (key, value, timestamp, headers)
        val headers = RecordHeaders().apply { add("source", "scheduled".toByteArray()) }
        context.forward(Record("k3", "v3", context.currentWallClockTime(), headers))
    },
)
```

```java [Java]
import io.stoatflow.core.processor.Record;
import org.apache.kafka.common.header.internals.RecordHeaders;

builder.<String, String>scheduled(
    null,
    Duration.ofSeconds(10),
    PunctuationType.WALL_CLOCK_TIME,
    null,
    context -> {
        // simplest: key + value, timestamped now
        context.forward("k1", "v1");

        // explicit event-time timestamp
        context.forward("k2", "v2", context.currentWallClockTime());

        // full control via Record (key, value, timestamp, headers)
        var headers = new RecordHeaders();
        headers.add("source", "scheduled".getBytes());
        context.forward(new Record<>("k3", "v3", context.currentWallClockTime(), headers));
    });
```
::

### Reading state stores

The context exposes `getStateStore(name)` for **read-only** access. Write operations (`put`, `delete`) on a store obtained this way throw `UnsupportedOperationException`. This makes the scheduled source a natural fit for periodically scanning state and emitting derived events — for example, scanning pending items and emitting a "publish" event once their scheduled time has passed.

::callout{color="info" icon="i-lucide-shield-check"}
**The write refusal covers every store family.** Plain key-value, window and session stores, and equally the *timestamped*, *versioned* and *headers-aware* ones — including the timestamped store a DSL `Materialized` aggregation or table produces, which is the common case for exactly the "scan state, emit derived events" pattern above. Whatever store type you ask for, you get an object of that type back, and its writes throw.
::

The store you get back keeps its declared type, so `getStateStore<TimestampedKeyValueStore<String, Long>>("counts")` returns something that really is a `TimestampedKeyValueStore` — reads work as usual, and only the write methods throw. This mirrors Kafka Streams, whose global-store decorators work the same way.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.state.ReadOnlyKeyValueStore

builder.scheduled<String, String>(
    named = Named.`as`("publish-scanner"),
    interval = Duration.ofMinutes(1),
    type = PunctuationType.WALL_CLOCK_TIME,
    emitter = { context ->
        val now = context.currentWallClockTime()
        val pending = context.getStateStore<ReadOnlyKeyValueStore<String, Long>>("pending-publishes")
        pending.all().use { it ->
            it.forEach { entry ->
                if (entry.value <= now) {
                    context.forward(entry.key, "published")
                }
            }
        }
    },
)
```

```java [Java]
import io.stoatflow.core.state.ReadOnlyKeyValueStore;
import io.stoatflow.core.state.KeyValueIterator;
import io.stoatflow.core.state.KeyValue;

builder.<String, String>scheduled(
    Named.as("publish-scanner"),
    Duration.ofMinutes(1),
    PunctuationType.WALL_CLOCK_TIME,
    null,
    context -> {
        long now = context.currentWallClockTime();
        ReadOnlyKeyValueStore<String, Long> pending = context.getStateStore("pending-publishes");
        try (KeyValueIterator<String, Long> it = pending.all()) {
            while (it.hasNext()) {
                KeyValue<String, Long> entry = it.next();
                if (entry.value <= now) {
                    context.forward(entry.key, "published");
                }
            }
        }
    });
```
::

`all()`, `range()`, `prefixScan()`, and the reverse variants all return a `KeyValueIterator<K, V>`, which is `Closeable` — close it (Kotlin `use {}`, Java try-with-resources) to release resources. See [State stores](https://stoatflow.io/docs/building/state-stores) for the full read-only query surface.

::callout{color="info" icon="i-lucide-info"}
For the time helpers: `currentWallClockTime()` and `currentSystemTimeMs()` return system time; `currentWatermarkMs()` (also `currentStreamTimeMs()`) returns the global event-time watermark. For a `STREAM_TIME` source, `currentWatermarkMs()` is the watermark value that triggered the firing; for a `WALL_CLOCK_TIME` source it's whatever the global watermark currently is (`Long.MIN_VALUE` if no events have been processed yet).
::

## Processing the result

`scheduled()` returns a `KStream<K, V>` — there's nothing special about it downstream. Map, filter, group, aggregate, join, or merge it with a Kafka-sourced stream, all with the operators from [KStream & KTable](https://stoatflow.io/docs/building/kstream-ktable).

::code-tabs{group="lang"}
```kotlin [Kotlin]
val ticks = builder.scheduled<String, Int>(
    interval = Duration.ofSeconds(5),
    type = PunctuationType.WALL_CLOCK_TIME,
    emitter = { context ->
        context.forward("a", 1)
        context.forward("b", 2)
    },
)

ticks
    .filter { _, value -> value > 1 }
    .mapValues { value -> value * 10 }
    .to("output", Produced.with(Serdes.String(), Serdes.Integer()))
```

```java [Java]
import io.stoatflow.core.topology.KStream;

KStream<String, Integer> ticks = builder.<String, Integer>scheduled(
    null,
    Duration.ofSeconds(5),
    PunctuationType.WALL_CLOCK_TIME,
    null,
    context -> {
        context.forward("a", 1);
        context.forward("b", 2);
    });

ticks
    .filter((key, value) -> value > 1)
    .mapValues(value -> value * 10)
    .to("output", Produced.with(Serdes.String(), Serdes.Integer()));
```
::

To combine a scheduled source with a Kafka topic, keep a reference to each `KStream` and `merge` them:

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.topology.Consumed

val scheduled = builder.scheduled<String, String>(
    interval = Duration.ofSeconds(5),
    type = PunctuationType.WALL_CLOCK_TIME,
    emitter = { context -> context.forward("scheduled", "tick") },
)
val kafka = builder.stream<String, String>("input", Consumed.with(Serdes.String(), Serdes.String()))

scheduled
    .merge(kafka)
    .to("output", Produced.with(Serdes.String(), Serdes.String()))
```

```java [Java]
import io.stoatflow.core.topology.Consumed;
import io.stoatflow.core.topology.KStream;

KStream<String, String> scheduled = builder.<String, String>scheduled(
    null,
    Duration.ofSeconds(5),
    PunctuationType.WALL_CLOCK_TIME,
    null,
    context -> context.forward("scheduled", "tick"));
KStream<String, String> kafka =
    builder.stream("input", Consumed.with(Serdes.String(), Serdes.String()));

scheduled
    .merge(kafka)
    .to("output", Produced.with(Serdes.String(), Serdes.String()));
```
::

## Testing scheduled sources

The [`TopologyTestDriver`](https://stoatflow.io/docs/building/testing) drives scheduled sources deterministically — no clock waiting. Three controls:

- **`triggerScheduledSource(name)`** — fires a source immediately, once, by name. With no `Named`, the auto name is `scheduled-source-0`, `-1`, … in declaration order.
- **`advanceWallClockTime(duration)`** — advances the test clock; interval (`WALL_CLOCK_TIME`) and cron sources fire for every boundary crossed.
- **`advanceWatermark(timestamp)`** — advances the event-time watermark; `STREAM_TIME` interval sources fire for each interval crossed. Cron sources do **not** fire on watermark advance.

Pass `initialWallClockTime` to `fromBuilder` to pin the starting instant for cron and wall-clock assertions. In Kotlin it's a named, defaulted parameter. From Java, `fromBuilder` is `@JvmOverloads` over `(builder, config, initialWallClockTime)` — to set the instant you must also pass the `StreamsConfig` (there is no two-argument `(builder, Instant)` overload), so construct one explicitly.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.testutils.TopologyTestDriver
import java.time.Instant

val builder = StreamsBuilder()
builder
    .scheduled<String, String>(
        interval = Duration.ofSeconds(10),
        type = PunctuationType.WALL_CLOCK_TIME,
        emitter = { context -> context.forward("key1", "tick") },
    )
    .to("output", Produced.with(Serdes.String(), Serdes.String()))

val driver = TopologyTestDriver.fromBuilder(builder, initialWallClockTime = Instant.ofEpochMilli(0L))
try {
    val output = driver.createOutputTopic("output", Serdes.String(), Serdes.String())

    // Option A: fire once, immediately, by name
    driver.triggerScheduledSource("scheduled-source-0")
    output.readRecord()?.key shouldBe "key1"

    // Option B: advance the clock and let the schedule fire
    driver.advanceWallClockTime(Duration.ofSeconds(10))
    output.readRecord()?.value shouldBe "tick"
} finally {
    driver.close()
}
```

```java [Java]
import io.stoatflow.core.config.StreamsConfig;
import io.stoatflow.testutils.TopologyTestDriver;
import io.stoatflow.testutils.TestOutputTopic;

import java.time.Duration;
import java.time.Instant;

StreamsBuilder builder = new StreamsBuilder();
builder
    .<String, String>scheduled(
        null,
        Duration.ofSeconds(10),
        PunctuationType.WALL_CLOCK_TIME,
        null,
        context -> context.forward("key1", "tick"))
    .to("output", Produced.with(Serdes.String(), Serdes.String()));

// fromBuilder is @JvmOverloads over (builder, config, initialWallClockTime);
// pass the config explicitly to set the starting instant from Java.
StreamsConfig config = StreamsConfig.builder("test-app", "localhost:9092").build();
TopologyTestDriver driver =
    TopologyTestDriver.fromBuilder(builder, config, Instant.ofEpochMilli(0L));
try {
    TestOutputTopic<String, String> output =
        driver.createOutputTopic("output", Serdes.String(), Serdes.String());

    // Option A: fire once, immediately, by name
    driver.triggerScheduledSource("scheduled-source-0");

    // Option B: advance the clock and let the schedule fire
    driver.advanceWallClockTime(Duration.ofSeconds(10));
} finally {
    driver.close();
}
```
::

## Execution semantics

A few behaviours worth knowing when you write an emitter:

- **One run at a time.** Each scheduled source uses locked-run semantics: if a previous invocation is still running when the next fire is due, the new fire is skipped rather than overlapped. Keep emitters reasonably quick, or expect skipped ticks under load.
- **Records flow through the engine like any source.** Emitted records participate in the same exactly-once commit boundaries as Kafka-sourced records — see [Exactly-once semantics](https://stoatflow.io/docs/concepts/exactly-once) and the [architecture overview](https://stoatflow.io/docs/concepts/architecture).
- **State access is read-only.** Use a scheduled source to *observe* state and emit derived events; mutate state from the processors that handle those events, not from the emitter. Enforced for every store family — plain key-value, window and session as well as timestamped, versioned and headers-aware (see [Reading state stores](https://stoatflow.io/#reading-state-stores)).
- **`STREAM_TIME` is data-driven.** A `STREAM_TIME` source only fires as the watermark advances, so it won't fire at all when no records are flowing. Choose `WALL_CLOCK_TIME` or a cron schedule when you need firing independent of throughput.

## See also

- [Building topologies](https://stoatflow.io/docs/building/streams-builder) — `StreamsBuilder` entry points and operator naming.
- [Event time & watermarks](https://stoatflow.io/docs/concepts/event-time-and-watermarks) — what `STREAM_TIME` vs `WALL_CLOCK_TIME` mean.
- [Processor API](https://stoatflow.io/docs/building/processor-api) — punctuators and timers, the per-record analogue of scheduled sources.
- [Testing](https://stoatflow.io/docs/building/testing) — the full `TopologyTestDriver` reference.
- [Kafka Streams compatibility matrix](https://stoatflow.io/docs/reference/ks-compatibility-matrix) — where scheduled sources sit relative to the Kafka Streams DSL.


# Serdes and Avro

A **serde** (serializer + deserializer) tells StoatFlow how to turn your keys and values into bytes on the wire and back. Every source, sink, grouping, and state store needs a key serde and a value serde. StoatFlow uses the standard Kafka `org.apache.kafka.common.serialization.Serde<T>` type, so any serde you already use with Kafka Streams works unchanged.

This page covers where serdes are resolved (defaults vs. per-operator), the built-in serdes, writing your own, serializing windowed keys, and the Avro + Schema Registry integration.

::tldr-panel
- **Default serdes** are set once on the runtime via `streamsConfigOverrides { defaultKeySerde(...); defaultValueSerde(...) }` (or the `default-key-serde` / `default-value-serde` YAML keys).
- **Per-operator serdes** override the defaults via `Consumed`, `Produced`, `Grouped`, and `Materialized` — `.withKeySerde(...)` / `.withValueSerde(...)`.
- **Built-ins** live in `org.apache.kafka.common.serialization.Serdes` (`Serdes.String()`, `Serdes.Long()`, …).
- **Windowed keys** use `WindowedSerdes.timeWindowedSerdeFrom(inner)` / `sessionWindowedSerdeFrom(inner)`.
- **Avro:** use the Confluent `SpecificAvroSerde`, point it at Schema Registry via `schema-registry-url`, and `configure(...)` any serde you build by hand.
::

## How serdes are resolved

StoatFlow resolves the serde for each operator in a fixed order:

1. **An explicit serde on the operator's config object** — `Consumed`, `Produced`, `Grouped`, or `Materialized`. This always wins.
2. **The default key/value serde** configured on the runtime — used whenever the operator config leaves the serde unset (`null`).

If neither is set, serialization for that operator fails at startup. The default serde is the convenient base case when most of your topology uses the same types (e.g. `String` keys); reach for per-operator serdes wherever an individual topic or store differs.

### Boundary key serdes (lane assignment)

There is one more place StoatFlow needs a key serde that Kafka Streams does not: a **sub-topology boundary**. Where a re-keyed record reaches an operator that needs the new key's lane affinity — an aggregation, a join, `toTable()`, or an explicit `repartition()` — records are re-dispatched to key-affinity lanes, and the new key is serialized to bytes for the lane hash, in memory, with no repartition topic.

Since 1.0.0 there are **fewer** of these boundaries than there used to be: a re-key alone no longer opens one, and neither does a Processor API node (`process()` / `processValues()`), matching Kafka Streams — see [Lanes and parallelism](https://stoatflow.io/docs/concepts/lanes-and-parallelism#where-the-handoff-actually-happens) and, for what that costs after a many-to-one re-key, [Key affinity after a re-key](https://stoatflow.io/docs/building/processor-api#key-affinity-after-a-re-key). A topology whose re-key feeds only stateless or Processor API operators now needs no boundary key serde at all.

StoatFlow resolves the boundary key serde from whatever your topology declares for **that key**, searching upstream first and then downstream, always within the region where the key keeps its type:

1. **Upstream**: the nearest ancestor declaration — the source's `Consumed` key serde, or a `repartition(Repartitioned.withKeySerde(...))` the key flowed through.
2. **Downstream**: the nearest declaration for the *new* key — an explicit `Repartitioned`, a `Grouped`, a sink's `Produced`, or an aggregation's `Materialized.withKeySerde(...)`.
3. **Last resort**: if you never configured a default key serde, the nearest declaration from *before* the re-key — inherited across it. See [Last-resort inheritance](https://stoatflow.io/#last-resort-inheritance) below.
4. **The configured default key serde** otherwise.

Two rules keep steps 1 and 2 predictable:

- A declaration only counts for the key it actually describes. After a re-key, upstream serdes no longer apply; a `Grouped` serde describes the *grouping* key, never the key arriving at the `groupBy`; and two **adjacent** re-keys find nothing in either step — no declaration sits between them to describe the intermediate key, so that boundary falls through to the last two steps.
- If two different serde classes are reachable for the same key, StoatFlow declines to guess.

### Last-resort inheritance

Steps 1 and 2 are strict: they only ever return a serde that was declared for *this* key. When both come up empty, StoatFlow walks upstream a second time, **through** the re-key, and takes the nearest declaration from the previous key region.

That is safe because a boundary key serde is used for **lane hashing only**. The bytes feed a MurmurHash3 lane assignment and nothing else — they are never stored, never sent to Kafka, and never deserialized back. So the serde only has to encode your key deterministically; it does not have to be the *right* serde for it in any other sense. (This is exactly why the same leniency is **not** applied to keyed timers, whose key bytes *are* persisted and read back.)

It works when the re-key preserves the declared key type — the common case for a re-key that reshapes a `String` id. When the type changes, the inherited serde cannot encode the new key and the boundary still fails on the first record, with the inherited serde named in the message.

A warning always names the edge (`'parent' → 'child'`) and the inherited serde class, because the topology now starts where it previously died — and if the types happen not to line up, you want to know before the first record. It is raised when the topology compiles, so you see it from your own tests as well as at startup. Silence it, or make it fatal, with the [`topology.validation`](https://stoatflow.io/#validation-severities) rule `inherited-boundary-key-serde`.

### When nothing resolves

If you *have* configured a default key serde, that default wins outright — step 3 never runs, because your explicit choice is the answer for this key. This is the escape hatch for a genuinely `byte[]`-keyed boundary: set `default.key.serde` to `Serdes.ByteArray()` explicitly and inheritance stays out of the way. Without that, StoatFlow cannot tell a deliberate byte-array key from an unconfigured default — they are the same value. It is the *presence* of the setting that decides, not its value, so any route that sets one closes inheritance: the `StreamsConfig` constructor, `.copy()`, the builder, `fromMap` / `fromProperties`, or the YAML key. Having answered, you are also out of the `unresolved-boundary-key-serde` rule below at every severity — including `error` — because there is nothing left to report.

Otherwise the boundary falls back to the raw `ByteArray` default and a warning names the edge when the topology compiles. If your keys are not raw bytes, that boundary will fail on the first record with `LaneKeySerializationException` — declare a serde at any of the places above, or set `default.key.serde`. The warning stays a warning by default because a genuinely byte-array-keyed boundary is valid and indistinguishable at build time; the `unresolved-boundary-key-serde` rule below lets you make it fatal anyway.

### Validation severities

Both boundary conditions above are rules in the `topology.validation` family, and each can be turned off or promoted to a build failure:

```yaml
stoatflow:
  topology:
    validation:
      unresolved-boundary-key-serde: error   # off | warn | error (default: warn)
      inherited-boundary-key-serde: off
```

`error` fails **topology compilation** with a single message naming every offending boundary at once, so a topology with several takes one build-run cycle to clean up rather than several.

All three severities are decided at that one point, when the topology compiles — not at engine start — so `TopologyTestDriver` reaches them as well. At the default `warn` that means the per-boundary line appears in your own test suite's output, with no broker; at `error` the driver refuses to build; at `off` neither happens, in tests or in production.

See the [configuration reference](https://stoatflow.io/docs/reference/configuration-reference) for the full rule list.

## Default serdes

Set the application-wide default key and value serdes through `streamsConfigOverrides` when you build the runtime. These apply to every operator that doesn't specify its own serde.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.runtime.StoatFlowRuntime
import org.apache.kafka.common.serialization.Serdes

val runtime = StoatFlowRuntime.fromConfig(
    topologyBuilder = { buildTopology(it) },
    configure = {
        streamsConfigOverrides {
            defaultKeySerde(Serdes.String())
            defaultValueSerde(Serdes.String())
        }
    },
)
```

```java [Java]
import io.stoatflow.runtime.StoatFlowRuntime;
import org.apache.kafka.common.serialization.Serdes;

var runtime = StoatFlowRuntime.fromConfig(
    Main::buildTopology,
    builder -> builder.streamsConfigOverrides(cfg -> {
        cfg.defaultKeySerde(Serdes.String());
        cfg.defaultValueSerde(Serdes.String());
    })
);
```
::

You can also set the defaults declaratively in `application.yaml` using the fully-qualified serde class name. The class must have a no-argument constructor; for parameterised serdes (e.g. Avro), set the instance in code instead.

```yaml
stoatflow:
  default-key-serde: org.apache.kafka.common.serialization.Serdes$StringSerde
  # default-value-serde: ...
```

::callout{color="info" icon="i-lucide-info"}
Defaults set in `streamsConfigOverrides` take precedence over the YAML class-name form — use the YAML keys for serdes with a no-arg constructor, and the code form whenever you need to pass a configured serde instance.
::

## Per-operator serdes

Each operator that touches Kafka or a state store takes a config object that carries serdes. All four follow the same shape: a `with(keySerde, valueSerde)` factory plus chained `.withKeySerde(...)` / `.withValueSerde(...)` builders.

| Config                | Operators                             | Factory                    | Builders                           |
| --------------------- | ------------------------------------- | -------------------------- | ---------------------------------- |
| `Consumed<K,V>`       | `stream`, `table`, `globalTable`      | `Consumed.with(k, v)`      | `.withKeySerde`, `.withValueSerde` |
| `Produced<K,V>`       | `to`                                  | `Produced.with(k, v)`      | `.withKeySerde`, `.withValueSerde` |
| `Repartitioned<K,V>`  | `repartition`                         | `Repartitioned.with(k, v)` | `.withKeySerde`, `.withValueSerde` |
| `Grouped<K,V>`        | `groupByKey`, `groupBy`               | `Grouped.with(k, v)`       | `.withKeySerde`, `.withValueSerde` |
| `Materialized<K,V,S>` | `count`, `reduce`, `aggregate`, joins | `Materialized.with(k, v)`  | `.withKeySerde`, `.withValueSerde` |

### Source and sink

`Consumed` controls how a source topic is deserialized; `Produced` controls how a sink topic is serialized.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.topology.Consumed
import io.stoatflow.core.topology.Produced
import org.apache.kafka.common.serialization.Serdes

// Source: String keys, Long values
val numbers = builder.stream(
    "input",
    Consumed.with(Serdes.String(), Serdes.Long()),
)

// Sink: String keys, Long values
numbers.to(
    "output",
    Produced.with(Serdes.String(), Serdes.Long()),
)
```

```java [Java]
import io.stoatflow.core.topology.Consumed;
import io.stoatflow.core.topology.KStream;
import io.stoatflow.core.topology.Produced;
import org.apache.kafka.common.serialization.Serdes;

// Source: String keys, Long values
KStream<String, Long> numbers = builder.stream(
    "input",
    Consumed.with(Serdes.String(), Serdes.Long()));

// Sink: String keys, Long values
numbers.to(
    "output",
    Produced.with(Serdes.String(), Serdes.Long()));
```
::

When you only need to override one of the two serdes (the other coming from the defaults), use the named factory or the builder form. `Consumed.as(...)` / `Produced.as(...)` set only the operator name and leave both serdes on the defaults — that is the pattern the [first app](https://stoatflow.io/docs/getting-started/first-app) uses, where the sink overrides just the value serde because counts are `Long`:

::code-tabs{group="lang"}
```kotlin [Kotlin]
.to(
    "word-counts",
    Produced.`as`<String, Long>("sink")
        .withKeySerde(Serdes.String())
        .withValueSerde(Serdes.Long()),
)
```

```java [Java]
.to(
    "word-counts",
    Produced.<String, Long>as("sink")
        .withKeySerde(Serdes.String())
        .withValueSerde(Serdes.Long()));
```
::

### Grouping and aggregation

A `groupBy` / `groupByKey` re-serializes records as it re-keys them, so the grouping serdes must match the post-grouping key and value types. `Grouped` carries those serdes; if a downstream aggregation materializes state, `Materialized` carries the store's serdes (which may differ — the aggregate value type is often not the input value type).

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.state.StateStore
import io.stoatflow.core.topology.Grouped
import io.stoatflow.core.topology.Materialized
import org.apache.kafka.common.serialization.Serdes

stream
    .groupBy(
        { _, value -> value.category },
        Grouped.with(Serdes.String(), productSerde),
    )
    .count(
        Materialized.`as`<String, Long, StateStore>("counts-by-category")
            .withKeySerde(Serdes.String())
            .withValueSerde(Serdes.Long()),
    )
```

```java [Java]
import io.stoatflow.core.state.StateStore;
import io.stoatflow.core.topology.Grouped;
import io.stoatflow.core.topology.Materialized;
import org.apache.kafka.common.serialization.Serdes;

stream
    .groupBy(
        (key, value) -> value.category(),
        Grouped.with(Serdes.String(), productSerde))
    .count(
        Materialized.<String, Long, StateStore>as("counts-by-category")
            .withKeySerde(Serdes.String())
            .withValueSerde(Serdes.Long()));
```
::

`Materialized` carries far more than serdes (store type, changelog, caching, retention). See [State stores](https://stoatflow.io/docs/building/state-stores) for the rest; this page covers only its serde configuration.

## Built-in serdes

The standard Kafka serdes cover the common primitive and byte types. They're factory methods on `org.apache.kafka.common.serialization.Serdes`:

| Type         | Factory               |
| ------------ | --------------------- |
| `String`     | `Serdes.String()`     |
| `Long`       | `Serdes.Long()`       |
| `Integer`    | `Serdes.Integer()`    |
| `Short`      | `Serdes.Short()`      |
| `Float`      | `Serdes.Float()`      |
| `Double`     | `Serdes.Double()`     |
| `byte[]`     | `Serdes.ByteArray()`  |
| `ByteBuffer` | `Serdes.ByteBuffer()` |
| `Bytes`      | `Serdes.Bytes()`      |
| `UUID`       | `Serdes.UUID()`       |
| `Void`       | `Serdes.Void()`       |

These are plain Kafka client types — the same ones you'd reach for in a Kafka Streams app. The full list is whatever your `kafka-clients` version exposes on `Serdes`.

## Custom serdes

For your own value types — JSON, Protobuf, a hand-rolled binary format — implement `org.apache.kafka.common.serialization.Serde<T>`, or build one from a `Serializer<T>` / `Deserializer<T>` pair with `Serdes.serdeFrom(serializer, deserializer)`.

A serde is just a serializer and a deserializer bundled together:

::code-tabs{group="lang"}
```kotlin [Kotlin]
import com.fasterxml.jackson.databind.ObjectMapper
import org.apache.kafka.common.serialization.Deserializer
import org.apache.kafka.common.serialization.Serde
import org.apache.kafka.common.serialization.Serdes
import org.apache.kafka.common.serialization.Serializer

data class Order(val id: String, val amount: Double)

class JsonOrderSerializer : Serializer<Order> {
    private val mapper = ObjectMapper()
    override fun serialize(topic: String?, data: Order?): ByteArray? =
        data?.let { mapper.writeValueAsBytes(it) }
}

class JsonOrderDeserializer : Deserializer<Order> {
    private val mapper = ObjectMapper()
    override fun deserialize(topic: String?, data: ByteArray?): Order? =
        data?.let { mapper.readValue(it, Order::class.java) }
}

val orderSerde: Serde<Order> =
    Serdes.serdeFrom(JsonOrderSerializer(), JsonOrderDeserializer())
```

```java [Java]
import com.fasterxml.jackson.databind.ObjectMapper;
import org.apache.kafka.common.serialization.Deserializer;
import org.apache.kafka.common.serialization.Serde;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.Serializer;

record Order(String id, double amount) {}

class JsonOrderSerializer implements Serializer<Order> {
    private final ObjectMapper mapper = new ObjectMapper();
    @Override
    public byte[] serialize(String topic, Order data) {
        if (data == null) return null;
        try {
            return mapper.writeValueAsBytes(data);
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
    }
}

class JsonOrderDeserializer implements Deserializer<Order> {
    private final ObjectMapper mapper = new ObjectMapper();
    @Override
    public Order deserialize(String topic, byte[] data) {
        if (data == null) return null;
        try {
            return mapper.readValue(data, Order.class);
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
    }
}

Serde<Order> orderSerde =
    Serdes.serdeFrom(new JsonOrderSerializer(), new JsonOrderDeserializer());
```
::

Use `orderSerde` exactly like a built-in — pass it to `Consumed.with(...)`, `Produced.with(...)`, `Materialized.withValueSerde(...)`, or set it as the default value serde.

::callout{color="warning" icon="i-lucide-triangle-alert"}
A serde must round-trip a `null` cleanly — return `null` for `null` input. KTable changelogs and tombstones rely on `null` values to signal deletion, so a serde that throws on `null` will break stateful operators.
::

## Windowed keys

Windowed aggregations (`windowedBy(...).count()`, session windows, sliding windows) produce keys of type `Windowed<K>` — the original key plus the window's start and end. To **write those keys to a topic** (or read them back), you need a serde that encodes the window bounds alongside the key bytes. `WindowedSerdes` builds those for you from the inner key serde.

The **state store** doesn't need one: windowed `Materialized` takes the **plain** key serde (`Serdes.String()`) — StoatFlow wraps it into a `Serde<Windowed<K>>` internally, exactly like Kafka Streams. So the windowed serde belongs on the **output** (`Produced`), not the `Materialized`:

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.state.StateStore
import io.stoatflow.core.topology.Materialized
import io.stoatflow.core.topology.Produced
import io.stoatflow.core.topology.WindowedSerdes
import org.apache.kafka.common.serialization.Serdes

// Encodes Windowed<String> keys for a topic — used on the output, not the store.
val windowedKeySerde = WindowedSerdes.timeWindowedSerdeFrom(Serdes.String())

stream
    .groupByKey()
    .windowedBy(/* window definition */)
    .count(
        // Plain key serde — StoatFlow wraps it into Serde<Windowed<String>> for the result table.
        Materialized.`as`<String, Long, StateStore>("windowed-counts")
            .withKeySerde(Serdes.String())
            .withValueSerde(Serdes.Long()),
    ).toStream()
    .to("windowed-counts", Produced.with(windowedKeySerde, Serdes.Long()))
```

```java [Java]
import io.stoatflow.core.state.StateStore;
import io.stoatflow.core.topology.Materialized;
import io.stoatflow.core.topology.Produced;
import io.stoatflow.core.topology.WindowedSerdes;
import io.stoatflow.core.topology.Windowed;
import org.apache.kafka.common.serialization.Serde;
import org.apache.kafka.common.serialization.Serdes;

// Encodes Windowed<String> keys for a topic — used on the output, not the store.
Serde<Windowed<String>> windowedKeySerde =
    WindowedSerdes.timeWindowedSerdeFrom(Serdes.String());

stream
    .groupByKey()
    .windowedBy(/* window definition */)
    .count(
        // Plain key serde — StoatFlow wraps it into Serde<Windowed<String>> for the result table.
        Materialized.<String, Long, StateStore>as("windowed-counts")
            .withKeySerde(Serdes.String())
            .withValueSerde(Serdes.Long()))
    .toStream()
    .to("windowed-counts", Produced.with(windowedKeySerde, Serdes.Long()));
```
::

Use `sessionWindowedSerdeFrom(inner)` instead for session windows. Both factories also exist as constructors — `WindowedSerdes.TimeWindowedSerde(inner)` and `WindowedSerdes.SessionWindowedSerde(inner)` — and expose the unwrapped inner serde via `keySerde()`.

::callout{color="info" icon="i-lucide-info"}
**KS deviation:** the factory methods take a `Serde<T>` for the inner key (not a `Class<T>` as in Kafka Streams), and there is no window-size argument — the window bounds are encoded directly in the serialized bytes. See [How StoatFlow differs from Kafka Streams](https://stoatflow.io/docs/concepts/how-stoatflow-differs-from-ks).
::

## Avro with Schema Registry

Avro is the most common production serialization format with Kafka. StoatFlow has no Avro serde of its own — you use the Confluent `SpecificAvroSerde` (or the generic/reflection variants), the same one you'd use with Kafka Streams. StoatFlow's only role is to hand Schema Registry's URL to your default serdes; serdes you build yourself, you configure yourself.

### Build setup

Generate Java classes from your `.avsc` schemas (the `com.github.davidmc24.gradle.plugin.avro` plugin against `org.apache.avro:avro` is one common choice), then depend on the Confluent serde from the Confluent Maven repository:

```kotlin
// build.gradle.kts
repositories {
    mavenCentral()
    maven("https://packages.confluent.io/maven/")
}

dependencies {
    implementation("io.stoatflow:stoatflow-runtime:<stoatflow-version>")
    implementation("io.confluent:kafka-streams-avro-serde:8.0.0")
}
```

::callout{color="warning" icon="i-lucide-triangle-alert"}
The Confluent serde transitively pulls `kafka-clients:8.0.0-ccs`, but StoatFlow targets the Apache `kafka-clients` 4.x APIs. Pin the Apache version so Confluent's `-ccs` build doesn't win dependency resolution:

```kotlin
configurations.all {
    resolutionStrategy.eachDependency {
        if (requested.group == "org.apache.kafka" && requested.name == "kafka-clients") {
            useVersion("4.3.1") // your Apache kafka-clients version
        }
    }
}
```
::

### Pointing serdes at Schema Registry

Set the registry URL once in `application.yaml`. StoatFlow propagates it to the **default** key and value serdes at startup, so an Avro serde set as the default value serde is configured for you:

```yaml
stoatflow:
  application-id: news-article-publish-processor
  bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVERS:-localhost:9092}
  schema-registry-url: ${SCHEMA_REGISTRY_URL:-http://localhost:8081}
  default-key-serde: org.apache.kafka.common.serialization.Serdes$StringSerde
```

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.confluent.kafka.streams.serdes.avro.SpecificAvroSerde
import io.stoatflow.runtime.StoatFlowRuntime
import org.apache.kafka.common.serialization.Serdes

val runtime = StoatFlowRuntime.fromConfig(
    topologyBuilder = { buildTopology(it) },
    configure = {
        streamsConfigOverrides {
            defaultKeySerde(Serdes.String())
            // Configured automatically with schema-registry-url from YAML
            defaultValueSerde(SpecificAvroSerde<Article>())
        }
    },
)
```

```java [Java]
import io.confluent.kafka.streams.serdes.avro.SpecificAvroSerde;
import io.stoatflow.runtime.StoatFlowRuntime;
import org.apache.kafka.common.serialization.Serdes;

var runtime = StoatFlowRuntime.fromConfig(
    Main::buildTopology,
    builder -> builder.streamsConfigOverrides(cfg -> {
        cfg.defaultKeySerde(Serdes.String());
        // Configured automatically with schema-registry-url from YAML
        cfg.defaultValueSerde(new SpecificAvroSerde<>());
    }));
```
::

::callout{color="info" icon="i-lucide-info"}
The automatic propagation applies **only to the default serdes**. Any Avro serde you build yourself and pass to a `Consumed`, `Produced`, or `Materialized` must be configured by hand — StoatFlow never touches it.
::

### Per-operator Avro serdes

When different topics carry different Avro types — or you want the value serde on a specific source rather than as the global default — build each serde and call `configure(...)` with the registry URL before handing it to the operator. The boolean second argument is `isKey` (`true` for key serdes, `false` for value serdes):

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.confluent.kafka.streams.serdes.avro.SpecificAvroSerde
import org.apache.kafka.common.serialization.Serde

fun createStockTickSerde(schemaRegistryUrl: String): Serde<StockTick> {
    val serde = SpecificAvroSerde<StockTick>()
    serde.configure(mapOf("schema.registry.url" to schemaRegistryUrl), false)
    return serde
}
```

```java [Java]
import io.confluent.kafka.streams.serdes.avro.SpecificAvroSerde;
import org.apache.kafka.common.serialization.Serde;
import java.util.Map;

static Serde<StockTick> createStockTickSerde(String schemaRegistryUrl) {
    SpecificAvroSerde<StockTick> serde = new SpecificAvroSerde<>();
    serde.configure(Map.of("schema.registry.url", schemaRegistryUrl), false);
    return serde;
}
```
::

Then wire the configured serde into the source or sink exactly like any other serde:

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.topology.Consumed
import io.stoatflow.core.topology.Produced
import org.apache.kafka.common.serialization.Serdes

val stockTickSerde = createStockTickSerde(schemaRegistryUrl)

builder
    .stream("stock-ticks", Consumed.with(Serdes.String(), stockTickSerde))
    // ... processing ...
    .to(
        "filtered-stock-ticks",
        Produced.with(Serdes.String(), stockTickSerde),
    )
```

```java [Java]
import io.stoatflow.core.topology.Consumed;
import io.stoatflow.core.topology.Produced;
import org.apache.kafka.common.serialization.Serde;
import org.apache.kafka.common.serialization.Serdes;

Serde<StockTick> stockTickSerde = createStockTickSerde(schemaRegistryUrl);

builder
    .stream("stock-ticks", Consumed.with(Serdes.String(), stockTickSerde))
    // ... processing ...
    .to(
        "filtered-stock-ticks",
        Produced.with(Serdes.String(), stockTickSerde));
```
::

### Testing Avro topologies

For unit tests against the in-memory [test driver](https://stoatflow.io/docs/building/testing), configure your Avro serdes against Confluent's **mock** Schema Registry (`mock://...`) instead of a live one — it keeps schemas in-process, so no broker or registry is needed:

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.confluent.kafka.serializers.AbstractKafkaSchemaSerDeConfig
import io.confluent.kafka.streams.serdes.avro.SpecificAvroSerde

fun stockTickSerde(): SpecificAvroSerde<StockTick> {
    val serde = SpecificAvroSerde<StockTick>()
    serde.configure(
        mapOf(AbstractKafkaSchemaSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG to "mock://schema-registry"),
        false,
    )
    return serde
}
```

```java [Java]
import io.confluent.kafka.serializers.AbstractKafkaSchemaSerDeConfig;
import io.confluent.kafka.streams.serdes.avro.SpecificAvroSerde;
import java.util.Map;

static SpecificAvroSerde<StockTick> stockTickSerde() {
    SpecificAvroSerde<StockTick> serde = new SpecificAvroSerde<>();
    serde.configure(
        Map.of(AbstractKafkaSchemaSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, "mock://schema-registry"),
        false);
    return serde;
}
```
::

When `schema-registry-url` is configured, the runtime also exposes a [Schema Registry health check](https://stoatflow.io/docs/concepts/architecture#failure-modes-and-observability) on `/health/ready` that auto-enables itself — no extra wiring required.

## Next steps

- **[State stores](https://stoatflow.io/docs/building/state-stores)** — the rest of `Materialized`: store types, changelog, caching, retention.
- **[Windowing](https://stoatflow.io/docs/building/windowing)** — where `Windowed<K>` keys come from and how windows close.
- **[Error handling and DLQ](https://stoatflow.io/docs/building/error-handling-dlq)** — what happens when a record fails to deserialize.
- **[Testing](https://stoatflow.io/docs/building/testing)** — drive a topology end-to-end in-memory with the test driver.


# How configuration works

The `:runtime` module loads configuration from a layered set of sources: a bundled `application.yaml`, external overlay files, environment variables, and JVM system properties — merged in a fixed precedence order. Non-serializable settings (serdes, exception handlers, callbacks) are supplied in code through a `streamsConfigOverrides` block. This page shows each source, the override forms, and the order in which they win.

::tldr-panel
- **Base config:** `application.yaml` (or `application.yml`) on the classpath — usually `src/main/resources/`.
- **Overlay files:** point `STOATFLOW_CONFIG_FILES` at one or more files; later files win, all must exist.
- **Env vars:** `STOATFLOW__LANES__COUNT=32` — double underscore = dot, single underscore = word boundary (kebab-case).
- **Order (highest wins):** env vars → overlay files → classpath YAML → data-class defaults. Code `streamsConfigOverrides` always wins last.
::

For *why* the model is layered this way — and how it maps onto the engine config versus the runtime infrastructure config — see [the configuration model](https://stoatflow.io/docs/concepts/configuration-model). For the actual keys, see [core config](https://stoatflow.io/docs/configuration/core-config) and [runtime config](https://stoatflow.io/docs/configuration/runtime-config).

## The config tree

All configuration lives under two top-level keys (plus an optional `logging`):

- **`stoatflow:`** — the stream-processing engine: Kafka connection, lanes, commit barriers, state, changelog, watermarks, license.
- **`runtime:`** — the runtime infrastructure: HTTP server, metrics, health checks, endpoint visibility.
- **`logging:`** — per-logger levels, overriding `logback.xml`.

A minimal `application.yaml` only needs an `application-id`; everything else has a default. Place it on the classpath at `src/main/resources/application.yaml`:

```yaml
stoatflow:
  application-id: word-count
  bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVERS:-localhost:9092}

  license:
    key: ${STOATFLOW_LICENSE_KEY}

runtime:
  http:
    enabled: true
    port: ${HTTP_PORT:-8080}
  metrics:
    enabled: true
```

::callout{color="info" icon="i-lucide-info"}
`${VAR}` and `${VAR:-default}` are resolved by the config loader at startup — handy for keeping secrets and environment-specific values out of the committed YAML. This is the placeholder syntax, distinct from the env-var *override* mechanism described below.
::

The loader reads both `application.yml` and `application.yaml` from the classpath if present; `.yml` takes precedence over `.yaml` when both exist. Both are optional — a config built entirely from environment variables and overlay files is valid as long as `stoatflow.application-id` resolves.

## Layered precedence

When the same key is set in more than one place, the highest-priority source wins. From highest to lowest:

| # | Source                                                        | Form                                         |
| - | ------------------------------------------------------------- | -------------------------------------------- |
| 1 | Environment variables, `__` path separator                    | `STOATFLOW__LANES__COUNT=32`                 |
| 2 | Overlay files from `STOATFLOW_CONFIG_FILES` (later file wins) | `STOATFLOW_CONFIG_FILES=base.yaml,prod.yaml` |
| 3 | Bundled `application.yml` on the classpath                    | `src/main/resources/application.yml`         |
| 4 | Bundled `application.yaml` on the classpath                   | `src/main/resources/application.yaml`        |
| 5 | Default values in the config data classes                     | e.g. `runtime.http.port = 8080`              |

This is the order applied by the config loader (`ConfigLoader.kt`). The two env-var conventions in rows 1-2 are layered together as the highest-priority sources, ahead of files.

::callout{color="info" icon="i-lucide-info"}
**System properties.** A handful of settings — notably the [license](https://stoatflow.io/docs/getting-started/license-configuration) — are read as JVM system properties (`-Dstoatflow.license.key=...`). For the license specifically, env vars and `-D` system properties take precedence over the YAML `stoatflow.license.*` fields. System properties are not a general override layer for the whole config tree; use environment variables or overlay files for that.
::

## Environment variables

The recommended convention uses **`__` (double underscore) as the path separator** and a **single `_` as a word boundary** within a property name (converted to kebab-case). The first segment must be a top-level key — `STOATFLOW`, `RUNTIME`, or `LOGGING`:

```bash
STOATFLOW__LANES__COUNT=32                                    # stoatflow.lanes.count
STOATFLOW__COMMIT_BARRIER__INTERVAL_MS=5000                   # stoatflow.commit-barrier.interval-ms
STOATFLOW__VALIDATION__PRE_FLIGHT_CONSUMER_GROUP_CHECK=false  # stoatflow.validation.pre-flight-consumer-group-check
RUNTIME__HTTP__PORT=9090                                      # runtime.http.port
```

The double-underscore form is the one to use because it can express multi-word property names (`pre-flight-consumer-group-check`). The convention matches Docker Compose and Quarkus (`RelaxedEnvironmentPropertySource.kt`).

### Kafka client and logging maps

Kafka client properties and logging levels are `Map<String, String>` entries whose keys must keep their dots and word boundaries (`linger.ms`, `org.apache.kafka.clients`). Those are read directly from the environment with their own conventions:

```bash
STOATFLOW__KAFKA__PRODUCER__LINGER_MS=5                       # producer linger.ms = 5
STOATFLOW__KAFKA__CONSUMER__MAX_POLL_RECORDS=1000            # consumer max.poll.records = 1000
LOGGING__LEVEL__ORG_APACHE_KAFKA_CLIENTS=DEBUG               # org.apache.kafka.clients = DEBUG
```

Inside these segments, `_` becomes `.` — so `LINGER_MS` resolves to `linger.ms` and env-var entries take precedence over the same key set in YAML (`ConfigMapper.kt`). See [Kafka client config](https://stoatflow.io/docs/configuration/kafka-client-config) for the full surface.

## Overlay files

For deployment-specific configuration that you don't want baked into the application JAR, set `STOATFLOW_CONFIG_FILES` to a comma-separated list of file paths. The files are deep-merged on top of the bundled classpath YAML, and **later files in the list override earlier ones**:

```bash
# Single external file
export STOATFLOW_CONFIG_FILES=/etc/myapp/application.yaml

# Multiple files — last wins
export STOATFLOW_CONFIG_FILES=/etc/myapp/base.yaml,/etc/myapp/prod-overrides.yaml
```

Supported extensions are `.yaml` and `.yml`. Every file listed is **mandatory** — if any path doesn't exist (or isn't a regular file) the runtime fails to start with a clear error naming the missing path (`ConfigLoader.kt`). This is the recommended way to inject a Kubernetes `ConfigMap` or a mounted secrets file into the runtime.

## Programmatic overrides

YAML and environment variables cover everything that can be expressed as a string, a number, or a class name. Things that can't — live serde instances, exception-handler objects, callbacks — are configured in code through the `streamsConfigOverrides` block on the runtime builder. These overrides are applied **last**, taking precedence over every YAML and env-var source (`ConfigMapper.kt`):

::code-tabs{group="lang"}
```kotlin [Kotlin]
val runtime = StoatFlowRuntime.fromConfig(
    topologyBuilder = { buildTopology(it) },
    configure = {
        streamsConfigOverrides {
            defaultKeySerde(Serdes.String())
            defaultValueSerde(Serdes.String())
        }
    },
)
runtime.start()
runtime.awaitTermination()
```

```java [Java]
var runtime = StoatFlowRuntime.fromConfig(
    Main::buildTopology,
    builder -> builder.streamsConfigOverrides(cfg -> {
        cfg.defaultKeySerde(Serdes.String());
        cfg.defaultValueSerde(Serdes.String());
    })
);
runtime.start();
runtime.awaitTermination();
```
::

`streamsConfigOverrides` exposes the same `StreamsConfig.Builder` the YAML mapping populates — so anything you can set in YAML you can also set here, and your code wins. Reserve it for the non-serializable types and for values you'd rather compute at runtime than declare statically.

### Class-name configuration in YAML

Some non-serializable types have a middle ground: if the class has a no-arg constructor, you can name it in YAML as a fully qualified class name and the runtime instantiates it for you. These are applied above the YAML defaults but below `streamsConfigOverrides`:

```yaml
stoatflow:
  default-key-serde: org.apache.kafka.common.serialization.Serdes$StringSerde
  deserialization-exception-handler: io.stoatflow.core.exception.LogAndContinueDeserializationExceptionHandler
```

The supported class-name keys are `default-key-serde`, `default-value-serde`, `deserialization-exception-handler`, `production-exception-handler`, `processing-exception-handler`, and `rocks-db-config-setter` (`ConfigMapper.kt`). Anything requiring a constructor argument must go through `streamsConfigOverrides` instead.

## Inspecting the merged result

The runtime exposes the fully merged, masked configuration at the `/config` HTTP endpoint — useful for confirming that an env-var override or overlay file actually landed where you expected. Sensitive values (passwords, secrets, tokens) are masked in the output:

```bash
curl -s localhost:8080/config
```

See the [REST API](https://stoatflow.io/docs/runtime/rest-api) for the full endpoint reference.

## Next steps

- **[Configuration model](https://stoatflow.io/docs/concepts/configuration-model)** — why config is split into `stoatflow` (engine) and `runtime` (infrastructure), and how it composes.
- **[Core config](https://stoatflow.io/docs/configuration/core-config)** — the `stoatflow.*` keys: lanes, barriers, state, changelog, watermarks.
- **[Runtime config](https://stoatflow.io/docs/configuration/runtime-config)** — the `runtime.*` keys: HTTP server, metrics, health, endpoints.
- **[Defaults and presets](https://stoatflow.io/docs/configuration/defaults-and-presets)** — what the out-of-the-box defaults are and the RocksDB presets.
- **[Configuration reference](https://stoatflow.io/docs/reference/configuration-reference)** — every key in one table.


# Engine configuration (:core)

These are the `:core` engine settings — the stream-processing knobs that exist whether you embed `stoatflow-core` directly or run on the batteries-included `stoatflow-runtime`. On the runtime they live under the `stoatflow:` block of `application.yaml`; in plain `:core` code you set the same values on `StreamsConfig` (or its `Builder`). This page groups the ones you'll actually reach for, with their defaults.

::tldr-panel
- **Required:** `application-id` (consumer group + transactional id) is the only mandatory key. `bootstrap-servers` defaults to `localhost:9092` — you'll almost always override it.
- **Cadence:** the commit barrier self-tunes between `commit-barrier.min-interval-ms` and `max-interval-ms`, seeded by `interval-ms`.
- **Parallelism:** `lanes.count` defaults to `max(2, CPU cores)`; raise it for per-key parallelism.
- **Guarantee:** `processing-guarantee` is `EXACTLY_ONCE` by default; switch to `AT_LEAST_ONCE` for lower commit latency.
- **Full key list:** the exhaustive reference is [Configuration reference](https://stoatflow.io/docs/reference/configuration-reference).
::

::callout{color="info" icon="i-lucide-info"}
This page covers the settings most apps touch. Many more `:core` knobs exist (dispatch tuning, restoration sizing, RocksDB presets, FK-join, caching, hot-standby HA). For every key with its type and bounds, see [Configuration reference](https://stoatflow.io/docs/reference/configuration-reference). For the layered merge model behind YAML overrides and Kafka client passthrough, see [Configuration model](https://stoatflow.io/docs/concepts/configuration-model) and [Kafka client config](https://stoatflow.io/docs/configuration/kafka-client-config).
::

## Identity and connection

Only `application-id` is required — it has no default, so the engine won't start without it. `bootstrap-servers` defaults to `localhost:9092`, which is fine for local development but you'll set it explicitly for any real broker.

| YAML key                      | `StreamsConfig` field | Default          | Notes                                                                                                                     |
| ----------------------------- | --------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------- |
| `stoatflow.application-id`    | `applicationId`       | — (required)     | Used for the consumer group **and** the transactional id. Pick a stable value; changing it starts a fresh consumer group. |
| `stoatflow.bootstrap-servers` | `bootstrapServers`    | `localhost:9092` | Comma-separated broker list. Override for any non-local broker.                                                           |

```yaml
stoatflow:
  application-id: order-processor
  bootstrap-servers: kafka-1:9092,kafka-2:9092,kafka-3:9092
```

::callout{color="info" icon="i-lucide-info"}
StoatFlow is single-instance — exactly one process owns all partitions. Startup validates complete partition assignment and (optionally) checks for other live group members. See [Lanes and parallelism](https://stoatflow.io/docs/concepts/lanes-and-parallelism) and the `validation.*` block in the [reference](https://stoatflow.io/docs/reference/configuration-reference).
::

## Default serdes

When an operator doesn't specify a serde via `Consumed`, `Produced`, `Materialized`, etc., these defaults apply. Both default to `ByteArray` — set them to the type your topology mostly uses.

| YAML key                        | `StreamsConfig` field | Default              |
| ------------------------------- | --------------------- | -------------------- |
| `stoatflow.default-key-serde`   | `defaultKeySerde`     | `Serdes.ByteArray()` |
| `stoatflow.default-value-serde` | `defaultValueSerde`   | `Serdes.ByteArray()` |

In YAML the values are fully qualified serde class names; in code you pass `Serde` instances:

::code-tabs{group="lang"}
```kotlin [Kotlin]
val config = StreamsConfig.builder("order-processor", "localhost:9092")
    .defaultKeySerde(Serdes.String())
    .defaultValueSerde(Serdes.String())
    .build()
```

```java [Java]
var config = StreamsConfig.builder("order-processor", "localhost:9092")
    .defaultKeySerde(Serdes.String())
    .defaultValueSerde(Serdes.String())
    .build();
```
::

```yaml
stoatflow:
  default-key-serde: org.apache.kafka.common.serialization.Serdes$StringSerde
  default-value-serde: org.apache.kafka.common.serialization.Serdes$StringSerde
```

::callout{color="info" icon="i-lucide-info"}
On the runtime, the cleaner way to set typed default serdes is `streamsConfigOverrides { ... }` in code — see [Your first app](https://stoatflow.io/docs/getting-started/first-app). Using Schema Registry serdes? Set `stoatflow.schema-registry-url`; it's propagated to default serdes automatically. More in [Serdes](https://stoatflow.io/docs/building/serdes).
::

::callout{color="neutral" icon="i-lucide-lightbulb"}
Setting `default-key-serde` explicitly does one extra thing: it turns off [last-resort boundary key-serde inheritance](https://stoatflow.io/docs/building/serdes#last-resort-inheritance). StoatFlow cannot distinguish a deliberate `Serdes.ByteArray()` from an unconfigured default — they are the same value — so "did you configure one at all" is the signal it uses. If you genuinely key a sub-topology boundary with raw bytes, say so here and nothing will second-guess it.
::

## Topology

How the compiler shapes and checks your topology. All of these are StoatFlow-only — Kafka Streams has no equivalent.

| YAML key                                        | `StreamsConfig` field      | Default  |
| ----------------------------------------------- | -------------------------- | -------- |
| `stoatflow.topology.sub-topology-split`         | `subTopologySplit`         | `lazy`   |
| `stoatflow.topology.processor-api-key-affinity` | `processorApiKeyAffinity`  | `off`    |
| `stoatflow.topology.validation.<rule-id>`       | `validationSeverity(rule)` | per rule |

**`sub-topology-split`** selects where sub-topology boundaries are opened. `lazy` (the default) opens one only where a downstream operator genuinely needs the re-keyed record's lane affinity — a grouped aggregation, any join, `toTable()` — plus every explicit `repartition()`. That is the Kafka Streams `repartitionRequired` model, so `describe()` lines up with a KS original. `eager` restores the pre-1.0.0 boundary at **every** key-changing node.

Changing it renumbers sub-topology ids, which show up in thread names, the `sub_topology` metric tag and the topology endpoints. See [Lanes and parallelism](https://stoatflow.io/docs/concepts/lanes-and-parallelism#where-the-handoff-actually-happens) — in particular the skew note, if you relied on a re-key to spread work across lanes.

**`processor-api-key-affinity`** selects whether a Processor API node's key affinity is *presumed* from the adapter's shape. `off` (the default) matches Kafka Streams, which never repartitions before `process()` / `processValues()` even with connected stores. `presumed` restores the pre-1.0.0 boundary. It is inert under `sub-topology-split: eager`, which splits at every key-changing node regardless.

`off` extends the lazy split's skew trade to Processor API nodes, and after a **many-to-one** re-key it gives up key affinity there — a real trade-off, spelled out in [Key affinity after a re-key](https://stoatflow.io/docs/building/processor-api#key-affinity-after-a-re-key). StoatFlow refuses to compile the provable case (`papi-key-affinity-presumed`, below). It also puts the node in its **upstream** sub-topology, so it inherits that sub-topology's lane count rather than starting its own — re-check any per-sub-topology `numberOfLanes` override you set deliberately.

**`validation.<rule-id>`** turns individual topology checks off, or promotes them to build failures:

```yaml
stoatflow:
  topology:
    sub-topology-split: lazy
    processor-api-key-affinity: off
    validation:
      unresolved-boundary-key-serde: error   # off | warn | error
      inherited-boundary-key-serde: off
      papi-key-affinity-presumed: error      # the default for this one
```

| Rule                               | Default     | Fires when                                                                                                                                                                                                                                                                           |
| ---------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `unresolved-boundary-key-serde`    | `warn`      | A sub-topology boundary resolved no key serde for the key crossing it **and** you configured no default key serde, so the placeholder `ByteArray` default is used for lane hashing. With a default configured, there is nothing to report and the rule stays quiet at every severity |
| `inherited-boundary-key-serde`     | `warn`      | The boundary resolved only by inheriting a serde from before the re-key                                                                                                                                                                                                              |
| `papi-key-affinity-presumed`       | **`error`** | A **proven** re-key (`selectKey`, `map`, `groupBy`, …) feeds a Processor API node connecting a **writable** store, with no boundary between them. Read-only stores (KIP-813) are excluded — they cannot be written, so there is no update to lose                                    |
| `papi-key-affinity-presumed-chain` | `warn`      | The same shape, but the feeder is itself a Processor API node. `process()` / `addProcessor()` declare a key change unconditionally, so the re-key may not exist at all — refusing the build would refuse topologies Kafka Streams runs happily                                       |

The two boundary-serde rules and the chain rule default to `warn`, so upgrading changes nothing about *what* you are told — only where. `papi-key-affinity-presumed` is the one that defaults to `error`, because the shape it names is silent data loss at runtime rather than a diagnostic. All severities are decided at **topology compilation** rather than at engine start, so `TopologyTestDriver` reaches them with no broker: at `error` the driver refuses to build, naming every offending site in one message rather than failing on the first; at `warn` the same line lands in your own test output. `off` silences the condition outright. An unknown rule id is rejected at startup rather than silently ignored.

## Lanes (parallelism)

Lanes are the engine's virtual partitions — records with the same key always route to the same lane, preserving per-key order while running many keys in parallel. See [Lanes and parallelism](https://stoatflow.io/docs/concepts/lanes-and-parallelism) for the why.

| YAML key                         | `StreamsConfig` field | Default             | Notes                                                                                                                                                                                            |
| -------------------------------- | --------------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `stoatflow.lanes.count`          | `numLanes`            | `max(2, CPU cores)` | More lanes = more per-key parallelism, at the cost of more queues. Start at the default; I/O-bound workloads commonly go 4×–8× cores — see [Tuning](https://stoatflow.io/docs/operating/tuning). |
| `stoatflow.lanes.queue-capacity` | `laneQueueCapacity`   | `300`               | Per-lane queue depth (backpressure). Minimum 32.                                                                                                                                                 |
| `stoatflow.lanes.thread-type`    | `laneThreadType`      | `VIRTUAL`           | `VIRTUAL` (Project Loom) or `PLATFORM` (OS threads). Keep `VIRTUAL` unless you've switched the RocksDB backend to `JNI` and pinning is a concern.                                                |

```yaml
stoatflow:
  lanes:
    count: 32
    queue-capacity: 500
```

## Commit-barrier cadence

The commit barrier is what makes StoatFlow exactly-once: when a barrier flows through the topology, state flushes and the Kafka transaction commits as one atomic unit. The cadence **self-tunes** at runtime within the bounds you set — `interval-ms` is the seed, and the engine adapts between `min-interval-ms` and `max-interval-ms`. You set the envelope, not the algorithm; how the engine schedules within it is an implementation concern (see [Architecture](https://stoatflow.io/docs/concepts/architecture)).

| YAML key                                     | `StreamsConfig` field    | Default             | Notes                                                                                                                                                                      |
| -------------------------------------------- | ------------------------ | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `stoatflow.commit-barrier.interval-ms`       | `barrierIntervalMs`      | `500`               | Seed interval. The runtime adjusts from here.                                                                                                                              |
| `stoatflow.commit-barrier.min-interval-ms`   | `barrierMinIntervalMs`   | `150`               | Floor on barrier-to-barrier time.                                                                                                                                          |
| `stoatflow.commit-barrier.max-interval-ms`   | `barrierMaxIntervalMs`   | `5000`              | Ceiling on barrier-to-barrier time.                                                                                                                                        |
| `stoatflow.commit-barrier.interval-factor`   | `barrierIntervalFactor`  | `1.5`               | Multiplier applied to recent commit duration when spacing barriers, clamped to `[min, max]` (`>= 1.0`). Rarely changed.                                                    |
| `stoatflow.commit-barrier.timeout-ms`        | `barrierTimeoutMs`       | `60000`             | Bounds the commit path. Must be **greater than** `interval-ms`. Also cascades to the Kafka producer's `transaction.timeout.ms`, `max.block.ms`, and `delivery.timeout.ms`. |
| `stoatflow.commit-barrier.max-epoch-records` | `barrierMaxEpochRecords` | unset (no hard cap) | Optional hard cap on records committed per barrier. Leave unset unless you need a strict bound.                                                                            |

```yaml
stoatflow:
  commit-barrier:
    interval-ms: 500
    min-interval-ms: 150
    max-interval-ms: 5000
    timeout-ms: 60000
```

::callout{color="info" icon="i-lucide-gauge"}
**Tighter latency vs. lower overhead.** Lowering `min-interval-ms` and `max-interval-ms` commits more often (lower end-to-end latency, more transaction overhead). Raising them batches more work per commit (higher throughput, higher tail latency). The defaults are a balanced starting point — tune from observed commit duration. See [Tuning](https://stoatflow.io/docs/operating/tuning).
::

::callout{color="warning" icon="i-lucide-shield-alert"}
A stalled commit is not silent: if a barrier can't complete within `timeout-ms`, the commit aborts and the process restarts cleanly. On restart, state recovers to the last committed barrier — no loss, no duplication under exactly-once. The barrier scheduling cadence and the commit protocol's internals are engine concerns; see [Architecture](https://stoatflow.io/docs/concepts/architecture) for the boundary, and [Exactly-once](https://stoatflow.io/docs/concepts/exactly-once) for the guarantee.
::

There are several finer self-tuning knobs under `commit-barrier.*` (e.g. `commit-duration-ema-alpha`, `initial-max-epoch-records`). They have sensible defaults and rarely need changing — the full list is in the [reference](https://stoatflow.io/docs/reference/configuration-reference).

## Processing guarantee

Controls the commit protocol. Both modes use barriers for state-flush and offset-commit synchronization; the difference is the transaction boundary.

| YAML key                         | `StreamsConfig` field | Default        |
| -------------------------------- | --------------------- | -------------- |
| `stoatflow.processing-guarantee` | `processingGuarantee` | `EXACTLY_ONCE` |

| Value                    | Behaviour                                                                                                                                                                |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `EXACTLY_ONCE` (default) | Kafka transactions commit output, changelog, and consumer offsets atomically. No duplicates or loss on crash recovery. Higher per-commit cost.                           |
| `AT_LEAST_ONCE`          | Non-transactional producer with `consumer.commitSync()`. Lower commit latency. Output records may be duplicated on crash recovery; windowed aggregations may over-count. |

```yaml
stoatflow:
  processing-guarantee: EXACTLY_ONCE
```

See [Exactly-once](https://stoatflow.io/docs/concepts/exactly-once) for the full semantics and the at-least-once recovery contract.

## State and changelog

State stores are RocksDB-backed by default and durably backed by Kafka changelog topics, so state survives restarts. See [State stores](https://stoatflow.io/docs/building/state-stores) and [State and thread safety](https://stoatflow.io/docs/concepts/state-and-thread-safety).

### State directory and memory

| YAML key                                     | `StreamsConfig` field           | Default                       | Notes                                                                                                                                                                                                                                                                                                                                                                                              |
| -------------------------------------------- | ------------------------------- | ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `stoatflow.state.dir`                        | `stateDir`                      | `${java.io.tmpdir}/stoatflow` | Put this on fast local storage (SSD). In Kubernetes, mount a PV — the default temp dir is wiped on pod restart, forcing full restoration.                                                                                                                                                                                                                                                          |
| `stoatflow.state.uncommitted-max-bytes`      | `stateStoreUncommittedMaxBytes` | `268435456` (256 MiB)         | Global cap on buffered uncommitted state; exceeding it triggers an early barrier.                                                                                                                                                                                                                                                                                                                  |
| `stoatflow.state.segment-checkpoint-records` | `segmentCheckpointRecords`      | `100000`                      | Periodic checkpoint cadence (record count) for segmented window/session stores, which run WAL-off and would otherwise be durable only on `close()`. Bounds how much changelog an ungraceful restart — or a [hot-standby](https://stoatflow.io/docs/operating/high-availability) promotion — re-applies. Whichever-first with `segment-checkpoint-ms`.                                              |
| `stoatflow.state.segment-checkpoint-ms`      | `segmentCheckpointMs`           | `30000` (30 s)                | The same checkpoint cadence by wall-clock time; whichever-first with `segment-checkpoint-records`.                                                                                                                                                                                                                                                                                                 |
| `stoatflow.rocks-db.preset`                  | `rocksDbConfig`                 | `DEFAULT` (256 MiB)           | `DEFAULT` / `LOW_MEMORY` (64 MiB) / `HIGH_PERFORMANCE` (1 GiB) total RocksDB memory.                                                                                                                                                                                                                                                                                                               |
| `stoatflow.rocks-db.backend`                 | `rocksDbBackendType`            | `AUTO`                        | `AUTO` picks `FFM` (Foreign Function & Memory — no virtual-thread pinning) on the JVM and `JNI` under a native image; set `FFM` or `JNI` to force one.                                                                                                                                                                                                                                             |
| `stoatflow.state.format-downgrade`           | `stateFormatDowngrade`          | `refuse`                      | What to do when a store's on-disk format is newer than the topology asks for — today, [record headers](https://stoatflow.io/docs/building/state-stores#record-headers-in-state-stores) turned off over a store that carries them. `refuse` fails startup with an actionable message and touches nothing; `wipe-and-restore` acknowledges the downgrade and rebuilds that store from its changelog. |

```yaml
stoatflow:
  state:
    dir: /var/lib/stoatflow/state
    uncommitted-max-bytes: 268435456
  rocks-db:
    preset: DEFAULT
```

### Changelog topics

| YAML key                                         | `StreamsConfig` field             | Default               | Notes                                                                                 |
| ------------------------------------------------ | --------------------------------- | --------------------- | ------------------------------------------------------------------------------------- |
| `stoatflow.changelog.enabled`                    | `changelogEnabled`                | `true`                | When on, state mutations are written to changelog topics inside the same transaction. |
| `stoatflow.changelog.create-topics-if-not-exist` | `createChangelogTopicsIfNotExist` | `true`                | Auto-creates compacted changelog topics on startup.                                   |
| `stoatflow.changelog.replication-factor`         | `changelogReplicationFactor`      | `-1` (broker default) | Set explicitly (e.g. `3`) for production durability.                                  |
| `stoatflow.changelog.num-partitions`             | `changelogNumPartitions`          | `1`                   | One partition is usually sufficient for single-instance.                              |
| `stoatflow.state.restoration-enabled`            | `stateRestorationEnabled`         | `true`                | Restore state from changelog on startup when local state is missing.                  |

```yaml
stoatflow:
  changelog:
    enabled: true
    replication-factor: 3
```

::callout{color="warning" icon="i-lucide-database"}
For production, set `changelog.replication-factor` to at least `3` and put `state.dir` on a persistent volume — together they keep restoration fast (delta, not full) across pod restarts. See [Kubernetes](https://stoatflow.io/docs/operating/kubernetes) and the [Production checklist](https://stoatflow.io/docs/operating/production-checklist).
::

## Event-time and watermarks

Watermarks track event-time progress for windowing, late-record handling, and event-time timers. The engine auto-detects whether your topology needs event-time tracking and bypasses it on the hot path when it doesn't. See [Event time and watermarks](https://stoatflow.io/docs/concepts/event-time-and-watermarks).

| YAML key                                           | `StreamsConfig` field       | Default          | Notes                                                                                                                                                                   |
| -------------------------------------------------- | --------------------------- | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `stoatflow.watermark.max-out-of-orderness-ms`      | `maxOutOfOrderness`         | `10000` (10 s)   | How far out of order events may arrive before being treated as late.                                                                                                    |
| `stoatflow.watermark.idleness-timeout-ms`          | `watermarkIdlenessTimeout`  | `300000` (5 min) | Partitions idle this long are excluded from the global watermark. `0` disables.                                                                                         |
| `stoatflow.watermark.auto-interval-ms`             | `autoWatermarkIntervalMs`   | `200`            | Periodic watermark emission interval.                                                                                                                                   |
| `stoatflow.processing.timestamp-coordination-mode` | `timestampCoordinationMode` | `AUTO`           | `AUTO` (derive from topology), `ENABLED` (force on), `DISABLED` (force off, benchmarking).                                                                              |
| `stoatflow.processing.max-task-idle-ms`            | `maxTaskIdleMs`             | `0`              | Cross-partition timestamp-ordering wait. `-1` disables (max throughput); `0` waits only while the broker has data; `>0` adds extra wait. Mirrors KS `max.task.idle.ms`. |

```yaml
stoatflow:
  watermark:
    max-out-of-orderness-ms: 10000
    idleness-timeout-ms: 300000
  processing:
    timestamp-coordination-mode: AUTO
```

::callout{color="info" icon="i-lucide-info"}
`timestamp-coordination-mode: AUTO` is correct for almost all topologies — override it only for debugging or benchmarking. Per-source watermark behaviour is configured on `Consumed` with a `WatermarkStrategy`; these settings are the engine-wide defaults.
::

## Putting it together

A representative production `stoatflow:` block:

```yaml
stoatflow:
  application-id: order-processor
  bootstrap-servers: kafka-1:9092,kafka-2:9092,kafka-3:9092

  default-key-serde: org.apache.kafka.common.serialization.Serdes$StringSerde
  default-value-serde: org.apache.kafka.common.serialization.Serdes$StringSerde

  processing-guarantee: EXACTLY_ONCE

  lanes:
    count: 32
    queue-capacity: 500

  commit-barrier:
    interval-ms: 500
    min-interval-ms: 150
    max-interval-ms: 5000
    timeout-ms: 60000

  state:
    dir: /var/lib/stoatflow/state
    uncommitted-max-bytes: 268435456

  changelog:
    enabled: true
    replication-factor: 3

  watermark:
    max-out-of-orderness-ms: 10000
    idleness-timeout-ms: 300000
```

## Next steps

- **[Configuration reference](https://stoatflow.io/docs/reference/configuration-reference)** — every `:core` key with type, default, and bounds.
- **[Runtime config ( :runtime )](https://stoatflow.io/docs/configuration/runtime-config)** — the `runtime.*` block (HTTP, metrics, endpoints, health).
- **[Kafka client config](https://stoatflow.io/docs/configuration/kafka-client-config)** — passthrough consumer/producer properties and the layered merge order.
- **[Defaults and presets](https://stoatflow.io/docs/configuration/defaults-and-presets)** — the opinionated defaults and how to override them.
- **[Tuning](https://stoatflow.io/docs/operating/tuning)** — picking lane count, barrier cadence, and memory caps from observed behaviour.


# Runtime configuration (:runtime)

The `:runtime` module reads its operational settings from the `runtime.*` block of `application.yaml`: the built-in HTTP server, Micrometer metrics, the `/info` and `/config` endpoint visibility, and the broker/Schema-Registry health checks. This page is a task-oriented walkthrough of that block. Two related top-level blocks — `logging.*` and the Kafka client passthrough under `stoatflow.kafka.*` — round out what you need to operate a runtime app.

::tldr-panel
- **`runtime.http`** — the admin/metrics HTTP server (host, port, backlog, debug endpoints). Defaults to `0.0.0.0:8080`.
- **`runtime.metrics`** — Prometheus export via `/metrics` (prefix, common tags, JVM metrics).
- **`runtime.endpoints`** — what `/info` shows and whether `/config` is reachable.
- **`runtime.health`** — broker and Schema-Registry health-check timeouts.
- **`logging.level`** — per-logger levels, applied to Logback at startup.
- Every key has a default — a minimal `runtime:` block is enough to get going. Full key list: [Configuration reference](https://stoatflow.io/docs/reference/configuration-reference).
::

::callout{color="info" icon="i-lucide-info"}
`runtime.*` is the **infrastructure** half of the config. The **engine** half (lanes, commit barriers, state, watermarks) lives under `stoatflow.*` — see [Core configuration](https://stoatflow.io/docs/configuration/core-config). For the conceptual split, read [Configuration model](https://stoatflow.io/docs/concepts/configuration-model).
::

## Where runtime config comes from

`StoatFlowRuntime.fromConfig(...)` loads `application.yaml` from the classpath, layers environment variables on top, and starts the HTTP + metrics server before your topology. The `runtime.*` block is only consulted by the `:runtime` module — a `:core`-only application has no HTTP server, no metrics endpoint, and no `runtime:` block to configure.

Every `runtime.*` key maps to an environment variable using the same `__`-separated convention as the engine config: double underscore (`__`) is a path separator, single underscore is a word boundary (kebab-case). The first segment is the config root, and it must name the block being targeted — for `runtime.*` keys that root is `runtime`, so `runtime.http.port` becomes `RUNTIME__HTTP__PORT`. (Engine keys under `stoatflow.*` use the `STOATFLOW__` root; logging levels use `LOGGING__LEVEL__`.) A variable that does not start with one of these three recognised roots is ignored — `STOATFLOW__RUNTIME__HTTP__PORT` would resolve to the nonexistent path `stoatflow.runtime.http.port` and silently fail to set the port. Environment variables take precedence over YAML — convenient for per-environment overrides without rebuilding the image.

## The full runtime.\* shape

Every key shown here is at its default value, so this block is equivalent to omitting `runtime:` entirely. Trim it down to just the keys you actually change.

```yaml
runtime:
  http:
    enabled: true        # serve the HTTP admin + metrics server
    host: 0.0.0.0        # bind address (all interfaces)
    port: 8080
    backlog: 50          # max pending socket connections
    debug:
      enabled: true      # register /debug/threads and /debug/barriers

  metrics:
    enabled: true        # expose /metrics in Prometheus format
    prefix: stoatflow    # prefix for all metric names
    recording-level: info  # info | debug | trace
    bind-jvm-metrics: true # JVM memory, GC, threads
    common-tags: {}      # tags applied to every metric

  endpoints:
    info:
      show-app: true
      show-java: true
      show-kafka: true
      show-uptime: true
      show-state: true
      show-watermarks: true
      show-topology: true
      show-endpoints: true
      show-offsets: true
      show-consumer: true
    config:
      enabled: true      # serve the /config endpoint

  health:
    kafka-broker:
      timeout-ms: 2000
    schema-registry:
      timeout-ms: 5000
```

## HTTP server (runtime.http)

The runtime starts a JDK built-in HTTP server on a virtual-thread executor. It serves health probes, metrics, topology introspection, and the pause/unpause controls. See [REST API](https://stoatflow.io/docs/runtime/rest-api) for the full endpoint catalogue.

| Key                          | Default   | Purpose                                                                                                                                                                           |
| ---------------------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `runtime.http.enabled`       | `true`    | Set `false` to run with no HTTP server at all. You lose health probes, `/metrics`, and the admin endpoints — only do this for an embedded/library use case.                       |
| `runtime.http.host`          | `0.0.0.0` | Bind address. `0.0.0.0` binds all interfaces (the right choice inside a container). Use `127.0.0.1` to restrict to loopback.                                                      |
| `runtime.http.port`          | `8080`    | Listen port.                                                                                                                                                                      |
| `runtime.http.backlog`       | `50`      | Maximum pending connections in the socket backlog.                                                                                                                                |
| `runtime.http.debug.enabled` | `true`    | Registers `/debug/threads` and `/debug/barriers`. When `false`, both return 404. Disable to reduce attack surface on hardened deployments — there is no hot-path cost either way. |

A typical production override binds a fixed port and turns the debug endpoints off:

```yaml
runtime:
  http:
    port: 8080
    debug:
      enabled: false
```

::callout{color="info" icon="i-lucide-shield"}
`/debug/threads` and `/debug/barriers` are pull-only snapshots used to diagnose a stuck commit pipeline. They are safe to leave on, but if your threat model limits what the admin port exposes, gating them with `runtime.http.debug.enabled: false` is the supported way to remove them. See [REST API](https://stoatflow.io/docs/runtime/rest-api).
::

## Metrics (runtime.metrics)

Metrics are collected with Micrometer and exposed at `/metrics` in Prometheus text format. Scrape it from your existing monitoring stack — there is no agent to install. For the metric catalogue, see [Metrics](https://stoatflow.io/docs/runtime/metrics).

| Key                                          | Default             | Purpose                                                                                                                                                                                                                                                |
| -------------------------------------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `runtime.metrics.enabled`                    | `true`              | Master switch for metric collection and the `/metrics` endpoint.                                                                                                                                                                                       |
| `runtime.metrics.prefix`                     | `stoatflow`         | Prefix applied to every metric name. Change it if your conventions require a different namespace.                                                                                                                                                      |
| `runtime.metrics.recording-level`            | `info`              | Detail level: `info` (essential operational metrics, production-safe), `debug` (higher-cardinality troubleshooting metrics), `trace` (deep internals, development only).                                                                               |
| `runtime.metrics.bind-jvm-metrics`           | `true`              | Binds JVM memory, GC, and thread metrics into the registry.                                                                                                                                                                                            |
| `runtime.metrics.naming`                     | `stoatflow`         | Metric-name families to emit: `stoatflow` (native only), `kafka-streams` (KS-named only, mapped native families hidden), or `both`. See [Reusing your Kafka Streams dashboards](https://stoatflow.io/docs/migration/reusing-kafka-streams-dashboards). |
| `runtime.metrics.ks-compat.shape`            | `micrometer-binder` | Prometheus shape for KS-named series: `micrometer-binder` (`kafka_stream_*`) or `jmx-exporter` (`kafka_streams_*`). Read only when `naming` ≠ `stoatflow`.                                                                                             |
| `runtime.metrics.ks-compat.sample-window-ms` | `30000`             | Trailing window for KS `-rate`/`-avg`/`-max` twins (min `5000`).                                                                                                                                                                                       |
| `runtime.metrics.ks-compat.kafka-version`    | `4.1.0`             | Value of the synthetic `kafka_version` tag (micrometer-binder shape).                                                                                                                                                                                  |
| `runtime.metrics.common-tags`                | `{}`                | A map of tags applied to every metric — useful for environment, region, or service identification.                                                                                                                                                     |

Common tags are how you keep one Prometheus/Grafana setup readable across many deployments:

```yaml
runtime:
  metrics:
    prefix: stoatflow
    recording-level: info
    common-tags:
      env: production
      region: eu-west-1
      service: order-processor
```

## Endpoint visibility (runtime.endpoints)

This block controls what the human-facing `/info` endpoint discloses and whether `/config` is served. It does not affect health, metrics, or topology endpoints.

### /info fields

`runtime.endpoints.info.*` are individual booleans (all `true` by default) that include or omit a section from the `/info` JSON response:

| Key               | Includes                                            |
| ----------------- | --------------------------------------------------- |
| `show-app`        | Application id                                      |
| `show-java`       | Java runtime version and vendor                     |
| `show-kafka`      | Bootstrap servers                                   |
| `show-uptime`     | Process uptime                                      |
| `show-state`      | StoatFlow application state with transition history |
| `show-watermarks` | Global and per-partition watermark state            |
| `show-topology`   | Topology traits and active optimizations            |
| `show-endpoints`  | The list of available HTTP endpoints                |

### /config endpoint

| Key                                | Default | Purpose                                                                                                                                                       |
| ---------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `runtime.endpoints.config.enabled` | `true`  | Serves the `/config` endpoint, which returns the merged effective configuration with sensitive values masked. Supports JSON and YAML via content negotiation. |

To trim what an exposed admin port reveals — for example, hide the bootstrap servers from `/info` and turn off the config dump entirely:

```yaml
runtime:
  endpoints:
    info:
      show-kafka: false
    config:
      enabled: false
```

## Health checks (runtime.health)

The readiness and liveness probes aggregate a set of health indicators. Two of them have configurable timeouts here. See [Health checks](https://stoatflow.io/docs/runtime/health-checks) for how the indicators map to `/health/live` and `/health/ready`.

| Key                                         | Default | Purpose                                                                                                                           |
| ------------------------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `runtime.health.kafka-broker.timeout-ms`    | `2000`  | Timeout for the broker connectivity check (via AdminClient).                                                                      |
| `runtime.health.schema-registry.timeout-ms` | `5000`  | Timeout for the Schema Registry connectivity check. This indicator auto-enables only when `stoatflow.schema-registry-url` is set. |

```yaml
runtime:
  health:
    kafka-broker:
      timeout-ms: 3000
    schema-registry:
      timeout-ms: 5000
```

## Logging levels (logging.level)

`logging.level` is a top-level block (a sibling of `stoatflow` and `runtime`, not nested under either). It maps logger names to levels and applies them to Logback at startup, overriding whatever `logback.xml` set. Use `ROOT` for the root logger.

```yaml
logging:
  level:
    ROOT: INFO
    io.stoatflow: DEBUG
    org.apache.kafka: WARN
    org.apache.kafka.clients.consumer: INFO
```

Valid levels are `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`, and `OFF`. Per-logger levels can also be set via environment variables with the `LOGGING__LEVEL__` prefix — for example `LOGGING__LEVEL__ORG_APACHE_KAFKA_CLIENTS=DEBUG` sets `org.apache.kafka.clients` to DEBUG. The single-underscore segments become dots in the logger name. Environment variables take precedence over YAML when both name the same logger.

## Kafka client passthrough (stoatflow\.kafka)

Kafka client tuning lives under `stoatflow.kafka.*` (the engine half), not under `runtime.*` — but it is part of operating a runtime app, so it is worth knowing where it is. The three maps pass properties straight through to the underlying Kafka clients, applied on top of StoatFlow's framework defaults:

```yaml
stoatflow:
  kafka:
    consumer:
      max.poll.records: 1000
      fetch.min.bytes: 65536
    producer:
      compression.type: lz4
      linger.ms: 50
    restoration-consumer:
      max.poll.records: 5000
```

A few forced overrides (for example `bootstrap.servers`, `group.id`, `enable.auto.commit=false`, and the transactional settings under exactly-once) cannot be overridden here — they are required for correctness. The full resolution order and the list of forced values are on [Kafka client configuration](https://stoatflow.io/docs/configuration/kafka-client-config).

::callout{color="warning" icon="i-lucide-triangle-alert"}
These maps preserve dotted Kafka property names verbatim (`linger.ms`, `max.poll.records`). When setting them via environment variables, use the dedicated `STOATFLOW__KAFKA__PRODUCER__*` / `STOATFLOW__KAFKA__CONSUMER__*` form (e.g. `STOATFLOW__KAFKA__PRODUCER__LINGER_MS=50`) — the generic env path would mangle the dotted keys. Details on [Kafka client configuration](https://stoatflow.io/docs/configuration/kafka-client-config).
::

### Map-valued blocks and environment variables

Three config blocks are maps whose *keys* are data — a Kafka property name, a logger name, a validation rule id — rather than a fixed schema. The generic `__` path form cannot carry those keys intact, so each has a dedicated environment-variable prefix that preserves them:

| Block                                                            | Environment form                                                      | Key fold                          |
| ---------------------------------------------------------------- | --------------------------------------------------------------------- | --------------------------------- |
| `stoatflow.kafka.{consumer,producer,restoration-consumer,admin}` | `STOATFLOW__KAFKA__CONSUMER__MAX_POLL_RECORDS=1000`                   | `_` → `.` (dotted Kafka property) |
| `logging.level`                                                  | `LOGGING__LEVEL__ORG_APACHE_KAFKA_CLIENTS=DEBUG`                      | `_` → `.` (dotted logger name)    |
| `stoatflow.topology.validation`                                  | `STOATFLOW__TOPOLOGY__VALIDATION__INHERITED_BOUNDARY_KEY_SERDE=error` | `_` → `-` (kebab rule id)         |

Note the fold differs: Kafka properties and logger names are dotted, validation rule ids are kebab-case. Environment variables win over YAML for the same key in all three.

## License (stoatflow\.license)

The license block is also under `stoatflow.*`, not `runtime.*`, but the `:runtime` module bridges it to the engine and exposes license state via the `/license` endpoint and the license health indicator. The minimal form references the key from an environment variable:

```yaml
stoatflow:
  license:
    key: ${STOATFLOW_LICENSE_KEY}
    environment: prod          # required for the Production tier
    cache-dir: /var/lib/stoatflow/state/license-cache
```

The full set of license keys, environment variables, and system properties — plus the production-tier requirements — is documented on [License configuration](https://stoatflow.io/docs/getting-started/license-configuration).

## Worked example

A production-leaning `runtime` + `logging` block: fixed port, debug endpoints off, environment tags on every metric, and a quieter Kafka client logger.

```yaml
runtime:
  http:
    host: 0.0.0.0
    port: 8080
    debug:
      enabled: false
  metrics:
    enabled: true
    recording-level: info
    common-tags:
      env: production
      region: eu-west-1
  endpoints:
    config:
      enabled: false
  health:
    kafka-broker:
      timeout-ms: 3000

logging:
  level:
    io.stoatflow: INFO
    org.apache.kafka: WARN
```

## Next steps

- [Core configuration](https://stoatflow.io/docs/configuration/core-config) — the `stoatflow.*` engine half (lanes, barriers, state, watermarks).
- [Kafka client configuration](https://stoatflow.io/docs/configuration/kafka-client-config) — the full consumer/producer/restoration passthrough and resolution order.
- [Defaults and presets](https://stoatflow.io/docs/configuration/defaults-and-presets) — what you get out of the box and the RocksDB presets.
- [REST API](https://stoatflow.io/docs/runtime/rest-api) and [Metrics](https://stoatflow.io/docs/runtime/metrics) — what the HTTP server and `/metrics` actually serve.
- [Configuration reference](https://stoatflow.io/docs/reference/configuration-reference) — the curated key reference with defaults (the JSON schema is the authoritative, always-current list).


# Defaults, adaptivity, and presets

StoatFlow ships opinionated defaults so a near-empty `application.yaml` runs a correct, bounded-memory application out of the box. This page explains what those defaults give you, which behaviour adapts at runtime versus what you set explicitly, and the RocksDB presets and bound knobs available when you do need to tune. For the layered resolution rules behind these settings (env / system property / YAML / programmatic precedence), see [Configuration model](https://stoatflow.io/docs/concepts/configuration-model); for a workload-driven walkthrough, see [Tuning](https://stoatflow.io/docs/operating/tuning).

::tldr-panel
- **Minimal config works** — only `application-id` is required; every other key has a sensible default.
- **Bounded by default** — RocksDB memory, uncommitted state, and lane queues all have caps out of the box, unlike Kafka Streams' unbounded defaults.
- **Commit cadence and epoch size self-tune** within the interval and memory bounds you set — you configure the envelope, the runtime adapts inside it.
- **RocksDB has three presets** (`DEFAULT`, `LOW_MEMORY`, `HIGH_PERFORMANCE`) plus a config-setter escape hatch for full control.
::

## A minimal config that works

The only required key is `stoatflow.application-id`. Everything else falls back to a default. A complete, runnable configuration can be as short as this:

```yaml
stoatflow:
  application-id: my-app
  bootstrap-servers: localhost:9092
```

Add the runtime's HTTP and metrics endpoints and the license, and you have what [Your first app](https://stoatflow.io/docs/getting-started/first-app) uses:

```yaml
stoatflow:
  application-id: word-count
  bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVERS:-localhost:9092}
  license:
    key: ${STOATFLOW_LICENSE_KEY}

runtime:
  http:
    enabled: true
    port: 8080
  metrics:
    enabled: true
```

This runs because the defaults already make the right choices for most workloads. The ones worth knowing:

| Concern              | Default               | Why                                                                                      |
| -------------------- | --------------------- | ---------------------------------------------------------------------------------------- |
| Processing guarantee | `EXACTLY_ONCE`        | Atomic commit of output, changelog, and offsets in one Kafka transaction.                |
| Lane count           | `max(2, CPU cores)`   | Parallelism scales with the machine you run on.                                          |
| Lane queue capacity  | `300`                 | Per-lane backpressure bound.                                                             |
| State store backend  | RocksDB (FFM)         | Persistent, on-disk, no virtual-thread pinning.                                          |
| RocksDB memory       | \~256 MiB total       | Bounded by default — Kafka Streams is unbounded.                                         |
| Changelog topics     | enabled, auto-created | State survives restart; topics created with `cleanup.policy=compact`.                    |
| Explicit naming      | enforced              | Every operator and store needs a stable name (KIP-1111); fails fast otherwise.           |
| State restoration    | enabled               | Local state rebuilds from changelog on cold start.                                       |
| Default serdes       | `ByteArray`           | Set `default-key-serde` / `default-value-serde` (or per-operator serdes) for real types. |

::callout{color="info" icon="i-lucide-shield-check"}
**Bounded by default is the deliberate difference from Kafka Streams.** RocksDB memory (\~256 MiB), total uncommitted state (256 MiB), and lane queue depth (300) all have caps without you setting anything. The goal is that the default config does not OOM a modestly sized container under load — you raise the caps when you have headroom, rather than discovering an unbounded default the hard way.
::

## What adapts, and what you set

Some runtime behaviour is **adaptive**: the engine measures its own commit cost and state-write rate and adjusts within bounds. The rest is **fixed** to what you configure. The split matters because you tune the two kinds differently — for adaptive behaviour you set the *envelope*, for fixed behaviour you set the *value*.

### Commit cadence self-tunes within an interval band

The commit barrier interval is not a fixed sleep. The runtime estimates how long a commit takes and schedules the next barrier relative to that estimate, so a topology with fast commits barriers more often and a topology with slow commits backs off — both staying inside the band you configure. You set the band; the runtime picks the cadence inside it.

| YAML key                         | Default | What it sets                                                                                                                          |
| -------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| `commit-barrier.interval-ms`     | `500`   | Seed interval used before the runtime has measured a commit.                                                                          |
| `commit-barrier.min-interval-ms` | `150`   | Floor — barriers never fire closer together than this.                                                                                |
| `commit-barrier.max-interval-ms` | `5000`  | Ceiling — barriers never fire further apart than this.                                                                                |
| `commit-barrier.interval-factor` | `1.5`   | Cadence target as a multiple of measured commit duration.                                                                             |
| `commit-barrier.timeout-ms`      | `60000` | Hard timeout for a commit; also cascades to the Kafka producer's `transaction.timeout.ms`, `max.block.ms`, and `delivery.timeout.ms`. |

```yaml
stoatflow:
  commit-barrier:
    min-interval-ms: 100     # allow tighter commits for lower latency
    max-interval-ms: 2000    # but never let an epoch run longer than 2s
    interval-factor: 1.5
```

Lower the band for lower end-to-end latency (more frequent, smaller commits); raise it for higher throughput (fewer, larger commits). The `interval-factor` controls how aggressively the cadence tracks commit cost: `1.0` is back-to-back, higher values leave more idle gap between commits.

### Epoch size self-tunes within memory and count bounds

An **epoch** is the batch of records between two commit barriers. The runtime sizes each epoch automatically, tracking the dispatch rate and the per-record state-write cost so it can keep uncommitted state bounded — small for high-fan-out stateful topologies (FK joins, multi-way joins, where one input record cascades into many state writes) and large for lightweight stateless ones. You don't set the epoch size; you set the bounds it operates within.

| YAML key                                   | Default               | What it sets                                                                                                                                                                                           |
| ------------------------------------------ | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `state.uncommitted-max-bytes`              | `268435456` (256 MiB) | Hard limit on total uncommitted state bytes across all stores. The dominant bound for stateful, high-fan-out topologies.                                                                               |
| `commit-barrier.max-epoch-records`         | unset (no hard cap)   | Optional hard ceiling on records per epoch.                                                                                                                                                            |
| `commit-barrier.initial-max-epoch-records` | `4096`                | Conservative cap for the first epoch(s), before the runtime has measured the topology. `-1` disables it.                                                                                               |
| `caching.max-estimated-bytes`              | `268435456` (256 MiB) | **Per-store** cap on estimated emission-cache bytes; exceeding it fires an early cache-pressure commit. The global `state.uncommitted-max-bytes` ceiling measures a superset and normally fires first. |
| `caching.max-entries`                      | unbounded (opt-in)    | Optional **per-store** entry-count cap — a lever for high-cardinality workloads that the byte estimate can't express. Setting a finite value makes workloads exceeding it early-commit.                |

```yaml
stoatflow:
  state:
    uncommitted-max-bytes: 268435456   # 256 MiB — raise on large containers, lower on small
  commit-barrier:
    initial-max-epoch-records: 4096    # cap startup epochs before adaptation kicks in
    max-epoch-records: 50000           # optional hard ceiling
```

::callout{color="info" icon="i-lucide-info"}
The adaptive sizing **algorithm** — how the runtime converges on an epoch size from its measurements — is an implementation concern and lives in the source. What you configure are the public bounds: the memory limit it must not exceed, the optional record ceiling, and the startup cap. Within those, it self-tunes; you do not hand-set epoch sizes per workload.
::

Sizing your `uncommitted-max-bytes` to the container is the main lever for stateful topologies. Uncommitted state is held **on the heap**, so the budget to size against is the computed `-Xmx`, not the container limit. The [container-aware entrypoint](https://stoatflow.io/docs/runtime/docker#container-aware-heap-sizing) sets `-Xmx` to the container limit minus a non-heap reservation — the JVM baseline (25% of the limit, capped at 512 MiB) plus RocksDB's 256 MiB — so on a 4 GiB container the heap is `4096 − (512 + 256) = 3328 MiB`. Nothing is left over: the reservation is what the JVM and RocksDB spend outside the heap, not spare room.

As a starting point, leave at least half the heap free for your own objects, the lane queues and GC:

| Container memory | Computed `-Xmx` | `uncommitted-max-bytes` | Rationale                                                          |
| ---------------- | --------------- | ----------------------- | ------------------------------------------------------------------ |
| 4 GiB            | 3328 MiB        | 1 GiB (`1073741824`)    | Leaves \~2.3 GiB of heap after the producer's 256 MiB send buffer. |
| 8 GiB            | 7424 MiB        | 1–2 GiB                 | More room for GC and Kafka client buffers.                         |
| 16 GiB           | 15616 MiB       | 2–4 GiB                 | Recommended for heavy stateful topologies.                         |

Those `-Xmx` figures assume RocksDB's default 256 MiB reservation. Change the preset and the reservation moves with it — `STOATFLOW_ROCKSDB_MB` is what the entrypoint subtracts.

The `/metrics` endpoint exposes per-epoch counters — `stoatflow.barrier.epoch.actual.records`, `stoatflow.barrier.epoch.memory.cap`, and a `stoatflow.barrier.trigger.total` counter tagged by what ended each epoch (time, record count, memory pressure, or per-store cache pressure) — so you can see which bound is binding and adjust the right one. See [Metrics](https://stoatflow.io/docs/runtime/metrics).

### Fixed knobs you set directly

These do not adapt; the value you set is the value used.

| YAML key                            | Default             | Notes                                                                                                                                              |
| ----------------------------------- | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `lanes.count`                       | `max(2, CPU cores)` | Per-key parallelism. Higher = more concurrency, more queues.                                                                                       |
| `lanes.queue-capacity`              | `300`               | Per-lane backpressure bound.                                                                                                                       |
| `processing-guarantee`              | `EXACTLY_ONCE`      | Set `AT_LEAST_ONCE` for lower commit latency where duplicates are tolerable — see [Exactly-once](https://stoatflow.io/docs/concepts/exactly-once). |
| `watermark.max-out-of-orderness-ms` | `10000`             | Event-time slack — see [Event time and watermarks](https://stoatflow.io/docs/concepts/event-time-and-watermarks).                                  |

## RocksDB presets and tuning

State stores default to RocksDB with a **bounded \~256 MiB memory budget** (block cache + memtables), as opposed to Kafka Streams' unbounded RocksDB defaults. You select a memory profile with a single key.

### Presets

```yaml
stoatflow:
  rocks-db:
    preset: DEFAULT   # DEFAULT | LOW_MEMORY | HIGH_PERFORMANCE
    backend: AUTO     # AUTO | FFM | JNI
```

| Preset             | Total memory | Composition                             | Use for                                                                               |
| ------------------ | ------------ | --------------------------------------- | ------------------------------------------------------------------------------------- |
| `LOW_MEMORY`       | 64 MiB       | 32 MiB block cache + 32 MiB memtables   | Development, CI, constrained containers. More frequent compaction and cache eviction. |
| `DEFAULT`          | 256 MiB      | 128 MiB block cache + 128 MiB memtables | Most production workloads with moderate state.                                        |
| `HIGH_PERFORMANCE` | 1 GiB        | 512 MiB block cache + 512 MiB memtables | High-throughput workloads with large state. Needs more memory.                        |

The memory budget is **shared across all stores** — one block cache and one write-buffer manager bound total RocksDB memory regardless of how many state stores the topology declares.

### Backend

`backend: AUTO` (default) picks each runtime's faster path: the Foreign Function & Memory API (`FFM`) on the JVM — lower per-call overhead, no virtual-thread pinning — and the `rocksdbjni` bindings (`JNI`) under a native image. Set `FFM` or `JNI` explicitly to force a specific backend. Both require the `--enable-native-access=ALL-UNNAMED` JVM flag (the `io.stoatflow` Gradle plugin sets it — see [Installation](https://stoatflow.io/docs/getting-started/installation)).

### Beyond the presets

The three presets cover the common cases. For finer control there are two escape hatches, neither exposed as additional YAML keys:

- **Custom total-memory budget (programmatic).** When constructing the config in code rather than YAML, `RocksDBConfig.withTotalMemory(bytes)` produces a config with cache, memtables, and file sizes scaled proportionally to any budget (minimum 32 MiB): :code-tabs[```kotlin \[Kotlin\]
  import io.stoatflow.core.state.rocksdb.RocksDBConfig

  // 512 MiB total, scaled proportionally
  val rocks = RocksDBConfig.withTotalMemory(512L * 1024 * 1024)
  ``````java \[Java\]
  import io.stoatflow.core.state.rocksdb.RocksDBConfig;

  // 512 MiB total, scaled proportionally
  RocksDBConfig rocks = RocksDBConfig.withTotalMemory(512L * 1024 * 1024);
  ```]{group="lang"}
- **Per-store RocksDB options (config setter).** For per-store tuning — different settings for write-heavy versus read-heavy stores, custom compression, compaction strategy — set `rocks-db-config-setter` to the fully qualified name of a class implementing `RocksDBConfigSetter`. Its `configure(storeName, options)` runs for every store with `options` pre-populated from the active preset, so you can branch on the store name and override per-store knobs such as `options.cf.maxWriteBufferNumber`, `options.cf.writeBufferSize`, or `options.table.useRibbonFilter`. Shared memory resources (the block cache and write-buffer manager) are framework-managed and **cannot** be overridden per store — set the total memory budget globally via the preset or `RocksDBConfig.withTotalMemory(...)`.
  ```yaml
  stoatflow:
    rocks-db-config-setter: com.example.MyRocksDBConfigSetter
  ```

::callout{color="warning" icon="i-lucide-triangle-alert"}
Ribbon filters (`useRibbonFilter`, default on) only take effect on the **FFM** backend — the JNI bindings fall back to Bloom filters. If you switch to `backend: JNI`, expect a slightly larger filter memory footprint at the same false-positive rate.
::

### Where the RocksDB budget sits in the pod

RocksDB's 256 MiB is one of four caps the defaults set. This is all of them against a 4 GiB container, drawn to scale, each with the knob that moves it and what fires when it fills:

![A 4 GiB pod's memory drawn as one stacked bar, to scale. At the bottom, two bands are held back off-heap before the container-aware entrypoint sets -Xmx: a 512 MiB JVM baseline — 25% of the limit capped at 512, which STOATFLOW\_NON\_HEAP\_MB replaces wholesale together with the RocksDB reservation — and RocksDB's 256 MiB from the DEFAULT preset — 128 MiB block cache plus 128 MiB memtables shared by every store, which on overflow evicts and compacts rather than firing a barrier. That leaves an -Xmx of 3328 MiB, holding a 256 MiB producer send buffer (kafka.producer.buffer.memory, which applies producer-side backpressure when full), a 256 MiB uncommitted-state ceiling (state.uncommitted-max-bytes, which fires an early commit barrier tagged MEMORY\_PRESSURE), and 2816 MiB of headroom for your objects, lane queues and GC, which nothing caps. An inset shows that each caching store has its own 256 MiB caching.max-estimated-bytes cap tagged CACHE\_PRESSURE, but their sum is bounded by the single uncommitted-state band, so the global ceiling normally fires first. In total the defaults reserve 1280 MiB of the 4096 and leave 2816 MiB free; the caps are absolute and do not scale with the container.](https://stoatflow.io/assets/docs/configuration/memory-budget_20260727.svg)

Two things follow from the shape of it. The caps are **absolute**, so a larger container buys headroom and nothing else — which is why the `uncommitted-max-bytes` sizing table above raises the ceiling with the container rather than leaving it at the default. And `-Xmx` is **derived**, not set: the container-aware entrypoint subtracts the RocksDB reservation and a JVM baseline from the container's own memory limit, so you size the container and the image sizes the heap. See [Docker → Container-aware heap sizing](https://stoatflow.io/docs/runtime/docker#container-aware-heap-sizing) for that calculation and [Tuning](https://stoatflow.io/docs/operating/tuning) for which knob to reach for from which symptom.

## Where to go next

- [Configuration model](https://stoatflow.io/docs/concepts/configuration-model) — how env vars, system properties, YAML, and programmatic config layer and override each other.
- [Tuning](https://stoatflow.io/docs/operating/tuning) — workload-driven tuning walkthroughs for latency, throughput, and heavy-state topologies.
- [Configuration reference](https://stoatflow.io/docs/reference/configuration-reference) — every key, default, and env override in one table.
- [Metrics](https://stoatflow.io/docs/runtime/metrics) — the per-epoch and commit-cadence counters that show which bound is binding.


# Kafka client configuration

StoatFlow constructs its own Kafka consumer, producer, and restoration consumers. You configure those clients by passing standard Kafka client properties through your `application.yaml` under `stoatflow.kafka.*`. This page covers the passthrough form, the opinionated defaults StoatFlow layers on top, and the small set of properties it forces for exactly-once correctness.

::tldr-panel
- **Passthrough:** put any Kafka consumer / producer property under `stoatflow.kafka.consumer` / `stoatflow.kafka.producer`. Restoration consumers tune separately under `stoatflow.kafka.restoration-consumer`.
- **Defaults:** StoatFlow applies single-instance, high-throughput defaults (bigger fetches, bigger batches) over the Kafka defaults — all overridable.
- **Forced:** a few keys (`bootstrap.servers`, `group.id`, `enable.auto.commit`, and the EOS producer keys like `transactional.id`, `enable.idempotence`, `acks`) are set by the framework and cannot be overridden.
::

## Passthrough form

Every property you set under `stoatflow.kafka.consumer`, `stoatflow.kafka.producer`, `stoatflow.kafka.restoration-consumer` or `stoatflow.kafka.admin` is passed straight to the corresponding Kafka client, using the exact Kafka property names (dotted keys):

```yaml
stoatflow:
  bootstrap-servers: localhost:9092
  kafka:
    consumer:
      max.poll.records: 1000
      auto.offset.reset: latest   # StoatFlow defaults this to earliest; override to latest here
      fetch.min.bytes: 65536
    producer:
      batch.size: 524288
      linger.ms: 50
      compression.type: lz4
```

You don't need to list every property — only the ones you want to change. Anything you omit falls back to StoatFlow's framework default, and anything StoatFlow doesn't set falls back to the Kafka client default.

::callout{color="info" icon="i-lucide-info"}
These are standard Kafka client properties. Use the names from the [Apache Kafka consumer](https://kafka.apache.org/documentation/#consumerconfigs){rel=""nofollow""} and [producer](https://kafka.apache.org/documentation/#producerconfigs){rel=""nofollow""} configuration reference verbatim. StoatFlow does not rename or alias them.
::

For where these keys sit in the wider configuration tree (and how YAML, environment variables, and system properties compose), see the [configuration model](https://stoatflow.io/docs/concepts/configuration-model) and the [core configuration](https://stoatflow.io/docs/configuration/core-config) reference.

## How consumer properties resolve

Consumer properties are resolved by merging layers, each overwriting the previous:

| Layer | Source                     | Purpose                                                                              |
| ----- | -------------------------- | ------------------------------------------------------------------------------------ |
| 1     | Kafka client defaults      | Implicit defaults from the Kafka client library                                      |
| 2     | Framework defaults         | StoatFlow's opinionated defaults (see below)                                         |
| 3     | `stoatflow.kafka.consumer` | Your overrides                                                                       |
| 4     | `main.consumer.` keys      | Main-consumer-only overrides — reachable only from a KS-keyed `Properties`, not YAML |
| 5     | Forced overrides           | Set by the framework, cannot be overridden                                           |

### Framework defaults

| Property                    | StoatFlow default | Kafka default | Rationale                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| --------------------------- | ----------------- | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `max.poll.records`          | 500               | 500           | Matches the Kafka default                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `fetch.max.bytes`           | 50 MiB            | 50 MiB        | Matches the Kafka default                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `max.partition.fetch.bytes` | 10 MiB            | 1 MiB         | A single instance consumes all partitions; larger per-partition fetches reduce network round trips                                                                                                                                                                                                                                                                                                                                                                         |
| `auto.offset.reset`         | `earliest`        | `latest`      | Matches Kafka Streams (which also defaults its main consumer to `earliest`) — a fresh `application.id` processes the topic from the beginning. Overridable.                                                                                                                                                                                                                                                                                                                |
| `max.poll.interval.ms`      | 10 min            | 5 min         | StoatFlow is single-instance, so a coordinator partition revocation is **always fatal** (no peer to rebalance to). Genuine hangs are already caught by the commit-pipeline watchdog (`commit-stall-threshold-ms`, 45 s) and the stall-aware liveness probe, so a short interval only causes *spurious* fatal shutdowns under transient slowness (GC pauses, CPU contention, a slow blocking downstream). A generous default removes that false-positive risk. Overridable. |

### Forced overrides

These are always set by the framework. Setting them under `stoatflow.kafka.consumer` has no effect.

| Property             | Value                                    | Reason                                                                                                                                                                                                                              |
| -------------------- | ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `bootstrap.servers`  | from `stoatflow.bootstrap-servers`       | Must match the cluster                                                                                                                                                                                                              |
| `group.id`           | the application ID                       | Static group membership for the single instance                                                                                                                                                                                     |
| `group.instance.id`  | the application ID                       | Static membership (KIP-345) for instant partition re-assignment on restart                                                                                                                                                          |
| `enable.auto.commit` | `false`                                  | Offsets are committed by StoatFlow's commit protocol, not by the Kafka client                                                                                                                                                       |
| `isolation.level`    | `read_committed` — **exactly-once only** | An exactly-once application must not ingest a transactional upstream's aborted batches. Matches Kafka Streams. Under at-least-once the key is **not** forced, so you can still set it yourself if you read a transactional upstream |

`auto.offset.reset` defaults to `earliest` (matching Kafka Streams) rather than the raw Kafka client's `latest`. It is **not** forced — override it globally under `stoatflow.kafka.consumer`, or per source topic with `Consumed.withOffsetResetPolicy(...)`, which seeks explicitly and takes precedence for that topic.

One exception: with [hot-standby HA](https://stoatflow.io/docs/operating/high-availability) enabled, `group.instance.id` is **omitted** (and any user-set value removed) — HA pods use `assign()` with no consumer-group membership, so static membership does not apply.

## How producer properties resolve

Producer properties resolve through the same layered merge:

| Layer | Source                                     | Purpose                                                                                                         |
| ----- | ------------------------------------------ | --------------------------------------------------------------------------------------------------------------- |
| 1     | Kafka client defaults                      | Implicit defaults from the Kafka client library                                                                 |
| 2     | Framework defaults                         | StoatFlow's opinionated defaults (see below)                                                                    |
| 3     | `stoatflow.kafka.consumer` security subset | The consumer's `security.*` / `ssl.*` / `sasl.*` settings, so a secured cluster needs its credentials only once |
| 4     | `stoatflow.kafka.producer`                 | Your overrides                                                                                                  |
| 5     | Forced overrides                           | Set by the framework, cannot be overridden                                                                      |

::callout{color="warning" icon="i-lucide-triangle-alert"}
**Shadow a security family completely, or not at all.** Layer 4 overrides Layer 3 **per key**, so a producer block naming only *some* keys of a `security.` / `ssl.` / `sasl.` family inherits the rest and ends up mixed. A consumer on `OAUTHBEARER` plus a producer setting only `sasl.mechanism: PLAIN` and `sasl.jaas.config` still inherits the consumer's `sasl.login.callback.handler.class`, and the producer then fails to construct at startup. StoatFlow warns when it detects a partial shadow and names the inherited keys.
::

### Framework defaults

| Property                                | StoatFlow default | Kafka default | Rationale                                                                                                                                                              |
| --------------------------------------- | ----------------- | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `batch.size`                            | 256 KiB           | 16 KiB        | StoatFlow produces output across many keys at once; larger batches improve throughput                                                                                  |
| `linger.ms`                             | 20                | 0             | Wait up to 20 ms to fill batches — paired with the larger `batch.size`                                                                                                 |
| `retry.backoff.ms`                      | 50                | 100           | Halves the per-retry wait on the transaction coordinator's transient `CONCURRENT_TRANSACTIONS` response, shrinking the commit-latency tail at frequent commit cadences |
| `buffer.memory`                         | 256 MiB           | 32 MiB        | Headroom for the larger batches without producer-side backpressure                                                                                                     |
| `enable.idempotence`                    | `true`            | `true`        | Explicit baseline locked to the StoatFlow version, insulating you from upstream default changes. Also forced in EOS mode (below).                                      |
| `acks`                                  | `all`             | `all`         | Explicit baseline locked to the StoatFlow version. Also forced in EOS mode.                                                                                            |
| `max.in.flight.requests.per.connection` | `5`               | `5`           | Explicit baseline locked to the StoatFlow version. Also forced in EOS mode.                                                                                            |

::callout{color="info" icon="i-lucide-info"}
**Memory note.** `buffer.memory` (256 MiB) is producer-side buffering and is separate from RocksDB and state-store memory. On small-heap deployments (under \~2 GiB) with both heavy state writes and heavy sink output, consider reducing `buffer.memory` — see the [memory-constrained](https://stoatflow.io/#memory-constrained-environments) scenario below and the [tuning guide](https://stoatflow.io/docs/operating/tuning).
::

### Forced overrides

| Property              | Value                              | Reason                                                                                                                       |
| --------------------- | ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| `bootstrap.servers`   | from `stoatflow.bootstrap-servers` | Must match the cluster                                                                                                       |
| `retries`             | `Integer.MAX_VALUE`                | Unlimited retries are required for idempotent / transactional producer correctness                                           |
| `max.block.ms`        | the configured barrier timeout     | Bounds how long a `send()` can block on a slow broker, so a broker slowdown surfaces as a timeout rather than a silent stall |
| `delivery.timeout.ms` | the configured barrier timeout     | Bounds the end-to-end delivery path on the same budget                                                                       |

The barrier timeout is the `stoatflow.commit-barrier.timeout-ms` knob — see the [core configuration](https://stoatflow.io/docs/configuration/core-config) reference.

#### Exactly-once mode only

When the [processing guarantee](https://stoatflow.io/docs/concepts/exactly-once) is `EXACTLY_ONCE`, StoatFlow additionally forces the transactional producer properties:

| Property                                | Value                          | Reason                                                                                                       |
| --------------------------------------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------ |
| `transactional.id`                      | `{applicationId}-producer`     | Transaction-coordination identity                                                                            |
| `transaction.timeout.ms`                | the configured barrier timeout | Must align with the commit budget                                                                            |
| `enable.idempotence`                    | `true`                         | Required for the transactional producer — disabling it breaks exactly-once                                   |
| `acks`                                  | `all`                          | `acks < all` weakens durability under broker failures, which is incompatible with the exactly-once guarantee |
| `max.in.flight.requests.per.connection` | `5`                            | The maximum that still preserves the idempotent producer's ordering guarantee                                |

In **at-least-once** mode (`AT_LEAST_ONCE`), `enable.idempotence`, `acks`, and `max.in.flight.requests.per.connection` stay at the framework-default layer and remain overridable via `stoatflow.kafka.producer`, so you can trade throughput against durability. In **exactly-once** mode they are forced: any value you set under `stoatflow.kafka.producer` is overwritten by the framework's forced override, so the client never sees a configuration that would break exactly-once. To see what the producer actually receives, check the resolved configuration (see [Inspecting the resolved configuration](https://stoatflow.io/#inspecting-the-resolved-configuration) below).

## Restoration consumer

State stores recover from changelog topics on startup using dedicated **restoration consumers**, which are configured independently of the processing consumer under `stoatflow.kafka.restoration-consumer`. They inherit the processing `stoatflow.kafka.consumer` as a shared baseline, then layer restoration-specific defaults and overrides on top:

| Layer | Source                                 | Purpose                                                               |
| ----- | -------------------------------------- | --------------------------------------------------------------------- |
| 1     | Kafka client defaults                  | Implicit defaults from the Kafka client library                       |
| 2     | `stoatflow.kafka.consumer`             | Processing-consumer config — shared baseline inherited by restoration |
| 3     | Framework defaults                     | Restoration-specific defaults (see below)                             |
| 4     | `stoatflow.kafka.restoration-consumer` | Your restoration overrides                                            |
| 5     | Forced overrides                       | Set by the framework, cannot be overridden                            |

### Framework defaults

Tuned for bulk changelog reads during recovery:

| Property                    | Default | Kafka default | Rationale                                             |
| --------------------------- | ------- | ------------- | ----------------------------------------------------- |
| `max.poll.records`          | 1,000   | 500           | Larger batches reduce poll overhead during bulk reads |
| `fetch.max.bytes`           | 50 MiB  | 50 MiB        | Matches the Kafka default (already large)             |
| `max.partition.fetch.bytes` | 10 MiB  | 1 MiB         | Faster per-partition fetches during restoration       |

### Forced overrides

| Property             | Value                                             | Reason                                                                       |
| -------------------- | ------------------------------------------------- | ---------------------------------------------------------------------------- |
| `bootstrap.servers`  | from `stoatflow.bootstrap-servers`                | Must match the cluster                                                       |
| `group.id`           | `{applicationId}-restoration-{timestamp}`         | Unique per restoration run                                                   |
| `enable.auto.commit` | `false`                                           | Restoration offsets are tracked alongside the state, not by the Kafka client |
| `auto.offset.reset`  | `earliest`                                        | Restoration must read the changelog from the beginning                       |
| `isolation.level`    | `read_committed` (EOS) / `read_uncommitted` (ALO) | Must match the processing guarantee                                          |

Restoration tuning:

```yaml
stoatflow:
  kafka:
    restoration-consumer:
      max.poll.records: 5000
      fetch.max.bytes: 104857600   # 100 MiB
```

## Common tuning scenarios

### High throughput

For maximum throughput with high event rates:

```yaml
stoatflow:
  kafka:
    consumer:
      max.poll.records: 2000
      fetch.min.bytes: 65536        # 64 KiB — wait for larger fetches
      fetch.max.wait.ms: 200        # max wait for fetch.min.bytes
    producer:
      batch.size: 524288            # 512 KiB
      linger.ms: 50                 # allow more time to fill batches
      compression.type: lz4         # reduce network bandwidth
```

### Low latency

For latency-sensitive workloads:

```yaml
stoatflow:
  kafka:
    consumer:
      fetch.min.bytes: 1            # return immediately (Kafka default)
      max.poll.records: 100         # smaller batches for lower processing latency
    producer:
      linger.ms: 0                  # send immediately
      batch.size: 16384             # Kafka default (16 KiB)
```

### Large records

When individual records are large (e.g. complex Avro > 10 KB):

```yaml
stoatflow:
  kafka:
    consumer:
      max.partition.fetch.bytes: 20971520   # 20 MiB
      fetch.max.bytes: 104857600            # 100 MiB
    producer:
      max.request.size: 10485760            # 10 MiB
      buffer.memory: 536870912              # 512 MiB
```

### Memory-constrained environments

When heap is limited:

```yaml
stoatflow:
  kafka:
    consumer:
      max.poll.records: 100
      max.partition.fetch.bytes: 1048576    # 1 MiB (Kafka default)
      fetch.max.bytes: 10485760             # 10 MiB
    producer:
      batch.size: 32768                     # 32 KiB
      buffer.memory: 33554432               # 32 MiB (Kafka default)
```

### Fast restoration

For large state stores where recovery time matters:

```yaml
stoatflow:
  kafka:
    restoration-consumer:
      max.poll.records: 5000
      receive.buffer.bytes: 1048576         # 1 MiB socket buffer
```

### Low-memory restoration

When restoration causes GC pressure on a constrained heap:

```yaml
stoatflow:
  kafka:
    restoration-consumer:
      max.poll.records: 200
      max.partition.fetch.bytes: 1048576    # 1 MiB (Kafka default)
      fetch.max.bytes: 10485760             # 10 MiB
```

## Setting properties via environment variables

Kafka client properties can also be set via environment variables, which take precedence over YAML. The format is `STOATFLOW__KAFKA__{CONSUMER|PRODUCER|RESTORATION_CONSUMER|ADMIN}__{PROPERTY}`, where the property name uses underscores instead of dots and is uppercased:

```bash
# Consumer properties
export STOATFLOW__KAFKA__CONSUMER__MAX_POLL_RECORDS=1000
export STOATFLOW__KAFKA__CONSUMER__FETCH_MAX_BYTES=104857600

# Producer properties
export STOATFLOW__KAFKA__PRODUCER__BATCH_SIZE=524288
export STOATFLOW__KAFKA__PRODUCER__LINGER_MS=50

# Restoration consumer properties
export STOATFLOW__KAFKA__RESTORATION_CONSUMER__MAX_POLL_RECORDS=5000
```

So `max.poll.records` becomes `STOATFLOW__KAFKA__CONSUMER__MAX_POLL_RECORDS`. See the [configuration model](https://stoatflow.io/docs/concepts/configuration-model) for the full precedence rules.

## SASL and SSL

Authentication and encryption are plain Kafka client properties — pass them through like any other:

**Set them once, under `consumer`.** StoatFlow propagates the consumer's `security.*` / `ssl.*` / `sasl.*` subset to every other client it builds — the producer, the admin clients (changelog topic management, broker health check, consumer-group preflight), the restoration consumers, and the HA metadata-log clients:

```yaml
stoatflow:
  bootstrap-servers: broker-1:9093,broker-2:9093
  kafka:
    consumer:
      security.protocol: SASL_SSL
      sasl.mechanism: PLAIN
      sasl.jaas.config: ${KAFKA_SASL_JAAS_CONFIG}
      ssl.truststore.location: /etc/kafka/truststore.jks
      ssl.truststore.password: ${KAFKA_TRUSTSTORE_PASSWORD}
```

Keep secrets out of committed files — reference them with environment-variable placeholders (`${...}`) as shown.

If a particular client authenticates as a **different principal**, override it on that client's own map — `stoatflow.kafka.producer`, `stoatflow.kafka.admin` or `stoatflow.kafka.restoration-consumer` — which always wins over the inherited subset. Spell out **every** key of the family you are overriding: the override is per key, so anything you leave out is still inherited (see the warning above).

```yaml
stoatflow:
  kafka:
    consumer:
      security.protocol: SASL_SSL
      sasl.mechanism: PLAIN
      sasl.jaas.config: ${KAFKA_SASL_JAAS_CONFIG}
    admin:
      # only when the admin principal differs from the consumer's
      sasl.jaas.config: ${KAFKA_ADMIN_SASL_JAAS_CONFIG}
```

::callout{color="info" icon="i-lucide-info"}
**Porting from Kafka Streams?** A KS-keyed `Properties` normally spells these **unprefixed** (`security.protocol` at the top level, no `consumer.` prefix). That works: StoatFlow routes a bare client key to every client it is valid for, exactly as Kafka Streams does, with any prefixed spelling taking precedence. See the [KS compatibility matrix](https://stoatflow.io/docs/reference/ks-compatibility-matrix).
::

## Inspecting the resolved configuration

The merged application configuration is available at runtime from the `/config` HTTP endpoint, with sensitive values (passwords, secrets, tokens) masked. It serves JSON or YAML by content negotiation. See the [REST API](https://stoatflow.io/docs/runtime/rest-api) reference.

The resolved configuration is also logged at startup when the engine logger is at `DEBUG`:

```yaml
logging:
  level:
    io.stoatflow.core.runtime.StreamProcessingEngine: DEBUG
```


# The runtime

`stoatflow-runtime` is the batteries-included wrapper around the bare `stoatflow-core` engine. You hand it a topology and a YAML config; it loads the config, wires metrics into the engine, starts an HTTP admin server with health and Prometheus endpoints, bridges your license configuration, runs your plugins, and manages the whole lifecycle — including graceful shutdown on `SIGTERM`. This is the module most applications depend on.

::tldr-panel
- **Entry point:** `StoatFlowRuntime.fromConfig(topologyBuilder, configure)` — loads `application.yaml`, builds your topology, returns a runtime ready to `start()`.
- **Lifecycle:** `start()` boots everything and returns; `awaitTermination()` blocks the main thread until shutdown; `stop()` shuts down gracefully. A JVM shutdown hook calls `stop()` on `SIGTERM`.
- **What it adds over `:core`:** YAML config loading, an HTTP admin server (health, metrics, topology, debug), Micrometer/Prometheus metrics, license-config bridge, and a plugin + lifecycle-listener system.
::

## What the runtime adds over `:core`

`stoatflow-core` is the DSL and the processing engine — you can embed it directly with `StoatFlow.fromBuilder(config, builder)` and drive `start()` / `close()` yourself (the KS-shaped `close(Duration)` / `close(CloseOptions)` overloads are available too — see the [KS compatibility matrix](https://stoatflow.io/docs/reference/ks-compatibility-matrix#lifecycle)). The runtime wraps that engine and adds the production scaffolding you'd otherwise hand-build:

| Capability               | What it does                                                                                                                                                                                                                                                                      |
| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **YAML config loading**  | Reads `application.yaml` / `application.yml` from the classpath, applies overlay files and `STOATFLOW__`-prefixed environment overrides, and maps it all onto the engine's `StreamsConfig`. See [Runtime config](https://stoatflow.io/docs/configuration/runtime-config).         |
| **HTTP admin server**    | A JDK-built-in HTTP server on a virtual-thread executor, exposing health, metrics, topology, state, and debug endpoints. See [REST API](https://stoatflow.io/docs/runtime/rest-api).                                                                                              |
| **Metrics**              | Micrometer instrumentation with a Prometheus scrape endpoint and native Kafka-client metrics bound in. See [Metrics](https://stoatflow.io/docs/runtime/metrics).                                                                                                                  |
| **Health checks**        | Liveness and readiness indicators for the engine, the Kafka broker, Schema Registry (auto-enabled when configured), and the license. See [Health checks](https://stoatflow.io/docs/runtime/health-checks).                                                                        |
| **License bridge**       | Translates `stoatflow.license.*` YAML into the system properties the engine reads, before the engine resolves the license key at start (validation itself is deferred \~5 minutes). See [License configuration](https://stoatflow.io/docs/getting-started/license-configuration). |
| **Plugin system**        | Register `RuntimePlugin`s and lifecycle listeners to extend the runtime — custom endpoints, metrics, health indicators, and startup/shutdown hooks. See [Plugins](https://stoatflow.io/docs/runtime/plugins).                                                                     |
| **Lifecycle management** | Coordinated startup ordering, a `SIGTERM` shutdown hook, and a watcher that shuts the runtime down if the engine terminates on its own.                                                                                                                                           |

If you only need the DSL and engine with no scaffolding, depend on `stoatflow-core` directly. Everything below assumes `stoatflow-runtime`.

## Creating a runtime from YAML

`StoatFlowRuntime.fromConfig(...)` is the primary entry point. It loads configuration from `application.yaml` on the classpath and applies it before building your topology. You pass two things:

- **`topologyBuilder`** — a function that receives a `StreamsBuilder` and defines your topology.
- **`configure`** — an optional block for everything that can't live in YAML: serdes, exception handlers, and other non-serializable settings, supplied via `streamsConfigOverrides { ... }`.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.topology.StreamsBuilder
import io.stoatflow.runtime.StoatFlowRuntime
import org.apache.kafka.common.serialization.Serdes

fun main() {
    val runtime = StoatFlowRuntime.fromConfig(
        topologyBuilder = { builder -> buildTopology(builder) },
        configure = {
            streamsConfigOverrides {
                defaultKeySerde(Serdes.String())
                defaultValueSerde(Serdes.String())
            }
        },
    )

    runtime.start()             // boots engine + HTTP + metrics, then returns
    runtime.awaitTermination()  // block until shutdown
}

private fun buildTopology(builder: StreamsBuilder) {
    builder.stream<String, String>("input")
        .mapValues { it.uppercase() }
        .to("output")
}
```

```java [Java]
import io.stoatflow.core.topology.StreamsBuilder;
import io.stoatflow.runtime.StoatFlowRuntime;
import org.apache.kafka.common.serialization.Serdes;

public class Main {
    public static void main(String[] args) {
        var runtime = StoatFlowRuntime.fromConfig(
            Main::buildTopology,
            builder -> builder.streamsConfigOverrides(cfg -> {
                cfg.defaultKeySerde(Serdes.String());
                cfg.defaultValueSerde(Serdes.String());
            })
        );

        runtime.start();             // boots engine + HTTP + metrics, then returns
        runtime.awaitTermination();  // block until shutdown
    }

    private static void buildTopology(StreamsBuilder builder) {
        builder.<String, String>stream("input")
            .mapValues(v -> v.toUpperCase())
            .to("output");
    }
}
```
::

The Java overloads accept `Consumer<StreamsBuilder>` / `Consumer<StoatFlowRuntime.Builder>` so void-returning lambdas work without the awkward `return Unit.INSTANCE`.

### Supplying a pre-loaded config

There's a second `fromConfig` overload that takes an already-loaded `ApplicationConfig`. Use it when you need to load configuration from a custom source, or to inspect or modify it before the runtime is created:

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.runtime.config.ConfigLoader

val appConfig = ConfigLoader.loadOrThrow()
// inspect or adjust appConfig here

val runtime = StoatFlowRuntime.fromConfig(
    appConfig = appConfig,
    topologyBuilder = { builder -> buildTopology(builder) },
    configure = {
        streamsConfigOverrides {
            defaultKeySerde(Serdes.String())
            defaultValueSerde(Serdes.String())
        }
    },
)
```

```java [Java]
import io.stoatflow.runtime.config.ConfigLoader;

var appConfig = ConfigLoader.INSTANCE.loadOrThrow();
// inspect or adjust appConfig here

var runtime = StoatFlowRuntime.fromConfig(
    appConfig,
    Main::buildTopology,
    builder -> builder.streamsConfigOverrides(cfg -> {
        cfg.defaultKeySerde(Serdes.String());
        cfg.defaultValueSerde(Serdes.String());
    })
);
```
::

## Creating a runtime with the builder

`fromConfig` is the recommended path because it pulls everything from YAML. If you'd rather configure the runtime entirely in code — no `application.yaml` — use the builder. You're then responsible for constructing the engine's `StreamsConfig` yourself, and for any HTTP / metrics config you want to override from defaults.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.config.StreamsConfig
import io.stoatflow.runtime.StoatFlowRuntime
import io.stoatflow.runtime.http.HttpServerConfig

val runtime = StoatFlowRuntime.builder()
    .streamsConfig(
        StreamsConfig(
            applicationId = "my-app",
            bootstrapServers = "localhost:9092",
        ),
    )
    .httpConfig(HttpServerConfig(port = 8080))
    .topology { builder ->
        builder.stream<String, String>("input")
            .mapValues { it.uppercase() }
            .to("output")
    }
    .build()

runtime.start()
runtime.awaitTermination()
```

```java [Java]
import io.stoatflow.core.config.StreamsConfig;
import io.stoatflow.runtime.StoatFlowRuntime;
import io.stoatflow.runtime.http.HttpServerConfig;

var runtime = StoatFlowRuntime.Companion.builder()
    .streamsConfig(
        StreamsConfig.builder("my-app", "localhost:9092").build())
    .httpConfig(new HttpServerConfig(8080, "0.0.0.0", 50, true, true))
    .topology(builder ->
        builder.<String, String>stream("input")
            .mapValues(v -> v.toUpperCase())
            .to("output"))
    .build();

runtime.start();
runtime.awaitTermination();
```
::

Metrics aren't configured here — when you omit `metricsConfig(...)`, the builder enables metrics by default and derives the application ID from the `StreamsConfig`. The builder also exposes `metricsConfig(...)`, `addPlugin(...)`, `addLifecycleListener(...)`, `addHealthIndicator(name, indicator)`, and config setters for `infoConfig`, `configConfig`, and `healthConfig`. The `streamsConfigOverrides { ... }` block is only used with `fromConfig` — when you pass a `StreamsConfig` directly, set everything on that object instead.

::callout{color="info" icon="i-lucide-lightbulb"}
`fromConfig` and the builder produce the same `StoatFlowRuntime`. The difference is where configuration comes from: `fromConfig` reads YAML and lets `streamsConfigOverrides` supply the non-serializable bits; the builder takes a fully-formed `StreamsConfig` in code. Most applications use `fromConfig`.
::

## Lifecycle

The runtime moves through a small set of states — `CREATED` → `STARTING` → `RUNNING` → `STOPPING` → `STOPPED` (or `ERROR`) — and exposes three lifecycle methods. Query the current state any time with `state()`.

### start()

`start()` boots the runtime and **returns** (it does not block). Internally it initialises metrics, builds and starts the engine in parallel with HTTP-server setup, starts your plugins, registers the JVM shutdown hook, and transitions to `RUNNING`. Health endpoints come up early — while the engine is still restoring state — so orchestrator probes get answers during a slow cold start.

`start()` can only be called from the `CREATED` state. If engine startup fails, the runtime cleans up everything it had brought up, transitions to `ERROR`, and rethrows.

### awaitTermination()

`awaitTermination()` blocks the calling thread until the runtime stops. This is what keeps `main` alive while the engine processes records. If the runtime terminated because of an error, this method rethrows that error — so a non-zero exit on failure falls out naturally:

::code-tabs{group="lang"}
```kotlin [Kotlin]
runtime.start()
runtime.awaitTermination()  // returns on clean stop; throws on error
```

```java [Java]
runtime.start();
runtime.awaitTermination();  // returns on clean stop; throws on error
```
::

### stop()

`stop()` shuts the runtime down gracefully: it closes the engine (which drains in-flight records and commits a final barrier), shuts plugins down in reverse registration order, stops the HTTP server, closes the metrics registry, and transitions to `STOPPED`. You rarely call `stop()` yourself — the registered JVM shutdown hook calls it on `SIGTERM` (the signal Kubernetes sends on pod termination), and a watcher thread calls it if the engine terminates on its own. It's idempotent and safe to call from any state.

### Lifecycle events

Each transition emits a [lifecycle event](https://stoatflow.io/docs/runtime/plugins) — `PRE_START`, `POST_START`, `PRE_STOP`, `POST_STOP` — to any registered `RuntimeLifecycleListener`. Use these to start or stop dependent services alongside the runtime.

## Pause and unpause

Processing can be paused and resumed at runtime through the HTTP admin endpoints — `POST /pause` and `POST /unpause`. Pausing stops the runtime consuming new records while keeping the process alive and its endpoints responsive; unpausing resumes from where it left off. This is operated over HTTP rather than as a code method, so you can drive it from an operator, a script, or a runbook without redeploying. See [Pause / unpause](https://stoatflow.io/docs/runtime/pause-unpause).

## The HTTP admin surface

When `runtime.http.enabled` is `true` (the default), `start()` brings up an HTTP server and registers the operational endpoints. The full catalogue is in the [REST API reference](https://stoatflow.io/docs/reference/rest-api-reference); the highlights:

- **Health** — `/health/live` and `/health/ready` for orchestrator probes. See [Health checks](https://stoatflow.io/docs/runtime/health-checks).
- **Metrics** — `/metrics` for Prometheus scraping. See [Metrics](https://stoatflow.io/docs/runtime/metrics).
- **Introspection** — `/info`, `/config` (masked), `/topology`, `/topology/ks`, `/topology/compiled`, `/state`, `/watermarks`, `/offsets`, `/consumer`, and `/license`.
- **Control** — `/pause`, `/unpause`; plus the `/ha/*` status and role-switch endpoints when [hot standby](https://stoatflow.io/docs/operating/high-availability) is enabled.
- **Debug** — `/debug/threads` and `/debug/barriers` for live diagnostics when a topology misbehaves; gated by `runtime.http.debug.enabled`.

`start()` returns a handle on the same runtime, so you can chain into it. A couple of accessors are handy from code: `httpPort()` returns the bound port (useful when you let the server pick one), and `meterRegistry()` returns the Micrometer registry for custom instrumentation.

## Extending the runtime

The runtime is designed to be extended without modifying it:

- **Plugins** — implement `RuntimePlugin` to register custom HTTP handlers, metrics, or health indicators during startup, and clean them up on shutdown. Register with `addPlugin(...)` on the builder, or have `fromConfig`'s `configure` block add them.
- **Lifecycle listeners** — implement `RuntimeLifecycleListener` (a functional interface) to hook the four lifecycle events.
- **Custom health indicators** — implement `HealthIndicator` and register via `addHealthIndicator(name, indicator)` or `PluginContext.healthIndicatorRegistry()`.

The plugin contract, the `PluginContext` it receives, and worked examples are in [Plugins](https://stoatflow.io/docs/runtime/plugins).

## Where to go next

- **[Runtime config](https://stoatflow.io/docs/configuration/runtime-config)** — every `stoatflow.*` and `runtime.*` YAML key.
- **[REST API](https://stoatflow.io/docs/runtime/rest-api)** — the full admin-endpoint catalogue.
- **[Health checks](https://stoatflow.io/docs/runtime/health-checks)** — liveness vs readiness, and the built-in indicators.
- **[Metrics](https://stoatflow.io/docs/runtime/metrics)** — the Prometheus surface and what to scrape.
- **[Plugins](https://stoatflow.io/docs/runtime/plugins)** — extend the runtime with custom endpoints, metrics, and lifecycle hooks.
- **[Docker](https://stoatflow.io/docs/runtime/docker)*&#x2A; and &#x2A;*[Native image](https://stoatflow.io/docs/runtime/native-image)** — package the runtime for deployment.


# The REST API

The `:runtime` module starts a small HTTP server alongside your topology. It serves operational endpoints — metadata, topology views, live state and progress, health probes, control actions, license state, Prometheus metrics, and freeze diagnostics. Everything below is what an operator reaches for from a shell, a dashboard, or a Kubernetes probe.

::tldr-panel
- **Server:** JDK built-in HTTP server on `0.0.0.0:8080` by default — enable and configure it under `runtime.http`.
- **Read endpoints:** `GET` — JSON (and a few negotiate text/YAML via the `Accept` header).
- **Control endpoints:** `POST /pause` and `POST /unpause`.
- **Probes:** `GET /health/live` and `GET /health/ready` — `200` UP, `503` DOWN.
- **Reference:** every field and response shape lives in the [REST API reference](https://stoatflow.io/docs/reference/rest-api-reference).
::

## Enabling the server

The HTTP server is configured under `runtime.http`. It's enabled by default; metrics live under `runtime.metrics`:

```yaml
runtime:
  http:
    enabled: true
    port: 8080          # default
    host: 0.0.0.0       # default — all interfaces
    debug:
      enabled: true     # default — /debug/* endpoints
  metrics:
    enabled: true
```

A quick liveness check from the shell:

```bash
curl -s localhost:8080/health/ready
```

::callout{color="warning" icon="i-lucide-shield"}
Several endpoints expose internals or customer-identifying data — `/license` (license ID, customer name, machine counts), `/config`, `/info`, the `/topology/*` views, and `/debug/*`. Expose only `/health/live` and `/health/ready` through internet-facing ingress; keep the rest on the in-cluster `Service`. See [Kubernetes](https://stoatflow.io/docs/operating/kubernetes) and [Probes](https://stoatflow.io/docs/operating/probes).
::

All read endpoints are `GET`. Calling one with the wrong method returns `405`. Before the engine has finished starting, endpoints that depend on it return `503` with a short JSON error.

## Metadata

Use these to confirm what's deployed and how it's configured.

| Endpoint  | Method | Purpose                                                                                          |
| --------- | ------ | ------------------------------------------------------------------------------------------------ |
| `/info`   | GET    | Application ID, JVM and Kafka info, uptime, current state, and the list of registered endpoints. |
| `/config` | GET    | The merged effective configuration, with secrets masked.                                         |

`/info` is the fastest way to see what's running and which endpoints this build exposes — the `endpoints` array is generated from the actual handler registration:

```bash
curl -s localhost:8080/info
```

`/config` content-negotiates: it returns YAML by default (more readable), or JSON if you ask for it. Passwords, secrets, and tokens are masked in the output.

```bash
# YAML (default)
curl -s localhost:8080/config

# JSON
curl -s -H 'Accept: application/json' localhost:8080/config
```

## Topology

Three views of the same topology, for different audiences.

| Endpoint             | Method | Purpose                                                                                            |
| -------------------- | ------ | -------------------------------------------------------------------------------------------------- |
| `/topology`          | GET    | StoatFlow-native view with rich metadata (scheduled sources, store names, sub-topology structure). |
| `/topology/ks`       | GET    | Kafka Streams-compatible view, matching `Topology#describe()` output.                              |
| `/topology/compiled` | GET    | The compiled, runtime-internal representation produced by the topology compiler.                   |

`/topology` and `/topology/ks` return a human-readable text description by default, or JSON when you request it:

```bash
# Text (default)
curl -s localhost:8080/topology

# JSON
curl -s -H 'Accept: application/json' localhost:8080/topology

# Kafka Streams-compatible description
curl -s localhost:8080/topology/ks
```

`/topology/compiled` reflects how the engine actually compiled and laid out your topology, which can differ from the logical DSL view — most visibly in where sub-topology boundaries fall (StoatFlow opens one where a re-keyed record reaches an operator that needs the new key's lane affinity, [not at every key change](https://stoatflow.io/docs/concepts/lanes-and-parallelism#where-the-handoff-actually-happens)). It's primarily a diagnostic view; it returns `503` until the application has started.

```bash
curl -s -H 'Accept: application/json' localhost:8080/topology/compiled
```

## State and progress

These show whether the application is making progress and where it's at — lifecycle state, consumer assignment, offset lag, and event-time watermarks.

| Endpoint      | Method | Purpose                                                                                                      |
| ------------- | ------ | ------------------------------------------------------------------------------------------------------------ |
| `/state`      | GET    | Current lifecycle state, when it was entered, and the recent state-transition history.                       |
| `/consumer`   | GET    | Consumer group metadata, source subscription, last commit time, and per-partition buffer fill + pause state. |
| `/offsets`    | GET    | Per-partition committed offsets and lag for source topics, plus committed changelog offsets per state store. |
| `/watermarks` | GET    | Global and per-partition event-time watermarks, idle flags, and last event times.                            |

`/state` is the lifecycle picture — is the app `RUNNING`, `DRAINING`, `PAUSED`, restoring, shutting down?

```bash
curl -s localhost:8080/state
```

`/offsets` answers "are we falling behind?" — it reports per-partition lag and a `totalLag` summary, derived from locally cached values (no Kafka RPCs on the request path):

```bash
curl -s localhost:8080/offsets
```

`/consumer` adds the consumer-group angle (member ID, generation, subscription) and shows per-partition buffer utilisation, so you can see whether any partition is paused under backpressure. `/watermarks` exposes event-time progress — useful when windowed or timer-driven logic seems to be waiting.

```bash
curl -s localhost:8080/consumer
curl -s localhost:8080/watermarks
```

See [Event time and watermarks](https://stoatflow.io/docs/concepts/event-time-and-watermarks) for how watermarks advance, and [Observability](https://stoatflow.io/docs/operating/observability) for turning these into dashboards.

## Health probes

Kubernetes-style liveness and readiness, with separate indicator sets.

| Endpoint        | Method | Purpose                                                  |
| --------------- | ------ | -------------------------------------------------------- |
| `/health/live`  | GET    | Liveness — is the process healthy enough to keep alive?  |
| `/health/ready` | GET    | Readiness — is the app ready to receive/process traffic? |

Both return `200` with `{"status":"UP", ...}` when all relevant indicators pass, or `503` with `status` `DOWN` otherwise. Liveness stays UP through transient states (startup, shutdown, and license issues a restart can't fix) so orchestrators don't needlessly restart the process; readiness goes DOWN during startup, state restoration, and shutdown.

```bash
curl -i localhost:8080/health/live
curl -i localhost:8080/health/ready
```

The response lists each component indicator (engine state, Kafka broker connectivity, Schema Registry if configured, license state) with its own `status` and `details`. See [Health checks](https://stoatflow.io/docs/runtime/health-checks) for the indicators and [Probes](https://stoatflow.io/docs/operating/probes) for wiring them to Kubernetes.

## Control

Pause and resume processing at runtime — for maintenance, draining, or controlled investigation.

| Endpoint   | Method | Purpose                                                          |
| ---------- | ------ | ---------------------------------------------------------------- |
| `/pause`   | POST   | Pause processing — drains in-flight work, then reaches `PAUSED`. |
| `/unpause` | POST   | Resume processing — returns to `RUNNING`.                        |

```bash
curl -s -X POST localhost:8080/pause
# -> {"status":"pausing","state":"DRAINING"}

curl -s -X POST localhost:8080/unpause
# -> {"status":"resumed","state":"RUNNING"}
```

Both are `POST`. A request that can't be honoured from the current state returns `400` with an explanatory error; if the engine isn't initialised yet you get `503`. See [Pause and unpause](https://stoatflow.io/docs/runtime/pause-unpause) for the state semantics and what "draining" guarantees.

## High availability

Registered only when [hot standby](https://stoatflow.io/docs/operating/high-availability) is enabled (`ha.mode != off`); otherwise these paths return `404`.

| Endpoint      | Method | Purpose                                                                                                                |
| ------------- | ------ | ---------------------------------------------------------------------------------------------------------------------- |
| `/ha/status`  | GET    | This pod's view of the cluster: role, replication lag, ready-standby count, promotion-token epoch, and observed peers. |
| `/ha/switch`  | POST   | Swap roles — the active drains and the peer promotes.                                                                  |
| `/ha/promote` | POST   | Promote a specific standby (`?pod=<id>`).                                                                              |
| `/ha/demote`  | POST   | Demote a specific active (`?pod=<id>`).                                                                                |

```bash
curl -s localhost:8080/ha/status
curl -s -X POST localhost:8080/ha/switch
# -> {"command":"SWITCH","target":null,"commandOffset":42}
```

The write endpoints publish a command and return `202 Accepted` with the command's offset as an idempotency token; the targeted pod observes it and acts. If no caught-up (`READY_STANDBY`) target exists, they reject with `409 Conflict` — handing off to a not-ready peer is a self-inflicted outage; `?force=true` overrides the gate (a forced promotion still fully restores state before processing). See [High availability](https://stoatflow.io/docs/operating/high-availability) for the operational guide and [REST API reference](https://stoatflow.io/docs/reference/rest-api-reference) for full response shapes.

## License

| Endpoint   | Method | Purpose                                   |
| ---------- | ------ | ----------------------------------------- |
| `/license` | GET    | Current runtime license-validation state. |

`/license` reports validation `status` (`PENDING` during the first \~5 minutes after start — the deferred-validation window — then `VALID` / `GRACE_PERIOD`, or `EXPIRED` / `REVOKED` / `INVALID` / `NO_KEY`), tier, expiry and days remaining, machine usage against the allowance, and heartbeat/grace state.

```bash
curl -s localhost:8080/license
```

::callout{color="warning" icon="i-lucide-shield"}
The response contains customer-identifying fields (`licenseId`, `customerName`, machine counts). Treat it as in-cluster only — never expose `/license` through internet ingress.
::

The same state also surfaces through the license health indicator (readiness goes DOWN on expired/revoked/invalid licenses) and as metrics. See [License configuration](https://stoatflow.io/docs/getting-started/license-configuration).

## Metrics

| Endpoint   | Method | Purpose                                                           |
| ---------- | ------ | ----------------------------------------------------------------- |
| `/metrics` | GET    | Prometheus text-exposition scrape of JVM and application metrics. |

```bash
curl -s localhost:8080/metrics
```

The endpoint serves the Prometheus text format (`text/plain; version=0.0.4`). It returns `503` if metrics are disabled (`runtime.metrics.enabled: false`). The full meter catalogue — commit, changelog, restoration, dispatch, and license meters — is in [Metrics](https://stoatflow.io/docs/runtime/metrics).

## Debug

Two pull-only diagnostic snapshots for investigating a stalled or frozen commit pipeline.

| Endpoint          | Method | Purpose                                                                                                          |
| ----------------- | ------ | ---------------------------------------------------------------------------------------------------------------- |
| `/debug/threads`  | GET    | Snapshot of engine-owned and platform thread state and stacks, for freeze diagnosis.                             |
| `/debug/barriers` | GET    | Commit-pipeline snapshot: in-flight commit progress, which lanes are outstanding, and a derived `phase` summary. |

Both are gated by `runtime.http.debug.enabled` (default `true`). When disabled, they return `404`. They are snapshots of state already kept for other purposes, so there's no cost when nobody is calling them.

```bash
# Thread snapshot (optional: depth, filter=all|stuck|known)
curl -s 'localhost:8080/debug/threads?filter=stuck'

# Commit-pipeline snapshot — see the derived phase
curl -s localhost:8080/debug/barriers
```

`/debug/threads` accepts optional query params: `depth` (max stack frames per thread, default 40) and `filter` (`all` default, `stuck` for non-runnable threads, `known` for engine threads only). `/debug/barriers` summarises the commit pipeline into a single `phase` string so you can tell at a glance whether a commit is waiting on lanes, queued, in transaction, or idle.

::callout{color="info" icon="i-lucide-info"}
The debug endpoints expose internal engine state and are meant for diagnosis, not routine monitoring. Leave them on (they're free at rest), but keep them off public ingress. For hardened deployments that limit attack surface, set `runtime.http.debug.enabled: false`.
::

## Full reference

This page is a tour by use case. For the exhaustive list — every response field, status code, and content-negotiation rule — see the &#x2A;*[REST API reference](https://stoatflow.io/docs/reference/rest-api-reference)**.


# Health checks

The `:runtime` module exposes two HTTP health endpoints — `/health/live` and `/health/ready` — backed by a registry of health indicators. Each indicator answers two independent questions: &#x2A;is the process alive?* (liveness) and &#x2A;is the process ready to serve traffic?* (readiness). This page covers the built-in indicators, exactly what flips each one, and how the probes behave while state is restoring.

::tldr-panel
- **`/health/live`** → keep the process running. Returns `503` only on a fatal condition where a restart is the right remedy.
- **`/health/ready`** → route traffic / count as "in service". Returns `503` during startup, **restoration**, pause, and shutdown.
- **Built-in indicators:** app state, Kafka broker, Schema Registry (auto-enabled when configured), and license. The aggregate is UP only if **every** indicator is UP.
- **k8s wiring:** map `livenessProbe` → `/health/live` and `readinessProbe` → `/health/ready`. See [Probes](https://stoatflow.io/docs/operating/probes).
::

## Liveness vs readiness

The two probes serve different orchestrator decisions, so they have different failure semantics:

| Probe         | Endpoint        | Orchestrator action on `503`                     | Use it to answer                                        |
| ------------- | --------------- | ------------------------------------------------ | ------------------------------------------------------- |
| **Liveness**  | `/health/live`  | **Restart** the pod                              | "Is this process wedged in a way only a restart fixes?" |
| **Readiness** | `/health/ready` | **Stop routing traffic** to the pod (no restart) | "Can this process serve / process right now?"           |

The split matters because **a restart does not fix every problem**. A pod that is still starting up, restoring its state stores, paused, or whose license has expired should *not* be restarted — restarting either wastes the work in progress or loops forever without resolving the issue. Those conditions flip readiness DOWN (stop sending it traffic) while keeping liveness UP (leave it running).

Both endpoints return the same JSON shape — an overall status plus one entry per indicator:

```json
{
  "status": "UP",
  "components": [
    { "name": "stoatflow", "status": "UP", "details": { "state": "RUNNING" } },
    { "name": "license", "status": "UP", "details": { "status": "VALID", "tier": "production", "daysRemaining": 281 } },
    { "name": "kafka-broker", "status": "UP", "details": { "broker": "reachable" } }
  ]
}
```

(In the first \~5 minutes after start the `license` component reports `"status": "PENDING"` — the deferred-validation window — still `UP`.)

HTTP status follows the overall status: `200` when `UP`, `503` when `DOWN`. The format is compatible with Spring Boot Actuator, so existing dashboards and probe tooling that parse Actuator health work unchanged.

::callout{color="info" icon="i-lucide-info"}
The aggregate is `UP` **only if every registered indicator is `UP`**. One DOWN indicator flips the whole endpoint to `503`. The `components` array always lists every indicator with its individual status, so you can see *which* one failed.
::

## The built-in indicators

`StoatFlowRuntime` registers four indicators automatically at startup (a fifth, `ha`, when [hot standby](https://stoatflow.io/docs/operating/high-availability) is enabled). Each is registered for **both** liveness and readiness, but most return *different* verdicts for the two checks — that is the whole point of the split.

| Indicator       | `name`            | What it checks                                                                 |
| --------------- | ----------------- | ------------------------------------------------------------------------------ |
| App state       | `stoatflow`       | The engine's lifecycle state (`RUNNING`, `RESTORING`, `PAUSED`, …)             |
| Kafka broker    | `kafka-broker`    | Broker connectivity via an `AdminClient` call                                  |
| Schema Registry | `schema-registry` | HTTP reachability — **only registered if a Schema Registry URL is configured** |
| License         | `license`         | Runtime license validity                                                       |
| Hot standby     | `ha`              | HA role and standby catch-up — &#x2A;*only registered when `ha.mode != off`**  |

### App state (`stoatflow`)

The canonical indicator — it reads the engine's lifecycle state and maps it to liveness and readiness:

| Engine state       | Liveness | Readiness | Why                               |
| ------------------ | -------- | --------- | --------------------------------- |
| `CREATED`          | DOWN     | DOWN      | Not started yet                   |
| `STARTING`         | UP       | DOWN      | Alive, not yet serving            |
| `VALIDATING_STATE` | UP       | DOWN      | Startup in progress               |
| `RESTORING`        | UP       | DOWN      | Startup in progress — see below   |
| `RUNNING`          | UP\*     | UP        | Ready to process                  |
| `DRAINING`         | UP       | DOWN      | Pausing, draining in-flight work  |
| `PAUSED`           | UP       | DOWN      | Paused, not accepting new traffic |
| `STOPPING`         | UP       | DOWN      | Shutting down, no new traffic     |
| `STOPPED`          | DOWN     | DOWN      | Terminated                        |
| `ERROR`            | DOWN     | DOWN      | Fatal error                       |

Readiness is `UP` in exactly one state — `RUNNING`. Every transient lifecycle state (starting, restoring, draining, pausing, stopping) is readiness DOWN but liveness UP, so the orchestrator routes traffic away without killing a process that is doing legitimate work.

\*The asterisk on `RUNNING` liveness: if the engine's commit pipeline stalls past a configured threshold, liveness flips DOWN so the orchestrator can restart a wedged process. This is an independent safety net layered on top of the in-process supervision the engine already runs — the probe exists so Kubernetes has its own recovery path even if internal recovery is itself stuck. The threshold is the engine's `commitStallThresholdMs` field (default 45 000 ms; set `0` to disable the check), set in YAML via `stoatflow.commit-stall.threshold-ms`. When it fires, the liveness response carries `stall_age_ms`, `stall_threshold_ms`, and related detail fields so the cause is visible in the probe payload.

### Kafka broker (`kafka-broker`)

Verifies broker connectivity by listing topics through a dedicated `AdminClient`. A StoatFlow app **cannot make progress without its broker**, so this indicator returns the *same* verdict for both probes: unreachable broker → DOWN for liveness *and* readiness. Loss of broker connectivity is treated as fatal — a restart is a reasonable response.

The check is bounded by a timeout (`runtime.health.kafka-broker.timeout-ms`, default 2 000 ms). On success the detail is `{ "broker": "reachable" }`; on failure it is `{ "broker": "unreachable", "error": "…" }`.

### Schema Registry (`schema-registry`)

**Self-enabling** — this indicator is registered only when a Schema Registry URL is configured (`stoatflow.schema-registry-url`). If you don't use Schema Registry, it never appears in the `components` array. When present, it does an HTTP `GET /subjects` against the registry; like the broker check it returns the same verdict for liveness and readiness (unreachable → DOWN for both). Its timeout is `runtime.health.schema-registry.timeout-ms` (default 5 000 ms).

### License (`license`)

The license indicator is the clearest example of the liveness/readiness split:

| License status | Liveness | Readiness |
| -------------- | -------- | --------- |
| `VALID`        | UP       | UP        |
| `GRACE_PERIOD` | UP       | UP        |
| `PENDING`      | UP       | UP        |
| `EXPIRED`      | UP       | **DOWN**  |
| `REVOKED`      | UP       | **DOWN**  |
| `INVALID`      | UP       | **DOWN**  |
| `NO_KEY`       | UP       | **DOWN**  |

Readiness goes DOWN the moment the license is no longer operational — `EXPIRED`, `REVOKED`, `INVALID`, or `NO_KEY` — so orchestrators stop routing traffic to a pod whose license has been pulled. `VALID` and `GRACE_PERIOD` are both treated as operational (the grace period exists precisely so a transient validation outage doesn't take you down). `PENDING` is the deferred-validation window — the first \~5 minutes after start, before the activating validation has run — and is treated as ready so a fresh pod takes traffic immediately.

Liveness stays **UP regardless of license status**. Restarting a pod will not fix an expired or revoked license — it would just produce a restart loop that hides the real problem. The readiness payload carries the `status` and `tier`; the operational (UP) readiness payload additionally carries `daysRemaining`.

Within the renewal-warning window, the **liveness** payload adds a `licenseRenewalWarning` detail (e.g. `"expires in 9 days"`) so the nudge is visible on the liveness response too — even though liveness itself stays UP.

::callout{color="info" icon="i-lucide-shield"}
The license **state** also drives a readiness check at the cluster level and is mirrored on the dedicated `/license` HTTP endpoint and license metrics. See the [REST API](https://stoatflow.io/docs/runtime/rest-api) for `/license`, and [Metrics](https://stoatflow.io/docs/runtime/metrics) for the license gauges and counters.
::

### Hot standby (`ha`)

Registered only when [hot standby](https://stoatflow.io/docs/operating/high-availability) is enabled (`ha.mode != off`). It encodes the readiness contract that makes the rolling update order-independent:

- **Readiness** is UP when the pod is the active **and** processing, or a standby **and** caught up. A still-catching-up standby is *not* ready, so Kubernetes won't advance the roll until the restarted pod is a caught-up standby. With more than one standby it also enforces the redundancy floor — a caught-up standby is ready to roll only if rolling it leaves the active plus at least one other caught-up standby (`ha.desired-standbys`).
- **Liveness** stays UP while a standby is catching up (so it isn't killed mid-catch-up); it goes DOWN only when the pod is genuinely down or a restart is required (for example, after a source topic's partition count changes).

The payload carries `role` and `ha_state` details; readiness also reports `replication_lag_records`. See [High availability](https://stoatflow.io/docs/operating/high-availability) for the full operational picture.

## How the probes behave during restoration

State restoration is the case the readiness/liveness split is designed for. On startup — and again after a crash recovery — StoatFlow rebuilds its state stores from their changelog topics before it begins processing. While that runs, the engine is in `RESTORING` (preceded by `STARTING` / `VALIDATING_STATE`):

- **Readiness is DOWN** for the entire restoration. The app is alive but not yet processing, so the orchestrator must not count it as in service or route admin traffic to it as "ready".
- **Liveness is UP** throughout. Restoration is expected, legitimate work — restarting mid-restore would throw away progress and start over, potentially never converging for a large state store.

This is why you map the **readiness** probe to load-balancer membership and the **liveness** probe to restart decisions. A correctly-wired deployment lets a restoring pod take all the time it needs (within the readiness probe's `failureThreshold × periodSeconds` budget) without being killed. Size the readiness probe's startup tolerance to your largest expected restore — see [Probes](https://stoatflow.io/docs/operating/probes) for the `startupProbe` / `initialDelaySeconds` guidance.

```bash
# During restoration: ready is 503, live is 200.
curl -s -o /dev/null -w "ready=%{http_code}\n" localhost:8080/health/ready   # ready=503
curl -s -o /dev/null -w "live=%{http_code}\n"  localhost:8080/health/live    # live=200

# The readiness body shows why:
curl -s localhost:8080/health/ready | jq '.components[] | select(.name=="stoatflow")'
# { "name": "stoatflow", "status": "DOWN", "details": { "state": "RESTORING" } }
```

The same pattern applies to **pause/unpause**: a paused engine (`PAUSED` / `DRAINING`) is readiness DOWN, liveness UP — traffic is steered away but the process is left running so you can `/unpause` it. See [Pause and unpause](https://stoatflow.io/docs/runtime/pause-unpause).

## The indicator registry

Indicators are held in a registry that keeps **separate collections for liveness and readiness**. Registering an indicator with `register(name, indicator)` adds it to both; the registry also supports liveness-only and readiness-only registration. `/health/live` evaluates every liveness indicator's `livenessHealth()`; `/health/ready` evaluates every readiness indicator's `health()`. The overall status is `UP` only when all indicators in that collection report `UP`.

### Adding your own indicator

A health indicator implements one interface — `health()` for readiness and an optional `livenessHealth()` (defaults to the same verdict as `health()`):

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.runtime.health.Health
import io.stoatflow.runtime.health.HealthIndicator

class DownstreamApiHealthIndicator(
    private val client: DownstreamClient,
) : HealthIndicator {
    // Readiness: stop routing traffic if the dependency is down.
    override fun health(): Health =
        if (client.ping()) {
            Health.up().withDetail("downstream", "reachable").build()
        } else {
            Health.down().withDetail("downstream", "unreachable").build()
        }

    // Liveness: a downstream outage should NOT restart us — leave the default,
    // or override to return UP explicitly.
    override fun livenessHealth(): Health =
        Health.up().withDetail("downstream", "checked-on-readiness-only").build()
}
```

```java [Java]
import io.stoatflow.runtime.health.Health;
import io.stoatflow.runtime.health.HealthIndicator;

public class DownstreamApiHealthIndicator implements HealthIndicator {

    private final DownstreamClient client;

    public DownstreamApiHealthIndicator(DownstreamClient client) {
        this.client = client;
    }

    // Readiness: stop routing traffic if the dependency is down.
    @Override
    public Health health() {
        return client.ping()
            ? Health.up().withDetail("downstream", "reachable").build()
            : Health.down().withDetail("downstream", "unreachable").build();
    }

    // Liveness: a downstream outage should NOT restart us.
    @Override
    public Health livenessHealth() {
        return Health.up().withDetail("downstream", "checked-on-readiness-only").build();
    }
}
```
::

Register it on the runtime builder with `addHealthIndicator(name, indicator)`, which puts it in both the liveness and readiness collections alongside the built-ins:

::code-tabs{group="lang"}
```kotlin [Kotlin]
val runtime = StoatFlowRuntime.fromConfig(
    topologyBuilder = { buildTopology(it) },
    configure = {
        addHealthIndicator("downstream-api", DownstreamApiHealthIndicator(client))
    },
)
```

```java [Java]
var runtime = StoatFlowRuntime.fromConfig(
    Main::buildTopology,
    builder -> builder.addHealthIndicator("downstream-api", new DownstreamApiHealthIndicator(client))
);
```
::

::callout{color="info" icon="i-lucide-plug"}
[Plugins](https://stoatflow.io/docs/runtime/plugins) can register health indicators too — `PluginContext.healthIndicatorRegistry()` gives a plugin the same registry, including the liveness-only / readiness-only registration methods for finer control than `addHealthIndicator`.
::

## Configuration

The health checks themselves need no configuration to work — the four built-in indicators are wired automatically. The only tunables are the connectivity-check timeouts and the broader HTTP-server enablement:

```yaml
stoatflow:
  # Schema Registry health auto-enables when this is set:
  schema-registry-url: ${SCHEMA_REGISTRY_URL:-}

runtime:
  http:
    enabled: true              # /health/live and /health/ready live on the runtime HTTP server
    port: ${HTTP_PORT:-8080}
  health:
    kafka-broker:
      timeout-ms: 2000         # broker connectivity check timeout (default 2000)
    schema-registry:
      timeout-ms: 5000         # schema-registry check timeout (default 5000)
```

::callout{color="warning" icon="i-lucide-info"}
The commit-stall liveness check is governed by the **engine** setting `stoatflow.commit-stall.threshold-ms` (default 45 000 ms; set `0` to disable the check), not a `runtime.health.*` key. It must be less than the barrier timeout. See the [Configuration reference](https://stoatflow.io/docs/reference/configuration-reference) for the full key list.
::

## Quick reference

```bash
# Liveness — should the orchestrator restart this pod?
curl -s localhost:8080/health/live  | jq .

# Readiness — should the orchestrator route traffic to this pod?
curl -s localhost:8080/health/ready | jq .

# Just the HTTP status (what a probe actually checks):
curl -s -o /dev/null -w "%{http_code}\n" localhost:8080/health/ready
```

| You see                                                          | Likely cause                           | Right action                                                                                 |
| ---------------------------------------------------------------- | -------------------------------------- | -------------------------------------------------------------------------------------------- |
| `ready=503`, `live=200`, state `RESTORING`                       | Rebuilding state stores on startup     | Wait — this is normal; size your readiness/startup budget for it                             |
| `ready=503`, `live=200`, state `PAUSED`                          | App was paused via `/pause`            | `/unpause` when ready ([Pause and unpause](https://stoatflow.io/docs/runtime/pause-unpause)) |
| `ready=503`, `live=200`, license `EXPIRED`/`REVOKED`             | License no longer operational          | Renew the license — a restart won't help                                                     |
| `ready=503`, `live=503`, `kafka-broker` DOWN                     | Broker unreachable                     | Fix connectivity; restart is a valid response                                                |
| `ready=503`, `live=200`, state `RUNNING`, `stall_age_ms` present | Commit pipeline stalled past threshold | Investigate; the orchestrator restart is the safety net                                      |

## Next steps

- **[Probes](https://stoatflow.io/docs/operating/probes)** — the Kubernetes `livenessProbe` / `readinessProbe` / `startupProbe` wiring, with budgets sized for restoration.
- **[REST API](https://stoatflow.io/docs/runtime/rest-api)** — the full endpoint catalogue, including `/state`, `/license`, and `/pause`.
- **[Metrics](https://stoatflow.io/docs/runtime/metrics)** — Prometheus gauges and counters, including engine state and license metrics.
- **[Pause and unpause](https://stoatflow.io/docs/runtime/pause-unpause)** — how pausing interacts with readiness.


# Metrics

The `:runtime` module exports metrics in Prometheus text format on `GET /metrics`. Internally it's a Micrometer `PrometheusMeterRegistry` — the engine reports events to a Micrometer instrumentation callback, the runtime binds JVM and Kafka-client metrics to the same registry, and the handler serves the registry's scrape output. Point your existing Prometheus stack at the endpoint and the StoatFlow meters appear alongside everything else.

::tldr-panel
- **Endpoint:** `GET /metrics` — Prometheus text format (`text/plain; version=0.0.4; charset=utf-8`).
- **Naming:** dotted `stoatflow.*` names; Prometheus rewrites `.` to `_` (e.g. `stoatflow.barrier.commit.latency` → `stoatflow_barrier_commit_latency`).
- **Common tag:** every meter carries `application` (the registry common tag); StoatFlow meters also carry `application_id`.
- **Detail level:** `runtime.metrics.recording-level` — `info` (default), `debug`, `trace`.
- **Always on:** JVM metrics (memory, GC, threads, classloader, processor) bind by default.
::

## Enable and scrape

Metrics are enabled by default and live under `runtime.metrics`:

```yaml
runtime:
  http:
    enabled: true
    port: 8080            # default
  metrics:
    enabled: true         # default
    prefix: stoatflow     # default — prefix on all metric names
    recording-level: info # info | debug | trace (default: info)
    bind-jvm-metrics: true # default — JVM memory/GC/thread binders
    common-tags:          # extra tags applied to every meter
      environment: production
      service: word-count
```

Scrape it from the shell:

```bash
curl -s localhost:8080/metrics | head -40
```

A minimal Prometheus scrape config:

```yaml
scrape_configs:
  - job_name: stoatflow
    metrics_path: /metrics
    static_configs:
      - targets: ["my-app:8080"]
```

The endpoint returns the standard exposition format, so any Prometheus-compatible collector (Prometheus, Grafana Agent, OpenTelemetry Collector with the Prometheus receiver, the VictoriaMetrics agent) scrapes it without extra translation. On Kubernetes, add `prometheus.io/scrape` / `prometheus.io/path` pod annotations or a `ServiceMonitor` pointing at the HTTP port — see [Observability](https://stoatflow.io/docs/operating/observability).

::callout{color="info" icon="i-lucide-info"}
Disabling metrics (`runtime.metrics.enabled: false`) un-registers the `/metrics` endpoint — it returns `404` — and switches the engine to no-op instrumentation, so there's no measurement overhead when metrics are off.
::

## Recording levels

`recording-level` controls how much detail the engine reports, mirroring Kafka Streams' levels. A higher level is a superset of the ones below it.

| Level   | What it adds                                                                                                                   | Use                       |
| ------- | ------------------------------------------------------------------------------------------------------------------------------ | ------------------------- |
| `info`  | Essential operational meters — throughput, barrier/commit latency, watermark progress, restoration, consumer/producer, errors. | Production default.       |
| `debug` | Per-store, per-partition, per-processor detail and dispatch internals. Higher cardinality.                                     | Targeted troubleshooting. |
| `trace` | Deepest internals; highest overhead.                                                                                           | Development only.         |

Tags like `lane_id`, `store_name`, `topic`, and `partition` multiply series count. On large topologies, keep production at `info` and switch to `debug` only while investigating.

## Naming and tags

::callout{color="info" icon="i-lucide-line-chart"}
**Migrating from Kafka Streams?** These native `stoatflow.*` names are not KS names. Set `runtime.metrics.naming: both` to *also* emit Kafka Streams-named, KS-shaped series derived from these meters so your existing KS dashboards work — see **[Reusing your Kafka Streams dashboards](https://stoatflow.io/docs/migration/reusing-kafka-streams-dashboards)**. (One side effect while it's on: the mapped native timers' `_seconds_max` tracks a \~30 s window instead of the \~2 min default, and `stoatflow.e2e.latency` gains a `{quantile="0.0"}` series.)
::

All StoatFlow meters use a dotted name with the configured prefix (`stoatflow` by default). The Prometheus registry rewrites dots to underscores and appends the conventional suffix (`_total` for counters; `_seconds_count`/`_seconds_sum`/`_seconds_max` for timers). So `stoatflow.barrier.completed.total` is scraped as `stoatflow_barrier_completed_total`, and the timer `stoatflow.barrier.commit.latency` produces `stoatflow_barrier_commit_latency_seconds_*` series. The latency timers publish client-side percentiles as `{quantile="0.5|0.95|0.99"}` labels on the base `_seconds` name rather than `_bucket` histograms, so use `rate(_sum)/rate(_count)` (or the `{quantile=...}` series) — `histogram_quantile()` over `_bucket` won't match.

Two application tags are present on StoatFlow meters:

| Tag              | Source                    | Notes                                                                                         |
| ---------------- | ------------------------- | --------------------------------------------------------------------------------------------- |
| `application`    | Registry-level common tag | Carries `application-id`; applied to **every** meter, including JVM and Kafka-client metrics. |
| `application_id` | Per-meter tag             | Applied by the StoatFlow instrumentation callback.                                            |

Anything you add under `common-tags` (for example `environment`, `region`, `service`) is also applied to every meter — handy for selecting one deployment in a shared Prometheus. Per-meter tags vary by group: `lane_id`, `store_name`, `topic`, `partition`, `error_type`, `trigger`, `type`, `result`, and others are listed with each meter below.

::callout{color="info" icon="i-lucide-info"}
Lane IDs follow the Kafka Streams task-ID convention `{subtopology}_{lane}` — e.g. `0_0`, `0_15`, `1_7`. Aggregate across a sub-topology in PromQL with a regex matcher: `sum(stoatflow_lane_records_processed_total{lane_id=~"1_.*"})`.
::

::callout{color="warning" icon="i-lucide-triangle-alert"}
**Sub-topology ids shifted in 1.0.0.** StoatFlow now inserts a sub-topology boundary only where an operator genuinely needs the re-keyed record's lane affinity, rather than at every key change ([ADR-138](https://stoatflow.io/docs/concepts/lanes-and-parallelism#where-the-handoff-actually-happens)). Re-keying topologies therefore have **fewer** sub-topologies, and the ids renumber — which silently changes any dashboard, alert or recording rule that pins a `sub_topology` / `subtopology_id` value or a `lane_id=~"N_.*"` regex. Re-check those selectors against a live scrape after upgrading. Setting `stoatflow.topology.sub-topology-split: eager` restores the previous numbering. **Processor-API topologies renumber a second time** in 1.0.0: `process()` / `processValues()` no longer open a boundary after a re-key (`stoatflow.topology.processor-api-key-affinity: off`, matching Kafka Streams); `presumed` restores those.
::

## Key meters

The dotted names below are the canonical metric IDs the engine emits. Remember the Prometheus rewrite (`.` → `_`, plus the type suffix) when writing queries. This is a working set, not the exhaustive list — the full catalogue is on the [metrics reference](https://stoatflow.io/docs/reference/metrics-reference), and your live scrape stays authoritative for exact tag sets.

Two families are deliberately left to the reference rather than repeated here: &#x2A;*`stoatflow.rocksdb.*`** (\~43 series per store, opt-in via `stoatflow.rocks-db.metrics.enabled&#x60; — reach for it when tuning RocksDB, not on a general dashboard) and &#x2A;*`stoatflow.suppress.*`** (a suppression-tuning signal).

### Throughput and end-to-end latency

| Metric                                   | Type    | Level | Tags      | Meaning                                                                                                          |
| ---------------------------------------- | ------- | ----- | --------- | ---------------------------------------------------------------------------------------------------------------- |
| `stoatflow.lane.records.processed.total` | Counter | info  | `lane_id` | Records processed by a lane.                                                                                     |
| `stoatflow.lane.bytes.processed.total`   | Counter | info  | `lane_id` | Bytes processed by a lane.                                                                                       |
| `stoatflow.lane.process.latency`         | Timer   | info  | `lane_id` | Per-record processing latency.                                                                                   |
| `stoatflow.lane.queue.size`              | Gauge   | info  | `lane_id` | Current lane queue depth (backpressure signal).                                                                  |
| `stoatflow.lane.queue.capacity`          | Gauge   | info  | `lane_id` | Configured lane queue capacity.                                                                                  |
| `stoatflow.lane.queue.full.total`        | Counter | info  | `lane_id` | Times a lane queue was full (backpressure events).                                                               |
| `stoatflow.e2e.latency`                  | Timer   | info  | —         | Per-record end-to-end latency, source timestamp to processing completion (analogous to KS `record-e2e-latency`). |

```promql
# Total processing throughput (records/sec)
sum(rate(stoatflow_lane_records_processed_total[1m]))

# Average per-record processing latency (this timer emits _sum/_count/_max, not _bucket)
sum(rate(stoatflow_lane_process_latency_seconds_sum[5m])) / sum(rate(stoatflow_lane_process_latency_seconds_count[5m]))
```

### Barrier commits

Commit barriers are StoatFlow's exactly-once mechanism (see [Exactly-once](https://stoatflow.io/docs/concepts/exactly-once)). These meters track barrier lifecycle and commit-phase latency. The latency timers publish client-side P50/P95/P99 percentiles.

| Metric                                            | Type                | Level | Tags           | Meaning                                                                                                                                                       |
| ------------------------------------------------- | ------------------- | ----- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `stoatflow.barrier.initiated.total`               | Counter             | info  | —              | Barriers initiated.                                                                                                                                           |
| `stoatflow.barrier.completed.total`               | Counter             | info  | —              | Barriers successfully completed.                                                                                                                              |
| `stoatflow.barrier.failed.total`                  | Counter             | info  | `error_type`   | Barrier failures, by error type.                                                                                                                              |
| `stoatflow.barrier.in.flight`                     | Gauge               | info  | —              | Currently pending barriers.                                                                                                                                   |
| `stoatflow.barrier.latency`                       | Timer               | info  | —              | Initiation to completion.                                                                                                                                     |
| `stoatflow.barrier.alignment.latency`             | Timer               | info  | —              | Time for all lanes to receive the barrier.                                                                                                                    |
| `stoatflow.lane.holdback.records`                 | DistributionSummary | info  | `sub_topology` | Records held back and replayed per barrier at a sub-topology boundary (receiver-side epoch hold-back — keeps the commit a consistent cut across the cascade). |
| `stoatflow.lane.holdback.drains.total`            | Counter             | info  | `sub_topology` | Non-empty hold-back drains (a receiving lane replayed stashed ahead-of-epoch records after crossing its barrier).                                             |
| `stoatflow.barrier.commit.latency`                | Timer               | info  | —              | Commit latency (producer drain + Kafka TX + state flush). Comparable to KS `commit-latency`.                                                                  |
| `stoatflow.barrier.commit.changelog.latency`      | Timer               | info  | —              | Aggregate changelog serialization+send latency per commit epoch.                                                                                              |
| `stoatflow.barrier.commit.state.flush.latency`    | Timer               | info  | —              | Aggregate state-store flush latency per commit epoch.                                                                                                         |
| `stoatflow.barrier.commit.producer.flush.latency` | Timer               | info  | —              | Pre-commit producer flush latency.                                                                                                                            |
| `stoatflow.barrier.commit.send.offsets.latency`   | Timer               | info  | —              | `sendOffsetsToTransaction` RPC latency.                                                                                                                       |
| `stoatflow.barrier.records.committed.total`       | Counter             | info  | —              | Records committed across barriers.                                                                                                                            |
| `stoatflow.barrier.bytes.committed.total`         | Counter             | info  | —              | Bytes committed across barriers.                                                                                                                              |
| `stoatflow.barrier.trigger.total`                 | Counter             | info  | `trigger`      | Barrier creation count by trigger — `TIME`, `RECORD_COUNT`, `MEMORY_PRESSURE`, `CACHE_PRESSURE` (state-cache cap forced an early commit).                     |
| `stoatflow.transaction.commit.total`              | Counter             | info  | —              | Kafka transactions committed.                                                                                                                                 |
| `stoatflow.transaction.abort.total`               | Counter             | info  | —              | Kafka transactions aborted.                                                                                                                                   |
| `stoatflow.transaction.commit.latency`            | Timer               | info  | —              | Transaction commit latency.                                                                                                                                   |

```promql
# Commit rate vs. failure rate
rate(stoatflow_barrier_completed_total[1m])
rate(stoatflow_barrier_failed_total[5m])
```

::callout{color="info" icon="i-lucide-info"}
The commit cadence and epoch size self-tune within their configured bounds; the scheduling gauges (`stoatflow.barrier.interval.target.ms`, `stoatflow.barrier.epoch.max.records`, and related) report what the engine is currently doing so you can see the cadence on a dashboard. The bound knobs themselves are documented under [Tuning](https://stoatflow.io/docs/operating/tuning) and the [configuration reference](https://stoatflow.io/docs/reference/configuration-reference).
::

### Watermarks

Event-time progress for windowing and late-event detection (see [Event time and watermarks](https://stoatflow.io/docs/concepts/event-time-and-watermarks)).

| Metric                                   | Type    | Level | Tags                 | Meaning                                           |
| ---------------------------------------- | ------- | ----- | -------------------- | ------------------------------------------------- |
| `stoatflow.watermark.current`            | Gauge   | info  | —                    | Current global watermark (epoch ms).              |
| `stoatflow.watermark.lag.ms`             | Gauge   | info  | —                    | Wall-clock time minus watermark (event-time lag). |
| `stoatflow.watermark.advance.total`      | Counter | info  | —                    | Watermark advancement events.                     |
| `stoatflow.watermark.late.records.total` | Counter | debug | `topic`, `partition` | Records arriving after the watermark.             |

```promql
# Alert when the watermark stalls while the app is running (RUNNING == 4)
delta(stoatflow_watermark_current[5m]) == 0 and stoatflow_client_state == 4
```

### Changelog writes and state restoration

Changelog write counters track state durability traffic; restoration meters track recovery progress on startup (see [State stores](https://stoatflow.io/docs/building/state-stores) and [State and thread safety](https://stoatflow.io/docs/concepts/state-and-thread-safety)).

| Metric                                            | Type    | Level | Tags                       | Meaning                                                                                                                                                                          |
| ------------------------------------------------- | ------- | ----- | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `stoatflow.store.changelog.records.written.total` | Counter | info  | `store_name`               | Changelog records written.                                                                                                                                                       |
| `stoatflow.store.changelog.bytes.written.total`   | Counter | info  | `store_name`               | Changelog bytes written.                                                                                                                                                         |
| `stoatflow.store.expired.records`                 | Counter | info  | `store_name`, `store_type` | Records event-time-expired past retention from an in-memory window or session store. A sudden climb usually means watermarks jumped — worth checking during an incident.         |
| `stoatflow.fk.enumeration.behind.scans`           | Counter | info  | `store_name`               | Foreign-key join evaluations that had to scan the subscription store as-of an earlier committed epoch. Sustained non-zero means FK joins are paying the slower enumeration path. |
| `stoatflow.restoration.in.progress`               | Gauge   | info  | —                          | 1 while restoration is running, 0 otherwise.                                                                                                                                     |
| `stoatflow.restoration.stores.total`              | Gauge   | info  | —                          | Total stores to restore.                                                                                                                                                         |
| `stoatflow.restoration.stores.completed`          | Gauge   | info  | —                          | Stores that have finished restoring.                                                                                                                                             |
| `stoatflow.restoration.records.restored.total`    | Counter | info  | `store_name`               | Records restored, per store.                                                                                                                                                     |
| `stoatflow.restoration.bytes.restored.total`      | Counter | info  | `store_name`               | Bytes restored, per store.                                                                                                                                                       |
| `stoatflow.restoration.duration.seconds`          | Timer   | info  | `store_name`               | Per-store restoration duration.                                                                                                                                                  |

```promql
# Restoration progress as a fraction of stores completed
stoatflow_restoration_stores_completed / stoatflow_restoration_stores_total
```

### Consumer, producer, and dispatch

| Metric                                      | Type    | Level | Tags                 | Meaning                        |
| ------------------------------------------- | ------- | ----- | -------------------- | ------------------------------ |
| `stoatflow.consumer.records.consumed.total` | Counter | info  | `topic`, `partition` | Records consumed.              |
| `stoatflow.consumer.bytes.consumed.total`   | Counter | info  | `topic`, `partition` | Bytes consumed.                |
| `stoatflow.consumer.lag.records`            | Gauge   | info  | `topic`, `partition` | Consumer lag, in records.      |
| `stoatflow.consumer.poll.latency`           | Timer   | info  | —                    | Consumer poll latency.         |
| `stoatflow.consumer.assigned.partitions`    | Gauge   | info  | —                    | Number of assigned partitions. |
| `stoatflow.producer.records.produced.total` | Counter | info  | `topic`              | Records produced.              |
| `stoatflow.producer.bytes.produced.total`   | Counter | info  | `topic`              | Bytes produced.                |
| `stoatflow.dispatcher.dispatch.latency`     | Timer   | info  | —                    | Per-batch dispatch latency.    |

```promql
# Worst-case consumer lag across partitions
max(stoatflow_consumer_lag_records)
```

### Errors

Error classification for alerting (see [Error handling](https://stoatflow.io/docs/building/error-handling-dlq)).

| Metric                                               | Type    | Level | Tags                     | Meaning                                                                                                                                                                                                                                                                          |
| ---------------------------------------------------- | ------- | ----- | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `stoatflow.error.total`                              | Counter | info  | `error_type`, `topic`    | Errors by type (and topic where applicable).                                                                                                                                                                                                                                     |
| `stoatflow.error.typed.total`                        | Counter | info  | `category`, `error_type` | Categorised errors (e.g. deserialization, processing, production).                                                                                                                                                                                                               |
| `stoatflow.queue.offer.failure.total`                | Counter | info  | `queue_type`             | Internal queue offer rejections.                                                                                                                                                                                                                                                 |
| `stoatflow.dlq.poison.replays.total`                 | Counter | info  | —                        | Poison-epoch replays — in-place engine restarts that reprocess an aborted epoch with the poison skipped. &#x2A;*The one to alert on:** the replay budget is in-memory, so it bounds `continue` per process, not end to end, and a non-converging poison will crash-loop the pod. |
| `stoatflow.dlq.poison.quarantined.total`             | Counter | info  | —                        | Source records quarantined as poison ahead of a replay. Each one loses *its own* outputs across all its sinks — bounded loss, not zero.                                                                                                                                          |
| `stoatflow.dlq.poison.replay.budget.exhausted.total` | Counter | info  | —                        | The replay budget ran out and the instance shut down with the offsets held. &#x2A;*Terminal — alert on it too:** `…replays.total` counts replays, so it goes quiet precisely when a non-converging poison starts crash-looping the pod.                                          |
| `stoatflow.dlq.poison.quarantine.size`               | Gauge   | info  | —                        | Source offsets a replay is currently skipping. `0` in steady state; non-zero only between a poison abort and the replay's first clean commit.                                                                                                                                    |

### Client state

| Metric                            | Type    | Level | Tags                               | Meaning                                                                                                                                                                 |
| --------------------------------- | ------- | ----- | ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `stoatflow.client.state`          | Gauge   | info  | —                                  | Application state ordinal: `CREATED=0`, `STARTING=1`, `VALIDATING_STATE=2`, `RESTORING=3`, `RUNNING=4`, `DRAINING=5`, `PAUSED=6`, `STOPPING=7`, `STOPPED=8`, `ERROR=9`. |
| `stoatflow.lanes`                 | Gauge   | info  | —                                  | Configured lane count.                                                                                                                                                  |
| `stoatflow.client.lane.alive`     | Gauge   | info  | —                                  | Currently running lanes.                                                                                                                                                |
| `stoatflow.client.uptime.seconds` | Gauge   | info  | —                                  | Seconds since the client was created.                                                                                                                                   |
| `stoatflow.engine.restart.total`  | Counter | info  | `disposition`, `target`, `trigger` | Successful in-place engine restarts (fault recovery via `REPLACE_THREAD`, HA demote-in-place).                                                                          |

### High availability

Bound only when [hot standby](https://stoatflow.io/docs/operating/high-availability) is enabled (`ha.mode != off`).

| Metric                                             | Type    | Level | Tags      | Meaning                                                                                                                                                          |
| -------------------------------------------------- | ------- | ----- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `stoatflow.ha.role`                                | Gauge   | info  | —         | `1` when this pod is the active, `0` when standby.                                                                                                               |
| `stoatflow.ha.standby.replication.lag.records`     | Gauge   | info  | —         | Standby replication lag in records (`0` when active or caught up).                                                                                               |
| `stoatflow.ha.standby.replication.applied.records` | Counter | info  | —         | Changelog records the standby applier has applied. `rate()` gives standby replication throughput — a flat line while lag is non-zero means the applier is stuck. |
| `stoatflow.ha.standby.replication.lag.ms`          | Gauge   | info  | —         | Standby replication lag in milliseconds.                                                                                                                         |
| `stoatflow.ha.peer.freshness.ms`                   | Gauge   | info  | —         | Time since the freshest observed peer's last heartbeat.                                                                                                          |
| `stoatflow.ha.peers`                               | Gauge   | info  | —         | Number of observed HA peers.                                                                                                                                     |
| `stoatflow.ha.redundancy.ready.standbys`           | Gauge   | info  | —         | Caught-up (`READY_STANDBY`) standby count, including self.                                                                                                       |
| `stoatflow.ha.redundancy.below.desired`            | Gauge   | info  | —         | `1` when the active has fewer ready standby spares than `desired-standbys`.                                                                                      |
| `stoatflow.ha.token.epoch`                         | Gauge   | info  | —         | Current promotion-token epoch (increments once per election).                                                                                                    |
| `stoatflow.ha.election.outcome`                    | Counter | info  | `outcome` | Cumulative election outcomes (`won` / `reclaimed` / `stood_down_fresh` / `stood_down_lost`).                                                                     |
| `stoatflow.ha.election.latency.ms`                 | Gauge   | info  | —         | Last claim→ACTIVE election latency in ms (`-1` until the first promotion).                                                                                       |
| `stoatflow.ha.restart.required`                    | Gauge   | info  | —         | `1` when this pod requires a restart (e.g. source-topic partition count changed).                                                                                |
| `stoatflow.ha.restarting`                          | Gauge   | info  | —         | `1` while this pod is restarting its engine in place to absorb a processing fault (it stays `ACTIVE`, so `stoatflow.ha.role` does not move).                     |

## License metrics

When a license is configured, the runtime binds license meters to the same registry (see [License configuration](https://stoatflow.io/docs/getting-started/license-configuration)). These use the `<prefix>.license` prefix (default `stoatflow.license`, tracking `runtime.metrics.prefix`) and carry the registry common tags (`application`, plus your `common-tags`). They're exported even when `recording-level` is `info`.

| Metric                                                       | Type    | Tags     | Meaning                                                                                                                                                                                |
| ------------------------------------------------------------ | ------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `stoatflow.license.valid`                                    | Gauge   | `tier`   | 1 when operating under a VALID or GRACE\_PERIOD license, else 0 — including the deferred-validation `PENDING` window (the first \~5 minutes after every start).                        |
| `stoatflow.license.expiry_timestamp_seconds`                 | Gauge   | —        | Unix epoch seconds of the entitlement deadline.                                                                                                                                        |
| `stoatflow.license.days_remaining`                           | Gauge   | —        | Whole days until expiry; negative within the post-expiry grace window.                                                                                                                 |
| `stoatflow.license.grace_remaining_seconds`                  | Gauge   | —        | Seconds left in the offline-grace budget; 0 when exhausted.                                                                                                                            |
| `stoatflow.license.heartbeat_consecutive_failures`           | Gauge   | —        | Heartbeat ticks failed since the last success; resets to 0 on success.                                                                                                                 |
| `stoatflow.license.heartbeat_last_success_timestamp_seconds` | Gauge   | —        | Unix epoch seconds of the most recent successful heartbeat.                                                                                                                            |
| `stoatflow.license.validations_total`                        | Counter | `result` | Validation outcomes, by result label — `success`, authoritative reject codes, plus the deferred-validation outcomes `network_failure`, `environment_required`, and `unexpected_error`. |
| `stoatflow.license.heartbeats_total`                         | Counter | `result` | Heartbeat tick outcomes, by result label.                                                                                                                                              |

```promql
# Alert when the license is not valid, or when expiry is within 14 days.
# Use a `for:` clause of at least 10m on the valid==0 rule: every deploy
# reads 0 for the ~5-minute deferred-validation PENDING window.
stoatflow_license_valid == 0
stoatflow_license_days_remaining < 14
```

::callout{color="warning" icon="i-lucide-shield"}
License meters carry tier and timing detail. Scrape `/metrics` only from your in-cluster monitoring stack, not through internet-facing ingress. See [Kubernetes](https://stoatflow.io/docs/operating/kubernetes).
::

## JVM and Kafka-client metrics

With `bind-jvm-metrics: true` (the default), the runtime binds Micrometer's standard JVM binders to the registry, so the usual `jvm_*` and `system_*` series appear on the same endpoint:

- `jvm_memory_used_bytes` / `jvm_memory_max_bytes` (heap and non-heap)
- `jvm_gc_pause_seconds` and related GC metrics
- `jvm_threads_live_threads`, `jvm_threads_states_threads`
- `jvm_classes_loaded_classes`
- `system_cpu_usage`, `process_cpu_usage`, `system_load_average_1m`

These carry the registry common tags (`application` + your `common-tags`) but not `application_id`. Native Kafka client metrics are also surfaced through Micrometer, so producer/consumer client internals appear alongside the StoatFlow meters.

## Dashboards and alerts

A starting overview combines throughput, latency, backpressure, and commit health:

```promql
# Throughput
sum(rate(stoatflow_lane_records_processed_total[1m]))

# Backpressure — total queued vs. capacity
sum(stoatflow_lane_queue_size) / sum(stoatflow_lane_queue_capacity)

# Commit health
rate(stoatflow_barrier_completed_total[1m])
rate(stoatflow_barrier_failed_total[5m])

# Event-time lag
stoatflow_watermark_lag_ms
```

Useful alert conditions:

| Alert             | Condition                                                                                              | Severity |
| ----------------- | ------------------------------------------------------------------------------------------------------ | -------- |
| Barrier failures  | `rate(stoatflow_barrier_failed_total[5m]) > 0`                                                         | Critical |
| License invalid   | `stoatflow_license_valid == 0` (`for: 10m` — every deploy reads 0 during the \~5-min `PENDING` window) | Critical |
| High consumer lag | `max(stoatflow_consumer_lag_records) > 100000`                                                         | Warning  |
| Watermark stall   | `delta(stoatflow_watermark_current[5m]) == 0 and stoatflow_client_state == 4`                          | Warning  |
| Backpressure      | `sum(stoatflow_lane_queue_size) > 0.8 * sum(stoatflow_lane_queue_capacity)`                            | Warning  |

## Next steps

- **[Observability](https://stoatflow.io/docs/operating/observability)** — wiring `/metrics`, logs, and probes into a monitoring stack.
- **[Health checks](https://stoatflow.io/docs/runtime/health-checks)** — the readiness and liveness signals that complement metrics.
- **[Tuning](https://stoatflow.io/docs/operating/tuning)** — which bound knobs the self-tuning cadence respects.
- **[The REST API](https://stoatflow.io/docs/runtime/rest-api)** — the other operational endpoints on the same HTTP server.


# Pause and resume

Pause stops the instance from dispatching new records and tracking new offsets, while keeping the consumer session alive and the lane threads warm — so you can resume instantly. It's a runtime control for short operational gaps (a maintenance window on a downstream system, transient backpressure), not a substitute for shutting the instance down.

::tldr-panel
- **Pause:** `curl -X POST localhost:8080/pause` — drains in-flight work, commits it, then sits idle without consuming new records.
- **Resume:** `curl -X POST localhost:8080/unpause` — picks up exactly where it left off.
- **While paused:** liveness stays UP, readiness goes DOWN. The Kafka consumer keeps heartbeating, so no rebalance and no restart.
- **Use it for:** a downstream system that's temporarily unavailable, or a quick maintenance task — anything where a full stop/start is heavier than you want.
::

## What pause does

Calling pause moves the instance through two states:

1. **`DRAINING`** — record dispatch stops immediately. No new records are handed to processing; timers and punctuators are paused; records already consumed but not yet dispatched are held, not dropped. Work already in flight finishes, and a final commit flushes that work and its offsets. This is a graceful drain — nothing in flight is dropped.
2. **`PAUSED`** — reached automatically once the in-flight work has drained and committed. The instance now sits idle: the Kafka consumer keeps polling for heartbeats (so the group membership and session stay alive), processing threads stay warm and ready, but no new records are consumed and no new offsets are tracked.

Because the consumer keeps heartbeating, pausing does **not** trigger a consumer-group rebalance and does **not** require a restart. Resume is near-instant — the threads never went cold.

::callout{color="info" icon="i-lucide-info"}
Pause is in-memory state, not persisted. If the process restarts while paused, it comes back up in the normal `RUNNING` flow and resumes processing — there is no "stays paused across restarts" behaviour.
::

## What unpause does

Unpause returns the instance to `RUNNING`:

- Record dispatch resumes — any held records drain back into processing and consumption continues.
- Timers and punctuators resume.

You can call unpause from either `PAUSED` (resume after the drain finished) or `DRAINING` (cancel the drain that's still in progress and resume immediately).

## Pause and readiness

This is the key operational interaction. While the instance is `DRAINING` or `PAUSED`:

| Probe           | State    | Meaning                                           |
| --------------- | -------- | ------------------------------------------------- |
| `/health/live`  | **UP**   | The process is healthy — do not restart it.       |
| `/health/ready` | **DOWN** | The instance is intentionally not taking traffic. |

Liveness staying UP is deliberate: a paused instance is working as intended, so an orchestrator must not kill and restart it. Readiness going DOWN reflects that it isn't processing. See [Health checks](https://stoatflow.io/docs/runtime/health-checks) and [Probes](https://stoatflow.io/docs/operating/probes) for how Kubernetes consumes these.

::callout{color="warning" icon="i-lucide-triangle-alert"}
StoatFlow runs as a **single active instance** — there is no second replica to pick up traffic while this one is paused. Pausing halts the pipeline; consumer lag will grow on the source topics for as long as you stay paused. Pause is for short, deliberate gaps, not load shedding. (Even with [hot standby](https://stoatflow.io/docs/operating/high-availability) enabled, pausing the active does not promote the standby — pause is a deliberate stop, not a failover.)
::

## The endpoints

Both endpoints are `POST` and take no body. They're served by the runtime's HTTP server (`runtime.http.enabled`, default port `8080`).

### `POST /pause`

```bash
curl -X POST localhost:8080/pause
```

| Response            | Code  | Body                                      |
| ------------------- | ----- | ----------------------------------------- |
| Pause initiated     | `200` | `{"status":"pausing","state":"DRAINING"}` |
| Wrong state         | `400` | `{"error":"Cannot pause from state ..."}` |
| Not initialized yet | `503` | `{"error":"StoatFlow not initialized"}`   |

Pause is only valid from `RUNNING`. Calling it during startup, while already paused, or while shutting down returns `400`. The `200` response reports `DRAINING` — the instance reaches `PAUSED` a moment later once the drain completes; poll [`/state`](https://stoatflow.io/docs/runtime/rest-api) to confirm.

### `POST /unpause`

```bash
curl -X POST localhost:8080/unpause
```

| Response            | Code  | Body                                        |
| ------------------- | ----- | ------------------------------------------- |
| Resumed             | `200` | `{"status":"resumed","state":"RUNNING"}`    |
| Wrong state         | `400` | `{"error":"Cannot unpause from state ..."}` |
| Not initialized yet | `503` | `{"error":"StoatFlow not initialized"}`     |

Unpause is valid from `PAUSED` or `DRAINING`. Calling it from `RUNNING` (or any other state) returns `400`.

## A maintenance sequence

A typical "pause, do something, resume" loop using the HTTP API:

```bash
# 1. Pause and watch it settle into PAUSED
curl -X POST localhost:8080/pause
#    -> {"status":"pausing","state":"DRAINING"}

# 2. Wait until it's fully drained (readiness flips DOWN; /state shows PAUSED)
until curl -s localhost:8080/state | grep -q '"PAUSED"'; do sleep 1; done

# ... perform maintenance on the downstream system ...

# 3. Resume
curl -X POST localhost:8080/unpause
#    -> {"status":"resumed","state":"RUNNING"}
```

The current lifecycle state is always available at [`/state`](https://stoatflow.io/docs/runtime/rest-api).

## Programmatic control (core)

The same pause/drain semantics are exposed as methods on the core `StoatFlow` handle — useful when you embed the engine via `:core` directly rather than driving it over HTTP. `pause()` returns once the drain has been initiated (state `DRAINING`); use `awaitState(...)` to block until `PAUSED`.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.StoatFlow
import java.time.Duration

stoatflow.pause()
stoatflow.awaitState(StoatFlow.State.PAUSED, Duration.ofSeconds(30))

// ... perform maintenance ...

stoatflow.unpause()
```

```java [Java]
import io.stoatflow.core.StoatFlow;
import java.time.Duration;

stoatflow.pause();
stoatflow.awaitState(StoatFlow.State.PAUSED, Duration.ofSeconds(30));

// ... perform maintenance ...

stoatflow.unpause();
```
::

`pause()` throws `IllegalStateException` if the instance is not `RUNNING`; `unpause()` throws if it is not `PAUSED` or `DRAINING` — the same state rules the HTTP handlers turn into `400` responses.

::callout{color="info" icon="i-lucide-info"}
When you run on the batteries-included `:runtime` (`StoatFlowRuntime`), pause and resume are driven over HTTP via `/pause` and `/unpause`. The programmatic `pause()` / `unpause()` / `awaitState()` methods above are on the underlying `StoatFlow` engine handle, which you hold directly when using the `:core` module.
::

## When to use it

- **Downstream backpressure.** A sink the topology writes to (a database, an external API, another topic's consumer) goes unavailable or slows down. Rather than letting failures pile up, pause, let the downstream recover, then unpause — the consumer session stays alive throughout, so you avoid a rebalance and a cold restart.
- **Short maintenance windows.** You need to touch the broker, the downstream store, or the host briefly and want a clean, committed boundary. Pause drains and commits in-flight work first, so you resume from a consistent point with no partial in-flight state.

For anything longer-lived or anything that changes the topology or configuration, stop the instance and start it again — pause is for keeping a single instance warm across a brief, deliberate gap, not for long outages. See [Production checklist](https://stoatflow.io/docs/operating/production-checklist) for operational guidance.

## Next steps

- **[REST API](https://stoatflow.io/docs/runtime/rest-api)** — the full endpoint reference, including `/state`.
- **[Health checks](https://stoatflow.io/docs/runtime/health-checks)** — how `DRAINING` / `PAUSED` map to liveness and readiness.
- **[Probes](https://stoatflow.io/docs/operating/probes)** — wiring the probes into Kubernetes so a paused instance isn't restarted.


# Plugins and lifecycle hooks

The `:runtime` module is a thin, extensible wrapper around the engine. Two extension points let you add your own behaviour without forking it: a `RuntimePlugin` runs at startup with access to the runtime's shared components (metrics registry, HTTP server, health registry), and a `RuntimeLifecycleListener` observes the runtime as it starts and stops.

::tldr-panel
- **`RuntimePlugin`** — `id`, `initialize(context)`, optional `shutdown()`. Register custom metrics, HTTP handlers, and health indicators at startup.
- **`PluginContext`** — what a plugin gets: `meterRegistry()`, `httpServer()`, `metricsPrefix()`, `applicationId()`, `healthIndicatorRegistry()`.
- **`RuntimeLifecycleListener`** — a callback fired on `PRE_START` / `POST_START` / `PRE_STOP` / `POST_STOP`.
- **Register both** via the runtime builder (`addPlugin`, `addLifecycleListener`) or the `configure { ... }` block on `StoatFlowRuntime.fromConfig`.
::

::callout{color="info" icon="i-lucide-info"}
`PluginContext` exposes the runtime's HTTP server, metrics registry, and health registry — **not** the stream-processing engine or a state-store handle. Plugins extend the operational surface (endpoints, metrics, probes); they cannot run interactive state queries against your topology.
::

## The RuntimePlugin interface

A plugin is a class implementing `RuntimePlugin`. It has a stable `id`, an `initialize(context)` method called once during startup, and an optional `shutdown()` for cleanup.

```kotlin
interface RuntimePlugin {
    val id: String
    fun initialize(context: PluginContext)
    fun shutdown() {}
}
```

- **`id`** — a short, descriptive name without spaces (e.g. `"prometheus"`, `"otel"`, `"audit-log"`). Used for logging and diagnostics. Plugins initialize in registration order.
- **`initialize(context)`** — called during startup, **before** the HTTP server starts and **before** stream processing begins. The metrics registry exists, the HTTP server is created but not yet running. This is where you register HTTP handlers, metrics, and health indicators.
- **`shutdown()`** — called during shutdown, **after** stream processing has stopped. The metrics registry is still available; the HTTP server may still be running. Release anything you acquired in `initialize`.

## The PluginContext surface

`initialize(context)` receives a `PluginContext`. These are the only members:

| Member                      | Returns                   | Purpose                                                                                                                                                                                                  |
| --------------------------- | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `meterRegistry()`           | `MeterRegistry?`          | Micrometer registry for custom metrics. `null` if metrics are disabled.                                                                                                                                  |
| `httpServer()`              | `RuntimeHttpServer?`      | The HTTP server, for registering custom endpoints. `null` if HTTP is disabled. Handlers must be registered during `initialize` — once the server starts, registration throws `IllegalArgumentException`. |
| `metricsPrefix()`           | `String`                  | The configured metrics prefix (e.g. `"stoatflow"`). Prefix your own meters with it for consistency.                                                                                                      |
| `applicationId()`           | `String`                  | The `stoatflow.application-id` from configuration.                                                                                                                                                       |
| `healthIndicatorRegistry()` | `HealthIndicatorRegistry` | Register custom health indicators for the liveness / readiness probes.                                                                                                                                   |

::callout{color="warning" icon="i-lucide-triangle-alert"}
`meterRegistry()` and `httpServer()` are nullable — they return `null` when metrics or the HTTP server are disabled in config. Always guard the call (`?.let { ... }`) so a plugin degrades gracefully instead of throwing on a misconfigured runtime.
::

## A complete plugin

This plugin registers a custom counter, a custom HTTP endpoint, and a custom health indicator — exercising every part of the `PluginContext`.

::code-tabs{group="lang"}
```kotlin [Kotlin]
package com.example

import io.stoatflow.runtime.health.Health
import io.stoatflow.runtime.health.HealthIndicator
import io.stoatflow.runtime.plugin.PluginContext
import io.stoatflow.runtime.plugin.RuntimePlugin

class AuditPlugin : RuntimePlugin {
    override val id = "audit"

    override fun initialize(context: PluginContext) {
        val prefix = context.metricsPrefix()

        // 1. Custom metric — prefix it to match the built-in meters.
        context.meterRegistry()?.let { registry ->
            registry.counter("$prefix.audit.events").increment()
        }

        // 2. Custom HTTP endpoint (com.sun.net.httpserver.HttpHandler).
        //    Must be registered now — before the server starts.
        context.httpServer()?.registerHandler("/audit") { exchange ->
            val body = """{"application":"${context.applicationId()}"}"""
                .toByteArray()
            exchange.responseHeaders.add("Content-Type", "application/json")
            exchange.sendResponseHeaders(200, body.size.toLong())
            exchange.responseBody.use { it.write(body) }
        }

        // 3. Custom health indicator — surfaces in /health/ready and /health/live.
        context.healthIndicatorRegistry().register("audit-sink", AuditHealthIndicator())
    }

    override fun shutdown() {
        // Release anything acquired in initialize().
    }
}

class AuditHealthIndicator : HealthIndicator {
    override fun health(): Health =
        Health.up().withDetail("audit-sink", "connected").build()
}
```

```java [Java]
package com.example;

import io.stoatflow.runtime.health.Health;
import io.stoatflow.runtime.health.HealthIndicator;
import io.stoatflow.runtime.plugin.PluginContext;
import io.stoatflow.runtime.plugin.RuntimePlugin;

import java.io.OutputStream;
import java.nio.charset.StandardCharsets;

public class AuditPlugin implements RuntimePlugin {

    @Override
    public String getId() {
        return "audit";
    }

    @Override
    public void initialize(PluginContext context) {
        String prefix = context.metricsPrefix();

        // 1. Custom metric — prefix it to match the built-in meters.
        var registry = context.meterRegistry();
        if (registry != null) {
            registry.counter(prefix + ".audit.events").increment();
        }

        // 2. Custom HTTP endpoint (com.sun.net.httpserver.HttpHandler).
        //    Must be registered now — before the server starts.
        var server = context.httpServer();
        if (server != null) {
            server.registerHandler("/audit", exchange -> {
                byte[] body = ("{\"application\":\"" + context.applicationId() + "\"}")
                    .getBytes(StandardCharsets.UTF_8);
                exchange.getResponseHeaders().add("Content-Type", "application/json");
                exchange.sendResponseHeaders(200, body.length);
                try (OutputStream os = exchange.getResponseBody()) {
                    os.write(body);
                }
            });
        }

        // 3. Custom health indicator — surfaces in /health/ready and /health/live.
        context.healthIndicatorRegistry().register("audit-sink", new AuditHealthIndicator());
    }

    @Override
    public void shutdown() {
        // Release anything acquired in initialize().
    }
}

class AuditHealthIndicator implements HealthIndicator {
    @Override
    public Health health() {
        return Health.up().withDetail("audit-sink", "connected").build();
    }
}
```
::

::callout{color="info" icon="i-lucide-info"}
In Kotlin the `id` is an abstract `val`; in Java it surfaces as `getId()`. The HTTP handler is the JDK's `com.sun.net.httpserver.HttpHandler` — a functional interface taking an `HttpExchange`, so it accepts a lambda directly.
::

### Registering a custom metric

Call `context.meterRegistry()` and use the standard Micrometer API (`counter`, `gauge`, `timer`, …). Prefix meter names with `context.metricsPrefix()` so they sit alongside the built-in StoatFlow meters and inherit the runtime's common tags. The new meters appear on the [`/metrics`](https://stoatflow.io/docs/runtime/metrics) scrape endpoint. See [Metrics](https://stoatflow.io/docs/runtime/metrics) for the built-in catalogue.

### Registering an HTTP handler

Call `context.httpServer()?.registerHandler(path, handler)` during `initialize`. The path is a context root (e.g. `/audit`); the underlying JDK `HttpServer` routes by longest-prefix match, so `/audit` also catches `/audit/anything`. The handler is a `com.sun.net.httpserver.HttpHandler`; the server runs each request on a virtual thread. Registration **must** happen during `initialize` — calling `registerHandler` after the server has started throws `IllegalArgumentException`. See [REST API](https://stoatflow.io/docs/runtime/rest-api) for the built-in endpoints.

### Registering a health indicator

Call `context.healthIndicatorRegistry().register(name, indicator)`. A `HealthIndicator` implements `health()` (readiness) and may override `livenessHealth()` (liveness — defaults to the same as `health()`). The registry offers three registration methods:

| Method                               | Effect                                         |
| ------------------------------------ | ---------------------------------------------- |
| `register(name, indicator)`          | Registers for **both** liveness and readiness. |
| `registerReadiness(name, indicator)` | Readiness probe only.                          |
| `registerLiveness(name, indicator)`  | Liveness probe only.                           |

Use readiness-only when a dependency being down should take the instance out of the load-balancer pool but **not** trigger a restart; use liveness-only for conditions where a restart is the right remedy. See [Health checks](https://stoatflow.io/docs/runtime/health-checks) for the built-in indicators and the probe semantics.

## Lifecycle hooks

A `RuntimeLifecycleListener` is a single-method callback (a Kotlin `fun interface`) invoked as the runtime transitions through its lifecycle. Unlike a plugin, it has no `PluginContext` — it receives the `LifecycleEvent` and the `StoatFlowRuntime` instance (for state queries).

```kotlin
fun interface RuntimeLifecycleListener {
    fun onEvent(event: LifecycleEvent, runtime: StoatFlowRuntime)
}
```

`LifecycleEvent` has four values, emitted in order:

| Event        | When                     | State at this point                                                |
| ------------ | ------------------------ | ------------------------------------------------------------------ |
| `PRE_START`  | Before initialization    | Config validated; no resources allocated yet.                      |
| `POST_START` | After successful startup | HTTP server running, metrics collecting, stream processing active. |
| `PRE_STOP`   | Before shutdown begins   | Shutdown requested; processing still active.                       |
| `POST_STOP`  | After shutdown completes | Processing stopped, HTTP server stopped, metrics registry closed.  |

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.runtime.LifecycleEvent
import io.stoatflow.runtime.StoatFlowRuntime
import io.stoatflow.runtime.plugin.RuntimeLifecycleListener

val listener = RuntimeLifecycleListener { event, runtime ->
    when (event) {
        LifecycleEvent.POST_START -> println("Started: ${runtime.state()}")
        LifecycleEvent.PRE_STOP -> println("Stopping: ${runtime.state()}")
        else -> {}
    }
}
```

```java [Java]
import io.stoatflow.runtime.LifecycleEvent;
import io.stoatflow.runtime.plugin.RuntimeLifecycleListener;

RuntimeLifecycleListener listener = (event, runtime) -> {
    switch (event) {
        case POST_START -> System.out.println("Started: " + runtime.state());
        case PRE_STOP -> System.out.println("Stopping: " + runtime.state());
        default -> { }
    }
};
```
::

::callout{color="neutral" icon="i-lucide-lightbulb"}
Use a **lifecycle listener** for one-off side effects tied to startup/shutdown transitions (notify an external system, flush a buffer). Use a **plugin** when you need the `PluginContext` to register endpoints, metrics, or health indicators. A plugin's `initialize`/`shutdown` and the `POST_START`/`POST_STOP` events overlap but differ in timing — `initialize` runs before the HTTP server starts; `POST_START` fires after it is running.
::

## Registering plugins and listeners

Register both on the runtime builder. With the batteries-included `StoatFlowRuntime.fromConfig(...)` entry point, reach the builder through the `configure { ... }` block; with the explicit builder, call the methods directly.

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.runtime.StoatFlowRuntime

// Via fromConfig — configure block exposes the builder
val runtime = StoatFlowRuntime.fromConfig(
    topologyBuilder = { buildTopology(it) },
    configure = {
        addPlugin(AuditPlugin())
        addLifecycleListener(listener)
    },
)
runtime.start()
runtime.awaitTermination()
```

```java [Java]
import io.stoatflow.runtime.StoatFlowRuntime;

// Via fromConfig — configure block exposes the builder
var runtime = StoatFlowRuntime.fromConfig(
    Main::buildTopology,
    builder -> {
        builder.addPlugin(new AuditPlugin());
        builder.addLifecycleListener(listener);
    }
);
runtime.start();
runtime.awaitTermination();
```
::

The builder also offers `addHealthIndicator(name, indicator)` as a shortcut for registering a health indicator without writing a plugin — it registers for both liveness and readiness. For finer control (readiness-only, liveness-only, or dynamic registration), go through a plugin and `context.healthIndicatorRegistry()`.

## Next steps

- **[Metrics](https://stoatflow.io/docs/runtime/metrics)** — the built-in Micrometer catalogue and the `/metrics` scrape format your custom meters join.
- **[Health checks](https://stoatflow.io/docs/runtime/health-checks)** — built-in indicators and the liveness vs readiness semantics your custom indicators participate in.
- **[REST API](https://stoatflow.io/docs/runtime/rest-api)** — the built-in HTTP endpoints alongside which your custom handlers are served.
- **[Runtime configuration](https://stoatflow.io/docs/configuration/runtime-config)** — toggling the HTTP server and metrics that your plugin depends on.


# Docker images

The supported way to containerize a StoatFlow app is the StoatFlow build conventions' [Jib](https://github.com/GoogleContainerTools/jib){rel="&#x22;nofollow&#x22;"} integration — **no `Dockerfile`, no Docker daemon to assemble layers**. It produces a reproducible image wired with the JVM flags StoatFlow needs and an entrypoint that sizes the heap from the container's memory limit at startup. Both the [Gradle plugin](https://stoatflow.io/docs/getting-started/project-setup) and the [Maven](https://stoatflow.io/docs/reference/maven-reference) conventions build the same image.

::tldr-panel
- **Gradle:** `stoatflow { docker { enabled.set(true) } }`, then `./gradlew jibDockerBuild`.
- **Maven:** set `<stoatflow.docker.*>` properties, then `mvn -Pstoatflow-docker package` (one command).
- **Base image:** `eclipse-temurin:25-jre-noble`, built for the host architecture (`linux/arm64` or `linux/amd64`).
- **Set for you:** the `--enable-preview` / `--enable-native-access=ALL-UNNAMED` / `-XX:+UseG1GC` flags, port `8080` exposed, a non-root user, a heap-sizing entrypoint, and a **reproducible** `creationTime`.
- **Full knob list:** [Gradle plugin reference](https://stoatflow.io/docs/reference/gradle-plugin-reference#docker-image-defaults) · [Maven reference](https://stoatflow.io/docs/reference/maven-reference#properties-reference).
::

## Build the image (Jib)

Container support is opt-in. Apply the conventions (see [Project setup](https://stoatflow.io/docs/getting-started/project-setup)), set `mainClass`, and turn the `docker` build on:

::code-tabs{group="build"}
```kotlin [Gradle]
// build.gradle.kts
stoatflow {
    mainClass.set("com.example.MainKt")
    docker {
        enabled.set(true)
        imageName.set("acme/word-count")   // default: the Gradle project name
    }
}
```

```xml [Maven]
<!-- pom.xml -->
<properties>
  <stoatflow.mainClass>com.example.Main</stoatflow.mainClass>
  <stoatflow.docker.imageName>acme/word-count</stoatflow.docker.imageName>
</properties>
```
::

Then build the image into your local Docker daemon:

::code-tabs{group="build"}
```bash [Gradle]
./gradlew jibDockerBuild
```

```bash [Maven]
mvn -Pstoatflow-docker package   # fat JAR + image, one command
```
::

Jib reads your compiled classes and dependencies and assembles an OCI image directly — no `Dockerfile`, no intermediate fat JAR to manage. The image is tagged `latest` and the project version, named after `imageName` (or the project name if unset).

::callout{color="info" icon="i-lucide-info"}
`jibDockerBuild` / `mvn -Pstoatflow-docker package` needs a reachable Docker daemon to receive the image, but Jib does **not** run a `docker build` — it constructs the layers itself. On macOS the build resolves the `docker` executable from common install paths (`/usr/local/bin/docker`, `/opt/homebrew/bin/docker`) when the daemon's `PATH` doesn't include them.
::

## What the build configures

The Jib image has the following baked in:

| Aspect                | Value                                                                                          | Knob                    |
| --------------------- | ---------------------------------------------------------------------------------------------- | ----------------------- |
| **Base image**        | `eclipse-temurin:25-jre-noble`                                                                 | `baseImage`             |
| **Platform**          | `linux/arm64` on an `aarch64` host, otherwise `linux/amd64`                                    | —                       |
| **Tags**              | `latest` and the project version                                                               | —                       |
| **`creationTime`**    | HEAD commit timestamp (reproducible)                                                           | `reproducibleBuild`     |
| **Exposed port**      | `8080` — the runtime's HTTP + metrics server                                                   | `port`                  |
| **User**              | `1000:1000` (non-root)                                                                         | `user`                  |
| **Working directory** | `/app`                                                                                         | —                       |
| **Entrypoint**        | a heap-sizing script (`/app/stoatflow-entrypoint.sh`) that `exec`s `java`                      | —                       |
| **JVM flags**         | `--enable-preview`, `--enable-native-access=ALL-UNNAMED`, `-XX:+UseG1GC`, plus your `jvmFlags` | `jvmFlags`              |
| **Env**               | `STOATFLOW_STATE_DIR=/data/state`; `STOATFLOW_ROCKSDB_MB=256` when RocksDB is on the classpath | `stateDir`, `rocksdbMb` |

Every knob in the right column has the same name in the Gradle `docker { }` block and a `stoatflow.docker.<name>` Maven property — the full enumeration with types and defaults is in the [Gradle plugin reference](https://stoatflow.io/docs/reference/gradle-plugin-reference#docker-image-defaults) and the [Maven reference](https://stoatflow.io/docs/reference/maven-reference#properties-reference).

### JVM flags

The image always carries the flags StoatFlow requires at runtime, the same ones applied to the `run` task:

- `--enable-preview` — StoatFlow is built on JDK preview features.
- `--enable-native-access=ALL-UNNAMED` — RocksDB's state store uses the Foreign Function & Memory API.
- `-XX:+UseG1GC` — short, predictable GC pauses, set explicitly because the JVM only auto-selects G1 on "server-class" hardware (≥ 2 CPUs and ≥ \~2 GB memory). See [Installation](https://stoatflow.io/docs/getting-started/installation#_3-configure-the-jvm-toolchain) for the reasoning.

Anything you add via `jvmFlags` is appended after these (and `jvmFlags` is itself overridable — the two preview/native-access flags are always re-added on top):

::code-tabs{group="build"}
```kotlin [Gradle]
stoatflow {
    docker {
        enabled.set(true)
        jvmFlags.add("-XX:+ExitOnOutOfMemoryError")
        jvmFlags.add("-Dsome.property=value")
    }
}
```

```xml [Maven]
<properties>
  <stoatflow.docker.jvmFlags>-XX:+UseG1GC -XX:+ExitOnOutOfMemoryError</stoatflow.docker.jvmFlags>
</properties>
```
::

### Reproducible builds

By default the image is **reproducible**: its `creationTime` is pinned to your **HEAD commit timestamp**, not wall-clock — so identical source produces an identical digest with a meaningful `Created` date, friendly to layer caching and supply-chain verification. Outside a git repo the timestamp falls back to the Unix epoch. Set `reproducibleBuild = false` (Gradle) / `<stoatflow.docker.reproducibleBuild>false</…>` (Maven) to restore wall-clock timestamps.

## Container-aware heap sizing

The Jib image does **not** hardcode `-Xmx`. Its entrypoint computes the heap from the memory available to the container at startup, reserving space for the JVM's own overhead and for RocksDB's off-heap usage, then `exec`s `java` with the computed `-Xmx`. The same image right-sizes itself whether it runs with a 512 MiB limit or a 4 GiB limit — you set the **container** memory limit, not a JVM flag.

The memory limit is resolved from the first available source, in order:

1. cgroup v2 (`/sys/fs/cgroup/memory.max`) — modern containers (Ubuntu 22.04+, Kubernetes 1.25+).
2. cgroup v1 (`/sys/fs/cgroup/memory/memory.limit_in_bytes`) — older runtimes.
3. `/proc/meminfo` — bare-metal Linux.
4. If none report a limit, the entrypoint falls back to `-XX:MaxRAMPercentage=75.0` and lets the JVM decide.

When a limit is found, the heap is the total minus a non-heap budget: a JVM baseline (25% of total, capped at 512 MiB) plus the RocksDB reservation. Two environment variables tune this:

| Variable                | Default                                          | Effect                                                                                                                            |
| ----------------------- | ------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- |
| `STOATFLOW_ROCKSDB_MB`  | `256` when RocksDB is on the classpath, else `0` | Off-heap MiB reserved for RocksDB. Subtracted from the heap. Set via the `rocksdbMb` knob at build time, or override at run time. |
| `STOATFLOW_NON_HEAP_MB` | unset                                            | Explicit total non-heap budget (overrides the baseline + RocksDB calculation). Clamped to ≤ 50% of total.                         |

The entrypoint logs the source, total, non-heap budget, and the chosen `-Xmx` to stderr at startup, so the sizing decision is visible in the container logs. If the resolved memory is too small to leave a viable heap, it logs a warning and forces a 128 MiB floor (expect an OOMKill — give the container more memory).

::callout{color="warning" icon="i-lucide-triangle-alert"}
`STOATFLOW_ROCKSDB_MB` and `STOATFLOW_NON_HEAP_MB` must be **plain integers** (MiB) — e.g. `256`, not `256m`. The entrypoint fails fast on a non-numeric value.
::

::callout{color="info" icon="i-lucide-info"}
**RocksDB reservation is automatic.** Because `stoatflow-runtime`/`stoatflow-core` bring RocksDB onto the classpath, the image sets `STOATFLOW_ROCKSDB_MB=256` for you. In-memory-only apps omit it. Override with the `rocksdbMb` knob, or `0` to omit. The native [`runtime` image](https://stoatflow.io/docs/runtime/native-image#runtime-target) uses the same entrypoint and the same reservation.
::

## Run the container

The image exposes port `8080` — the runtime's HTTP admin + Prometheus metrics server (see [REST API](https://stoatflow.io/docs/runtime/rest-api) and [Metrics](https://stoatflow.io/docs/runtime/metrics)) — and reads its own configuration (bootstrap servers, license key, topics) from `application.yaml` and environment overrides (see [Runtime config](https://stoatflow.io/docs/configuration/runtime-config)). Mount a volume at `STOATFLOW_STATE_DIR` to persist RocksDB state across restarts:

```bash
docker run --rm \
  -e KAFKA_BOOTSTRAP_SERVERS=host.docker.internal:9092 \
  -e STOATFLOW_LICENSE_KEY="key/...your key..." \
  -v stoatflow-state:/data/state \
  -p 8080:8080 \
  acme/word-count:latest
```

A Kubernetes deployment then needs only a memory limit (the entrypoint sizes the heap from it) and a volume for the state dir — see [Kubernetes](https://stoatflow.io/docs/operating/kubernetes):

```yaml
resources:
  limits:
    memory: 2Gi          # the entrypoint sizes the heap from this
volumeMounts:
  - { name: state, mountPath: /data/state }   # matches stateDir
```

## Rolling your own Dockerfile

The Jib path is the supported, simplest route and right-sizes the heap automatically — prefer it. If your pipeline standardizes on `docker build`, build the fat JAR (`./gradlew shadowJar` / `mvn package`) and write a thin image over it: a `eclipse-temurin:25-jre-noble` base, `COPY` the `-all.jar`, and launch `java --enable-preview --enable-native-access=ALL-UNNAMED -XX:+UseG1GC -jar app.jar`. You lose the automatic container-aware heap sizing — pin `-Xmx` or use `-XX:MaxRAMPercentage` yourself.

For a **native** image (GraalVM), the containerized build *is* a `docker build` against a bundled, repo-independent `Dockerfile.native` — see [Native image](https://stoatflow.io/docs/runtime/native-image).

## Next steps

- **[Project setup](https://stoatflow.io/docs/getting-started/project-setup)** — the full build-conventions surface, including the `docker` and `nativeImage` blocks.
- **[Gradle plugin reference](https://stoatflow.io/docs/reference/gradle-plugin-reference)*&#x2A; · &#x2A;*[Maven reference](https://stoatflow.io/docs/reference/maven-reference)** — every property, task, and default.
- **[Native image](https://stoatflow.io/docs/runtime/native-image)** — the GraalVM native-image build for fast startup and low memory.
- **[Health checks](https://stoatflow.io/docs/runtime/health-checks)** — the liveness/readiness endpoints the container probes hit.


# GraalVM native image

A StoatFlow application compiles to a GraalVM **native image** — a standalone binary with fast start-up, a small footprint, and (with the right GC) steady-state CPU and tail latency on par with the JVM. The opt-in is one line, because the framework already ships the native-image metadata for everything *it* touches; the only per-app work is the metadata for your own serde value classes.

This is the most advanced part of the build. Set up a working JVM build first ([Project setup](https://stoatflow.io/docs/getting-started/project-setup)); for most deployments the [JVM container image](https://stoatflow.io/docs/runtime/docker) is the simpler, better-trodden path. Reach for native when cold-start time or per-instance memory is the binding constraint.

::tldr-panel
- **Opt in:** Gradle `nativeImage { enabled.set(true) }` · Maven activate `-Pstoatflow-native`.
- **Build it:** the **containerized** build is recommended — `./gradlew nativeDockerBuild` / `mvn stoatflow:native-docker-build`. It pins the build glibc, wires the GC + reproducible epoch, and bakes the container-aware heap entrypoint.
- **Use Oracle GraalVM + G1 for any stateful (RocksDB) workload** — Community Edition ships Serial GC only, which can't carry an allocation-heavy state path. The containerized build already uses the Oracle GraalVM builder.
- **The RocksDB backend auto-tunes:** FFM on the JVM, JNI under native image (`AUTO`, the default) — each runtime gets its faster path with no config.
- **You add only your serde value classes** to the native metadata; the framework covers its own engine, Kafka client, RocksDB, and runtime surface.
::

## One-line opt-in

A StoatFlow app doesn't start from a blank GraalVM configuration. The framework bundles its native-image metadata, discovered automatically by classpath. Turning native image on applies GraalVM's build plugin, the curated `--initialize-at-run-time` flags RocksDB and the Kafka SASL client need, and the `nativeDockerBuild` task:

::code-tabs{group="build"}
```kotlin [Gradle]
// build.gradle.kts
stoatflow {
    mainClass.set("com.example.MainKt")
    nativeImage {
        enabled.set(true)
        gc.set("G1")        // Oracle GraalVM; omit for Serial GC (stateless only)
    }
}
```

```xml [Maven]
<!-- pom.xml — native config lives in <properties>; activate with -Pstoatflow-native -->
<properties>
  <stoatflow.mainClass>com.example.Main</stoatflow.mainClass>
  <stoatflow.nativeImage.gc>G1</stoatflow.nativeImage.gc>
</properties>
```
::

## Build it

The **containerized build** is the recommended path for any image you'll actually run. It compiles the native binary *inside Docker* from your prebuilt fat JAR, against a generic, repo-independent `Dockerfile.native` the build stages for you — so it pins the build-host glibc, wires the GC and the reproducible `SOURCE_DATE_EPOCH`, stages the heap entrypoint, and applies the `runtimeTarget`:

::code-tabs{group="build"}
```bash [Gradle]
./gradlew nativeDockerBuild
# → <imageName>:native  +  <imageName>:<version>-native

# Quick local smoke-compile (host GraalVM, no Docker):
./gradlew nativeCompile
```

```bash [Maven]
mvn package                          # build the fat JAR first
mvn stoatflow:native-docker-build
# → <imageName>:native  +  <imageName>:<version>-native

# Quick local smoke-compile (host GraalVM, no Docker):
mvn -Pstoatflow-native package
```
::

The containerized build stages a build context (`build/stoatflow-native/` for Gradle) holding the prebuilt `app.jar`, the shared `native-image.args` argfile, the heap entrypoint, and a `pgo/` directory, then runs `native-image @native-image.args …` against the bundled `Dockerfile.native`. Detected platform: `linux/arm64` on Apple Silicon, otherwise `linux/amd64`.

The pipeline, stage by stage:

```mermaid
flowchart LR
    SRC["your app +<br/>nativeImage.enabled"] --> JAR["fat JAR"]
    JAR --> CTX["staged build context —<br/>app.jar · native-image.args ·<br/>heap entrypoint · pgo/"]
    CTX --> NI["docker build —<br/>native-image @native-image.args<br/>on the Oracle GraalVM builder"]
    NI --> IMG["distroless image —<br/>runtime (heap entrypoint)<br/>or runtime-lean"]
```

::callout{color="info" icon="i-lucide-info"}
**Why build in Docker?** A native binary links the build host's glibc and needs that version (or newer) at runtime. The bundled `Dockerfile.native` builds on a pinned **Oracle GraalVM** image (`native-image:25-ol9`, Oracle Linux 9 / glibc 2.34) and runs on a distroless `cc-debian12` base (glibc 2.36), so the binary doesn't crash-loop on a `GLIBC_2.xx not found` mismatch — and you get G1 + PGO, which Community Edition lacks. A raw host `nativeCompile` is best treated as a *does-it-compile* smoke check.
::

::callout{color="warning" icon="i-lucide-clock"}
Native compilation is resource-hungry: budget **8 GB+ RAM** and **15–20 minutes**. The containerized build needs only Docker — no local GraalVM install.
::

### Runtime target

`runtimeTarget` selects the final image stage:

- **`runtime`** (default) — a `cc-debian12:debug` distroless base with a busybox shell so the **container-aware heap entrypoint** can size `-Xmx` from the container memory limit, reserving `STOATFLOW_ROCKSDB_MB` for RocksDB off-heap (recommended).
- **`runtime-lean`** — a minimal `cc-debian12` distroless base, no shell, no entrypoint: SubstrateVM sizes the heap itself. Smallest possible image.

::code-tabs{group="build"}
```kotlin [Gradle]
stoatflow { nativeImage { runtimeTarget.set("runtime-lean") } }
```

```xml [Maven]
<properties>
  <stoatflow.nativeImage.runtimeTarget>runtime-lean</stoatflow.nativeImage.runtimeTarget>
</properties>
```
::

## The garbage collector matters most

The single most important native-image decision is the GC — far more than the foreign-function path or anything exotic.

- **GraalVM Community Edition ships only Serial GC** for native image. Serial is fine for a *stateless* transform, but it **cannot carry an allocation-heavy stateful (RocksDB) workload** — under load a stateful app on Serial falls behind the input and never catches up (we measured a 112× end-to-end latency blowup).
- **Oracle GraalVM** (free for production under the GFTC licence, and what the containerized build already uses) unlocks **G1** and **PGO**. Switching Serial → G1 — one flag — takes a stateful app straight back to JVM parity.

The rule: &#x2A;*stateless → Serial is acceptable; stateful → use G1.** Set `gc` to `G1` and the containerized build passes it through to `native-image`.

::callout{color="info" icon="i-lucide-info"}
**Memory is tunable, not free.** Under Serial GC a native image is tiny. Under G1 the heap reservation is sized from `-Xmx`, and the container entrypoint hands the JVM most of the container — so a *tiny-heap stateless* app can use more RSS than on the JVM. For those, pin a tight `-Xmx` (an env override, no rebuild) to recover the win. Stateful apps, whose live footprint is larger, already land **below** the JVM's total memory. See [Tuning](https://stoatflow.io/docs/operating/tuning) for how GC interacts with commit cadence.
::

## The RocksDB backend: `AUTO` (FFM ⇄ JNI)

StoatFlow talks to RocksDB's C API through two interchangeable backends, and the faster one **inverts** between runtimes:

- **On the JVM, FFM wins.** The JIT intrinsifies its `critical(false)` downcalls down to nearly a direct C call.
- **Under native image, JNI wins.** SubstrateVM has no JIT, so that intrinsification never fires and a plain JNI call ends up cheaper (\~8% less CPU, better P99 in our measurements).

So the default backend is &#x2A;*`AUTO`**: FFM on the JVM, JNI under native image. **You don't configure anything** — each runtime is routed to its faster path automatically, and the brittle FFM-under-native machinery is off the default path entirely.

## Where reachability metadata lives

GraalVM's closed-world analysis needs metadata for everything reached via reflection, JNI, resources, and FFM foreign downcalls. StoatFlow ships this metadata **inside the module JARs** — not in per-app config files:

| Module              | Metadata path                                 | Covers                                                                                                                     |
| ------------------- | --------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| `stoatflow-core`    | `META-INF/native-image/io.stoatflow/core/`    | Kafka serde reflection, the Kafka client, RocksDB (both the JNI bindings **and** the FFM foreign downcalls), JCTools, zstd |
| `stoatflow-runtime` | `META-INF/native-image/io.stoatflow/runtime/` | the YAML config binding, Logback, Micrometer/HdrHistogram, and the rest of the runtime stack                               |

Any app that depends on `stoatflow-runtime` (and transitively `stoatflow-core`) inherits the correct metadata automatically through GraalVM's classpath discovery. You copy or generate **nothing** for the StoatFlow surface — only for your own types, next.

## Custom serde metadata

The closed-world compiler still needs to know about **your** types — the value classes you serialize/deserialize — because anything reflective, JNI, or FFM must be declared up front, or it fails the first time that code path runs (never at build time).

The reliable way to generate it is the **GraalVM tracing agent**: run your app (or its tests) on the JVM with the agent attached, exercise every code path, and it writes the metadata. Drop the result under your module's resources so GraalVM discovers it by classpath:

```bash
# Run your app/tests on the JVM with the tracing agent, driving real traffic:
java -agentlib:native-image-agent=config-output-dir=src/main/resources/META-INF/native-image/com.example/word-count \
     --enable-preview --enable-native-access=ALL-UNNAMED -jar build/libs/word-count-all.jar
```

::callout{color="warning" icon="i-lucide-triangle-alert"}
**Drive every path before you trust the binary.** Closed-world compilation moves failures from build time to *first-record* time. A `MissingReflectionRegistrationError` only surfaces when its code path first executes — so exercise your serdes, punctuators, joins, and error paths under the agent.
::

## Snares & gotchas

The discoveries that cost an afternoon each:

- **Avro + Schema Registry is the worst offender.** Confluent's stack instantiates classes by name from its own config defaults (e.g. `NullContextNameStrategy`), Jackson-deserializes the registry's REST entities, and native image disables URL protocols by default — so the schema fetch fails until you add `--enable-url-protocols=http,https`. The tracing agent catches the config-default classes in one pass; it's the surer route for anything Avro.
- **`reachability-metadata.json` shadows the legacy `reflection-config.json`.** When both are present the newer format wins and silently drops the old file's registrations. Keep your metadata in the newer `reachability-metadata.json`; don't run the two side by side.
- **PGO profiles only dump on clean shutdown.** A streaming app runs forever, so extracting a `.iprof` takes plumbing — the bundled native entrypoint forwards `SIGTERM` to the child process so `kill -TERM 1` drains and dumps the profile. (See the [deep-dive](https://stoatflow.io/blog/native-image-g1-pgo-jni-vs-ffm).)

## Profile-Guided Optimisation (PGO)

PGO is the CPU lever — a release-time, two-pass build (compile an instrumented binary, run it under representative load to collect a `.iprof`, rebuild with the profile). It's purely a CPU optimisation (latency/throughput are already at parity under G1) and trims the residual AOT-vs-JIT gap by another \~13–17%. It's **Oracle-GraalVM-only** and adds real build time, so treat it as a release step, not a per-change one. The build context includes a `pgo/` directory where the collected profile is staged. The full recipe — instrumenting, extracting the profile from a long-running server, and rebuilding — is in the [engineering deep-dive](https://stoatflow.io/blog/native-image-g1-pgo-jni-vs-ffm).

## What the build passes to native-image

The static flags come from a single bundled `native-image.args` (identical for Gradle, Maven, and the containerized build); `--gc` and PGO are layered on dynamically. You write none of these by hand:

- `-H:+JNI`, `-H:+ReportExceptionStackTraces`, `-H:+AddAllCharsets`, the Kotlin resource includes, `--enable-preview`, `--no-fallback`.
- `--initialize-at-run-time` for `org.rocksdb`, `org.rocksdb.RocksDB`, `org.rocksdb.util.Environment`, the FFM RocksDB bindings, and the Kafka SASL client authenticator — these load native libraries or read runtime state and must initialize when the binary starts, not when it's compiled.

If your own code (or a dependency) needs run-time initialization, add it via `additionalInitializeAtRunTime` (Gradle) / extend the `native-maven-plugin` `<buildArgs>` (Maven); drop a curated entry with `removeInitializeAtRunTime`. The full enumeration is in the [Gradle plugin reference](https://stoatflow.io/docs/reference/gradle-plugin-reference#native-image-defaults). The typical trigger is a build-time error: `Classes that should be initialized at run time got initialized during image building` — add the named class and rebuild.

## Verify it runs

After building, run the image and check the same health and metrics endpoints as the JVM runtime:

```bash
docker run -d --name sf-native \
  -e KAFKA_BOOTSTRAP_SERVERS=host.docker.internal:9092 \
  -e STOATFLOW_LICENSE_KEY="key/...your key..." \
  -p 8080:8080 \
  my-app:native

curl -s localhost:8080/health/ready
curl -s localhost:8080/info
```

If the process exits at startup with `Classes that should be initialized at run time got initialized during image building` or a `MissingReflectionRegistrationError`, the fix is metadata: add the class to [`additionalInitializeAtRunTime`](https://stoatflow.io/#what-the-build-passes-to-native-image), or register the reflective access via the tracing agent, then rebuild.

## What's covered — and what isn't yet

**Covered today*&#x2A; (benchmarked under load): String, Protobuf, and Avro + Schema Registry serdes; stateless and stateful (RocksDB) topologies; exactly-once. With Oracle GraalVM + G1 + PGO, the apps we measured run within **−4% to +10% CPU** of the JVM, **latency equal-or-better everywhere**, and **memory below the JVM on the stateful apps**.

**Not yet mapped** — these pull in reflection/JNI surface that still needs native metadata:

- **Kafka client security** — SASL (SCRAM, GSSAPI/Kerberos), OAuth bearer, and mTLS.
- **Compression codecs beyond zstd** — snappy, lz4, gzip.

If you hit a `MissingReflectionRegistrationError` on one of these, the tracing agent is your first move — and [let us know](https://stoatflow.io/contact), so we can fold the metadata into the framework.

## Knob reference

| Capability                                | Gradle (`stoatflow { nativeImage { … } }`) | Maven (`<properties>` / build)                                 |
| ----------------------------------------- | ------------------------------------------ | -------------------------------------------------------------- |
| Enable                                    | `enabled.set(true)`                        | activate `-Pstoatflow-native`                                  |
| Garbage collector                         | `gc.set("G1")`                             | `<stoatflow.nativeImage.gc>G1</…>`                             |
| Runtime image stage                       | `runtimeTarget.set("runtime-lean")`        | `<stoatflow.nativeImage.runtimeTarget>…</…>`                   |
| Add `--initialize-at-run-time`            | `additionalInitializeAtRunTime.add("…")`   | extend the `native-maven-plugin` `<buildArgs>`                 |
| Drop a curated `--initialize-at-run-time` | `removeInitializeAtRunTime.add("…")`       | `<stoatflow.nativeImage.removeInitializeAtRunTime>…</…>` (CSV) |

## Further reading

- **[The engineering deep-dive](https://stoatflow.io/blog/native-image-g1-pgo-jni-vs-ffm)** — G1, PGO, why JNI beats FFM under AOT, and the full benchmark matrix.
- **[Benchmarks](https://stoatflow.io/product/benchmarks)** — the numbers behind the claims here.
- **[Gradle plugin reference](https://stoatflow.io/docs/reference/gradle-plugin-reference)*&#x2A; · &#x2A;*[Maven reference](https://stoatflow.io/docs/reference/maven-reference)** — every native knob, task, and default.
- **[Docker](https://stoatflow.io/docs/runtime/docker)** — the JVM container image (Jib), the simpler default deployment target.
- **[Kubernetes](https://stoatflow.io/docs/operating/kubernetes)*&#x2A; and &#x2A;*[Probes](https://stoatflow.io/docs/operating/probes)** — deploying the image and wiring liveness/readiness.


# Deploying and operating

This section is for the people who run StoatFlow in production: deploying it, sizing it, probing it, and keeping it healthy. The model is unusual in one way that shapes everything else — a StoatFlow application has exactly one **active** process. Most of operating it follows from that.

::tldr-panel
- **Deployment unit:** one JVM process. Exactly one *active* instance — never two processing the same source topics.
- **Why one active:** two active instances writing the same state corrupts it. The [Architecture](https://stoatflow.io/docs/concepts/architecture) page sets out the model in full.
- **High availability:** fast restart by default, or an opt-in [hot-standby cluster](https://stoatflow.io/docs/operating/high-availability) — one or more warm *passive* standbys — for near-instant, lag-aware failover. Choose per application.
- **What's here:** Kubernetes deployment, high availability, liveness/readiness probes, observability, tuning, and a production checklist.
::

## The deployment model

A StoatFlow application runs as exactly one JVM process. That process opens a Kafka consumer group with a single member — itself — and that member is assigned every partition of every source topic the topology reads from. There is no cluster, no scheduler, no worker pool, and no second instance sharing the load.

For an operator, this collapses a lot of moving parts. You deploy one pod, not a StatefulSet of N. There is no rebalancing to wait out on a rolling restart, no partition reassignment to reason about, no inter-instance coordination to monitor. The unit you scale, probe, restart, and observe is a single process.

You scale it **vertically** — more CPU and memory for the one process — not horizontally by adding replicas. Processing parallelism inside the process comes from lanes running on virtual threads, and lane count is decoupled from the partition count of your input topics. The full conceptual model is on [Architecture](https://stoatflow.io/docs/concepts/architecture); how lanes turn one process into many concurrent units of work is on [Lanes and parallelism](https://stoatflow.io/docs/concepts/lanes-and-parallelism).

## One active instance

This is the one rule that an operator must not break: &#x2A;*run exactly one *active* instance.** By default that means exactly one replica — not zero, not two.

Two *active* instances of the same application, pointed at the same source topics and writing the same state, will corrupt that state. By default StoatFlow runs a single process with no second instance to coordinate with, and the exactly-once commit protocol is built around that assumption. The one supported way to run more than one instance is the opt-in [hot-standby cluster](https://stoatflow.io/docs/operating/high-availability): the extra instances stay **passive** standbys — each follows the changelog and keeps its state warm, but never processes or commits — and the cluster fences a single active so that, under exactly-once, split-brain is impossible. Outside that mode, never run more than one.

Kafka's consumer-group semantics make the default-mode failure concrete. If you did start a second instance in the same group, Kafka would assign all partitions to one member and idle the other — at best you'd have a hot spare doing nothing useful, at worst a flapping reassignment if both keep trying to claim partitions. And because each instance also holds its own local state and writes its own changelog, two instances racing on the same changelog topics and the same transactional output produce interleaved, inconsistent writes that neither instance can reconcile. The result is corrupted state, not graceful degradation.

::callout{color="error" icon="i-lucide-alert-triangle"}
**Never run two *active* replicas of the same StoatFlow application against the same source topics.** With the default `ha.mode: off`, set your Kubernetes Deployment to `replicas: 1` and a rollout strategy where a new pod fully replaces the old one — not one that briefly runs both. For a hot-standby cluster, use [hot standby](https://stoatflow.io/docs/operating/high-availability) (`replicas: 2` or more, `ha.mode: active-standby`), which keeps the standbys passive. The [Kubernetes](https://stoatflow.io/docs/operating/kubernetes) page shows the exact manifest.
::

The practical consequences for an operator are small and specific:

- **Set `replicas: 1`** on the Deployment under the default mode, and use a rollout strategy that tears the old pod down before — or as — the new one comes up, so the two never process concurrently. (Hot standby is the exception: a readiness-gated cluster of `replicas: 2` or more — see [High availability](https://stoatflow.io/docs/operating/high-availability).) See [Kubernetes](https://stoatflow.io/docs/operating/kubernetes).
- **Don't run a manual instance against production topics** while the deployed one is live — for a debug session, point a throwaway instance at a different consumer group and, ideally, copies of the topics.
- **Treat the application id as identity.** Two processes sharing the `application-id` share a consumer group and changelog topics; that's the collision to avoid.

## High availability

StoatFlow offers two HA tiers, and you choose per application. Neither involves two *active* instances — that remains the rule.

- **Fast restart (the default).** The one instance comes back fast after it goes down, with no data loss and no duplicate output. Recovery is a restart plus a changelog restore. This is the right answer for most workloads; the reasoning and the trade-offs are on [Motivation](https://stoatflow.io/product/motivation).
- **Hot standby (opt-in).** One active plus one or more warm *passive* standbys that continuously follow the changelog; on failover a lag-aware election promotes the freshest in seconds — independent of state size. Choose it for large state or a tight recovery-time objective where a cold restore is too slow. The full how-to is on [High availability](https://stoatflow.io/docs/operating/high-availability).

The rest of this section describes the default, fast-restart tier. Three properties make fast restart a credible availability story rather than a downgrade:

- **No rebalancing tax.** A standard multi-instance stream processor pays a rebalance whenever an instance joins or leaves — partitions stop, get reassigned, and resume. With one instance and one group member, there is nothing to rebalance. Restart cost is restoration cost, full stop.
- **Bounded, parallel state recovery.** On restart, local state rebuilds from the Kafka changelog topics, and restoration runs in parallel across stores rather than one at a time. Restart time scales with state size; see [Benchmarks](https://stoatflow.io/product/benchmarks) for measured cold-start numbers on representative workloads.
- **Exactly-once across the restart.** The last successful commit barrier is the recovery point. Work the crashed process hadn't committed was aborted at the broker, so a restart resumes from the last committed offsets with no duplicate output downstream and no lost state. The mechanism is on [Exactly-once](https://stoatflow.io/docs/concepts/exactly-once).

For an operator this means availability is an orchestration concern, not an application-clustering one. Let Kubernetes restart the pod on failure; let the readiness probe hold traffic off until state has recovered. The probe contract is the lever that makes this clean — `/health/ready` returns 503 throughout restoration, so nothing routes to a process that isn't caught up yet. See [Probes](https://stoatflow.io/docs/operating/probes).

If your tolerance for restart-window downtime is tight, the levers are state size (smaller state restores faster), disk persistence (a warm local store on a persistent volume means restoration only reads the changelog gap since the last commit, not the whole topic), and broker read throughput. [Tuning](https://stoatflow.io/docs/operating/tuning) covers these.

## The operating pages

::path-cards
  :::path-card
  ---
  icon: i-lucide-ship
  title: Kubernetes
  to: https://stoatflow.io/docs/operating/kubernetes
  ---
  The single-replica StatefulSet manifest, rollout strategy, resource requests, persistent volumes for warm restart, and graceful shutdown on `SIGTERM`.
  :::

  :::path-card
  ---
  icon: i-lucide-copy
  title: High availability
  to: https://stoatflow.io/docs/operating/high-availability
  ---
  The two HA tiers, the opt-in hot-standby cluster (one active + one or more warm standbys), the exactly-once-versus-at-least-once failover trade-off, and the multi-replica StatefulSet that deploys it.
  :::

  :::path-card
  ---
  icon: i-lucide-heart-pulse
  title: Probes
  to: https://stoatflow.io/docs/operating/probes
  ---
  Liveness versus readiness, how readiness holds traffic off during state restoration, and how to wire the `/health/live` and `/health/ready` endpoints into an orchestrator.
  :::

  :::path-card
  ---
  icon: i-lucide-activity
  title: Observability
  to: https://stoatflow.io/docs/operating/observability
  ---
  The Prometheus metrics surface, the introspection and debug endpoints, structured logs, and what to alert on for a single-instance application.
  :::

  :::path-card
  ---
  icon: i-lucide-sliders-horizontal
  title: Tuning
  to: https://stoatflow.io/docs/operating/tuning
  ---
  Sizing CPU and memory, lane count, the commit-cadence bounds, and the levers that trade latency against throughput and restart time.
  :::

  :::path-card
  ---
  icon: i-lucide-clipboard-check
  title: Production checklist
  to: https://stoatflow.io/docs/operating/production-checklist
  ---
  The pre-flight list before going live — replica count, probes, resources, persistence, license, metrics, and the failure policies you've chosen.
  :::
::

## Where to go next

- **[Architecture](https://stoatflow.io/docs/concepts/architecture)** — the single-instance model in full: data flow, lanes, state, commit barriers, and the operational surface.
- **[The runtime](https://stoatflow.io/docs/runtime)** — the batteries-included `:runtime` module that exposes the health, metrics, and admin endpoints operators rely on.
- **[Kubernetes](https://stoatflow.io/docs/operating/kubernetes)** — start here when you're ready to deploy.
- **[High availability](https://stoatflow.io/docs/operating/high-availability)** — the opt-in hot-standby cluster, for when fast restart isn't fast enough.
- **[Motivation](https://stoatflow.io/product/motivation)** — why StoatFlow is single-active-instance, and the trade-offs that come with it.


# Running on Kubernetes

StoatFlow runs as exactly one active JVM process, so on Kubernetes you deploy it as a **single-replica StatefulSet** — not a Deployment. A StatefulSet gives you a stable identity, a per-pod persistent volume for the state directory, and an update strategy that tears the old pod down before the new one starts, which is exactly what running [one active instance](https://stoatflow.io/docs/operating#one-active-instance) requires. (For a hot-standby cluster, see [High availability](https://stoatflow.io/docs/operating/high-availability) — a readiness-gated StatefulSet of two or more replicas.)

::tldr-panel
- **Workload:** `StatefulSet` with `replicas: 1`. Never a `Deployment`, never `replicas > 1` — except the opt-in [hot-standby](https://stoatflow.io/docs/operating/high-availability) cluster (`replicas: 2` or more, `ha.mode: active-standby`).
- **State:** a `volumeClaimTemplates` PVC mounted at the state directory; set `stoatflow.state.dir` to that path.
- **Shutdown:** `terminationGracePeriodSeconds` ≥ your `stoatflow.shutdown.timeout-ms` (plus margin) so the final commit barrier completes.
- **License:** inject the key from a Kubernetes `Secret` — as `STOATFLOW_LICENSE_KEY` env, or as a mounted file. Never bake it into the image or a ConfigMap.
- **Probes:** `/health/live` and `/health/ready` on the HTTP port — see [Probes](https://stoatflow.io/docs/operating/probes).
::

## Why a StatefulSet, not a Deployment

A `Deployment` with `replicas: 1` and the default `RollingUpdate` strategy will, during a rollout, **start the new pod before terminating the old one** (`maxSurge: 1`). For a moment two processes are alive and pointed at the same source topics — which is precisely the [data-corruption scenario](https://stoatflow.io/docs/operating#one-active-instance) the engine cannot tolerate. (Hot standby is the supported way to run more than one replica: the standbys are *passive*, and a `RollingUpdate` is readiness-gated so they never process concurrently — see [High availability](https://stoatflow.io/docs/operating/high-availability).)

A `StatefulSet` with `replicas: 1` does the opposite: on an update it **terminates the old pod and waits for it to fully stop before creating the replacement**. Combined with `podManagementPolicy: OrderedReady`, you get an at-most-one-running guarantee across rollouts. That, plus a stable PVC for the state directory, is why StatefulSet is the right primitive here.

::callout{color="error" icon="i-lucide-alert-triangle"}
Do not deploy StoatFlow as a `Deployment`, and never set `replicas` above `1` — unless you are running the opt-in [hot-standby](https://stoatflow.io/docs/operating/high-availability) cluster (`replicas: 2` or more, `ha.mode: active-standby`), which keeps the standbys passive. Two *active* instances against the same source topics corrupt state. See [one active instance](https://stoatflow.io/docs/operating#one-active-instance).
::

## The state volume

State stores are RocksDB-backed on local disk under `stoatflow.state.dir` (default `${java.io.tmpdir}/stoatflow`). The state directory is **durable, not authoritative** — the changelog topics are the source of truth, and on restart the runtime rebuilds from them. A persistent volume doesn't change correctness; it changes **restart speed**. With a surviving state directory the runtime replays only the changelog records since the last commit (a delta restore) instead of reading the whole changelog from scratch.

Mount a PVC at a fixed path and point `stoatflow.state.dir` at it:

```yaml
# In the StatefulSet pod spec
volumeMounts:
  - name: state
    mountPath: /var/lib/stoatflow/state
```

```yaml
# In the StatefulSet spec
volumeClaimTemplates:
  - metadata:
      name: state
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: fast-ssd   # SSD/NVMe — RocksDB is latency-sensitive
      resources:
        requests:
          storage: 100Gi
```

Then set the directory to that mount, either in `application.yaml`:

```yaml
stoatflow:
  state:
    dir: /var/lib/stoatflow/state
```

…or via the env-var override (double underscore separates path segments):

```bash
STOATFLOW__STATE__DIR=/var/lib/stoatflow/state
```

::callout{color="info" icon="i-lucide-hard-drive"}
Size the PVC for your **state size plus RocksDB working headroom** (compaction, SST overhead) — not for the changelog. Use a fast `storageClassName` (local SSD or NVMe-backed): RocksDB read/write latency feeds directly into commit-barrier duration. See [Resource sizing](https://stoatflow.io/#resource-sizing).
::

## Graceful shutdown and the grace period

On `SIGTERM` (which Kubernetes sends on pod termination) the runtime stops accepting new records, drains in-flight work, injects one final commit barrier, commits it, and closes the consumer and producer cleanly. That final commit is what keeps the shutdown exactly-once — interrupting it just means the next start resumes from the previous barrier, but letting it finish avoids re-processing on restart.

The runtime bounds its own graceful shutdown with `stoatflow.shutdown.timeout-ms` (default `25000`). Kubernetes bounds it independently with `terminationGracePeriodSeconds` (default `30`): when that elapses, Kubernetes sends `SIGKILL` regardless of progress.

**Align the two so Kubernetes doesn't kill a still-progressing shutdown.** Set `terminationGracePeriodSeconds` comfortably above your shutdown timeout in seconds:

```yaml
spec:
  template:
    spec:
      terminationGracePeriodSeconds: 60   # > stoatflow.shutdown.timeout-ms / 1000, with margin
```

```yaml
stoatflow:
  shutdown:
    timeout-ms: 25000   # default; raise for very large in-flight epochs
```

If your topology runs large epochs or slow stateful commits, raise **both** together — `stoatflow.shutdown.timeout-ms` first, then `terminationGracePeriodSeconds` so it stays larger. The runtime also has a hard-exit fallback that guarantees a clean process exit if shutdown itself hangs on a stuck lock (`stoatflow.shutdown.hard-exit-on-shutdown-hang`, default `true`); leave it on so Kubernetes always observes a clean termination rather than relying solely on `SIGKILL`. See [Configuration reference](https://stoatflow.io/docs/reference/configuration-reference) for the full shutdown key set.

## Secure license injection

StoatFlow requires the license key to be **present** at startup (the activating validation itself runs \~5 minutes after start, on a background thread), so the pod needs the key — but the key is a secret and must never land in the image, a ConfigMap, or source control. Store it in a Kubernetes `Secret` and surface it one of two ways. Both map onto the [license resolution order](https://stoatflow.io/docs/getting-started/license-configuration): an explicit key (env var) beats a key file.

First, create the Secret:

```bash
kubectl create secret generic stoatflow-license \
  --from-literal=license-key="key/...your production key..."
```

### Option A — Secret as an environment variable

The simplest approach: project the secret value into `STOATFLOW_LICENSE_KEY`. The runtime reads the env var directly.

```yaml
# In the container spec
env:
  - name: STOATFLOW_LICENSE_KEY
    valueFrom:
      secretKeyRef:
        name: stoatflow-license
        key: license-key
  # Production REQUIRES an explicit environment label — set it per deployment:
  - name: STOATFLOW_LICENSE_ENVIRONMENT
    value: prod-eu-west
```

### Option B — Secret mounted as a file

Mount the secret as a file and point `STOATFLOW_LICENSE_FILE` at it. Useful when your platform standardises on mounted secrets over env vars.

```yaml
# In the container spec
env:
  - name: STOATFLOW_LICENSE_FILE
    value: /etc/stoatflow/license/license.key
  - name: STOATFLOW_LICENSE_ENVIRONMENT
    value: prod-eu-west
volumeMounts:
  - name: license
    mountPath: /etc/stoatflow/license
    readOnly: true
```

```yaml
# In the pod spec
volumes:
  - name: license
    secret:
      secretName: stoatflow-license
      items:
        - key: license-key
          path: license.key
      defaultMode: 0400   # owner read-only
```

::callout{color="warning" icon="i-lucide-key-round"}
**Production requires an explicit environment label.** The Production tier refuses to run if it has to auto-generate one (the refusal fires at the deferred validation, \~5 minutes after start) — set `STOATFLOW_LICENSE_ENVIRONMENT` (e.g. `prod`, `prod-eu-west`) per deployment so each gets its own node-locked seat. A mounted key file must be readable only by the runtime user (`defaultMode: 0400`); the runtime refuses an insecurely-permissioned file. Full details on [License configuration](https://stoatflow.io/docs/getting-started/license-configuration).
::

## Resource sizing

StoatFlow scales vertically — give the one process the CPU, memory, and disk it needs. As a starting point:

| Workload                                  | CPU      | Memory | Disk        |
| ----------------------------------------- | -------- | ------ | ----------- |
| Light (\~10K events/sec, \~1 GB state)    | 4 cores  | 8 GB   | 20 GB SSD   |
| Medium (\~100K events/sec, \~10 GB state) | 16 cores | 64 GB  | 100 GB SSD  |
| Heavy (\~500K events/sec, \~50 GB state)  | 32 cores | 128 GB | 500 GB NVMe |

Memory splits roughly into JVM heap (state cache), off-heap (RocksDB block cache, write buffers, producer buffers), and OS overhead — budget for all three, not just the heap. Set container `requests` equal to `limits` for the **Guaranteed** QoS class, so the kernel doesn't reclaim memory from the process under node pressure. These are starting points; [Tuning](https://stoatflow.io/docs/operating/tuning) covers how to refine lane count, commit cadence, and memory caps against your workload.

## A representative manifest

A complete single-replica StatefulSet: ConfigMap for `application.yaml`, Secret-injected license, persistent state volume, grace period aligned with shutdown, and both health probes. Adjust image, topic/broker config, resources, and the license-injection option to your environment.

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: stoatflow-config
data:
  application.yaml: |
    stoatflow:
      application-id: my-stream-app
      bootstrap-servers: kafka-broker:9092
      state:
        dir: /var/lib/stoatflow/state
      shutdown:
        timeout-ms: 25000
      license:
        # The literal key is injected from a Secret via STOATFLOW_LICENSE_KEY (below);
        # the explicit env var wins over anything in this file.
        environment: prod-eu-west
    runtime:
      http:
        enabled: true
        port: 8080
      metrics:
        enabled: true
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: my-stream-app
spec:
  replicas: 1                      # exactly one — never more
  serviceName: my-stream-app
  podManagementPolicy: OrderedReady
  updateStrategy:
    type: RollingUpdate            # StatefulSet rolling update stops the old pod before starting the new
  selector:
    matchLabels:
      app: my-stream-app
  template:
    metadata:
      labels:
        app: my-stream-app
    spec:
      terminationGracePeriodSeconds: 60   # > stoatflow.shutdown.timeout-ms / 1000, with margin
      containers:
        - name: app
          image: my-registry/my-stream-app:latest
          ports:
            - name: http
              containerPort: 8080
          env:
            - name: STOATFLOW_LICENSE_KEY
              valueFrom:
                secretKeyRef:
                  name: stoatflow-license
                  key: license-key
            # application.yaml is loaded from the mounted ConfigMap:
            - name: STOATFLOW_CONFIG_FILES
              value: /etc/stoatflow/application.yaml
          readinessProbe:
            httpGet:
              path: /health/ready
              port: http
            initialDelaySeconds: 30
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health/live
              port: http
            initialDelaySeconds: 60
            periodSeconds: 30
          resources:
            requests:
              cpu: "16"
              memory: 64Gi
            limits:
              cpu: "16"
              memory: 64Gi          # match requests → Guaranteed QoS
          volumeMounts:
            - name: config
              mountPath: /etc/stoatflow
              readOnly: true
            - name: state
              mountPath: /var/lib/stoatflow/state
      volumes:
        - name: config
          configMap:
            name: stoatflow-config
  volumeClaimTemplates:
    - metadata:
        name: state
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: fast-ssd
        resources:
          requests:
            storage: 100Gi
```

::callout{color="info" icon="i-lucide-info"}
`STOATFLOW_CONFIG_FILES` points the runtime at the mounted `application.yaml`. The container image is the one built by the StoatFlow build conventions (Jib, on Gradle or Maven) — it already sets the required JVM flags (`--enable-preview`, `--enable-native-access=ALL-UNNAMED`) and runs as a non-root user. See [Docker](https://stoatflow.io/docs/runtime/docker) for image build details.
::

## Verify the rollout

After applying the manifests, confirm the single pod comes up and reports ready:

```bash
kubectl rollout status statefulset/my-stream-app
kubectl get pods -l app=my-stream-app          # exactly one pod, 1/1 Running

# Port-forward the HTTP port and check readiness:
kubectl port-forward statefulset/my-stream-app 8080:8080 &
curl -s localhost:8080/health/ready            # {"status":"UP", ...} once state restoration completes
```

During cold start `/health/ready` returns `503` until state restoration finishes — that's expected and is exactly why the readiness probe gates traffic. On a rolling update, watch that the old pod reaches `Terminating` and stops **before** the replacement is created; the StatefulSet guarantees this ordering, but it's worth confirming on first deploy.

The rollout you should observe, end to end — for the multi-replica hot-standby roll, where readiness additionally enforces the redundancy floor, see [High availability](https://stoatflow.io/docs/operating/high-availability#how-readiness-gates-a-rolling-deploy):

```mermaid
sequenceDiagram
    autonumber
    participant K as StatefulSet controller
    participant O as Old pod
    participant N as New pod

    K->>O: SIGTERM
    O->>O: drain in-flight work, commit final barrier
    O-->>K: terminated (fully stopped)
    note over K: only now is the<br/>replacement created
    K->>N: create pod
    N->>N: restore state (delta replay<br/>if the PVC survived)
    note over N: /health/ready 503<br/>while restoring
    N-->>K: ready — rollout complete
```

## Next steps

- **[High availability](https://stoatflow.io/docs/operating/high-availability)** — the opt-in hot-standby cluster: a multi-replica StatefulSet for near-instant failover.
- **[Probes](https://stoatflow.io/docs/operating/probes)** — liveness vs. readiness, the JSON each returns, and how to tune probe timings.
- **[Observability](https://stoatflow.io/docs/operating/observability)** — scraping `/metrics`, the key metrics to alert on, and structured logs.
- **[Tuning](https://stoatflow.io/docs/operating/tuning)** — lane count, commit cadence, and memory caps for your workload.
- **[Production checklist](https://stoatflow.io/docs/operating/production-checklist)** — everything to verify before going live.
- **[License configuration](https://stoatflow.io/docs/getting-started/license-configuration)** — the full license reference, including CI/CD.


# High availability

StoatFlow gives you two ways to stay available, and you choose per application. By default an application is a single instance that recovers by **restarting** — Kubernetes brings the pod back and state rebuilds from the changelog. For workloads that can't absorb that restart window, you can opt into a **hot standby**: a cluster of one active instance plus one or more warm standbys, where a standby takes over in seconds. The single-instance default is unchanged; hot standby is an opt-in upgrade on top of it.

::tldr-panel
- **Two tiers:** fast restart (the default, one instance) or an opt-in hot standby (`stoatflow.ha.mode: active-standby`, one active plus one or more warm standbys).
- **Default is unchanged:** with `mode: off`, recovery is a restart plus a changelog restore. Most deployments need nothing more.
- **Hot standby:** each passive standby continuously follows the changelog and stays warm, so promotion is seconds — independent of how large your state is.
- **Still one active:** at most one instance processes and commits at a time; the standbys are passive. On failover a lag-aware election promotes the freshest one, and under exactly-once Kafka's transactional fencing makes split-brain impossible.
- **Shape:** a `StatefulSet` with two or more replicas and split health probes; a rolling deploy costs about one graceful handoff, and a graceful role swap no longer restarts a pod.
::

## The two HA tiers

| Tier                       | What you get                                                                                                                       | Failover time                                     | When to choose                                                                                |
| -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | --------------------------------------------------------------------------------------------- |
| **Fast restart** (default) | One instance. On failure Kubernetes restarts the pod and state rebuilds from the changelog.                                        | Restart + state restore — scales with state size. | Most workloads, where a restart-window gap is acceptable.                                     |
| **Hot standby** (opt-in)   | One active plus one or more warm standbys. Each standby follows the changelog; on failover the freshest is elected and takes over. | Seconds, independent of state size.               | Large state (tens of GB) or a tight recovery-time objective where a cold restore is too slow. |

Hot standby adds availability redundancy without changing the processing model, the exactly-once guarantees, or the single-active-instance shape. Everything the [Architecture](https://stoatflow.io/docs/concepts/architecture) page describes still holds; you're keeping one or more warm spares ready, nothing more. The simplest cluster is one active and one standby — a two-replica `StatefulSet`, the default; add standbys for more redundancy (see [`desired-standbys`](https://stoatflow.io/#configuration)).

::callout{color="info" icon="i-lucide-info"}
If a cold-restore window of seconds-to-minutes is acceptable for your workload, stay on the default. Hot standby exists to remove that window for large-state or tight-RTO applications — not because the single-instance default is unsafe. Stateless topologies gain nothing from it; use a plain rolling restart.
::

## How hot standby works

With `ha.mode: active-standby` you run two or more instances of the same application. At any moment one is the **active** — it owns the source partitions, processes records, and commits — and the rest are **passive standbys**. Each standby consumes the same changelog the active writes, keeping its own local state warm and a step behind, but it does not process source records or produce output. They coordinate through a compacted internal Kafka topic (auto-created; its name is configurable via `stoatflow.ha.coordination-topic`), exchanging a periodic heartbeat so each knows who is alive and which role they hold.

**How the standby stays warm.** A single changelog applier streams committed deltas into local state in real time — one consumer (in `assign()` mode, `read_committed` under exactly-once / `read_uncommitted` under at-least-once) and one virtual thread. It applies uniformly across RocksDB and in-memory stores and folds the derived-index rebuilds for foreign-key joins and suppression into the same loop, so the standby's state — primary stores and derived indices alike — tracks the active continuously rather than in periodic batches. Replication lag is read straight from that consumer's position, with no admin round-trip on the broker.

Promotion happens two ways:

- **Graceful handoff** — a rolling deploy, or an explicit `POST /ha/switch`. The active drains its in-flight work, commits, and hands the role over to an elected standby. This is an **in-place** role swap: the demoted active rebuilds its engine as a standby *inside the same pod* and rejoins warm — no process exit, no cold state open, no pod restart.
- **Failover** — the active stops heart-beating; the standbys notice, elect a successor, and it promotes itself.

Because the standby is already warm, taking over is a matter of seconds rather than a full cold restore. (A crash, a lost node, or a genuinely stuck restore still restarts the pod — the fence protects every path.)

The graceful path end to end — here triggered by an explicit `POST /ha/switch`; a rolling deploy runs the same sequence per pod:

```mermaid
sequenceDiagram
    autonumber
    participant O as Operator
    participant A as Pod A (active)
    participant B as Pod B (standby)
    participant K as Kafka

    O->>A: POST /ha/switch
    break no caught-up standby (and no force=true)
        A-->>O: 409 Conflict
    end
    A-->>O: 202 Accepted (commandOffset)
    A->>K: publish SWITCH on coordination topic
    A->>K: drain in-flight epoch, commit final barrier
    A->>A: rebuild engine as standby (in place, no pod restart)
    note over A: keeps heartbeating as<br/>ACTIVE (DRAINING)
    B->>K: claim promotion token (CAS)
    note right of B: allowed — a draining<br/>holder is unprotected
    B->>K: fence producer epoch
    B->>K: drain changelog tail to LSO (already warm — near zero)
    B->>K: heartbeat ACTIVE
    A->>A: sees peer ACTIVE — becomes STANDBY
```

**Drain-to-LSO before processing.** Whatever the trigger, a promoting pod first drains its applier to the committed end of the changelog — the log-stable-offset (LSO) — before it processes a single record. The departing active's producer is fenced eagerly, which freezes the LSO, so there is a fixed, finite target to drain to and no new committed deltas can appear past it. A caught-up standby drains almost nothing (it's already current); a standby that had fallen behind drains exactly the records it was missing — so it never resumes processing on stale state. This is what makes failover correct under exactly-once even when the standby was lagging at the moment the active died.

**Electing the successor.** With more than one standby, promotion is an election, not a free-for-all. Candidates are ranked by replication lag: a standby is eligible only while its total committed-changelog lag is at or below `acceptable-recovery-lag`, and the **freshest** eligible standby wins — an exact tie broken by a stable hash of the pod identity (nudged by `failover-priority`), so the choice always converges on exactly one. The winner claims a **promotion token** on the coordination topic *before* it fences the old active and restores; a standby that loses the claim simply **stands down** and stays a standby — no fence, no error, no restart. That ordering — elect and claim, *then* fence — is what stops two pods promoting at once in a multi-standby cluster. If the active is gone and no standby is within the threshold, the least-behind one is elected anyway after `promotion-grace-ms` (availability first) — still draining to the LSO before it processes. The producer-epoch fence stays the backstop: even if the election ever misfired, the broker still lets only one instance commit.

The same machinery under a crash — ungraceful failover from detection to takeover:

```mermaid
sequenceDiagram
    autonumber
    participant A as Pod A (active)
    participant B as Pod B (standby, freshest)
    participant C as Pod C (standby, behind)
    participant K as Kafka

    A--xK: heartbeats stop (crash)
    note over B,C: peer goes stale after the miss debounce<br/>(local-receipt freshness, not the sender's clock)
    B->>B: election — lag-ranked, freshest eligible wins
    C->>C: same election — not the winner, stands down
    B->>K: claim promotion token (CAS)
    K-->>B: token won
    B->>B: settle window — confirm sole claimant
    B->>K: fence producer epoch (freezes the LSO)
    B->>K: drain changelog to LSO (only what it was missing)
    B->>K: heartbeat ACTIVE
    note over B: processing resumes on caught-up state
```

## Processing guarantees under failover

How clean a failover is depends on the [processing guarantee](https://stoatflow.io/docs/concepts/exactly-once) the application runs under.

- **Exactly-once (the default).** Failover is split-brain-proof. Kafka's transactional producer fencing — a broker-enforced guarantee — ensures at most one instance can ever commit a given transaction. If a network partition or a false failure-detection briefly leaves both instances believing they are active, only one can commit; the broker fences the other, which then shuts itself down. Downstream consumers reading `read_committed` see exactly-once output across the handoff: no duplicates, no lost work.
- **At-least-once.** Failover is *bounded-duplicate*. The cluster still coordinates promotion safely — the promotion token is itself the fence under at-least-once — but without transactional fencing an ungraceful failover can re-process a small, bounded window of records, so a few duplicate output records are possible.

::callout{color="warning" icon="i-lucide-shield-alert"}
For split-brain-proof failover, run hot standby under exactly-once. Under at-least-once, make downstream consumers idempotent or duplicate-tolerant and size them for a bounded-duplicate window on an ungraceful failover.
::

## Application faults under hot standby

Losing the node is not the only way an active stops serving. A processor can throw, or a send can fail, and the [error-handling model](https://stoatflow.io/docs/concepts/error-handling-model) escalates that to a fatal. What happens next depends on whether the fault is one a rebuilt engine could survive.

**A restartable fault is absorbed in place — the role does not move.** If your application registers a KS-compatible `StreamsUncaughtExceptionHandler` and answers `REPLACE_THREAD`, a processing or production failure triggers an [in-place engine restart](https://stoatflow.io/blog/in-place-restart-multi-standby): the engine is torn down and rebuilt inside the live JVM, resuming from the last committed barrier. The pod keeps the active role throughout, the standby stays a standby, and recovery costs an engine rebuild rather than a promotion plus a pod round-trip and a re-catch-up. This works the same whether hot standby is on or off.

Restarts are budgeted — `stoatflow.commit-barrier.max-engine-restarts` (default 5) within `stoatflow.commit-barrier.engine-restart-window-ms` (default 5 minutes). When the budget is spent, the instance shuts down, and *that* is when the role moves: the shutdown is a graceful `DRAINING` hand-off, so a caught-up standby promotes on the fast path — the same path a `/ha/switch` takes, not the slower staleness backstop. So a recurring fault costs exactly one failover after a bounded number of local attempts, rather than a role transfer per occurrence.

::callout{color="info" icon="i-lucide-info"}
**The pod is unready while it rebuilds.** `/health/ready` reports DOWN for the duration — it genuinely is not processing, and any interactive-query endpoints served from it are unavailable. Two practical consequences: an in-flight StatefulSet rolling update stalls until the pod is ready again, and the pod drops out of Service endpoints. Liveness is unaffected; a successful in-place restart never reaches a terminal state.
::

**A fault the rebuild cannot survive still hands off.** Only processing and production failures are restartable. A record that cannot deserialize will not deserialize for a rebuilt engine either, so a deserialization `FAIL` goes straight to shutdown — as do punctuator failures and commit stalls. The handler is still consulted (your alerting and metrics fire), but a `REPLACE_THREAD` answer cannot be honoured and a `WARN` records that. The instance then drains, publishes its hand-off, and a standby promotes.

That hand-off is published from the terminal state transition itself, so it does not depend on anything calling `close()`. This matters for **embedded deployments that use `:core` directly** without the `:runtime` wrapper: before StoatFlow 1.0.0-beta.8 the hand-off only ran inside `close()`, which the runtime's watcher triggers but an embedded application may never call — so a dead engine kept heartbeating as a healthy active and the standby waited indefinitely. If you embed `:core`, upgrade to beta.8 or later, or make sure a liveness probe can reach the pod.

**While a pod is rebuilding, role-change commands are refused.** `POST /ha/switch`, `/ha/promote` and `/ha/demote` return `409` if a pod the command would move is mid-restart, because a role change cannot be serialised against an engine swap. Every command moves the **active**, so a `promote` is refused while the active it would supersede is rebuilding — not only when its own `?pod=` target is. `?force=true` does not override it — force exists to override the *readiness* judgement, not to make a structurally impossible operation succeed. The restart is bounded either way, so poll `GET /ha/status` until `selfRestarting` (or the peer's `restarting`) is false and re-issue. `stoatflow.ha.restarting` is the same signal as a metric — worth having on a dashboard, because the pod stays `ACTIVE` throughout and `stoatflow.ha.role` therefore does not move.

## Configuration

Hot standby is configured under `stoatflow.ha`. The only required key is `mode`; everything else has a sensible default.

| Key                                                     | Default                                      | Purpose                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| ------------------------------------------------------- | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `stoatflow.ha.mode`                                     | `off`                                        | `off` (single instance, fast restart) or `active-standby` (hot-standby cluster).                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `stoatflow.ha.pod-id`                                   | `POD_NAME` env → `HOSTNAME` → local hostname | Stable per-pod identity that distinguishes the peers.                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `stoatflow.ha.coordination-topic`                       | `__stoatflow_<applicationId>_ha`             | Single-partition, compacted, auto-created topic the cluster coordinates through.                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `stoatflow.ha.desired-standbys`                         | `1`                                          | How many caught-up standby spares the readiness gate protects. `1` is the classic active/passive pair; raise it (with `replicas`) for more redundancy. Drives the [redundancy floor](https://stoatflow.io/#how-readiness-gates-a-rolling-deploy) on rolling deploys.                                                                                                                                                                                                                                               |
| `stoatflow.ha.max-standbys`                             | `3`                                          | Hard ceiling on standbys, bounding changelog fan-out (each standby is one more changelog reader). Total instances ≤ `max-standbys + 1`.                                                                                                                                                                                                                                                                                                                                                                            |
| `stoatflow.ha.failover-priority`                        | `0`                                          | Per-pod tie-break hint: orders candidates *within* the equal-lag bucket (higher wins). Never overrides lag or the hash floor.                                                                                                                                                                                                                                                                                                                                                                                      |
| `stoatflow.ha.promotion-settle-window-ms`               | `500`                                        | After claiming the promotion token, how long the winner confirms it is the sole claimant before fencing — the single-promoter guarantee. Bounded `1..staleness-threshold-ms`.                                                                                                                                                                                                                                                                                                                                      |
| `stoatflow.ha.heartbeat-ms`                             | `1000`                                       | How often each pod publishes its liveness and role.                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `stoatflow.ha.staleness-threshold-ms`                   | `5000`                                       | How long without a peer heartbeat before it's considered stale. Must be ≥ `heartbeat-ms`.                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `stoatflow.ha.staleness-misses`                         | `3`                                          | Consecutive missed checks before a standby is elected to promote — a debounce against transient jitter.                                                                                                                                                                                                                                                                                                                                                                                                            |
| `stoatflow.ha.acceptable-recovery-lag`                  | `50000`                                      | Total committed-changelog lag (in records, summed across **all** changelog partitions) at or below which a standby is considered caught up: it reports `READY_STANDBY` and is eligible to promote without the grace fallback. This is a single **total sum**, not per-task like Kafka Streams' `acceptable.recovery.lag` (10000) — so the KS number would be far too strict, and \~one large epoch can approach it. Correctness never depends on it — restore-before-process does; it's mostly a *raise-me* lever. |
| `stoatflow.ha.promotion-grace-ms`                       | `5000`                                       | When the active is gone and no standby is within `acceptable-recovery-lag`, the least-behind standby auto-promotes after this bounded grace window (availability-first). Eligibility is re-checked continuously during the window.                                                                                                                                                                                                                                                                                 |
| `stoatflow.ha.promotion-restore-no-progress-timeout-ms` | `60000`                                      | Progress-aware deadline for restore-before-process (bringing local state to the committed changelog end before processing): fails only if a full window of this length elapses with **zero** records applied (genuinely stuck), **not** the commit-path barrier timeout. On expiry the pod self-terminates and Kubernetes restarts it (the fence already prevents split-brain).                                                                                                                                    |

```yaml
stoatflow:
  application-id: my-app
  bootstrap-servers: kafka:9092
  ha:
    mode: active-standby      # off (default) | active-standby
    desired-standbys: 2       # 1 (default) = active/passive pair; here 1 active + 2 standbys
    max-standbys: 3           # ceiling on standbys (bounds changelog fan-out)
    # pod-id, coordination-topic, and the timing knobs all have sensible
    # defaults — a two-replica pair sets only mode.
```

::callout{color="info" icon="i-lucide-info"}
The staleness knobs are a downtime-versus-false-positive trade, not a correctness lever — correctness comes from the fence, so a false positive costs at most one bounded restart, never corrupt state. Tune them conservatively. On Kubernetes, `pod-id` resolves automatically from the downward-API `POD_NAME`, so a typical deployment sets only `mode`.
::

### Coordination topic

The cluster coordinates through one compacted internal topic (`stoatflow.ha.coordination-topic`, default `__stoatflow_<applicationId>_ha`). It carries the per-pod heartbeats and the **promotion token** (the compare-and-set that serialises the election). It is **auto-created** on startup; the application's Kafka principal needs `Describe` + `Read` + `Write` on it, plus `Create` if you rely on auto-creation.

Two hard requirements:

- **Exactly one partition.** The single-partition total order is load-bearing for the promotion-token compare-and-set — never create it with more than one partition.
- **Exclusive to one application.** The default name is `applicationId`-scoped for exactly this reason. If you override `coordination-topic`, give every application its own topic — sharing one across applications intermixes pod-status records and collides on the token, breaking the election and fencing.

Under **at-least-once** the topic is correctness-critical: the promotion token is itself the fence (ALO has no broker-enforced producer-epoch fence), so size its durability (replication factor, `min.insync.replicas`) accordingly. Under **exactly-once** the token is an availability layer — the election — and the fence is the producer epoch.

If your cluster **denies topic auto-creation**, pre-create the topic before starting the application. StoatFlow logs the exact required spec at `ERROR` and fails startup if the topic is missing and cannot be created:

| Property                    | Value                                                                                      |
| --------------------------- | ------------------------------------------------------------------------------------------ |
| partitions                  | `1` (required — do not increase)                                                           |
| `cleanup.policy`            | `compact`                                                                                  |
| `retention.ms`              | `-1`                                                                                       |
| `min.insync.replicas`       | `2` (`1` on a single-broker cluster)                                                       |
| replication factor          | `3` (or the broker count, if smaller)                                                      |
| `segment.ms`                | `300000`                                                                                   |
| `min.compaction.lag.ms`     | `0`                                                                                        |
| `max.compaction.lag.ms`     | `300000` — forces the active segment to roll and compact, bounding the heartbeat-churn log |
| `min.cleanable.dirty.ratio` | `0.1`                                                                                      |

## Deploying the cluster on Kubernetes

Run the cluster as a **`StatefulSet` with two or more replicas** (not a `Deployment`), keeping everything from the single-replica [Kubernetes](https://stoatflow.io/docs/operating/kubernetes) manifest — the state volume, license injection, config — and adjusting these points:

- `replicas: N` — one active plus `N-1` passive standbys. `replicas: 2` is the classic pair (the default); the example below uses `replicas: 3` for one active + two standbys.
- `STOATFLOW__HA__DESIRED_STANDBYS` / `STOATFLOW__HA__MAX_STANDBYS` — set alongside `replicas` when you run more than one standby (`desired-standbys` should be `replicas - 1`).
- `updateStrategy: RollingUpdate` — the roll is readiness-gated and order-independent (see below).
- `volumeClaimTemplates` — each pod gets its own persistent volume, so every standby holds warm state across restarts.
- Split `/health/live` and `/health/ready` probes.
- `terminationGracePeriodSeconds` ≥ your graceful-handoff budget (epoch drain + final commit), so a draining active finishes its handoff before `SIGKILL`.
- `POD_NAME` from the downward API → `stoatflow.ha.pod-id`.
- A headless `Service` (the StatefulSet's `serviceName`).

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: my-stream-app
spec:
  serviceName: my-stream-app     # headless Service
  replicas: 3                    # one active + two passive standbys (replicas: 2 = the default pair)
  updateStrategy:
    type: RollingUpdate          # readiness-gated; roll order doesn't matter
  selector:
    matchLabels:
      app: my-stream-app
  template:
    metadata:
      labels:
        app: my-stream-app
    spec:
      # Must exceed the graceful-handoff budget so a draining active
      # completes its handoff before SIGKILL.
      terminationGracePeriodSeconds: 60
      containers:
        - name: app
          image: my-registry/my-stream-app:latest
          ports:
            - name: http
              containerPort: 8080
          env:
            # Stable per-pod identity → stoatflow.ha.pod-id
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: STOATFLOW__HA__MODE
              value: "active-standby"
            - name: STOATFLOW__HA__DESIRED_STANDBYS
              value: "2"           # replicas - 1 (omit for the default pair)
            - name: STOATFLOW__HA__MAX_STANDBYS
              value: "3"
          readinessProbe:          # gates the rolling update
            httpGet: { path: /health/ready, port: http }
            periodSeconds: 5
            failureThreshold: 3
          livenessProbe:           # stays UP while a standby catches up
            httpGet: { path: /health/live, port: http }
            periodSeconds: 10
            failureThreshold: 6
          volumeMounts:
            - name: state
              mountPath: /var/lib/stoatflow/state
  volumeClaimTemplates:
    - metadata:
        name: state
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 100Gi        # ≥ active state size; the standby needs the same capacity
```

::callout{color="warning" icon="i-lucide-triangle-alert"}
`replicas: 2` or more is valid **only** with `ha.mode: active-standby`. Multiple instances of the same application with `mode: off` will corrupt state — see [One active instance](https://stoatflow.io/docs/operating#one-active-instance). The standbys are passive precisely so the cluster can run more than one replica safely.
::

## How readiness gates a rolling deploy

The split health probes are what make the rolling update safe and order-independent.

| Role        | `/health/ready`               | `/health/live`                                |
| ----------- | ----------------------------- | --------------------------------------------- |
| **Active**  | UP while it is processing     | UP while running                              |
| **Standby** | UP only once it has caught up | UP while it is still catching up (not killed) |

Because the standby reports ready only after it has caught up, Kubernetes won't move on to the next pod in the roll until the one it just restarted is a caught-up standby. And because a still-catching-up standby keeps its liveness probe UP, Kubernetes won't kill it mid-catch-up. The result is a safe rolling deploy with no custom controller — whichever pod rolls first, the cluster never cuts over to an instance that isn't ready. See [Probes](https://stoatflow.io/docs/operating/probes) and [Health checks](https://stoatflow.io/docs/runtime/health-checks).

**The redundancy floor.** With more than one standby, readiness is gated on redundancy as well as catch-up: a caught-up standby reports ready to roll only if rolling it would still leave the active plus at least one other caught-up standby (governed by `desired-standbys`). So a rolling deploy takes one pod down at a time, standbys first and the active last, and never drops below an active and a warm spare while it runs. A two-replica pair has no other spare to preserve, so this reduces to the pair's behaviour.

```mermaid
sequenceDiagram
    participant K as StatefulSet controller
    participant R as Pod being rolled
    participant S as Other standby
    participant A as Active

    note over K,A: replicas 3, desired-standbys 2 —<br/>readiness gates every step, so roll order does not matter
    K->>R: SIGTERM
    R-->>K: terminated (fully stopped)
    note over S: now the only standby —<br/>stays Ready, a lone spare is exempt
    K->>R: create replacement
    R->>R: restore state, delta replay if the PVC survived
    note over R: ready 503, live UP —<br/>catching up, never liveness-killed
    R->>R: caught up (READY_STANDBY)
    alt another fresh standby is also caught up
        R-->>K: ready — the controller moves on
    else the other standby is behind
        R-->>K: still 503 — the roll stalls here
        note over K,A: never fewer than the active<br/>plus one warm spare
    end
    K->>A: SIGTERM, when the active's turn comes
    A->>A: drain in-flight epoch, commit final barrier
    S->>S: promote — token CAS, fence, drain to LSO
    A->>A: returns as a standby, restores, rejoins the floor
```

## Operating the cluster

When HA is enabled, each pod exposes a small set of HTTP endpoints (in-cluster). They are not registered when `ha.mode: off`.

| Endpoint               | Method | Purpose                                                                                                                                                                                                                                                                                                                      |
| ---------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `/ha/status`           | GET    | This pod's view of the cluster: its own role, total committed-changelog lag, the freshness of its coordination view (`tailFreshnessMs` / `tailCaughtUp`), whether it is restarting its engine in place (`selfRestarting`), the caught-up-standby count, the current promotion-token epoch/holder, and the peers it observes. |
| `/ha/switch`           | POST   | Swap roles — whichever pod is active drains and the peer promotes. Rejected `409` unless a caught-up (`READY_STANDBY`) peer exists (override `?force=true`), **or** while the active is restarting in place (not overridable).                                                                                               |
| `/ha/promote?pod=<id>` | POST   | Ask a specific standby pod to promote. Rejected `409` unless the target is caught up (override `?force=true`), **or** while the target or the current active is restarting in place (not overridable).                                                                                                                       |
| `/ha/demote?pod=<id>`  | POST   | Ask a specific active pod to demote. Rejected `409` unless a caught-up peer can take over (override `?force=true`), **or** while the target is restarting in place (not overridable).                                                                                                                                        |

```bash
curl -s localhost:8080/ha/status
```

```json
{
  "selfPodId": "my-stream-app-0",
  "selfRole": "ACTIVE",
  "selfState": "ACTIVE",
  "selfRestarting": false,
  "replicationLagRecords": 0,
  "replicationLagMs": 0,
  "tailFreshnessMs": 180,
  "tailCaughtUp": true,
  "restartRequired": null,
  "readyStandbyCount": 2,
  "desiredStandbys": 2,
  "redundancyBelowDesired": false,
  "tokenEpoch": 1,
  "tokenHolder": "my-stream-app-0",
  "peers": [
    {
      "podId": "my-stream-app-1",
      "role": "STANDBY",
      "state": "READY_STANDBY",
      "replicationLagRecords": 1240,
      "replicationLagMs": 85,
      "failoverPriority": 0,
      "generation": 7,
      "freshnessMs": 420,
      "restarting": false
    },
    {
      "podId": "my-stream-app-2",
      "role": "STANDBY",
      "state": "READY_STANDBY",
      "replicationLagRecords": 980,
      "replicationLagMs": 70,
      "failoverPriority": 0,
      "generation": 7,
      "freshnessMs": 310,
      "restarting": false
    }
  ]
}
```

The write endpoints are thin shims: they publish a command and return `202 Accepted` with the command's offset as an idempotency token; the targeted pod observes it and acts. `promote` and `demote` require a `?pod=<id>` target; `switch` targets whichever pod is currently active.

There are **two independent reasons** a command endpoint returns `409 Conflict`, and only one of them is overridable.

**No caught-up target — overridable.** To prevent a self-inflicted outage, `switch`/`promote`/`demote` are rejected unless a caught-up (`READY_STANDBY&#x60;) target exists. Override with &#x2A;*`?force=true`** when you deliberately want to hand off to a behind pod — a forced promotion still fully restores to the committed changelog before processing (correctness is never traded; `force` only skips the *readiness* check).

**A pod the command would move is restarting in place — not overridable.** See [Application faults](https://stoatflow.io/#application-faults-under-hot-standby) below: `?force=true` deliberately does not help here, because the conflict is structural rather than a judgement call. Poll `GET /ha/status` until `selfRestarting` (or the peer's `restarting`) is `false`, then re-issue. Since every command moves the active, this check covers the active for all three verbs — a `promote` is refused while the active it would supersede is mid-restart, not only when its own `?pod=` target is.

```bash
curl -X POST localhost:8080/ha/switch
# { "command": "SWITCH", "target": null, "force": false, "commandOffset": 42 }

curl -X POST 'localhost:8080/ha/promote?pod=my-stream-app-1&force=true'
# { "command": "PROMOTE", "target": "my-stream-app-1", "force": true, "commandOffset": 43 }
```

`tailFreshnessMs` / `tailCaughtUp` report how current this pod's view of the coordination topic is. A pod whose tail has gone stale (`tailCaughtUp: false`) defers failover decisions until its view is current again — correctness is unaffected (the fence is the split-brain backstop), but a persistently stale tail on a pod is worth investigating.

Every role change is recorded on the coordination topic, so its full history can be replayed with a console consumer if you need an audit trail. See [REST API](https://stoatflow.io/docs/runtime/rest-api) for the complete endpoint reference.

## Metrics and what to alert on

The runtime exposes HA metrics on `/metrics` (Prometheus) when HA is enabled.

| Metric                                             | Meaning                                                                                                                                                                                                               | Alert on                                                                                              |
| -------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- |
| `stoatflow.ha.role`                                | `1` when this pod is the active, `0` when standby.                                                                                                                                                                    | — (use for dashboards / "is there exactly one active?").                                              |
| `stoatflow.ha.standby.replication.lag.records`     | Gauge — true **total** committed-changelog lag in records, summed across all changelog partitions and read directly from the standby applier consumer's position; no admin round-trip (`0` when active or caught up). | Sustained growth, or staying above `acceptable-recovery-lag`.                                         |
| `stoatflow.ha.standby.replication.applied.records` | Counter — cumulative committed changelog deltas the streaming standby applier has applied; `rate(...)` gives apply throughput (resets on promotion/restart).                                                          | — (dashboards).                                                                                       |
| `stoatflow.ha.standby.replication.lag.ms`          | Gauge — standby replication lag in milliseconds.                                                                                                                                                                      | Sustained values above your replication-lag SLO.                                                      |
| `stoatflow.ha.peer.freshness.ms`                   | Time since the freshest peer's last heartbeat.                                                                                                                                                                        | Approaches `staleness-threshold-ms`.                                                                  |
| `stoatflow.ha.peers`                               | Number of observed peers.                                                                                                                                                                                             | Drops below `desired-standbys` (lost redundancy).                                                     |
| `stoatflow.ha.redundancy.ready.standbys`           | Gauge — count of caught-up (`READY_STANDBY`) standbys, including self.                                                                                                                                                | Below `desired-standbys`.                                                                             |
| `stoatflow.ha.redundancy.below.desired`            | `1` when the active has fewer ready standby spares than `desired-standbys` — a single point of failure.                                                                                                               | Equals `1`.                                                                                           |
| `stoatflow.ha.token.epoch`                         | Gauge — current promotion-token epoch. Increments once per election.                                                                                                                                                  | Climbing rapidly (repeated elections / flapping).                                                     |
| `stoatflow.ha.election.outcome`                    | Counter, tagged `outcome=won|reclaimed|stood_down_fresh|stood_down_lost` — cumulative election outcomes.                                                                                                              | Frequent `stood_down_lost` (contended elections).                                                     |
| `stoatflow.ha.election.latency.ms`                 | Gauge — last claim→ACTIVE promotion latency (`-1` until the first).                                                                                                                                                   | Above your failover SLO.                                                                              |
| `stoatflow.ha.restart.required`                    | `1` when this pod needs a restart (for example, after the source topic's partition count changes).                                                                                                                    | Equals `1`.                                                                                           |
| `stoatflow.ha.restarting`                          | `1` while this pod is rebuilding its engine in place to absorb a processing fault. It stays `ACTIVE` throughout.                                                                                                      | Sustained `1`, or a rising rate — repeated faults end in a shutdown once the restart budget is spent. |

A healthy cluster shows exactly one pod with `role = 1`, its standbys' replication lag hovering near zero, `peers = N-1`, and `redundancy.below.desired = 0`. See [Metrics](https://stoatflow.io/docs/runtime/metrics) and [Observability](https://stoatflow.io/docs/operating/observability) for the full surface.

## Failover characteristics

- **Graceful** (rolling deploy or `/ha/switch`): about one handoff — the active drains and commits, an elected standby promotes, and the old active rebuilds as a standby in place (no pod restart). No reprocessing under exactly-once.
- **Ungraceful** (the active crashes): bounded by detection (`staleness-misses × heartbeat-ms`) plus a near-instant promotion, since the elected standby is already warm. This is independent of state size — the cold-restore window is exactly what hot standby removes.
- **A restartable application fault: no failover at all.** The active absorbs it in place and keeps the role (see [Application faults under hot standby](https://stoatflow.io/#application-faults-under-hot-standby)). The role moves only if the restart budget is spent, and then it is a graceful hand-off — one failover, not one per fault.

Segmented window/session stores run with the WAL off and would otherwise be durable only on a clean `close()`. A periodic checkpoint bounds how much of the changelog a pod re-applies after an *ungraceful* restart — and how much a promoting standby drains — to at most one checkpoint interval. Tune it with `stoatflow.state.segment-checkpoint-records` (default `100000`) and `stoatflow.state.segment-checkpoint-ms` (default `30000`), whichever fires first; see the [configuration reference](https://stoatflow.io/docs/reference/configuration-reference#stoatflowstate).

For measured cold-start numbers on the default (fast-restart) tier — the cost you avoid with hot standby — see [Benchmarks](https://stoatflow.io/product/benchmarks).

## Limitations and what to consider

- **It's redundancy, not scale.** Hot standby keeps warm spares; it does not add throughput. There is still exactly one active instance. To go faster, scale the active vertically.
- **More standbys, more footprint and more changelog reads.** Each standby is another always-on instance with its own persistent volume, and another consumer streaming the changelog — so `K` standbys multiply that read fan-out `K`-fold. `max-standbys` bounds it; pick the redundancy you need, not the maximum. The standbys run under the same application licence — they don't add seats.
- **The standbys are not queryable.** While passive a standby only mirrors the changelog into local state; interactive queries (`readOnlyStore`, the state HTTP endpoints) are served by the active alone. A standby answers reads only once it has promoted.
- **Give the active CPU headroom.** A busy active under a fan-out workload needs its cores; standbys co-located with the active on one node compete for them, and a starved active can miss a commit window (handled cleanly, but it's still a restart). Spread the pods across nodes, or size the node for the whole cluster.
- **An extra internal topic.** The coordination topic is auto-created and compacted; account for it when planning topics and ACLs. See [Coordination topic](https://stoatflow.io/#coordination-topic) for the principal's permissions, the single-partition and exclusive-per-application requirements, and the pre-create spec for locked-down clusters.
- **At-least-once is bounded-duplicate**, not split-brain-proof — see [Processing guarantees under failover](https://stoatflow.io/#processing-guarantees-under-failover).
- **Increasing a source topic's partition count requires a restart** of the cluster; StoatFlow surfaces this via `stoatflow.ha.restart.required` and the liveness probe, so Kubernetes recycles the pod.
- **Single region.** Every pod must reach the same Kafka cluster; hot standby is not a multi-region or active-active mechanism.
- **The default is often the right answer.** If a restart-window gap is tolerable, fast restart is simpler and cheaper.

## Next steps

- **[Kubernetes](https://stoatflow.io/docs/operating/kubernetes)** — the full single-replica manifest this page extends.
- **[Probes](https://stoatflow.io/docs/operating/probes)** — liveness versus readiness and how the readiness gate works.
- **[Observability](https://stoatflow.io/docs/operating/observability)** — scraping metrics and what to alert on.
- **[Exactly-once](https://stoatflow.io/docs/concepts/exactly-once)** — the guarantee that makes failover split-brain-proof.


# Liveness and readiness probes

StoatFlow exposes two HTTP health endpoints designed to drive Kubernetes liveness and readiness probes: `/health/live` decides whether the pod should be kept alive, `/health/ready` decides whether it should receive traffic and count toward a rollout. This page shows how to wire them, what each one gates on, and how to set the probe periods.

::tldr-panel
- **Readiness** (`/health/ready`) is UP only while the engine is `RUNNING` — it returns 503 during startup, state restoration, pause, and shutdown, plus when the broker is unreachable or the license is non-operational. Wire it to `readinessProbe` so rollouts and load balancers wait.
- **Liveness** (`/health/live`) stays UP through normal transient states so the orchestrator doesn't restart a healthy-but-busy pod — but returns 503 if the commit pipeline has frozen, so Kubernetes restarts the wedged process.
- Both endpoints return HTTP **200 = UP**, **503 = DOWN**, with a JSON body listing each component. Default port is **8080**.
::

For the response shape and the full component breakdown, see [Health checks](https://stoatflow.io/docs/runtime/health-checks). This page is about wiring and tuning the probes.

## The two endpoints

Both endpoints aggregate every registered health indicator. The overall status is &#x2A;*UP (HTTP 200)** only when *every* component is UP; if any component is DOWN, the endpoint returns &#x2A;*DOWN (HTTP 503)**. The JSON body always lists the per-component status so you can see which one tripped.

| Endpoint        | Probe            | UP means                             | DOWN means                                      |
| --------------- | ---------------- | ------------------------------------ | ----------------------------------------------- |
| `/health/ready` | `readinessProbe` | Ready to process and serve traffic   | Don't route traffic / don't advance rollout yet |
| `/health/live`  | `livenessProbe`  | Process is healthy, leave it running | Process is wedged, restart it                   |

The built-in indicators are `stoatflow` (engine state), `license`, `kafka-broker`, and — when a Schema Registry URL is configured — `schema-registry`. Each contributes to both probes, but several of them deliberately report **different** liveness vs. readiness status (table below).

## What readiness gates

Readiness answers "should this pod receive traffic and count as available?" StoatFlow returns `/health/ready` = DOWN (503) until the application is genuinely able to process records:

- **State restoration.** On cold start or restart the engine rebuilds state stores from their changelog topics before processing begins. Readiness stays DOWN for the whole `STARTING` / `VALIDATING_STATE` / `RESTORING` window. During a rolling update this is what keeps Kubernetes from cutting traffic over to a pod that hasn't caught up yet. See [Architecture — lifecycle](https://stoatflow.io/docs/concepts/architecture) for the cold-start sequence.
- **Broker availability.** The `kafka-broker` indicator checks broker connectivity. If the broker is unreachable, readiness goes DOWN — the pod can't make progress, so it shouldn't be considered ready.
- **License.** The `license` indicator returns readiness DOWN when the runtime license is `EXPIRED`, `REVOKED`, `INVALID`, or absent — the orchestrator stops routing to a pod whose license has been pulled. (`VALID`, `GRACE_PERIOD`, and `PENDING` — the deferred-validation window in the first \~5 minutes after start — are treated as ready.) See [License configuration](https://stoatflow.io/docs/getting-started/license-configuration).
- **Schema Registry** (only when configured). If a Schema Registry URL is set and the registry is unreachable, readiness goes DOWN.
- **Pause and shutdown.** While paused (`DRAINING` / `PAUSED`) or shutting down (`STOPPING`), readiness is DOWN so load balancers drain the pod cleanly. See [Pause / unpause](https://stoatflow.io/docs/runtime/pause-unpause).

Readiness returns UP only when the engine is in the `RUNNING` state and every other component is healthy.

::callout{color="info" icon="i-lucide-copy"}
**Under [hot standby](https://stoatflow.io/docs/operating/high-availability)** readiness carries an extra condition: the active reports ready while processing, and a standby reports ready **only once it has caught up**. That's what makes the rolling update across the cluster order-independent — Kubernetes won't advance the roll until the restarted pod is a caught-up standby. A still-catching-up standby keeps its *liveness* UP so it isn't killed mid-catch-up. With more than one standby, readiness also holds the redundancy floor so a roll never drops below the active plus a warm spare.
::

## What liveness gates

Liveness answers a narrower question: "is this process broken in a way that only a restart can fix?" It deliberately stays UP through the normal transient states — `STARTING`, `VALIDATING_STATE`, `RESTORING`, `DRAINING`, `PAUSED`, `STOPPING` — so the orchestrator doesn't kill a pod that's simply busy restoring state or draining for a clean shutdown.

Liveness returns DOWN in two situations:

1. **Hard dependency failure** — the broker (or a configured Schema Registry) is unreachable. These can't be made healthy by anything the pod is doing, so a restart is the right response.
2. **A frozen commit pipeline** (see below).

Note one deliberate asymmetry: the **license** indicator never fails liveness. An expired or revoked license fails *readiness* (traffic stops) but keeps *liveness* UP — restarting the pod won't fix a license problem, and a restart loop would only hide it.

### Stall-aware liveness

The commit barrier is the heart of exactly-once processing: a transaction periodically commits state, output, and offsets together (see [Exactly-once semantics](https://stoatflow.io/docs/concepts/exactly-once)). If that pipeline were ever to freeze — a commit that starts but never finishes — the pod would sit there alive but making no progress, the worst kind of failure for an orchestrator to miss.

StoatFlow guards against this. The runtime tracks how long it has been since the last successful commit while commit work is pending. If that age exceeds a configured threshold, `/health/live` flips to DOWN — Kubernetes restarts the pod, and the restarted process resumes from the last successful barrier with no duplicates downstream. The threshold is `stoatflow.commit-stall.threshold-ms` (**default 45000 ms**; set to `0` to disable the check, which re-introduces the silent-freeze failure mode and is discouraged).

::callout{color="info" icon="i-lucide-info"}
This stall check is an independent recovery path. The runtime also detects and aborts a frozen commit internally, but the liveness probe gives Kubernetes its own way to recover even if the in-process recovery is itself wedged. Both read the same `threshold-ms`. When liveness trips on a stall, the JSON body includes a `stall_age_ms` detail and the `/debug/barriers` endpoint shows the stuck barrier.
::

## Liveness vs. readiness, per component

| Component                         | Liveness (`/health/live`)                                                                                                                          | Readiness (`/health/ready`)                                                                        |
| --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| `stoatflow` engine                | UP during transient states (`STARTING`, `RESTORING`, `DRAINING`, `PAUSED`, `STOPPING`); DOWN if the commit pipeline has stalled past the threshold | UP only in `RUNNING`                                                                               |
| `license`                         | UP regardless of license status                                                                                                                    | DOWN on `EXPIRED` / `REVOKED` / `INVALID` / absent key; UP on `VALID` / `GRACE_PERIOD` / `PENDING` |
| `kafka-broker`                    | DOWN if broker unreachable                                                                                                                         | DOWN if broker unreachable                                                                         |
| `schema-registry` (if configured) | DOWN if registry unreachable                                                                                                                       | DOWN if registry unreachable                                                                       |

## Kubernetes probe configuration

Point both probes at the HTTP server's port (default `8080`). A typical container spec:

```yaml
# Deployment / Pod container spec
ports:
  - name: http
    containerPort: 8080

readinessProbe:
  httpGet:
    path: /health/ready
    port: http
  periodSeconds: 5
  timeoutSeconds: 3
  failureThreshold: 3        # 3 × 5s = ~15s of consecutive failures before "not ready"

livenessProbe:
  httpGet:
    path: /health/live
    port: http
  periodSeconds: 10
  timeoutSeconds: 3
  failureThreshold: 3        # ~30s of consecutive failures before restart

startupProbe:
  httpGet:
    path: /health/ready
    port: http
  periodSeconds: 10
  failureThreshold: 60       # allow up to ~10 minutes for state restoration
```

::callout{color="warning" icon="i-lucide-clock"}
**Use a `startupProbe` for any topology with non-trivial state.** Cold-start restoration time scales with state size and can exceed a default liveness `initialDelaySeconds`. The `startupProbe` holds the liveness probe off until restoration completes — `failureThreshold × periodSeconds` is your total restoration budget. Raise `failureThreshold` for large state rather than inflating `initialDelaySeconds`. While the startup probe is running, readiness is already DOWN (the engine isn't `RUNNING` yet), so no traffic arrives.
::

### Tuning the periods

- **Readiness period** controls how quickly traffic is cut from a pod that goes unhealthy and restored when it recovers. A short `periodSeconds` (e.g. 5s) reacts fast at the cost of slightly more probe load; the probe is cheap (an in-process status aggregation), so erring short is fine.
- **Liveness period and `failureThreshold`** govern how long a wedged pod survives before restart: roughly `periodSeconds × failureThreshold`. Keep this comfortably **longer than the stall threshold** (`stoatflow.commit-stall.threshold-ms`, default 45s) so a genuine stall is what trips the restart, not a single slow probe — for example a 10s period × 3 failures (\~30s) layered on top of the stall threshold means a real freeze is reported by the endpoint well before the probe budget would have expired on its own.
- **`timeoutSeconds`** should leave headroom over the broker health-check timeout, since the readiness aggregation includes the `kafka-broker` connectivity check. The default broker check timeout is short, but set `timeoutSeconds` to at least 2-3s.

::callout{color="info" icon="i-lucide-shield"}
`stoatflow.commit-stall.threshold-ms` must be **less than** the commit transaction timeout (`stoatflow.commit-barrier.timeout-ms`) — validated at startup. This ordering ensures the in-process recovery has its chance to abort a stuck commit before the stall check escalates to a pod restart. See [Core configuration](https://stoatflow.io/docs/configuration/core-config).
::

## Verifying locally

With the app running, the endpoints answer over HTTP. A ready pod returns 200:

```bash
# Returns 200 + {"status":"UP",...} when RUNNING; 503 during restoration/pause/shutdown
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/health/ready

# Returns 200 unless the process is wedged (commit stall) or a hard dependency is down
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/health/live

# Inspect the component breakdown
curl -s localhost:8080/health/ready | jq
```

During state restoration `/health/ready` returns 503 with a body identifying the in-progress restorations; once the engine reaches `RUNNING` it flips to 200.

## See also

- [Health checks](https://stoatflow.io/docs/runtime/health-checks) — response format, all indicators, registering custom ones
- [High availability](https://stoatflow.io/docs/operating/high-availability) — how readiness gates the hot-standby rolling update
- [Kubernetes](https://stoatflow.io/docs/operating/kubernetes) — deploying StoatFlow on Kubernetes
- [Observability](https://stoatflow.io/docs/operating/observability) — metrics and the `/debug/*` endpoints
- [Pause / unpause](https://stoatflow.io/docs/runtime/pause-unpause) — how pause affects readiness
- [Core configuration](https://stoatflow.io/docs/configuration/core-config) — `commit-stall` and `commit-barrier` settings


# Observability

A StoatFlow application is one process, so observability is simpler than a clustered stream processor — there is one set of metrics, one log stream, and one HTTP surface to point your monitoring stack at. This page wires those three together: scraping `/metrics` into Prometheus and Grafana, the meters worth alerting on, reading and correlating the logs, and using the `/debug/*` endpoints when a pipeline goes quiet or slow.

::tldr-panel
- **Metrics:** `GET /metrics` in Prometheus text format — scrape it from your existing stack. The full meter catalogue is on the [metrics reference](https://stoatflow.io/docs/reference/metrics-reference).
- **Logs:** plain text to stdout out of the box; per-package levels are configurable. Enable MDC to stamp each line with the lane and barrier it belongs to, and switch the encoder to JSON if you want structured ingestion.
- **Diagnosis:** `/debug/threads` and `/debug/barriers` give a live snapshot of what every lane is doing and where a commit is stuck — for when the dashboards say "stopped" but not "why".
- **Exposure:** keep `/metrics`, `/debug/*`, and the introspection endpoints on the in-cluster network; only the health probes belong on public ingress. See [Kubernetes](https://stoatflow.io/docs/operating/kubernetes).
::

## The three signals

The runtime exposes everything an operator needs over the same HTTP server (default port `8080`):

| Signal                   | Surface                                     | Use it for                                                                         |
| ------------------------ | ------------------------------------------- | ---------------------------------------------------------------------------------- |
| **Metrics**              | `GET /metrics` (Prometheus)                 | Dashboards, alerting, trend lines, capacity planning.                              |
| **Logs**                 | stdout (plain text by default)              | Correlating a metric spike with a specific record, lane, or barrier; post-mortems. |
| **Diagnostic snapshots** | `GET /debug/threads`, `GET /debug/barriers` | On-demand "what is it doing right now" when a pipeline is frozen or slow.          |

Metrics and logs are continuous and cheap; the `/debug/*` snapshots are pull-on-demand and read state the engine already keeps, so there is no steady-state cost to having them enabled.

## Scrape metrics into Prometheus

Metrics are enabled by default under `runtime.metrics`. The endpoint serves the standard Prometheus exposition format, so any Prometheus-compatible collector scrapes it without translation.

```yaml
runtime:
  http:
    enabled: true
    port: 8080            # default
  metrics:
    enabled: true         # default
    recording-level: info # info | debug | trace (default: info)
    common-tags:          # applied to every meter — handy in a shared Prometheus
      environment: production
      service: word-count
```

A minimal static scrape config:

```yaml
scrape_configs:
  - job_name: stoatflow
    metrics_path: /metrics
    static_configs:
      - targets: ["my-app:8080"]
```

On Kubernetes, prefer service discovery over static targets. With a Prometheus Operator, point a `ServiceMonitor` at the HTTP port; without it, the pod annotations are enough for the default Kubernetes SD relabeling:

```yaml
# Pod template metadata — picked up by Prometheus pod service-discovery
metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/path: "/metrics"
    prometheus.io/port: "8080"
```

```yaml
# Or, with the Prometheus Operator: a ServiceMonitor selecting the app Service
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: word-count
spec:
  selector:
    matchLabels:
      app: word-count
  endpoints:
    - port: http        # the named container/Service port that maps to 8080
      path: /metrics
      interval: 15s
```

::callout{color="warning" icon="i-lucide-shield"}
`/metrics` carries license tier and timing detail when a license is configured. Scrape it from your in-cluster monitoring stack only — never expose the metrics port through internet-facing ingress. The deployment-level exposure rules are on [Kubernetes](https://stoatflow.io/docs/operating/kubernetes).
::

## Build the Grafana view

Grafana reads the same Prometheus series. A single-instance app means no per-replica aggregation to reason about — one target, one set of series, filtered by `application_id` (and your `common-tags`) when several apps share a Prometheus.

The names below are the canonical dotted metric IDs; Prometheus rewrites `.` to `_` and appends the type suffix (`_total` for counters, `_seconds_count` / `_seconds_sum` / `_seconds_max` for timers). The complete catalogue and every tag are on the [metrics reference](https://stoatflow.io/docs/reference/metrics-reference); recording levels and scrape setup are on [Metrics](https://stoatflow.io/docs/runtime/metrics) — this is the operator's working subset. Already have a Grafana view built for Kafka Streams? Skip the rebuild — turn on `runtime.metrics.naming: both` and point it at StoatFlow: [Reusing your Kafka Streams dashboards](https://stoatflow.io/docs/migration/reusing-kafka-streams-dashboards).

A starter overview row, top to bottom:

```promql
# Throughput — records processed per second across all lanes
sum(rate(stoatflow_lane_records_processed_total[1m]))

# End-to-end latency — mean over the scrape window, source timestamp to processing completion.
# e2e.latency is a plain timer: it exports _seconds_count / _seconds_sum / _seconds_max, no
# _bucket series — so use the sum/count ratio for the average and the _max gauge for the peak.
rate(stoatflow_e2e_latency_seconds_sum[5m]) / rate(stoatflow_e2e_latency_seconds_count[5m])
stoatflow_e2e_latency_seconds_max

# Commit health — barriers completing vs. failing
rate(stoatflow_barrier_completed_total[1m])
rate(stoatflow_barrier_failed_total[5m])

# Backpressure — total queued depth as a fraction of capacity
sum(stoatflow_lane_queue_size) / sum(stoatflow_lane_queue_capacity)

# Event-time progress — how far behind wall-clock the watermark is
stoatflow_watermark_lag_ms

# Consumer lag — worst partition
max(stoatflow_consumer_lag_records)
```

::callout{color="info" icon="i-lucide-info"}
True percentiles (P95/P99) are available only for the timers registered with percentile publishing — for example the barrier commit and alignment timers, which export a `{quantile="0.99"}` label series you read directly, e.g. `stoatflow_barrier_latency_seconds{quantile="0.99"}`. The end-to-end and per-record processing-latency timers are plain timers: they publish count, sum, and max only, so a `histogram_quantile(...)` over a `_bucket` series returns no data for them. Use the sum/count average and the `_max` gauge for those, as above. Which timers carry percentiles is listed on [Metrics](https://stoatflow.io/docs/runtime/metrics).
::

With `bind-jvm-metrics: true` (default) the standard `jvm_*` and `system_*` series land on the same endpoint, so a second row covers the runtime itself:

```promql
# Heap used vs. max
jvm_memory_used_bytes{area="heap"} / jvm_memory_max_bytes{area="heap"}

# GC pause rate
rate(jvm_gc_pause_seconds_sum[5m])

# CPU
process_cpu_usage
```

::callout{color="info" icon="i-lucide-info"}
Lane ids follow the Kafka Streams task convention `{subtopology}_{lane}` — `0_0`, `0_15`, `1_7`. To break a panel down per sub-topology, group by a regex matcher: `sum by (lane_id) (rate(stoatflow_lane_records_processed_total{lane_id=~"1_.*"}[1m]))`.
::

::callout{color="warning" icon="i-lucide-triangle-alert"}
**Sub-topology ids shifted in 1.0.0.** StoatFlow now inserts a sub-topology boundary only where an operator genuinely needs the re-keyed record's lane affinity, rather than at every key change ([ADR-138](https://stoatflow.io/docs/concepts/lanes-and-parallelism#where-the-handoff-actually-happens)). Re-keying topologies therefore have **fewer** sub-topologies, and the ids renumber — which silently changes any dashboard, alert or recording rule that pins a `sub_topology` / `subtopology_id` value or a `lane_id=~"N_.*"` regex. Re-check those selectors against a live scrape after upgrading. Setting `stoatflow.topology.sub-topology-split: eager` restores the previous numbering. **Processor-API topologies renumber a second time** in 1.0.0: `process()` / `processValues()` no longer open a boundary after a re-key (`stoatflow.topology.processor-api-key-affinity: off`, matching Kafka Streams); `presumed` restores those.
::

## Alert on the right meters

For a single-instance app, the alerts that matter are the ones that tell you the one process is stuck, falling behind, or about to be evicted — and the license, because an expired license takes the readiness probe down. Wire these into Alertmanager (or your equivalent); the full rationale for each meter is on [Metrics](https://stoatflow.io/docs/runtime/metrics).

| Alert                         | Condition                                                                     | Severity | What it means                                                                                                                                                                                                                          |
| ----------------------------- | ----------------------------------------------------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Barrier failures              | `rate(stoatflow_barrier_failed_total[5m]) > 0`                                | Critical | A commit transaction is failing — the process aborts and restarts on a fatal commit error. Pair with `/debug/barriers`.                                                                                                                |
| Not ready                     | `stoatflow_client_state != 4` for 5m (while expected up)                      | Critical | The app left `RUNNING` — validating, restoring, paused, or stopping.                                                                                                                                                                   |
| License invalid               | `stoatflow_license_valid == 0` with `for: 10m`                                | Critical | Readiness goes 503; renew before the grace budget runs out. The `for: 10m` matters: every deploy reads 0 for the \~5-minute deferred-validation `PENDING` window.                                                                      |
| License expiring              | `stoatflow_license_days_remaining < 14`                                       | Warning  | Lead time to renew.                                                                                                                                                                                                                    |
| Watermark stall               | `delta(stoatflow_watermark_current[5m]) == 0 and stoatflow_client_state == 4` | Warning  | Running but event time isn't advancing — windows won't close.                                                                                                                                                                          |
| High consumer lag             | `max(stoatflow_consumer_lag_records) > 100000`                                | Warning  | Falling behind the input; check throughput and backpressure.                                                                                                                                                                           |
| Backpressure                  | `sum(stoatflow_lane_queue_size) > 0.8 * sum(stoatflow_lane_queue_capacity)`   | Warning  | Lanes can't keep up; a slow operator or downstream call is filling the queues.                                                                                                                                                         |
| Engine restarts on faults     | `rate(stoatflow_engine_restart_total{trigger="replace_thread"}[10m]) > 0`     | Warning  | Processing faults are recovering via in-place engine restart (`REPLACE_THREAD`). Recoveries are clean, but a sustained rate approaches the restart budget and ends in a terminal shutdown — find the root-cause exception in the logs. |
| Standby lag (HA)              | `stoatflow_ha_standby_replication_lag_ms > 5000`                              | Warning  | Under [hot standby](https://stoatflow.io/docs/operating/high-availability), the standby is falling behind — promotion would have catch-up to do.                                                                                       |
| Redundancy below desired (HA) | `stoatflow_ha_redundancy_below_desired == 1`                                  | Critical | Under hot standby, the active has fewer caught-up standby spares than `desired-standbys` — reduced (or no) warm failover redundancy.                                                                                                   |

The standby alerts apply only when [hot standby](https://stoatflow.io/docs/operating/high-availability) is enabled. The `stoatflow.client.state` gauge reports the ordinal of the application lifecycle state. The states are numbered in the order the app moves through them on startup — `CREATED=0`, `STARTING=1`, `VALIDATING_STATE=2`, `RESTORING=3`, `RUNNING=4&#x60; — so **`== 4` is the "healthy and processing" predicate** the alerts above gate on, and any value below 4 means the app has not yet reached (or has left) `RUNNING`. The complete state-to-ordinal mapping and the readiness contract are on [Health checks](https://stoatflow.io/docs/runtime/health-checks) and [Probes](https://stoatflow.io/docs/operating/probes).

## Logs and correlation

The runtime logs to stdout. Out of the box that is **plain text** — Logback's pattern encoder, one line per event — which is fine for `kubectl logs` and local development. If you want structured ingestion into Loki, Elasticsearch, CloudWatch, or any pipeline that parses JSON, add a JSON encoder to your application's Logback configuration; StoatFlow does not impose one.

Set per-package log levels through config (applied to Logback at startup) without rebuilding or editing `logback.xml`:

```yaml
logging:
  level:
    root: INFO
    io.stoatflow: INFO
    # Quieten the Kafka clients in steady state:
    org.apache.kafka: WARN
```

### Correlating a log line to a lane and barrier

Lane and barrier identifiers are not in the log output by default. The engine can publish them via SLF4J's MDC (Mapped Diagnostic Context), but **MDC is off unless you opt in** — set `stoatflow.processing.mdc-enabled: true` in your config:

```yaml
stoatflow:
  processing:
    mdc-enabled: true   # default: false
```

With it enabled, the engine populates these MDC keys while processing:

- **`laneId`** — the lane handling the record or barrier, in the `{subtopology}_{lane}` form (`0_0`, `1_7`). This is the same value the metrics carry as the `lane_id` tag, so you can pivot from a per-lane Grafana panel to that lane's log lines — note the key name differs by case: MDC uses `laneId`, the metric tag is `lane_id`.
- **`barrierId`** — the monotonically increasing commit-barrier identifier. The same id appears in the `/debug/barriers` snapshot, so a slow commit can be followed across logs and the debug endpoint.
- **`sourceTopic`*&#x2A;, &#x2A;*`sourcePartition`*&#x2A;, &#x2A;*`sourceOffset`** — the input record's coordinates, for tracing a specific record through the topology.

These keys only reach your log lines if your Logback pattern references them — for example `%X{laneId}` and `%X{barrierId}` in a pattern encoder, or the equivalent fields in a JSON encoder. Operator names you set explicitly (`Named.as(...)`, `Consumed.as(...)`, …) appear in the topology-related log messages themselves, so a log entry can still be tied back to a specific node in the DAG without MDC.

::callout{color="info" icon="i-lucide-info"}
Correlation works because the identifiers are shared, not because of distributed tracing. Once MDC is enabled and your encoder emits the keys, filter logs to one `laneId` or one `barrierId` and you're looking at exactly the records and commit the corresponding metric series and `/debug/barriers` snapshot describe — no trace context to propagate.
::

## Diagnose a frozen or slow pipeline

Metrics tell you *that* throughput dropped to zero or a commit stopped advancing. The two `/debug/*` endpoints tell you *why*, by snapshotting state the engine already holds. Reach for them when a dashboard shows a stall — `stoatflow_barrier_completed_total` flat, `stoatflow_consumer_lag_records` climbing, throughput at zero — and you need to see what the process is actually doing right now.

Both are gated by `runtime.http.debug.enabled` (default `true`); when disabled they return `404`. They expose internal engine state, so keep them on the in-cluster network and off public ingress.

### `/debug/threads` — what every lane is doing

A JSON snapshot of engine-owned and platform threads with their current state and stack traces. Use it to answer "is a lane blocked, and on what?" — a lane parked in a user `Processor` waiting on a slow REST call looks different from one waiting on a commit, and the stack tells you which.

```bash
# Full snapshot
curl -s localhost:8080/debug/threads

# Only non-runnable threads (the usual starting point for a freeze)
curl -s 'localhost:8080/debug/threads?filter=stuck'

# Engine-named threads and lanes only, deeper stacks
curl -s 'localhost:8080/debug/threads?filter=known&depth=80'
```

Query params (both optional):

| Param    | Default | Effect                                                                                    |
| -------- | ------- | ----------------------------------------------------------------------------------------- |
| `depth`  | `40`    | Max stack frames per thread.                                                              |
| `filter` | `all`   | `all`, `stuck` (only non-`RUNNABLE` threads), or `known` (engine threads and lanes only). |

`filter=stuck` is the fast triage view: if throughput is zero, the threads that aren't `RUNNABLE` are where the work has piled up. The response also surfaces any threads the JVM has detected as deadlocked.

### `/debug/barriers` — where a commit is stuck

A JSON snapshot of the commit pipeline, built to answer "which barrier is stuck, at what phase, and what's holding it up?" from a single call. When `stoatflow_barrier_completed_total` has gone flat, this is the endpoint that tells you whether the commit is waiting on lanes, queued, mid-transaction, or simply idle.

```bash
curl -s localhost:8080/debug/barriers
```

The snapshot includes the last committed barrier and its age, any in-flight commit progress, and a derived &#x2A;*`phase`** field — a single-string summary of where the pipeline is:

| `phase`                   | Meaning                                                |
| ------------------------- | ------------------------------------------------------ |
| `IDLE`                    | No commit in progress; nothing pending.                |
| `AWAITING_LANE_ACKS`      | A barrier is in flight, waiting for lanes to reach it. |
| `WAITING_FOR_COMMIT_GATE` | A barrier is pending, waiting to enter the commit.     |
| `QUEUED_FOR_COMMIT`       | Barrier queued for the commit step.                    |
| `IN_TX_COMMIT`            | The Kafka transaction commit is executing.             |
| `AWAITING_ASYNC_FLUSH`    | The commit is waiting on the asynchronous state flush. |

A healthy app cycles through these quickly; a stuck app sits in one. If `phase` is `AWAITING_LANE_ACKS` and not moving, cross-reference `/debug/threads?filter=stuck` to find which lane is blocked and on what. If a commit is genuinely frozen, the runtime detects the stall, aborts the in-flight transaction, and exits so the orchestrator restarts the process — you'll see this as a spike in `stoatflow.barrier.failed.total` and the restarting instance entering its restoration phase.

::callout{color="info" icon="i-lucide-info"}
The `/debug/*` endpoints are for diagnosis, not routine monitoring — they dump internal engine state on demand. Leave them enabled (free at rest), keep them off public ingress, and for hardened deployments that minimise attack surface set `runtime.http.debug.enabled: false`. The exposure split for every endpoint is on [The REST API](https://stoatflow.io/docs/runtime/rest-api).
::

## A diagnosis workflow

When the dashboards say something is wrong, the order that gets to a cause fastest:

1. **Check `stoatflow.client.state`** — is the app `RUNNING` (`4`)? If it's validating, restoring, or stopping (any value below `4`), the answer is lifecycle, not a freeze. Confirm against `/health/ready`.
2. **Check throughput and lag** — `rate(stoatflow_lane_records_processed_total[1m])` at zero with rising `stoatflow_consumer_lag_records` means the process stopped making progress.
3. **Check commit progress** — flat `stoatflow_barrier_completed_total` points at the commit pipeline. Pull `/debug/barriers` and read the `phase`.
4. **Find the blocked lane** — if `phase` is waiting on lanes, `curl '…/debug/threads?filter=stuck'` and read the stacks for the lane that isn't making progress.
5. **Correlate with logs** — with MDC enabled, filter the logs to the offending `laneId` and `barrierId` to see the records and errors around the stall.

## RocksDB metrics

For persistent (RocksDB-backed) stores, StoatFlow can expose the same signals Kafka Streams does — cache hit ratios, compaction and flush activity, write stalls, memtable and SST sizes — under `stoatflow.rocksdb.*`, tagged `store_name`. They are **off by default**; two flags gate them:

```yaml
stoatflow:
  rocks-db:
    metrics:
      enabled: true             # property + block-cache gauges (recording-level info) — negligible cost
      statistics-enabled: true  # + ticker/histogram/ratio family (recording-level debug) — RocksDB write-path
                                # cost, commonly cited ~5–10% on write-heavy loads; wired at store open (restart to toggle)
```

The property/cache half (`enabled`) is cheap — a handful of native reads per store per minute — and is what confirms a sizing diagnosis: `stoatflow_rocksdb_shared_block_cache_usage_bytes` vs `..._capacity_bytes` tells you whether the block cache is full, and `stoatflow_rocksdb_estimate_num_keys` / `..._total_sst_files_size_bytes` track state growth. The statistics half (`statistics-enabled`) adds the health signals that need RocksDB's internal counters:

- `stoatflow_rocksdb_block_cache_hit_ratio` (+ `data`/`index`/`filter` variants) — **per-interval**; a falling ratio under load is the classic "cache too small" symptom (see [Tuning → RocksDB](https://stoatflow.io/docs/operating/tuning)).
- `stoatflow_rocksdb_write_stall_duration_avg_ms` and `..._total_ms` — writes stalling on compaction back-pressure.
- `rate(stoatflow_rocksdb_compaction_bytes_written_total[5m])` and the compaction/flush time histograms — compaction load, useful when segment-checkpoint churn is suspected.

Ratios and histogram `avg`/`min`/`max` are per-interval gauges; tickers are `.total` counters — use PromQL `rate()`. Two caveats: the counters **reset on an in-place engine restart** (a normal Prometheus counter reset, absorbed by `rate()`), and on the default (FFM) backend the histogram `*.min.ms` series is unavailable — use `avg`/`max`. The block cache is shared across all stores, so `stoatflow_rocksdb_shared_block_cache_*` (no `store_name`) is the global truth; the per-store `stoatflow_rocksdb_block_cache_*` gauges repeat it for Kafka Streams dashboard portability.

::callout{color="info" icon="i-lucide-line-chart"}
**Coming from Kafka Streams?** Rather than rebuild these panels on `stoatflow.*` names, turn on `runtime.metrics.naming: both` and StoatFlow also emits Kafka-Streams-named series (including the RocksDB rows above) so your existing KS dashboards light up. See [Reusing your Kafka Streams dashboards](https://stoatflow.io/docs/migration/reusing-kafka-streams-dashboards).
::

## Next steps

- **[Metrics](https://stoatflow.io/docs/runtime/metrics)** — scraping, recording levels, tags, and license metrics; the complete meter catalogue is on the [metrics reference](https://stoatflow.io/docs/reference/metrics-reference).
- **[Probes](https://stoatflow.io/docs/operating/probes)** — liveness and readiness, and how readiness holds traffic off during restoration.
- **[The REST API](https://stoatflow.io/docs/runtime/rest-api)** — every operational endpoint, with the in-cluster vs. public exposure split.
- **[Tuning](https://stoatflow.io/docs/operating/tuning)** — the bound knobs behind the self-tuning commit cadence, and the latency/throughput/restart trade-offs.
- **[High availability](https://stoatflow.io/docs/operating/high-availability)** — the hot-standby meters and what to alert on for the cluster.
- **[Production checklist](https://stoatflow.io/docs/operating/production-checklist)** — the pre-flight list before going live.


# Tuning under load

StoatFlow ships with defaults that work for most topologies, and the parts that need to adapt to your workload — commit cadence and epoch size — self-tune within the bounds you set. This page covers the knobs you do set: lane count, the commit-cadence bounds, the memory and uncommitted-state limits, and RocksDB sizing. Each section starts from a **symptom** so you know when to reach for it.

For the why behind these mechanics, see [Lanes and parallelism](https://stoatflow.io/docs/concepts/lanes-and-parallelism), [Exactly-once](https://stoatflow.io/docs/concepts/exactly-once), and [State and thread safety](https://stoatflow.io/docs/concepts/state-and-thread-safety). For where each value comes from and how it's merged, see the [Configuration model](https://stoatflow.io/docs/concepts/configuration-model) and [Defaults and presets](https://stoatflow.io/docs/configuration/defaults-and-presets).

::tldr-panel
- **Lane count** scales per-key parallelism — start at CPU cores, raise it when processing is I/O-bound, not CPU-bound.
- **Commit cadence self-tunes** between `commit-barrier.min-interval-ms` and `max-interval-ms`. You set the bounds; the engine picks the cadence inside them.
- **Uncommitted state** is bounded by `state.uncommitted-max-bytes` — the limit that protects you from OOM on stateful, high-fan-out topologies.
- **RocksDB memory** is a preset (`rocks-db.preset`): `DEFAULT` (256 MiB), `LOW_MEMORY` (64 MiB), `HIGH_PERFORMANCE` (1 GiB).
::

## Before you tune

Tune one thing at a time, and tune against a representative load, not a synthetic burst. The defaults are deliberately conservative; most workloads need at most one or two of the knobs below. Always confirm the symptom first — reach for `/metrics` and the `/debug/barriers` endpoint (see [REST API](https://stoatflow.io/docs/runtime/rest-api)) rather than guessing.

::callout{color="warning" icon="i-lucide-flask-conical"}
All values below are the **defaults from the engine config**. Every knob has a YAML key under `stoatflow.*` and an environment-variable override (`STOATFLOW__<SECTION>__<KEY>`). The full list — with types, ranges, and overrides — is in the [Configuration reference](https://stoatflow.io/docs/reference/configuration-reference).
::

## Lane count vs cores

**Knob:** `stoatflow.lanes.count` — default `max(2, CPU cores)`.

Lanes are the unit of per-key parallelism: every record is routed to a lane by its key, and each lane is a virtual thread that processes its keys in order. More lanes means more keys processed concurrently, at the cost of more queues and a little more dispatch overhead.

```yaml
stoatflow:
  lanes:
    count: 16
```

How to size it:

- **CPU-bound processing** (pure transforms, local aggregation): start at the number of CPU cores. Going much higher just adds scheduling overhead without more throughput, because the work is already saturating the cores.
- **I/O-bound processing** (enrichment calls, blocking lookups, external APIs): go well above core count — virtual threads park cheaply while blocked, so 4×–8× cores keeps cores busy while many lanes wait on I/O. This is the case where raising the lane count pays off most.
- **Few distinct keys:** lanes only help if keys spread across them. If your traffic is dominated by a handful of hot keys, extra lanes sit idle — the hot keys still serialize onto their assigned lanes. Fix the key distribution first.

| Symptom                                                                  | Likely cause                                      | Reach for                              |
| ------------------------------------------------------------------------ | ------------------------------------------------- | -------------------------------------- |
| Cores underused, throughput plateaus, processing involves blocking calls | Too few lanes for an I/O-bound topology           | Raise `lanes.count` (4×–8× cores)      |
| High CPU, lots of context switching, no throughput gain from more lanes  | Over-provisioned lanes for CPU-bound work         | Lower `lanes.count` toward core count  |
| A few keys dominate; most lanes idle                                     | Skewed key distribution, not a lane-count problem | Re-key upstream; lane count won't help |

::callout{color="info" icon="i-lucide-info"}
Lane count is fixed at startup and cannot be changed while running — it's part of the engine's single-instance topology. Changing it requires a restart. See [Lanes and parallelism](https://stoatflow.io/docs/concepts/lanes-and-parallelism).
::

### Lane queue capacity

**Knob:** `stoatflow.lanes.queue-capacity` — default `300`.

Each lane has a bounded queue; when a lane fills, the dispatcher applies backpressure. Raise it if you see bursty input stalling dispatch and you have memory headroom; the default is sufficient for steady-state load.

```yaml
stoatflow:
  lanes:
    queue-capacity: 500
```

## Commit cadence

StoatFlow commits in **epochs** — the work between two commit barriers becomes one Kafka transaction plus one state-store flush (see [Exactly-once](https://stoatflow.io/docs/concepts/exactly-once)). The cadence — how often a barrier fires — is **self-tuning**: the engine adapts it within the bounds you configure, so you set the envelope, not the exact interval. The algorithm itself stays in the source; what you control is the floor and ceiling.

**The bounds you set:**

| Knob             | YAML key                         | What it does                                              |
| ---------------- | -------------------------------- | --------------------------------------------------------- |
| Seed interval    | `commit-barrier.interval-ms`     | Starting interval before the cadence settles              |
| Interval factor  | `commit-barrier.interval-factor` | Target interval as a multiple of measured commit duration |
| Minimum interval | `commit-barrier.min-interval-ms` | Floor — barriers never fire closer together than this     |
| Maximum interval | `commit-barrier.max-interval-ms` | Ceiling — barriers never fire further apart than this     |

Defaults and `StreamsConfig` field names for every `commit-barrier.*` knob are in [Engine configuration → Commit-barrier cadence](https://stoatflow.io/docs/configuration/core-config#commit-barrier-cadence).

```yaml
stoatflow:
  commit-barrier:
    min-interval-ms: 150
    max-interval-ms: 5000
    interval-factor: 1.5
```

The trade-off is the classic latency-vs-throughput one:

- **Tighter, more frequent commits** (lower `max-interval-ms`, lower `interval-factor`) reduce end-to-end latency and bound how much work a crash replays — at the cost of more transaction and flush overhead per record.
- **Looser, less frequent commits** (higher bounds) amortize commit overhead over more records, raising throughput — at the cost of higher latency and a larger replay window after a restart.

| Symptom                                                | Likely cause                                         | Reach for                                          |
| ------------------------------------------------------ | ---------------------------------------------------- | -------------------------------------------------- |
| End-to-end latency too high; output lags input         | Cadence too loose; records wait for the next barrier | Lower `max-interval-ms` (and/or `interval-factor`) |
| Throughput capped; commit overhead dominates a profile | Cadence too tight for the volume                     | Raise `max-interval-ms`                            |
| Output is bursty in lockstep with commits              | Working as designed — output is released at barriers | Tighten cadence if the burstiness hurts downstream |

::callout{color="info" icon="i-lucide-info"}
`commit-barrier.timeout-ms` is a **safety bound**, not a cadence knob. It caps how long a commit may take; if a commit stalls past it, the engine aborts the in-flight transaction and the process restarts rather than freezing silently. Don't lower it to force faster commits — see [Probes](https://stoatflow.io/docs/operating/probes).
::

### Epoch size

You normally don't touch epoch size — it self-tunes within the bounds above. Two knobs exist for the edges:

| Knob                          | YAML key                                   | When to use                                                                                                          |
| ----------------------------- | ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------- |
| Hard cap on records per epoch | `commit-barrier.max-epoch-records`         | Unset by default (no cap) — set only if you need a deterministic upper bound on epoch size regardless of self-tuning |
| Initial cap before warmup     | `commit-barrier.initial-max-epoch-records` | Lower for conservative startup on heavy-state topologies; `-1` for an unlimited first epoch                          |

```yaml
stoatflow:
  commit-barrier:
    initial-max-epoch-records: 1024   # conservative startup for heavy-state topologies
    # max-epoch-records: 25000        # optional deterministic ceiling
```

## Memory and uncommitted state

Between commits, a stateful topology accumulates **uncommitted state** in memory — the writes buffered since the last barrier, held until the transaction commits (KIP-892 transactional state stores). On topologies where one input record fans out into many state writes (multi-way joins, foreign-key joins, high-cardinality aggregations), this is the value most likely to threaten container memory.

**Knob:** `stoatflow.state.uncommitted-max-bytes` — default `268435456` (256 MiB).

When buffered uncommitted bytes approach this limit, the engine triggers an early commit barrier to flush state and bound peak memory. This is the backstop that keeps a high-fan-out epoch from running the container out of memory.

```yaml
stoatflow:
  state:
    uncommitted-max-bytes: 268435456   # 256 MiB (default)
```

The global ceiling has two **per-store** companions under `stoatflow.caching`: `max-estimated-bytes` (default 256 MiB per caching store) and `max-entries` (default unbounded). Exceeding a per-store cap fires the same early commit barrier — tagged `CACHE_PRESSURE` on the barrier-trigger metric — so one hot store can't monopolise the buffer. In practice `uncommitted-max-bytes` measures a superset and fires first; leave `max-entries` unbounded unless a specific store must bound its per-epoch key cardinality.

Sizing it against the container:

| Container memory limit | `uncommitted-max-bytes` | Rationale                                                       |
| ---------------------- | ----------------------- | --------------------------------------------------------------- |
| 4 GiB                  | \~1 GiB                 | Leaves headroom for heap, RocksDB, Kafka client buffers, and GC |
| 8 GiB                  | 1–2 GiB                 | More room for GC and producer buffers                           |
| 16 GiB                 | 2–4 GiB                 | Production sizing for heavy stateful topologies                 |

::callout{color="warning" icon="i-lucide-triangle-alert"}
`uncommitted-max-bytes` is one of several memory consumers in the process — it does **not** account for the JVM heap, the RocksDB memory budget (next section), or the Kafka producer's send buffer. On small-heap deployments with both heavy state writes and heavy sink output, leave generous headroom and watch container memory before raising it. [Defaults and presets](https://stoatflow.io/docs/configuration/defaults-and-presets#where-the-rocksdb-budget-sits-in-the-pod) has all four caps drawn to scale against a container, which is the quickest way to see how much of the limit the defaults already spend.
::

| Symptom                                                          | Likely cause                                                                                                                                                          | Reach for                                                                                                                     |
| ---------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| Container memory spikes near the limit; OOMKills under load      | Uncommitted state per epoch too large for the container                                                                                                               | Lower `uncommitted-max-bytes`, or raise the container limit if you have hardware                                              |
| Very frequent commits; throughput suffers on a stateful topology | Early barriers firing on memory pressure (`MEMORY_PRESSURE` trigger) or a per-store cache cap (`CACHE_PRESSURE` trigger) — the barrier-trigger metric tells you which | Raise `uncommitted-max-bytes`, or the tripping store's `caching.max-estimated-bytes` (only if the container has the headroom) |
| Slow time-to-first-commit on a heavy-state topology at startup   | First epoch buffering too much before warmup                                                                                                                          | Lower `commit-barrier.initial-max-epoch-records`                                                                              |

## Segmented-store checkpoint cadence

Time-windowed and session stores are *segmented*, and their RocksDB segments run with the write-ahead log off — so they only become durable on a clean `close()`. A periodic checkpoint (flush dirty segments, then the offsets) makes them durable between commits and caps how much changelog a pod replays after an *ungraceful* restart, and how much a [hot-standby](https://stoatflow.io/docs/operating/high-availability) standby drains on promotion — to at most one checkpoint interval.

**Knobs:** `stoatflow.state.segment-checkpoint-records` (default `100000`) and `stoatflow.state.segment-checkpoint-ms` (default `30000`), whichever fires first.

The defaults are deliberately coarse: a checkpoint flushes SST files, so a tight cadence trades a smaller recovery window against more SST/compaction churn in steady state. Lower them only if your recovery-time objective needs a shorter ungraceful-restart catch-up than \~30 s / 100k records of windowed writes; raise them if you see checkpoint-driven write amplification and can absorb a longer catch-up.

## RocksDB for large state

Persistent state lives in RocksDB. Unlike Kafka Streams' unbounded default, StoatFlow caps RocksDB memory by default to keep the process within its container. You pick a **preset**; the framework wires the block cache and memtable budget for you.

**Knob:** `stoatflow.rocks-db.preset` — default `DEFAULT`.

| Preset             | Total RocksDB memory | Use for                                                  |
| ------------------ | -------------------- | -------------------------------------------------------- |
| `LOW_MEMORY`       | 64 MiB               | Constrained containers, small state                      |
| `DEFAULT`          | 256 MiB              | Most workloads                                           |
| `HIGH_PERFORMANCE` | 1 GiB                | Large state, read-heavy access patterns, high throughput |

```yaml
stoatflow:
  rocks-db:
    preset: HIGH_PERFORMANCE
```

When state grows large (millions of keys, hot read paths), `DEFAULT`'s 256 MiB block cache becomes a bottleneck: working-set reads start missing the cache and hit disk. `HIGH_PERFORMANCE` widens both the block cache and the memtable budget. Budget for it — it is real resident memory on top of the JVM heap and `uncommitted-max-bytes`, so size the container accordingly.

**Confirm the diagnosis before you resize.** Enable the [RocksDB metrics](https://stoatflow.io/docs/operating/observability#rocksdb-metrics) (`rocks-db.metrics.enabled` + `statistics-enabled`) and watch `stoatflow_rocksdb_block_cache_hit_ratio` under load: a ratio falling toward the floor while `stoatflow_rocksdb_shared_block_cache_usage_bytes` sits at capacity is the signal that the cache is too small — the metric that turns the symptom below from a guess into a measurement.

| Symptom                                                                             | Likely cause                              | Reach for                                                         |
| ----------------------------------------------------------------------------------- | ----------------------------------------- | ----------------------------------------------------------------- |
| Read latency climbs as state grows; disk I/O high (falling `block.cache.hit.ratio`) | Block cache too small for the working set | `rocks-db.preset: HIGH_PERFORMANCE`                               |
| Memory tight on a small container with modest state                                 | RocksDB budget larger than needed         | `rocks-db.preset: LOW_MEMORY`                                     |
| Need a budget between the presets                                                   | Preset granularity too coarse             | Set a custom total via a programmatic `RocksDBConfig` (see below) |

### Backend selection

**Knob:** `stoatflow.rocks-db.backend` — default `AUTO`.

`AUTO` picks each runtime's faster path: the `FFM` (Foreign Function & Memory API) backend on the JVM — lower per-call overhead than JNI and no virtual-thread pinning during native RocksDB calls — and `JNI` under a native image, where FFM's critical-downcall advantage doesn't apply. Leave it on `AUTO` unless you have a specific reason to pin a backend; if you force `JNI` on the JVM, also set `stoatflow.lanes.thread-type: PLATFORM` to avoid pinning carrier threads.

### Custom RocksDB budget (programmatic)

The YAML preset is the supported path for most deployments. If you need a memory budget between the presets, set `RocksDBConfig` directly when building the engine config — `RocksDBConfig.withTotalMemory(...)` splits the budget evenly between block cache and memtables:

::code-tabs{group="lang"}
```kotlin [Kotlin]
import io.stoatflow.core.config.StreamsConfig
import io.stoatflow.core.state.rocksdb.RocksDBConfig

val config = StreamsConfig.builder("my-app", "localhost:9092")
    .rocksDbConfig(RocksDBConfig.withTotalMemory(512L * 1024 * 1024)) // 512 MiB total
    .build()
```

```java [Java]
import io.stoatflow.core.config.StreamsConfig;
import io.stoatflow.core.state.rocksdb.RocksDBConfig;

var config = StreamsConfig.builder("my-app", "localhost:9092")
    .rocksDbConfig(RocksDBConfig.withTotalMemory(512L * 1024 * 1024L)) // 512 MiB total
    .build();
```
::

## Watching the effect

Tune against signals, not hunches. After each change, watch:

- **`/metrics`** — throughput, consumer lag, commit timing, and JVM/container memory, scraped by Prometheus. See [Metrics](https://stoatflow.io/docs/runtime/metrics) and [Observability](https://stoatflow.io/docs/operating/observability).
- **`/debug/barriers`** — the live commit-pipeline state, which tells you whether barriers are firing on schedule and what's triggering them. See [REST API](https://stoatflow.io/docs/runtime/rest-api).
- **`/health/ready`** — readiness flips DOWN if the engine can't keep up or a commit stalls. See [Probes](https://stoatflow.io/docs/operating/probes).

A clean tuning loop: change one knob → run representative load → read the metric that maps to the symptom → keep or revert. Resist changing several knobs at once; you won't know which one moved the needle.

## Next steps

- **[Defaults and presets](https://stoatflow.io/docs/configuration/defaults-and-presets)** — where every default comes from and how layers merge.
- **[Configuration reference](https://stoatflow.io/docs/reference/configuration-reference)** — every key, type, range, default, and environment override.
- **[Production checklist](https://stoatflow.io/docs/operating/production-checklist)** — the pre-flight list before you take a topology to production.
- **[Observability](https://stoatflow.io/docs/operating/observability)** — wiring metrics and dashboards so the symptoms above show up before users notice.


# Production checklist

Before you put a StoatFlow application in front of production traffic, walk this list. Each item is a decision or a wiring step you can verify, and each links to the page that covers it in full — this page is the index, not the explanation.

::tldr-panel
- **What this is:** a pre-flight checklist for going live — one row per decision, each linking to its canonical page.
- **Non-negotiables:** exactly one *active* instance, a deliberate HA tier, a valid license, readiness wired to hold traffic during restoration, and a deliberate error policy.
- **How to use it:** confirm each item against the linked page; nothing here repeats the detail, it points at it.
::

## Deployment and the HA tier

The one rule an operator must not break: run **exactly one *active* instance**. Two *active* instances pointed at the same source topics corrupt state. Then choose your HA tier — fast restart (default) or the opt-in hot-standby cluster — deliberately.

- **HA tier chosen deliberately.** Fast restart (default, `replicas: 1`) for most workloads, or the opt-in hot-standby cluster (`replicas: 2` or more, `ha.mode: active-standby`) for near-instant failover on large state / tight RTO. → [High availability](https://stoatflow.io/docs/operating/high-availability)
- **`replicas: 1`** on the workload (default mode), with a rollout strategy that tears the old pod down before the new one starts processing — never two *active* concurrently. Hot standby is the exception: a readiness-gated cluster of `replicas: 2` or more. → [Kubernetes](https://stoatflow.io/docs/operating/kubernetes), [High availability](https://stoatflow.io/docs/operating/high-availability)
- **Failure mode of two active instances understood**, and why the standby is passive. → [Architecture](https://stoatflow.io/docs/concepts/architecture), [Deploying and operating](https://stoatflow.io/docs/operating)
- **Stable `application-id`.** It is the application's identity — the consumer group and changelog topics derive from it. Two *active* processes sharing it collide. → [Core configuration](https://stoatflow.io/docs/configuration/core-config)
- **Graceful shutdown on `SIGTERM`.** The runtime drains in-flight records and commits a final barrier before exiting; give the orchestrator's termination grace period enough headroom (and, under hot standby, room for the handoff). → [Kubernetes](https://stoatflow.io/docs/operating/kubernetes)

::callout{color="error" icon="i-lucide-alert-triangle"}
**Never run two *active* replicas of the same StoatFlow application against the same source topics.** This is the single most important item on the list. The supported way to run more than one replica is the passive [hot-standby](https://stoatflow.io/docs/operating/high-availability) cluster. → [Deploying and operating](https://stoatflow.io/docs/operating)
::

## State persistence and restart

Fast restart — the default HA tier — depends on whether the new pod starts with a warm local store or rebuilds state from the changelog cold. (Hot standby sidesteps the restore entirely by keeping one or more warm standbys ready — see [High availability](https://stoatflow.io/docs/operating/high-availability).)

- **Persistent volume for warm restart.** A `PersistentVolumeClaim` retained across restarts means restoration only reads the changelog gap since the last commit, not the whole topic. → [Kubernetes](https://stoatflow.io/docs/operating/kubernetes)
- **Restart window understood.** Cold-start time scales with state size; the levers are state size, disk persistence, and broker read throughput. → [Tuning](https://stoatflow.io/docs/operating/tuning), [Benchmarks](https://stoatflow.io/product/benchmarks)
- **Recovery semantics confirmed** for your chosen processing guarantee — the last successful commit barrier is the recovery point. → [Exactly-once](https://stoatflow.io/docs/concepts/exactly-once)

## Processing guarantee

Pick exactly-once or at-least-once deliberately — it is the central reliability decision and it shapes your latency floor.

- **Exactly-once vs at-least-once chosen** on purpose, not by default. Exactly-once commits state, output, and offsets atomically on a barrier; at-least-once trades possible duplicates for a lower commit-cadence floor on latency. → [Exactly-once](https://stoatflow.io/docs/concepts/exactly-once)
- **Downstream consumers match the mode.** Under exactly-once, downstream reads with `read_committed` isolation see no duplicates; under at-least-once, downstream must be idempotent or duplicate-tolerant. → [Exactly-once](https://stoatflow.io/docs/concepts/exactly-once)

## License

A production instance needs a valid runtime license, injected as a secret — never baked into an image or committed.

- **License key injected** as an environment variable / secret, not hardcoded. → [License configuration](https://stoatflow.io/docs/getting-started/license-configuration)
- **License health is wired into readiness.** An expired, revoked, or missing license flips `/health/ready` to not-ready; the renewal window surfaces a warning before it bites. Note the timing: a missing key fails at boot, but invalid/expired keys surface at the *deferred* validation \~5 minutes after start (the pod runs `PENDING`-and-ready until then). → [License configuration](https://stoatflow.io/docs/getting-started/license-configuration), [Health checks](https://stoatflow.io/docs/runtime/health-checks)

## Probes

Readiness is the lever that keeps traffic off a process that hasn't caught up. Wire both probes.

- **`/health/ready` wired as the readiness probe.** It returns 503 throughout state restoration, so nothing routes to a process that isn't caught up. → [Probes](https://stoatflow.io/docs/operating/probes), [Health checks](https://stoatflow.io/docs/runtime/health-checks)
- **`/health/live` wired as the liveness probe.** Lets the orchestrator restart a genuinely stuck process. → [Probes](https://stoatflow.io/docs/operating/probes)
- **Probe timing tuned** so a long but legitimate restoration isn't killed by an aggressive liveness deadline. → [Probes](https://stoatflow.io/docs/operating/probes)

## Observability

You operate one process, so its metrics and introspection endpoints are your whole picture.

- **`/metrics` scraped** into your monitoring stack — JVM, Kafka-client, and StoatFlow internal counters in Prometheus format. → [Metrics](https://stoatflow.io/docs/runtime/metrics), [Observability](https://stoatflow.io/docs/operating/observability)
- **Alerts set** for the signals that matter on a single-instance app — consumer lag, commit stalls, error rates, restoration progress. → [Observability](https://stoatflow.io/docs/operating/observability)
- **Logs collected** — plain text by default; add a JSON encoder to your Logback config for structured ingestion, and enable MDC to correlate by lane and barrier. → [Observability](https://stoatflow.io/docs/operating/observability)
- **Introspection endpoints reachable** for diagnosis — `/topology`, `/state`, `/watermarks`, and the `/debug/*` views. → [REST API](https://stoatflow.io/docs/runtime/rest-api), [Observability](https://stoatflow.io/docs/operating/observability)

## Error handling

Decide what happens to a bad record before one arrives — silent skipping should be a choice, not a default.

- **Processing-exception handler chosen** — log-and-continue, log-and-fail, or route to a dead-letter queue. → [Error handling (DLQ)](https://stoatflow.io/docs/building/error-handling-dlq), [Error-handling model](https://stoatflow.io/docs/concepts/error-handling-model)
- **Deserialization-error handler chosen** for malformed source records, with its own policy. → [Error handling (DLQ)](https://stoatflow.io/docs/building/error-handling-dlq)
- **DLQ topic provisioned** if you route to one, and monitored — a filling DLQ is a signal. → [Error handling (DLQ)](https://stoatflow.io/docs/building/error-handling-dlq), [Observability](https://stoatflow.io/docs/operating/observability)

## Configuration and tuning

Start from the presets, then tune against your workload rather than guessing.

- **Defaults and presets reviewed** — you know which preset you're running and why. → [Defaults and presets](https://stoatflow.io/docs/configuration/defaults-and-presets)
- **Resources sized** — CPU and memory for the single process, lane count for your core count. → [Tuning](https://stoatflow.io/docs/operating/tuning)
- **Commit-cadence bounds set** appropriately for your latency-vs-throughput target. The cadence self-tunes within the configured bounds; you set the bounds. → [Tuning](https://stoatflow.io/docs/operating/tuning), [Configuration model](https://stoatflow.io/docs/concepts/configuration-model)
- **Kafka client config reviewed** — bootstrap servers, security, and any client overrides. → [Kafka client configuration](https://stoatflow.io/docs/configuration/kafka-client-config)
- **`-XX:+UseG1GC` set** so GC pauses stay short and predictable (the Gradle plugin applies it to the Docker image; set it explicitly otherwise). → [Installation](https://stoatflow.io/docs/getting-started/installation), [Docker](https://stoatflow.io/docs/runtime/docker)

## Custom processor correctness

If you wrote custom `Processor` code, make sure it holds under concurrent lane execution.

- **Thread safety understood** — key affinity makes per-key updates serial, but custom processors that read-modify-write across multiple keys need the runtime's key-lock utility. → [State and thread safety](https://stoatflow.io/docs/concepts/state-and-thread-safety)
- **No cross-lane state assumptions** in custom code. → [State and thread safety](https://stoatflow.io/docs/concepts/state-and-thread-safety), [Processor API](https://stoatflow.io/docs/building/processor-api)
- **Topology tested** with the in-memory test driver before deploying. → [Testing](https://stoatflow.io/docs/building/testing)

## Where to go next

- **[Kubernetes](https://stoatflow.io/docs/operating/kubernetes)** — the manifest that satisfies the deployment and persistence rows above.
- **[Observability](https://stoatflow.io/docs/operating/observability)** — what to scrape and what to alert on once you're live.
- **[Tuning](https://stoatflow.io/docs/operating/tuning)** — turn the sizing and cadence rows into concrete numbers for your workload.
- **[High availability](https://stoatflow.io/docs/operating/high-availability)** — the opt-in hot-standby cluster, if fast restart isn't fast enough.
- **[Deploying and operating](https://stoatflow.io/docs/operating)** — the operating section overview, including both HA tiers in full.


# Migrating from Kafka Streams

Moving a Kafka Streams application to StoatFlow is mostly a port, not a rewrite: the part you spend the most time on — the topology — carries over largely unchanged, and what changes is the runtime model around it. This page frames the migration conceptually: what stays the same, what differs operationally, and how to choose between starting clean and carrying state across. The two sub-pages then walk the concrete steps for each path.

::tldr-panel
- **The topology code carries over.** Same DSL, same Processor API, same `Consumed` / `Produced` / `Materialized` / `Grouped` config objects. The imports and the entry point change; the operators don't.
- **The runtime model changes.** Single instance instead of a rebalancing cluster, in-memory re-keying instead of repartition topics, one commit barrier instead of per-task transactions, global state instead of partition-scoped state.
- **The decision that drives everything: state.** A topology with no meaningful state, or one you can rebuild by reprocessing input, is a clean cutover. A topology whose correctness depends on accumulated state needs a state-carrying plan.
- **The port itself is mechanized.** The [automated port](https://stoatflow.io/docs/migration/automated-port) recipe rewrites the imports, the entry point and the build across Java and Kotlin, and flags every judgment call rather than guessing.
- **Two paths for your state:** [Without data migration](https://stoatflow.io/docs/migration/without-data-migration) (green-field cutover, reprocess from input) and [With data migration](https://stoatflow.io/docs/migration/with-data-migration) (carry existing state across).
::

## What carries over unchanged

StoatFlow implements the Kafka Streams DSL. The topology you already wrote — `StreamsBuilder`, `KStream` / `KTable` / `KGroupedStream`, the windowed and session variants, the joins, the Processor API, and the KS-compatible functional interfaces (`ValueMapper`, `KeyValueMapper`, `ValueJoiner`, `Reducer`, `Aggregator`, …) — is the same code on both engines. The method-by-method status is in the [KS compatibility matrix](https://stoatflow.io/docs/reference/ks-compatibility-matrix).

The same map-filter chain is written the same way against either engine:

::code-tabs{group="lang"}
```kotlin [Kotlin]
val intermediate =
    stream1
        .selectKey { _, _ -> "lala" }
        .map { key, value -> KeyValue(value.substring(0, 3), "$key:$value") }
        .filter { _, value -> value.length > 5 }
        .mapValues { value -> value.uppercase() }

intermediate.to(
    "output-topic",
    Produced.`as`<String, String>("sink1")
        .withKeySerde(Serdes.String())
        .withValueSerde(Serdes.String()),
)
```

```java [Java]
KStream<String, String> intermediate =
    stream1
        .selectKey((k, v) -> "lala")
        .map((key, value) -> KeyValue.pair(value.substring(0, 3), key + ":" + value))
        .filter((k, value) -> value.length() > 5)
        .mapValues(value -> value.toUpperCase());

intermediate.to(
    "output-topic",
    Produced.<String, String>as("sink1")
        .withKeySerde(Serdes.String())
        .withValueSerde(Serdes.String()));
```
::

Two things shift even on the unchanged-code path:

- **Imports.** StoatFlow's DSL lives under `io.stoatflow.core.topology.*` (and the runtime under `io.stoatflow.runtime.*`) rather than `org.apache.kafka.streams.*`. This is a package rename across your topology source, not a logic change.
- **Entry point.** Where Kafka Streams takes a `Topology` plus a `Properties` and a `KafkaStreams` object you `start()`, StoatFlow has two front doors — `StoatFlow.fromBuilder(config, builder)` on `:core` (DSL and engine only) and `StoatFlowRuntime.fromConfig(...)` on `:runtime` (the batteries-included wrapper that loads `application.yaml` and starts the HTTP admin, metrics, and health endpoints). See [Modules overview](https://stoatflow.io/docs/getting-started/modules-overview).

## What changes — and where it bites

The DSL is stable; the engine underneath is a different shape. None of these require topology rewrites, but they change how you configure, deploy, and reason about the application. Each is covered in depth on its concept page — this is the orientation.

| Area            | Kafka Streams                                         | StoatFlow                                                        | Migration impact                                                                    |
| --------------- | ----------------------------------------------------- | ---------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
| Deployment      | Multiple instances, rebalancing consumer group        | One JVM, single-member group, no rebalancing                     | You deploy one pod, sized by cores and memory — not a fleet sized by instance count |
| Exactly-once    | Per-task transactions, coordinated across the cluster | One commit barrier → one transaction, whole topology             | EOS is the default, not an opt-in you tune                                          |
| Config contract | `StreamsConfig` properties tuned per deployment       | Typed config / `application.yaml`, with KS-specific keys dropped | Some KS config keys have no meaning here and are removed (below)                    |

The same shift runs through parallelism (lane count is a config knob, not a topic-layout decision), re-keying (internal repartition topics disappear — `repartition()` becomes in-process), and state (global — no co-partitioning requirement, no partition routing for queries). The full aspect-by-aspect table, and the conceptual *why* behind each row, are on [How StoatFlow differs from Kafka Streams](https://stoatflow.io/docs/concepts/how-stoatflow-differs-from-ks#the-deltas-at-a-glance); the model in its own right is on [Architecture](https://stoatflow.io/docs/concepts/architecture).

### Config keys that go away

A subset of Kafka Streams configuration encodes assumptions that the single-instance model removes. These keys have no StoatFlow equivalent and are dropped during the port:

- **Instance- and rebalancing-related** — `num.stream.threads`, standby-replica settings, static-membership and cooperative-rebalancing tuning. There is no group to rebalance and no second *active* instance, so these KS knobs have nothing to act on. (StoatFlow's own opt-in [hot standby](https://stoatflow.io/docs/operating/high-availability) is a separate mechanism under `stoatflow.ha.*`, not a port of these.)
- **Exactly-once enablement** — the KS `processing.guarantee` opt-in and its associated transactional tuning. StoatFlow commits the whole topology under one barrier-linked transaction by default; you choose exactly-once or at-least-once as a mode, not by assembling the per-task transactional machinery yourself.
- **Partitioning that assumes a cluster** — `withPartition(...)` on interactive queries, partitioner settings on `TableJoined` (`withPartitioner` / `withOtherPartitioner`), and the repartition-topic partition counts. With one instance and global state there is nothing to route across.

::callout{color="info" icon="i-lucide-info"}
This is removal, not translation: there is no StoatFlow key that "replaces" `num.stream.threads`. Parallelism is expressed as lane count instead. The full key-by-key reference is in the [configuration reference](https://stoatflow.io/docs/reference/configuration-reference); the model is on [Configuration model](https://stoatflow.io/docs/concepts/configuration-model).
::

### API-surface differences

A handful of Kafka Streams APIs deliberately diverge. Some are accepted but inert because the single-instance model gives them nothing to act on — the partitioner methods on `TableJoined`, for example, compile and do nothing. Some are stricter than KS — a multicast `StreamPartitioner` on `repartition()` is rejected. One is *more* permissive: a `processor.wrapper.class` (KIP-1112) is honoured even when set only on the runtime config, where Kafka Streams silently ignores it as "too late". And a few have small shape changes — `VersionedRecord.validTo()` returns an `Optional`, exception handlers receive a `Record` instead of separate key/value. These are catalogued in the [KS compatibility matrix](https://stoatflow.io/docs/reference/ks-compatibility-matrix); the shape changes surface at compile time during the port, which makes them easy to find and fix.

## The decision: green-field cutover vs. carrying state

The single question that determines the shape of your migration is **what happens to your existing state**.

StoatFlow does not read Kafka Streams' changelog topics directly — the two engines lay out state differently on the wire. State still carries across: the &#x2A;*`stoatflow-migration-tool`** translates KS changelog topics into StoatFlow changelog topics offline and carries the input offsets over. Which path you take depends on whether your topology's correctness depends on state that you cannot cheaply rebuild.

The decision, compressed into one tree:

```mermaid
flowchart TD
    Q1{"Is the topology stateful?"}
    Q1 -->|"no — map / filter / route"| A1["Straight cutover:<br/>fresh application id,<br/>reprocess or start from latest"]
    Q1 -->|yes| Q2{"Can the state be rebuilt<br/>by reprocessing?<br/>(source retention covers it,<br/>catch-up window acceptable)"}
    Q2 -->|yes| A2["Rebuild by reprocessing:<br/>fresh application id, earliest,<br/>validate, then switch traffic"]
    Q2 -->|no| A3["State-carrying migration:<br/>translate the changelogs with<br/>the migration tool, verify, cut over"]
    A1 --> P1(["Without data migration"])
    A2 --> P1
    A3 --> P2(["With data migration"])
```

::path-cards
  :::path-card
  ---
  icon: i-lucide-wand-sparkles
  title: Automated port
  to: https://stoatflow.io/docs/migration/automated-port
  ---
  Start here, whichever state path you take. One OpenRewrite command rewrites the imports, the entry point and the Maven build across Java and Kotlin, and reports every residual it deliberately left for you.
  :::

  :::path-card
  ---
  icon: i-lucide-sparkles
  title: Without data migration
  to: https://stoatflow.io/docs/migration/without-data-migration
  ---
  Green-field cutover. Your topology is stateless, or its state can be rebuilt by reprocessing the input topics from the beginning. Point StoatFlow at the source topics, let it reprocess, and cut over. The simplest path — pick it whenever you can.
  :::

  :::path-card
  ---
  icon: i-lucide-database
  title: With data migration
  to: https://stoatflow.io/docs/migration/with-data-migration
  ---
  State-carrying migration. Your topology accumulates state you cannot afford to rebuild — input topics have aged out of retention, or a full reprocess is too slow or too expensive. The `stoatflow-migration-tool` translates your Kafka Streams changelog topics into StoatFlow changelogs offline, carries the input offsets, and the app resumes exactly where KS stopped.
  :::
::

### When a green-field cutover is the right call

Choose [Without data migration](https://stoatflow.io/docs/migration/without-data-migration) when **any** of these holds:

- The topology is **stateless** — `map` / `filter` / `flatMap` / routing, no aggregations or joins that accumulate. There is nothing to carry; reprocessing produces identical output.
- The state is &#x2A;*derivable from input still in retention.** If your source topics retain enough history to recompute every aggregate, count, and join from scratch, reprocessing rebuilds the state exactly, and the cutover is a config-and-deploy exercise.
- You can tolerate a &#x2A;*reprocessing window.** Reading the input topics from the beginning takes time proportional to their size; if you can run the new instance to catch-up before switching consumers over, this is the lowest-risk path.

This is the default recommendation. It avoids any state-format coupling between the two engines, and the result is provably correct because it is computed from the same input the KS app consumed.

### When you need a state-carrying migration

Choose [With data migration](https://stoatflow.io/docs/migration/with-data-migration) when reprocessing is not viable:

- **Input topics have aged out.** Compacted or time-limited source topics no longer hold the full history, so a reprocess would produce incomplete aggregates.
- **A full reprocess is too slow or too expensive.** Very large state, or long-running windowed aggregations spanning weeks, can make recomputation impractical even when the input technically survives.
- **You need continuity of in-flight state** — open sessions, long windows, running counters that downstream systems depend on without a recomputation gap.

This path is a **supported, self-service migration**: the [`stoatflow-migration-tool`](https://stoatflow.io/docs/migration/migration-tool) reads your KS changelog topics, seeds byte-translated StoatFlow changelogs, and carries the consumer-group offsets — with a preflight that catches every known silent-failure mode and a verify step that proves the seeding is faithful. The [strategy page](https://stoatflow.io/docs/migration/with-data-migration) covers the per-store support matrix and how to classify your stores; the [tool page](https://stoatflow.io/docs/migration/migration-tool) has the config reference and the cutover runbook.

::callout{color="info" icon="i-lucide-message-circle"}
Not sure which path your topology falls into? The deciding factor is almost always retention versus state size. And some shapes — suppress-heavy topologies, versioned stores at scale, custom partitioners — are still worth migrating hands-on: [get in touch](https://stoatflow.io/contact) and we'll work through it with you.
::

## Recommended order

A migration in practice tends to run in this order, regardless of which state path you take:

1. **Port the build and the topology.** Swap the dependency, rename the imports, change the entry point, and let the compiler surface the dropped APIs. Validate the ported topology with the in-memory [test driver](https://stoatflow.io/docs/building/testing) — no broker required, deterministic, and the fastest way to confirm the logic is intact.
2. **Clean up the config.** Remove the KS keys that have no meaning here (above) and express parallelism as lane count. See [Configuration model](https://stoatflow.io/docs/concepts/configuration-model).
3. **Pick the state path** — [without](https://stoatflow.io/docs/migration/without-data-migration) or [with](https://stoatflow.io/docs/migration/with-data-migration) data migration — and follow that page.
4. **Deploy as a single pod** and wire up the operational surface: health probes, metrics, and the debug endpoints described on [Architecture](https://stoatflow.io/docs/concepts/architecture#failure-modes-and-observability) and in [Operating](https://stoatflow.io/docs/operating).

## Where to go next

- [Without data migration](https://stoatflow.io/docs/migration/without-data-migration) — the green-field cutover path: reprocess from input, no state carried across
- [With data migration](https://stoatflow.io/docs/migration/with-data-migration) — carrying existing state across when reprocessing isn't viable
- [AI assistants](https://stoatflow.io/docs/getting-started/ai-assistants) — install the skills pack so your AI assistant drives the port (code and state) correctly instead of hallucinating Kafka Streams
- [How StoatFlow differs from Kafka Streams](https://stoatflow.io/docs/concepts/how-stoatflow-differs-from-ks) — the conceptual deltas the migration follows from
- [Comparison matrix](https://stoatflow.io/product/comparison-matrix) — feature-by-feature against Kafka Streams and Flink
- [KS compatibility matrix](https://stoatflow.io/docs/reference/ks-compatibility-matrix) — method-by-method DSL parity and the deliberate exceptions
- Migration in flight and want a second pair of eyes? [Get in touch](https://stoatflow.io/contact) — real people read every email during the alpha.


# Automated port

Most of a port from Kafka Streams to StoatFlow is import replacement. The `stoatflow-openrewrite-recipe` recipe does that replacement for you — on Java and Kotlin alike — rewrites the application entry point, swaps your Maven coordinates, and then tells you, precisely, about the handful of things it refused to change on your behalf.

::tldr-panel
- **One command, with a dry run first.** `rewriteDryRun` writes a diff and changes nothing; `rewriteRun` applies it.
- **It reads types, not text.** The recipe operates on a type-attributed syntax tree, so it never rewrites a package name that happens to appear inside a string literal or a comment — and it fixes the eight types a naive prefix sweep sends to a package that doesn't exist.
- **It never guesses.** Semantics that changed, types with no StoatFlow equivalent, an entry point whose builder it cannot resolve: each is flagged inline and listed in a CSV report, never silently rewritten.
- **Run it against a project that still compiles.** The rules match resolved types; anything that does not type-check is skipped and reported. JDK 17+ (25 included) with a current OpenRewrite plugin.
::

## Before you run it

Any JDK from 17 up — 25 included — runs the migration. One version caveat, for Kotlin codebases only:

::callout{color="warning" icon="i-lucide-triangle-alert"}
**On JDK 25, use a current OpenRewrite plugin.** Plugin releases from before February 2026 bundle a Kotlin compiler that throws on Java 25 — every Kotlin source then fails to parse, and because the migration rules match on resolved types they skip those files *without an error*: the run reports success and changes nothing. The plugin versions in the snippets below are past the fix. The recipe reports each unparsed file either way, so a silent skip is never invisible.
::

The recipe must also run against a project that **still compiles against Kafka Streams**. The rules match resolved types, so a source file that doesn't type-check is skipped. Migrate a green build.

Finally, you need repository access — the same credentials you use to depend on StoatFlow at all. Follow [Store your credentials](https://stoatflow.io/docs/getting-started/installation) if you have not already. For Maven, the repository must be visible to `<pluginRepositories>` as well as `<repositories>`, because the recipe resolves as a plugin dependency:

::code-tabs{group="build"}
```xml [Maven]
<!-- ~/.m2/settings.xml — add a profile so the repo is visible to BOTH lists -->
<profiles>
  <profile>
    <id>stoatflow</id>
    <repositories>
      <repository>
        <id>stoatflow-releases</id>
        <url>https://maven.stoatflow.io/releases</url>
      </repository>
    </repositories>
    <pluginRepositories>
      <pluginRepository>
        <id>stoatflow-releases</id>
        <url>https://maven.stoatflow.io/releases</url>
      </pluginRepository>
    </pluginRepositories>
  </profile>
</profiles>
<activeProfiles>
  <activeProfile>stoatflow</activeProfile>
</activeProfiles>
```

```kotlin [Gradle]
// rewrite.init.gradle.kts — no change to your build files at all.
initscript {
    repositories { maven { url = uri("https://plugins.gradle.org/m2") } }
    dependencies { classpath("org.openrewrite:plugin:7.39.0") }
}

rootProject {
    apply(plugin = "org.openrewrite.rewrite")
    repositories {
        mavenCentral()
        maven {
            url = uri("https://maven.stoatflow.io/releases")
            credentials {
                username = providers.gradleProperty("stoatflowRepoUser").get()
                password = providers.gradleProperty("stoatflowRepoToken").get()
            }
        }
    }
    dependencies { add("rewrite", "io.stoatflow:stoatflow-openrewrite-recipe:VERSION") }
    configure<org.openrewrite.gradle.RewriteExtension> {
        activeRecipe("io.stoatflow.rewrite.MigrateFromKafkaStreams")
    }
}
```
::

::callout{color="info" icon="i-lucide-key-round"}
**If resolution fails, check the credentials before the coordinates.** A 401 from a *plugin* repository surfaces as a generic `Could not resolve io.stoatflow:stoatflow-openrewrite-recipe`, with no mention of authentication.
::

## Run it

The recipe version is the StoatFlow version you are adopting — they ship in lockstep. The current version is :stoatflow-version.

::code-tabs{group="build"}
```bash [Maven]
# 1. Preview. Writes target/rewrite/rewrite.patch and changes nothing.
mvn org.openrewrite.maven:rewrite-maven-plugin:dryRun \
  -Drewrite.recipeArtifactCoordinates=io.stoatflow:stoatflow-openrewrite-recipe:VERSION \
  -Drewrite.activeRecipes=io.stoatflow.rewrite.MigrateFromKafkaStreams

# 2. Apply.
mvn org.openrewrite.maven:rewrite-maven-plugin:run \
  -Drewrite.recipeArtifactCoordinates=io.stoatflow:stoatflow-openrewrite-recipe:VERSION \
  -Drewrite.activeRecipes=io.stoatflow.rewrite.MigrateFromKafkaStreams
```

```bash [Gradle]
# 1. Preview. Writes build/reports/rewrite/rewrite.patch and changes nothing.
./gradlew --init-script rewrite.init.gradle.kts rewriteDryRun

# 2. Apply.
./gradlew --init-script rewrite.init.gradle.kts rewriteRun
```
::

## What it did, and what it left you

Read the diff, then read the `PortingResiduals` CSV the run exported alongside it. Every `~~>` marker in the diff, and every row in that table, names the entry in the [KS compatibility matrix](https://stoatflow.io/docs/reference/ks-compatibility-matrix) that explains what to do.

| The recipe rewrote                                                                                                                                         | The recipe flagged                                                                                                                                                                                        |
| ---------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| All `org.apache.kafka.streams.*` imports, including the eight types a prefix sweep mis-routes (`StateStore`, `TimestampExtractor`, `StreamPartitioner`, …) | The exactly-once default. Kafka Streams defaults to `at_least_once`; StoatFlow defaults to exactly-once. If you never set `processing.guarantee`, your ported application silently becomes transactional. |
| `new KafkaStreams(topology, props)` → `StoatFlow.fromBuilder(new StreamsConfig(props), builder)`, when the builder is in reach                             | Types with no StoatFlow equivalent: Interactive Queries v2, `StreamsMetrics`, `TaskMetadata`, `processor.To`, the record-metadata accessors.                                                              |
| `streams.state()` → `streams.kafkaStreamsState()`, so state comparisons keep Kafka Streams semantics                                                       | Shapes that compile but throw at runtime: `Suppressed.maxBytes`, multicast on `repartition()`, a hand-rolled `BytesStoreSupplier`.                                                                        |
| `JoinWindows` field access, `ValueTransformerWithKey.init`, `StateListener.onChange` parameter types                                                       | An entry point whose `StreamsBuilder` it could not resolve — because guessing would produce code that compiles and drops every processor.                                                                 |
| Maven dependency coordinates and the repository declaration                                                                                                |                                                                                                                                                                                                           |

Gradle build files are left alone by design; apply the five-line change yourself. Swap `org.apache.kafka:kafka-streams` for `io.stoatflow:stoatflow-core`, add the repository, and move to a JDK 25 toolchain with `--enable-preview` — the `io.stoatflow` Gradle plugin does the last part for you.

Then build. &#x2A;*Remaining compile errors should map one-to-one onto flagged rows.** Anything else is a recipe bug, and worth reporting.

## The one thing worth understanding

Kafka Streams has a single `KafkaStreams.State`. StoatFlow has two enums: a richer native `StoatFlow.State`, returned by `state()`, and a Kafka Streams-exact `KafkaStreamsState` mirror. The recipe maps `KafkaStreams.State` onto the mirror **and** redirects `state()` to `kafkaStreamsState()`, because either half alone is dangerous: `streams.state() == KafkaStreamsState.RUNNING` would then compare two unrelated enums. In Java that is a compile error. In Kotlin it compiles with only a warning and is **always false at runtime** — and `.equals(...)` is silently always false in both languages.

You do not have to think about this; the recipe handles it, and flags anything it could not. It is worth knowing because it is the one place a mechanical port could otherwise go quietly wrong.

## Kotlin null-key lambdas

One Kotlin-only thing the recipe can't fix, because it is a language artifact rather than an import: over a topic that may carry **null keys**, declare the key parameter of a stateless operator lambda nullable — `map { _: K?, v -> … }`, `filter { _: K?, v -> … }`, `selectKey { _: K?, v -> … }`. A non-nullable inferred key parameter (even `_`) makes the Kotlin compiler insert a `checkNotNullParameter` null-check, so a null key throws `NullPointerException: Parameter specified as non-null is null` inside your compiled lambda — **on Kafka Streams too**, not just StoatFlow. Both frameworks invoke your lambda with the raw (null) key; the nullable declaration suppresses the check. Aggregations (`groupBy(...).count/reduce/aggregate`) and the stream–table INNER join already drop null-key records before your lambda runs, matching Kafka Streams — no change needed there.

## After the port

You now have a compiling StoatFlow application and a decision to make about your existing state.

::path-cards
  :::path-card
  ---
  icon: i-lucide-sparkles
  title: Without data migration
  to: https://stoatflow.io/docs/migration/without-data-migration
  ---
  Reprocess from the input topics. The simplest path — pick it whenever you can.
  :::

  :::path-card
  ---
  icon: i-lucide-database
  title: With data migration
  to: https://stoatflow.io/docs/migration/with-data-migration
  ---
  Carry your accumulated state across rather than recomputing it.
  :::
::

If you prefer no tooling at all, the manual import table in [Without data migration](https://stoatflow.io/docs/migration/without-data-migration) remains a complete fallback.


# Migration without carrying state

This is the simplest way to move a topology from Kafka Streams to StoatFlow: you do **not** carry the existing Kafka Streams state across. You point a StoatFlow build at the same source topics under a **new** application id (a fresh consumer group), let the stateful operators rebuild their state — from the source data or from StoatFlow's own changelog topics, never from the Kafka Streams ones — and switch downstream traffic once the new app has caught up.

It suits most cases: stateless topologies (nothing to carry), topologies whose source topics are still fully retained, and any time you can tolerate a re-read of the input. If your state can only be reconstructed by replaying years of compacted input you can't afford to re-read, see [Migration with data migration](https://stoatflow.io/docs/migration/with-data-migration) instead.

::tldr-panel
- **Swap the dependency:** `org.apache.kafka:kafka-streams` → `io.stoatflow:stoatflow-runtime`; imports `org.apache.kafka.streams.*` → `io.stoatflow.core.topology.*`.
- **Swap the entry point:** `new KafkaStreams(topology, props).start()` → `StoatFlowRuntime.fromConfig(...).start()` with `application.yaml`.
- **New application id** = a fresh consumer group. Set `auto.offset.reset` to `earliest` so stateful operators rebuild from the start of the source.
- **Remove:** `num.stream.threads`, standby replicas, rebalancing / static-membership tuning — there's no cluster to rebalance.
- **Keep:** the topology code (same DSL), topic names, serdes, and most Kafka client properties.
::

::callout{color="info" icon="i-lucide-info"}
This page is the procedure. For *why* the model is different — single instance, lanes instead of tasks, in-memory re-keying, barrier-based exactly-once — read [How StoatFlow differs from Kafka Streams](https://stoatflow.io/docs/concepts/how-stoatflow-differs-from-ks) first. The DSL itself carries over unchanged.
::

## When this path applies

| Your topology                                                     | This path works because                                                                                   |
| ----------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
| Stateless (`map` / `filter` / `flatMap` / branching / re-routing) | There is no state to carry — the cutover is purely dependency + config.                                   |
| Stateful, with source topics still fully retained                 | Stateful operators rebuild by re-reading the source from the beginning.                                   |
| Stateful, backed by compacted input or rebuildable aggregates     | StoatFlow builds its **own** changelog topics as it processes; a one-time reprocess re-derives the state. |

The case this path does **not** cover: state you can only get by replaying source you no longer have (or can't afford to re-read). StoatFlow cannot restore from a Kafka Streams changelog topic — the changelog formats are not interchangeable. That scenario is [Migration with data migration](https://stoatflow.io/docs/migration/with-data-migration).

## The four-step cutover

1. **Swap the dependency and imports** — `io.stoatflow:stoatflow-runtime`, DSL imports under `io.stoatflow.core.topology.*`.
2. **Swap the entry point** — `KafkaStreams` → `StoatFlowRuntime`, `Properties` → `application.yaml`.
3. **Run the StoatFlow app alongside the old one** under a fresh application id so it builds an independent consumer group and (for stateful topologies) its own state. Let it catch up to the live offset.
4. **Switch downstream traffic** to the StoatFlow output, then retire the Kafka Streams app.

Steps 1 and 2 are the code change. Steps 3 and 4 are the operational cutover. The rest of this page walks each one, grounded in the `map-filter` example that ships in both flavours.

## Step 1 — dependency and build

Drop the Kafka Streams dependency and add `stoatflow-runtime` (the batteries-included module — it transitively pulls in `stoatflow-core`). If you've followed [Installation](https://stoatflow.io/docs/getting-started/installation), the private Maven repository and the `io.stoatflow` Gradle plugin are already wired up.

**Before — the Kafka Streams build:**

```kotlin [Gradle]
// build.gradle.kts (Kafka Streams)
plugins {
    kotlin("jvm")
    application
}

application {
    mainClass.set("com.example.MainKt")
}

dependencies {
    implementation("org.apache.kafka:kafka-streams:4.1.1")
    runtimeOnly("ch.qos.logback:logback-classic")
}
```

**After — the StoatFlow build:**

```kotlin [Gradle]
// build.gradle.kts (StoatFlow)
plugins {
    kotlin("jvm")
    id("io.stoatflow") version "<stoatflow-version>"
}

stoatflow {
    mainClass.set("com.example.MainKt")
}

dependencies {
    implementation("io.stoatflow:stoatflow-runtime:<stoatflow-version>")
    runtimeOnly("ch.qos.logback:logback-classic")
}
```

The current version is :stoatflow-version — substitute it for the `<stoatflow-version>` placeholder. The `io.stoatflow` Gradle plugin applies the JDK 25 toolchain, the `--enable-preview` and `--enable-native-access=ALL-UNNAMED` JVM flags, `-XX:+UseG1GC`, and a runnable fat-jar — so you don't hand-maintain any of the run wiring. See [Installation](https://stoatflow.io/docs/getting-started/installation) for the Maven equivalent and the plugin-free option.

::callout{color="warning" icon="i-lucide-triangle-alert"}
StoatFlow requires **JDK 25+** at both compile and run time (it uses JDK preview features). If your Kafka Streams app runs on an older JDK, the toolchain bump is part of this migration. See [Installation → Configure the JVM toolchain](https://stoatflow.io/docs/getting-started/installation).
::

## Step 2 — imports, entry point, and config

The topology code stays the same shape — same `StreamsBuilder`, same operators, same `Consumed` / `Produced` / `Named`. Two things change: the **imports** move from `org.apache.kafka.streams.*` to `io.stoatflow.core.topology.*`, and the **bootstrap** moves from a `KafkaStreams` instance + `Properties` to `StoatFlowRuntime.fromConfig(...)` + `application.yaml`.

### Imports

::callout{color="info" icon="i-lucide-wand-sparkles"}
**You do not have to do this by hand.** The [automated port](https://stoatflow.io/docs/migration/automated-port) applies this table, and the rest of the import surface, across your whole codebase — Java and Kotlin — and flags what it cannot safely rewrite. The table below is the no-tooling fallback, and a useful reference for reading the diff.
::

| Kafka Streams import                                                  | StoatFlow import                                                                                          |
| --------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
| `org.apache.kafka.streams.StreamsBuilder`                             | `io.stoatflow.core.topology.StreamsBuilder`                                                               |
| `org.apache.kafka.streams.kstream.KStream`                            | `io.stoatflow.core.topology.KStream`                                                                      |
| `org.apache.kafka.streams.kstream.Consumed`                           | `io.stoatflow.core.topology.Consumed`                                                                     |
| `org.apache.kafka.streams.kstream.Produced`                           | `io.stoatflow.core.topology.Produced`                                                                     |
| `org.apache.kafka.streams.kstream.Named`                              | `io.stoatflow.core.topology.Named`                                                                        |
| `org.apache.kafka.streams.kstream.Branched`                           | `io.stoatflow.core.topology.Branched`                                                                     |
| `org.apache.kafka.streams.KeyValue`                                   | `io.stoatflow.core.state.KeyValue`                                                                        |
| `org.apache.kafka.streams.state.*` (`Stores`, `KeyValueStore`, …)     | `io.stoatflow.core.state.*`                                                                               |
| `org.apache.kafka.streams.processor.api.*` (`Processor`, `Record`, …) | `io.stoatflow.core.processor.*`                                                                           |
| `org.apache.kafka.streams.errors.*` (exception handlers)              | `io.stoatflow.core.exception.*`                                                                           |
| `org.apache.kafka.streams.CloseOptions`                               | `io.stoatflow.core.CloseOptions&#x60; &#x2A;(only if you call `close(CloseOptions)`; see the note below)* |
| `org.apache.kafka.common.serialization.Serdes`                        | `org.apache.kafka.common.serialization.Serdes&#x60; &#x2A;(unchanged — Kafka serdes are reused)*          |

The Kafka **serde** classes (`Serdes.String()`, `Serdes.Long()`, your Avro/Protobuf/JSON serdes) come from the Kafka clients library and carry over verbatim — as does everything else under `org.apache.kafka.common.*` and `org.apache.kafka.clients.*` (`Headers`, `ConsumerRecord`, `ProducerRecord`, …). Only the Kafka Streams types move namespaces. See the [KS compatibility matrix](https://stoatflow.io/docs/reference/ks-compatibility-matrix) for the method-by-method DSL parity.

::callout{color="info" icon="i-lucide-info"}
**`close(Duration)` / `close(CloseOptions)` carry over.** `streams.close(Duration.ofSeconds(30))` works unchanged; `close(CloseOptions...)` uses the current top-level KIP-1153 shape via `io.stoatflow.core.CloseOptions` (`CloseOptions.timeout(d).withGroupMembershipOperation(...)`). If your app used the **deprecated nested** `KafkaStreams.CloseOptions` (KIP-812, `new CloseOptions().timeout(d).leaveGroup(true)`), rewrite that one line to the factories. See [Lifecycle](https://stoatflow.io/docs/reference/ks-compatibility-matrix#lifecycle).
::

### Entry point

In Kafka Streams you build a `Properties`, construct `KafkaStreams`, register a shutdown hook, and call `start()`. The map-filter example does exactly that:

::code-tabs{group="lang"}
```kotlin [Kotlin]
// Before — Kafka Streams
val props = Properties().apply {
    put(StreamsConfig.APPLICATION_ID_CONFIG, "map-filter-example-ks")
    put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, System.getenv("KAFKA_BOOTSTRAP_SERVERS") ?: "localhost:9092")
    put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String()::class.java)
    put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String()::class.java)
}

val builder = StreamsBuilder()
buildTopology(builder)

val streams = KafkaStreams(builder.build(), props)
Runtime.getRuntime().addShutdownHook(Thread { streams.close() })
streams.start()
Thread.currentThread().join()
```

```java [Java]
// Before — Kafka Streams
var props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "map-filter-example-ks");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG,
    System.getenv().getOrDefault("KAFKA_BOOTSTRAP_SERVERS", "localhost:9092"));
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());

var builder = new StreamsBuilder();
buildTopology(builder);

var streams = new KafkaStreams(builder.build(), props);
Runtime.getRuntime().addShutdownHook(new Thread(streams::close));
streams.start();
Thread.currentThread().join();
```
::

On StoatFlow, `StoatFlowRuntime.fromConfig(...)` loads `application.yaml`, starts the HTTP + metrics server, installs graceful shutdown, and runs the topology until terminated. Non-serializable settings — the default serdes, exception handlers — are set in code via `streamsConfigOverrides { ... }`; everything else lives in YAML:

::code-tabs{group="lang"}
```kotlin [Kotlin]
// After — StoatFlow runtime
import io.stoatflow.core.topology.StreamsBuilder
import io.stoatflow.runtime.StoatFlowRuntime
import org.apache.kafka.common.serialization.Serdes

fun main() {
    val runtime = StoatFlowRuntime.fromConfig(
        topologyBuilder = { buildTopology(it) },
        configure = {
            streamsConfigOverrides {
                defaultKeySerde(Serdes.String())
                defaultValueSerde(Serdes.String())
            }
        },
    )
    runtime.start()
    runtime.awaitTermination()
}
```

```java [Java]
// After — StoatFlow runtime
import io.stoatflow.core.topology.StreamsBuilder;
import io.stoatflow.runtime.StoatFlowRuntime;
import org.apache.kafka.common.serialization.Serdes;

public class Main {
    public static void main(String[] args) {
        var runtime = StoatFlowRuntime.fromConfig(
            Main::buildTopology,
            builder -> builder.streamsConfigOverrides(cfg -> {
                cfg.defaultKeySerde(Serdes.String());
                cfg.defaultValueSerde(Serdes.String());
            })
        );
        runtime.start();
        runtime.awaitTermination();
    }
}
```
::

The `application-id` and `bootstrap-servers` move out of `Properties` and into `application.yaml`:

```yaml
# src/main/resources/application.yaml
stoatflow:
  application-id: map-filter-stoatflow        # NEW id — a fresh consumer group
  bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVERS:-localhost:9092}

  license:
    key: ${STOATFLOW_LICENSE_KEY}

runtime:
  http:
    enabled: true
    port: ${HTTP_PORT:-8080}
  metrics:
    enabled: true
```

StoatFlow is licensed — `stoatflow.license.key` is required and has no Kafka Streams equivalent. See [License configuration](https://stoatflow.io/docs/getting-started/license-configuration).

::callout{color="info" icon="i-lucide-activity"}
**Porting a `setStateListener`.** A Kafka Streams `setStateListener` keyed on `KafkaStreams.State` is usually wired for readiness or alerting. Under the runtime, that surface is the operational one — `/health/ready`, `/health/live`, and `/state` (see [REST API](https://stoatflow.io/docs/runtime/rest-api)) — so most such listeners retire. If you build against `:core` directly, `StoatFlow.setStateListener(...)` is the KS-exact single overload (a method reference or lambda, no cast), and StoatFlow's richer 10-state lifecycle maps back onto the KS 7-state `KafkaStreams.State` at the boundary via `newState.toKafkaStreamsState()` / `app.kafkaStreamsState()`. See the [Lifecycle](https://stoatflow.io/docs/reference/ks-compatibility-matrix#lifecycle) rows in the compatibility matrix.
::

::callout{color="info" icon="i-lucide-database"}
The operators that the two map-filter examples have in common — the `selectKey` / `map` / `filter` / `mapValues` / `peek` / `split` / `to` chain — port across with only the import-namespace change. Compare the Kafka Streams example (`examples-ks/map-filter`) against the StoatFlow runtime example (`examples/map-filter-runtime`): the overlapping DSL is the same, line for line, aside from the namespace. The two examples are *not* identical — the StoatFlow one also exercises StoatFlow-only surface (a `scheduled` source, a `Processor` with key-based timers, an extra `table(...)` branch), and the Kafka Streams one keeps an explicit `repartition()` that StoatFlow's in-memory re-keying makes unnecessary. But that's added scope, not a rewrite of the shared chain. That's the point of the DSL parity: the operators you already wrote port mechanically.
::

### Residual manual fixes

The import swap gets a typical port compiling except for a short, enumerated set of deliberate divergences. Walk this checklist once — most are one-line fixes the compiler points at; the store-supplier and `Suppressed` rules throw when the topology is built, and the last two need no code change at all:

- **`JoinWindows.beforeMs` / `afterMs`** are getters here, not public fields: replace Java field access `jw.beforeMs` with `jw.getBeforeMs()` / `jw.getAfterMs()`. Kotlin property access is unaffected.
- **`VersionedRecord.validTo()`** returns `Optional<Long>` instead of `long` — use `.orElse(...)` / `.isPresent()`.
- **Exception handlers receive a `Record`**, not separate key/value: read `record.key()` / `record.value()` in `ProcessingExceptionHandler.handle(...)`. Production *serialization* failures also arrive through `handle(...)` — branch on `(context as ProductionContext).failedOn` instead of overriding a separate method.
- **`StreamPartitioner` import is mis-routed by a bulk rewrite**: it lives in `io.stoatflow.core.topology.StreamPartitioner`, not under `processor.*` as in Kafka Streams — fix that one import by hand. Multicast partitioners (KIP-837) are honoured on sinks but &#x2A;*rejected on `repartition()`**.
- **`ValueTransformerWithKey.init`** takes a `FixedKeyProcessorContext` instead of `ProcessorContext` — change the parameter type; state-store access is identical.
- **Custom store suppliers must come from `Stores.*` factories** — hand-rolled `*BytesStoreSupplier` implementations are rejected. `Stores.*WithHeadersBuilder(...)` needs a `*WithHeaders(...)` supplier, and an explicit `StreamJoined` store supplier must set `retainDuplicates = true`.
- **`Suppressed` `maxBytes` / `withMaxBytes` throw** (suppress buffers hold objects in memory, not bytes) — use `maxRecords(...)` or `unbounded()`.
- **`KStream.transform*` is absent** (removed in Kafka Streams 4.x too) — use `process` / `processValues`.
- **A re-key into a store-connected `process()` can fail the build.** Like Kafka Streams, StoatFlow does not repartition before `process()` / `processValues()` — so after a **many-to-one** re-key those records arrive on several lanes. In KS that fragments per-key state across tasks; in StoatFlow a store read-modify-write straddling a commit barrier can lose an update outright, and no lock closes the window. So the provable shape — a proven re-key (`selectKey`/`map`/`groupBy`) feeding a node with a **writable** store — is refused at topology-compile time, in your own test suite, with no broker. Fix with `repartition()` (in-memory here, not a topic), `topology.processor-api-key-affinity: presumed`, or by re-keying the store. A chain of Processor API nodes only **warns** (`papi-key-affinity-presumed-chain`), because `process()` declares a key change whether or not yours changes one. See [Key affinity after a re-key](https://stoatflow.io/docs/building/processor-api#key-affinity-after-a-re-key).
- **`processor.wrapper.class` (KIP-1112) works**, and — unlike Kafka Streams — is honoured even when set only on the runtime config. Your wrapper decorates every node, DSL and Processor API alike. Caveats: a DSL-internal node can be decorated, observed or short-circuited but not *replaced* (rejected at build time); `init`/timer/watermark callbacks on internal nodes bypass the wrapper; the supplier's `get()` runs once per **lane**, not once per task; and node names are StoatFlow's, so name pattern-matching finds nothing. See [Processor wrapping](https://stoatflow.io/docs/building/processor-api#processor-wrapping-kip-1112).
- **Boundary key serdes have no Kafka Streams counterpart**, and an app that sets `default.key.serde` — the standard Kafka Streams idiom — never meets them. Where a re-keyed record reaches an operator that needs key affinity, StoatFlow serializes the new key in memory to pick a lane; if nothing in your topology declares a serde for that key *and* you configured no default, it inherits the nearest declaration from before the re-key rather than failing. A port that carries per-topic rather than global serdes therefore starts where it would previously have died on the first record — the inherited serde holds as long as the re-key preserves the declared key type — and a warning raised when the topology compiles names the `'parent' → 'child'` edge and the serde it inherited. Make that fatal with `topology.validation.inherited-boundary-key-serde: error`. See [Boundary key serdes](https://stoatflow.io/docs/building/serdes#boundary-key-serdes-lane-assignment).

Two behavioural shifts need no code change but are worth knowing: `taskId()` is always `0_0` (single instance — don't branch on it), and `ProcessorContext.currentStreamTimeMs()` returns the **watermark** — global, and generally lower than KS per-task stream time. The full catalogue, including everything that needs *no* change, is the [KS compatibility matrix](https://stoatflow.io/docs/reference/ks-compatibility-matrix).

## Step 3 — what to remove, what to keep

The bulk of a Kafka Streams `Properties` block carries over as `stoatflow.kafka.*` passthrough or has a direct `stoatflow.*` equivalent. A handful of properties model a multi-instance cluster and have **no meaning** in StoatFlow's single-instance model — drop them.

::callout{color="info" icon="i-lucide-info"}
**Keeping your KS `Properties`?** If you build against `:core` directly, the KS-keyed config carries over wholesale: `new StreamsConfig(props)` / `StreamsConfig.fromProperties(props).build()` accepts the full Kafka Streams key surface. Recognised-but-inapplicable cluster keys (the table below) log a WARN and are ignored, client keys pass through under their `consumer.` / `producer.` prefixes, and truly-unknown keys warn (or fail fast with `stoatflow.config.strict=true`). The YAML shape on this page is the recommended end state — the adapter just lets you defer the config rewrite.
::

### Remove — these model a cluster that doesn't exist

| Kafka Streams property                                                            | Why it goes                                                                                                                                                                                                                                                                                                               |
| --------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `num.stream.threads`                                                              | There is no thread-per-task pool. Parallelism is set by `stoatflow.lanes.count`, which scales with cores rather than partition count. See [Lanes and parallelism](https://stoatflow.io/docs/concepts/lanes-and-parallelism).                                                                                              |
| `num.standby.replicas`                                                            | This KS knob speeds task hand-off during rebalancing; StoatFlow has no rebalancing, so it has no equivalent. HA is fast restart by default, or StoatFlow's own opt-in hot standby — configured under `stoatflow.ha.*`, not via this knob. See [High availability](https://stoatflow.io/docs/operating/high-availability). |
| `acceptable.recovery.lag`, `max.warmup.replicas`, `probing.rebalance.interval.ms` | All rebalancing / task-assignment tuning — no group to rebalance.                                                                                                                                                                                                                                                         |
| `group.instance.id` (if you set it manually)                                      | StoatFlow forces static membership using the application id. Setting it has no effect.                                                                                                                                                                                                                                    |
| `application.server`                                                              | Interactive-query host discovery is a multi-instance concern; state is global in one process.                                                                                                                                                                                                                             |
| `replication.factor` (KS internal-topic key)                                      | StoatFlow uses `stoatflow.changelog.replication-factor` for its changelog topics.                                                                                                                                                                                                                                         |

::callout{color="info" icon="i-lucide-info"}
You don't need to set `processing.guarantee`. StoatFlow defaults to `EXACTLY_ONCE` (`stoatflow.processing-guarantee`); if your Kafka Streams app ran `exactly_once_v2`, you already match. Mind the default flip, though: Kafka Streams defaults to `at_least_once`, so a KS app that never set the key was running at-least-once — on StoatFlow it now runs under exactly-once (stronger, but transactional output and a different commit-latency profile) unless you set `AT_LEAST_ONCE` explicitly. See [Exactly-once](https://stoatflow.io/docs/concepts/exactly-once).
::

### Keep — these carry over

| Kafka Streams concern                                                                                                                                    | Where it goes in StoatFlow                                                                                                                                                                                                                                                                                                                     |
| -------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Topology code (DSL operators, processors)                                                                                                                | Unchanged, modulo the import namespace (Step 2).                                                                                                                                                                                                                                                                                               |
| Topic names                                                                                                                                              | Unchanged — they're string literals in the topology.                                                                                                                                                                                                                                                                                           |
| Serdes (`Serdes.*`, Avro/Protobuf/JSON)                                                                                                                  | Unchanged — Kafka serde classes are reused as-is.                                                                                                                                                                                                                                                                                              |
| `application.id`                                                                                                                                         | Becomes `stoatflow.application-id` — but **pick a new value** (see below).                                                                                                                                                                                                                                                                     |
| `bootstrap.servers`                                                                                                                                      | Becomes `stoatflow.bootstrap-servers`.                                                                                                                                                                                                                                                                                                         |
| `default.key.serde` / `default.value.serde`                                                                                                              | `streamsConfigOverrides { defaultKeySerde(...) }` in code, or the YAML class-name keys. Setting the **key** serde also closes last-resort [boundary key serde inheritance](https://stoatflow.io/docs/building/serdes#last-resort-inheritance): a configured default is taken as your answer for every unresolved boundary, whatever its value. |
| `schema.registry.url`                                                                                                                                    | `stoatflow.schema-registry-url` — propagated to default serdes automatically. See [Serdes](https://stoatflow.io/docs/building/serdes).                                                                                                                                                                                                         |
| Raw consumer/producer tuning (`max.poll.records`, `fetch.min.bytes`, `compression.type`, `linger.ms`, `acks`, `auto.offset.reset`, security/SSL/SASL, …) | Passed straight through under `stoatflow.kafka.consumer` / `stoatflow.kafka.producer` using the **exact** Kafka property names. See [Kafka client config](https://stoatflow.io/docs/configuration/kafka-client-config).                                                                                                                        |

::callout{color="warning" icon="i-lucide-shield"}
A few Kafka client properties are **forced** by StoatFlow and can't be overridden through passthrough: `group.id` and `group.instance.id` (both set to the application id, for static single-member membership) and `enable.auto.commit=false` (offsets are committed by StoatFlow's commit protocol). Setting them under `stoatflow.kafka.consumer` has no effect — full list in [Kafka client config](https://stoatflow.io/docs/configuration/kafka-client-config#forced-overrides).
::

### Use a new application id

This is the one config value you must change rather than copy. The application id is StoatFlow's consumer group id (and transactional id). Reusing the Kafka Streams group id would have the StoatFlow app inherit the old committed offsets — it would resume mid-stream and **skip the historical records that stateful operators need to rebuild from**.

A fresh id gives the StoatFlow app an independent consumer group with no committed offsets, so it can read the source from the beginning. StoatFlow already defaults `auto.offset.reset` to `earliest` (matching Kafka Streams), so a stateful topology rebuilds from the start with no extra config — set it explicitly only to make the intent obvious or to choose otherwise:

```yaml
stoatflow:
  application-id: map-filter-stoatflow        # NOT the old KS application.id
  kafka:
    consumer:
      auto.offset.reset: earliest             # the default; shown here to make the intent explicit
```

::callout{color="info" icon="i-lucide-info"}
For a **stateless** topology, `auto.offset.reset` is your reprocessing choice, not a correctness requirement: `earliest` (StoatFlow's default) reprocesses the full history into the new output; setting `latest` (the raw Kafka client default) starts the StoatFlow app from the current tip, processing only new records. Pick whichever matches your cutover. For a **stateful** topology you almost always want `earliest`, so the aggregates are complete before you switch traffic over.
::

## Step 4 — run, catch up, and switch traffic

With the build and config swapped, run the StoatFlow app the same way as any StoatFlow app:

```bash
export KAFKA_BOOTSTRAP_SERVERS=localhost:9092
export STOATFLOW_LICENSE_KEY="key/...from your onboarding email..."

./gradlew run
```

Because the StoatFlow app uses a **new** application id and writes to its own state and (if stateful) its own changelog topics, it runs independently of the still-running Kafka Streams app — no coordination, no shared group. The recommended cutover:

1. **Start the StoatFlow app** pointed at the same source topics, with the new id and `auto.offset.reset: earliest`. It begins consuming from the start of the source and rebuilding state.
2. **Write to a parallel output** while validating. Either point `to(...)` at new output topics, or run a side-by-side comparison against the Kafka Streams output. This is the safe default — you confirm correctness before anything downstream depends on it.
3. **Watch it catch up.** Consumer lag falls toward zero as the StoatFlow app reaches the live tip. Readiness comes up once state restoration completes. &#x2A;What you see:* `/health/ready` returns 200, the `stoatflow.consumer.lag.records` metric trends to \~0, and `/state` shows stores fully restored. See [Metrics](https://stoatflow.io/docs/runtime/metrics) and the [REST API](https://stoatflow.io/docs/runtime/rest-api). Want to watch this on your **existing Kafka Streams dashboards** instead? Turn on `runtime.metrics.naming: both` — see [Reusing your Kafka Streams dashboards](https://stoatflow.io/docs/migration/reusing-kafka-streams-dashboards).
4. **Switch downstream traffic** to the StoatFlow output once lag is near zero and the outputs match.
5. **Retire the Kafka Streams app.** Stop it; once you're confident, delete its consumer group and its internal repartition/changelog topics.

The same cutover as a timeline — the two apps overlap until the switch:

```mermaid
flowchart LR
    KS(["Kafka Streams app —<br/>untouched, still serving"]) -.until the switch.-> S4
    S1["1. Start StoatFlow<br/>fresh id, earliest"] --> S2["2. Write to a<br/>parallel output"]
    S2 --> S3["3. Catch up —<br/>lag → ~0, ready"]
    S3 --> S4["4. Switch<br/>downstream traffic"]
    S4 --> S5["5. Retire the<br/>KS app"]
```

::callout{color="warning" icon="i-lucide-triangle-alert"}
**Don't point two writers at the same output topic during the overlap.** While both apps run, have the StoatFlow app write to a parallel output (or run it in compare-only mode). Switching the output topic is the cutover — do it once, deliberately, after validation. There is no protocol to coordinate a Kafka Streams app and a StoatFlow app writing the same output.
::

::callout{color="info" icon="i-lucide-info"}
StoatFlow re-keying (`selectKey` / `groupBy` / key-changing joins) happens **in memory** between lanes — there are no internal repartition topics to provision. The repartition topics your Kafka Streams app created are not used by StoatFlow and can be cleaned up when you retire the old app. See [How StoatFlow differs from KS → in-memory re-keying](https://stoatflow.io/docs/concepts/how-stoatflow-differs-from-ks).
::

## Verify before you cut over

A topology test driver runs the StoatFlow topology in-memory with no broker — use it to confirm the ported topology behaves identically to the Kafka Streams original before any live cutover. The harness mirrors the Kafka Streams `TopologyTestDriver`:

::code-tabs{group="lang"}
```kotlin [Kotlin]
val driver = TopologyTestDriver.fromBuilder(builder)
val input = driver.createInputTopic<String, String>("input-topic", Serdes.String(), Serdes.String())
val output = driver.createOutputTopic<String, String>("output-topic", Serdes.String(), Serdes.String())

input.pipeInput("k", "streaming")
assertThat(output.readRecord()?.value).isEqualTo("STREAMING")

driver.close()
```

```java [Java]
var driver = TopologyTestDriver.fromBuilder(builder);
var input = driver.createInputTopic("input-topic", Serdes.String(), Serdes.String());
var output = driver.createOutputTopic("output-topic", Serdes.String(), Serdes.String());

input.pipeInput("k", "streaming");
assertThat(output.readRecord().value()).isEqualTo("STREAMING");

driver.close();
```
::

Add the test dependency (`io.stoatflow:stoatflow-test-utils`, test scope — see [Installation](https://stoatflow.io/docs/getting-started/installation)). Full coverage of the driver, time control, and state-store assertions is in [Testing](https://stoatflow.io/docs/building/testing).

## Next steps

- **[Migration with data migration](https://stoatflow.io/docs/migration/with-data-migration)** — when you can't re-read the source and must reconstruct state another way.
- **[AI assistants](https://stoatflow.io/docs/getting-started/ai-assistants)** — install the skills pack so your AI assistant applies the import codemod and the divergence fixes for you.
- **[How StoatFlow differs from Kafka Streams](https://stoatflow.io/docs/concepts/how-stoatflow-differs-from-ks)** — the conceptual deltas behind every step here.
- **[Configuration reference](https://stoatflow.io/docs/reference/configuration-reference)** — every `stoatflow.*` key with type and default.
- **[KS compatibility matrix](https://stoatflow.io/docs/reference/ks-compatibility-matrix)** — method-by-method DSL parity.
- **[Reusing your Kafka Streams dashboards](https://stoatflow.io/docs/migration/reusing-kafka-streams-dashboards)** — turn on KS-compatible metrics so your existing Grafana dashboards and alerts light up.
- **[Production checklist](https://stoatflow.io/docs/operating/production-checklist)** — before the cutover goes live.
- Migrating something non-trivial? [Get in touch](https://stoatflow.io/contact) — real people read every email during the alpha.


# Migration carrying state

This page is about **stateful** topologies — aggregations, joins, windows, anything with a state store — and what happens to that state when you move to StoatFlow. There are two supported ways to bring the state across: **rebuild it by reprocessing** the input topics, or **translate it** with the `stoatflow-migration-tool` CLI, which reads your Kafka Streams changelog topics and seeds byte-translated StoatFlow changelogs plus the carried-over input offsets. A third path — **engage us** — remains for the shapes the tool deliberately does not cover.

If your topology has no state stores, you don't need any of this — see [Migration without state](https://stoatflow.io/docs/migration/without-data-migration).

::tldr-panel
- **StoatFlow → StoatFlow restart:** state is carried automatically. Local RocksDB is reused when valid; otherwise the changelog tops it back up. Nothing to do.
- **Kafka Streams → StoatFlow, state rebuildable:** **reprocess** — fresh `application-id`, replay the input, validate, cut over. Still the simplest path; pick it whenever you can.
- **Kafka Streams → StoatFlow, state too expensive to rebuild:** **translate** — the [`stoatflow-migration-tool`](https://stoatflow.io/docs/migration/migration-tool) converts KS changelog topics into StoatFlow changelog topics offline and carries the consumer-group offsets, so the app resumes exactly where KS stopped.
- **Suppress-heavy, exotic, or unsure:** **engage** — [talk to us](https://stoatflow.io/contact) and we'll plan the cutover with you.
::

## How StoatFlow carries its own state

StoatFlow's durability model is the same family as Kafka Streams: every state-store write is logged to a compacted **changelog topic** named `{application-id}-{store-name}-changelog`, local state lives in RocksDB (or in memory), and on every start the runtime decides per store whether to reuse, delta-restore, or full-restore. The full model — including the per-store restore decision — is on [Architecture: state and durability](https://stoatflow.io/docs/concepts/architecture#state-stores-and-durability) and [Lifecycle](https://stoatflow.io/docs/concepts/architecture#lifecycle-startup-restart-recovery).

The property that makes a state-carrying migration possible: &#x2A;*a first start over pre-seeded changelog topics is just a forced full restore.** The engine needs no migration mode — the migration tool writes StoatFlow-format changelogs, and the first start rebuilds RocksDB from them exactly as it would after losing a disk.

## The supported-path matrix

| Path          | When                                                                                                                                                      | What it involves                                                                                                                                                                                            |
| ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Reprocess** | State is derivable from input still in retention, and you can afford the catch-up window                                                                  | Fresh `application-id`, `earliest` reset, validate, cut over — see [Migration without state](https://stoatflow.io/docs/migration/without-data-migration)                                                    |
| **Translate** | State is too large, too old, or too expensive to rebuild — long windows, KTables over sources with lost history, high-volume sources with short retention | The [`stoatflow-migration-tool`](https://stoatflow.io/docs/migration/migration-tool): `plan` → `translate` → `seed-offsets` → `verify` inside one quiesce window, then first StoatFlow start = full restore |
| **Engage**    | Suppress-heavy topologies, versioned stores at scale, custom partitioners, or anything you're unsure how to classify                                      | [Get in touch](https://stoatflow.io/contact) — we plan the cutover with you                                                                                                                                 |

**Direct reuse of a Kafka Streams changelog — pointing StoatFlow at the KS topics — remains unsupported.** The two engines' on-wire changelog formats differ for several store types (window-key seqnums, versioned timestamp placement, the LEFT/OUTER join outer store, timestamped value envelopes), and the offset bookkeeping differs. That byte gap is exactly what the migration tool's translation closes — offline, verifiably, and outside the engine.

## What the translate path supports

Per store type (the tool's `type` values in parentheses):

| Store type                                                 | Verdict             | Notes                                                                                                         |
| ---------------------------------------------------------- | ------------------- | ------------------------------------------------------------------------------------------------------------- |
| Key-value, plain (`kv`)                                    | ✅ Supported         | Byte passthrough                                                                                              |
| Key-value, timestamped (`kv-timestamped`)                  | ✅ Supported         | The stripped timestamp is re-wrapped into the value                                                           |
| Window, incl. timestamped (`window`, `window-timestamped`) | ✅ Supported         | The KS seqnum suffix is handled either way                                                                    |
| Window with duplicates (`window-duplicates`)               | ✅ Supported         | Join window stores; seeded duplicates are collision-safe                                                      |
| Session (`session`)                                        | ✅ Supported         | Byte-identical on both sides                                                                                  |
| FK-join subscription store (`fk-subscription`)             | ✅ Supported         | Migrated foreign-key rows keep re-joining on foreign-table updates                                            |
| LEFT/OUTER join outer store (`outer-join`)                 | ✅ Supported         | Boundary-unmatched records still emit their null-side finals after cutover                                    |
| Headers-aware stores (KIP-1271)                            | ✅ Supported         | `recordHeaders: true` carries the native record headers; timestamped-KV/window + session families             |
| Emit-frontier companion (`emitFrontier: true`)             | ✅ Seeded on request | Prevents OnWindowClose aggregations from re-emitting every restored closed window on the first watermark tick |
| Versioned (KIP-889) (`versioned`)                          | ⚠️ Experimental     | Translation is verified; history-retention semantics around the restore boundary await real-world validation  |
| Suppress buffer                                            | ❌ Not in v1         | The KS buffer envelope is not byte-translatable — drain it at quiesce (see caveats) or accept the loss        |
| Source KTables                                             | Conditional         | Compacted source → free (StoatFlow rebuilds from the source topic); see the classification below              |

## Classifying your stores

The tool's config declares each store's `type`, and the classification follows the **StoatFlow-side store kind**, not just the KS operation. Two things matter:

1. **Name your stores explicitly on both sides** (`Materialized.as(...)`, `StreamJoined.withStoreName(...)`, `TableJoined.as(...)`). KS auto-generated names (`KSTREAM-AGGREGATE-STATE-STORE-0000000007`) are not guaranteed to match the ported topology's generated names; the config's `ksName → sfName` mapping is the escape hatch, not the plan.
2. **Get the `kv` vs `kv-timestamped` split right** — it is byte-silent if wrong. Both engines' changelogs carry bare value bytes for KV stores (KS strips the timestamp into the record timestamp), so the `type` tells the tool whether the **StoatFlow** store expects a timestamp-wrapped value.

| KS DSL operation                                                                                                                         | KS store                 | StoatFlow store                                                       | `type`                              |
| ---------------------------------------------------------------------------------------------------------------------------------------- | ------------------------ | --------------------------------------------------------------------- | ----------------------------------- |
| `count` / `reduce` / `aggregate` (KTable result)                                                                                         | timestamped KV           | timestamped KV                                                        | `kv-timestamped`                    |
| `builder.table(...)` source materialization                                                                                              | timestamped KV           | **plain KV**                                                          | `kv`                                |
| FK-join **result** materialization                                                                                                       | timestamped KV           | **plain KV**                                                          | `kv`                                |
| `windowedBy(TimeWindows/SlidingWindows).count/…`                                                                                         | timestamped window       | timestamped window                                                    | `window-timestamped`                |
| `windowedBy(SessionWindows).count/…`                                                                                                     | session                  | session                                                               | `session`                           |
| Stream-stream join window stores (`{name}-this-join-store`, `{name}-outer-other-join-store` with `StreamJoined.withStoreName("{name}")`) | plain window, duplicates | plain window, duplicates (`{name}-left-store` / `{name}-right-store`) | `window-duplicates`                 |
| LEFT/OUTER join shared outer store (`{name}-left-shared-join-store` / `-outer-shared-join-store`)                                        | list-valued KV           | per-entry outer store (`{name}-outer-store`)                          | `outer-join`                        |
| FK-join subscription store (`{join-name}-subscription-store` with `TableJoined.as("{join-name}")` — identical on both sides)             | subscription KV          | subscription KV                                                       | `fk-subscription`                   |
| Versioned KTable (`Stores.persistentVersionedKeyValueStore`)                                                                             | versioned                | versioned                                                             | `versioned`                         |
| Headers-aware stores (`Stores.persistent*WithHeaders`)                                                                                   | headers-aware            | headers-aware                                                         | family type + `recordHeaders: true` |

**Source KTables** resolve per the compaction of their source topic:

| Source topic  | KS app                                 | Path                                                                                                                |
| ------------- | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| Compacted     | any                                    | **Free** — StoatFlow rebuilds from the source topic; omit the store from the tool config                            |
| Non-compacted | unoptimized (a KS `-changelog` exists) | Translate the KS changelog as `kv`                                                                                  |
| Non-compacted | source-optimized (no KS changelog)     | Force `Consumed.materializeFromSourceTopic(true)` in the ported code — accepts the same retention gap KS itself had |

The tool's `plan` command resolves and prints this path per store, and structurally sanity-checks every declared type against sampled changelog records — see [the migration tool](https://stoatflow.io/docs/migration/migration-tool).

## Semantics caveats at the cutover boundary

- **Suppress final emissions.** StoatFlow's suppress buffer is changelog-backed and restorable — the boundary risk exists only because the KS buffer is not translated. Any final result still sitting in the KS suppress buffer at shutdown is lost to the StoatFlow side. Mitigation: cut over at a windows-closed point — quiesce input, let stream time advance past window-end + grace so KS emits, then stop. Note stream time only advances with records; a hard input stop *before* the drain leaves the buffer full forever. See the suppress notes on [Windowing](https://stoatflow.io/docs/building/windowing).
- **OnWindowClose duplicate re-emission.** With the `{store}-emitfrontier` companion left unseeded, StoatFlow's first watermark tick past a window close re-emits a final for **every restored closed window** within KS retention — duplicates of results KS already emitted, in one burst. Harmless for idempotent-upsert consumers; disruptive for incremental/append consumers. Fix: set `emitFrontier: true` in the tool config so `translate` seeds the frontier with the store's own max changelog timestamp.
- **ALO clean-shutdown requirement.** Under `at_least_once`, only a **clean** KS shutdown aligns changelog state with committed input offsets — a crashed ALO app's snapshot inherits at-least-once duplication into migrated state (double-counting on resume). Exactly-once snapshots are crash-safe. The tool warns loudly; never migrate a crashed ALO app's snapshot.
- **LEFT/OUTER join boundary.** Unmatched records buffered at cutover migrate via the `outer-join` rule and emit their null-side finals from StoatFlow's watermark scan once it passes window-end + grace. Optionally drain first (quiesce past join-window close) to minimize translated volume; INNER joins are unaffected. See [Joins](https://stoatflow.io/docs/building/joins).
- **Stream time vs watermark.** StoatFlow's stream time is the watermark; at resume it rebuilds from live records with the configured strategy (e.g. bounded out-of-orderness lag) — window closure and expiry around the boundary can differ from KS by up to the out-of-orderness bound. See [Event time and watermarks](https://stoatflow.io/docs/concepts/event-time-and-watermarks).
- **Retention boundaries.** Restored window/session records older than the StoatFlow store's retention are expired on the first post-restore watermark tick — same as native state, but worth stating.
- **Logging-disabled KS stores** have no changelog to translate from — reprocess or accept the loss.

## What is and is not supported

| Scenario                                                                    | Supported | Notes                                                                                                     |
| --------------------------------------------------------------------------- | --------- | --------------------------------------------------------------------------------------------------------- |
| StoatFlow → StoatFlow restart, RocksDB directory preserved                  | ✅         | Local state reused; fast delta replay tops it up                                                          |
| StoatFlow → StoatFlow restart, RocksDB directory gone                       | ✅         | Full rebuild from StoatFlow's own changelog topics                                                        |
| Translating KS changelogs into StoatFlow changelogs with the migration tool | ✅         | The supported state-carry path — offline, verifiable, re-runnable                                         |
| Carrying KS committed input offsets into the StoatFlow consumer group       | ✅         | The tool's `seed-offsets` step; the app resumes exactly where KS stopped                                  |
| StoatFlow reading a **Kafka Streams** changelog topic directly              | ❌         | Encodings and offset bookkeeping differ; translation exists precisely for this                            |
| Reusing the &#x2A;*Kafka Streams `application.id`** for the StoatFlow app   | ❌         | StoatFlow forces `group.id = application-id` AND derives changelog names from it — reuse collides on both |
| Copying RocksDB files from a Kafka Streams deployment onto a StoatFlow node | ❌         | The on-disk store layout is StoatFlow's; there is no file-level import                                    |
| Mixed KS + StoatFlow operation on one consumer group                        | ❌         | Cut over atomically inside the quiesce window                                                             |

## Where to go next

- **[The migration tool](https://stoatflow.io/docs/migration/migration-tool)** — config reference, the four commands, the full cutover runbook, and the rollback line.
- **[Migration without state](https://stoatflow.io/docs/migration/without-data-migration)** — the reprocess path when state is rebuildable.
- **[Architecture: state and durability](https://stoatflow.io/docs/concepts/architecture#state-stores-and-durability)** — the changelog + RocksDB model and how restoration works.
- **[Reusing your Kafka Streams dashboards](https://stoatflow.io/docs/migration/reusing-kafka-streams-dashboards)** — KS-compatible metrics for watching the cutover on the dashboards you already have.
- Suppress-heavy topology, versioned stores at scale, or unsure how to classify? [Get in touch](https://stoatflow.io/contact) — real people read every email.


# The migration tool

`stoatflow-migration-tool` is a standalone, offline CLI that carries Kafka Streams state into StoatFlow: it reads your KS app's changelog topics, writes byte-translated StoatFlow changelog topics under the new `application-id`, seeds the emit-frontier companions where asked, and carries the input consumer-group offsets over — so the StoatFlow app's first start is a normal full restore that resumes exactly where KS stopped. When to choose this path over reprocessing is covered on [Migration carrying state](https://stoatflow.io/docs/migration/with-data-migration); this page is the how-to.

::tldr-panel
- One YAML config drives four discrete, re-runnable commands: **`plan`** (verify everything, write nothing) → **`translate`** (seed the changelogs) → **`seed-offsets`** (carry the input offsets) → **`verify`** (compare the seeded topics against a re-read).
- All steps run inside **one quiesce window** with the KS app cleanly stopped.
- The tool only *reads* KS topics — before the first StoatFlow start, rollback is simply restarting the KS app.
- It needs a JVM 17+ host with network access to the brokers; it never touches the engine, RocksDB, or your application code.
::

## Getting the tool

Your Reposilite credentials (the same ones your build uses for StoatFlow artifacts) fetch the fat jar directly — match the tool version to the StoatFlow version you are migrating **to** (currently :stoatflow-version ):

```bash
curl -u customer-<slug>:<token> -O \
  "https://maven.stoatflow.io/releases/io/stoatflow/stoatflow-migration-tool/<version>/stoatflow-migration-tool-<version>-all.jar"

java -jar stoatflow-migration-tool-<version>-all.jar --help
```

Teams that prefer resolving through their build can depend on `io.stoatflow:stoatflow-migration-tool:<version>:all` and copy the artifact — it is a plain executable jar, JVM 17+.

## The migration config

One YAML file drives all commands:

```yaml
kafka:
  bootstrap.servers: broker:9092
  # + optional security client props passthrough (security.protocol, sasl.*, ssl.*)

source:                       # the Kafka Streams app being migrated FROM
  applicationId: my-ks-app
  processingGuarantee: exactly_once     # REQUIRED: exactly_once | at_least_once
  inputTopics: [orders, customers]      # topics whose group offsets carry over
  ignoreStores: []                      # KS stores deliberately left behind (e.g. a drained suppress buffer)

target:                       # the StoatFlow app being migrated TO
  applicationId: my-sf-app
  changelogNumPartitions: 1             # must match the SF app's config (default 1)
  replicationFactor: 3

stores:
  - ksName: order-totals                # KS store name (changelog: {ks-app}-order-totals-changelog)
    sfName: order-totals                # SF store name (default = ksName; set when the port renamed a store)
    type: kv-timestamped                # kv | kv-timestamped | window | window-timestamped | window-duplicates |
                                        #   session | versioned | fk-subscription | outer-join
    extraTopicConfigs: {}               # optional per-store topic-config overrides
    recordHeaders: false                # true iff the store is headers-aware (KIP-1271) — carries the
                                        #   KS changelog's native record headers through
    emitFrontier: true                  # OnWindowClose aggregations only: seed the {sfName}-emitfrontier
                                        #   companion so migrated closed windows aren't re-emitted
  - ksName: my-table-source
    type: kv                            # SF table() source stores are plain KV — see the classification table
    sourceTopic: customers              # declares a source-KTable materialization; plan resolves its path
```

The `type` values and the `ksName → sfName` mapping follow the [classification table](https://stoatflow.io/docs/migration/with-data-migration#classifying-your-stores) — get `kv` vs `kv-timestamped` right, it is byte-silent if wrong (and `plan`'s heuristics cannot catch that particular pair; the post-start spot-check below can).

**`source.processingGuarantee` is required** because it changes what a safe quiesce means: under `at_least_once`, a **clean shutdown is mandatory** — a crashed ALO app passes the group-empty check after its session timeout with changelog state *ahead* of its committed input offsets, and migrating that snapshot bakes at-least-once duplication into the carried state. Exactly-once snapshots are crash-safe.

## The four commands

```bash
java -jar stoatflow-migration-tool-<v>-all.jar plan         -c migration.yaml
java -jar stoatflow-migration-tool-<v>-all.jar translate    -c migration.yaml [--force]
java -jar stoatflow-migration-tool-<v>-all.jar seed-offsets -c migration.yaml
java -jar stoatflow-migration-tool-<v>-all.jar verify       -c migration.yaml
```

### `plan` — verify everything, write nothing

Runs every preflight and prints the migration plan. Each check exists because the failure it catches is otherwise **silent**:

| Check                                   | Failure it prevents                                                                                                                                                                                                                  |
| --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Changelog discovery cross-check         | An undeclared changelog (easily an auto-named FK `-subscription-store` or `OUTERSHARED` outer store) would silently not migrate — every discovered changelog must be declared or explicitly listed in `ignoreStores`                 |
| KS group is EMPTY                       | Migrating while the KS app still runs captures a moving snapshot                                                                                                                                                                     |
| Repartition-topic lag = 0               | In-flight re-keyed records would be lost — StoatFlow repartitions in memory and will never read KS repartition topics                                                                                                                |
| ALO clean-shutdown warning              | See above — a crashed ALO snapshot inherits duplication                                                                                                                                                                              |
| Structural type heuristics              | Samples each changelog and checks every record against the declared type (window/session key shapes, seqnum-vs-duplicates, FK CombinedKey bounds, outer-store list envelopes). A misclassified type is byte-silent at translate time |
| Source-KTable path resolution           | Prints free / translate / `materializeFromSourceTopic` per `sourceTopic` store                                                                                                                                                       |
| Target partition-count assert           | The engine does **not** validate a changelog topic's partition count against its config — a mismatch would mis-place keys silently                                                                                                   |
| `message.timestamp.type` = `CreateTime` | The timestamped/versioned rules consume record timestamps; `LogAppendTime` would substitute broker time                                                                                                                              |
| Headers-presence vs `recordHeaders`     | A headers-aware store seeded without `recordHeaders: true` restores with **all header data silently dropped**                                                                                                                        |
| Target preconditions                    | Target changelogs absent/empty, target group empty — catches leftovers from a previous attempt                                                                                                                                       |
| Consistency point                       | Prints per-partition changelog LSOs + the KS group's committed input offsets                                                                                                                                                         |

Run it as a dry run while the KS app is still up (expect only the "group not empty" error), and again for real inside the quiesce window.

### `translate` — seed the StoatFlow changelogs

Per store: creates `{sf-app-id}-{store}-changelog` with StoatFlow's exact topic configs, consumes the KS changelog (`read_committed`, from earliest to the captured stable offset), applies the store type's byte-translation rule, and produces with the default partitioner — reproducing StoatFlow's own changelog placement. For every store marked `emitFrontier: true` it additionally seeds the `{sfName}-emitfrontier` companion with that store's own max changelog record timestamp.

Re-run semantics: a re-run requires an empty target topic or `--force` (delete + recreate + reseed). There is no partial resume.

### `seed-offsets` — carry the input offsets

Copies the KS group's committed offsets for the configured `inputTopics` into the (empty) target group. Must run **before** the StoatFlow app's first start. Recommendation: set `AutoOffsetReset.none()` on migrated sources for the first start — a partial or failed seeding then fails loud instead of silently reprocessing from `earliest` or skipping to `latest`.

### `verify` — prove the seeding is faithful

Per store: record count + an order-independent streaming checksum of the seeded topic versus a **re-read of the KS source through the same translation**, plus per-partition spot byte-comparisons, the partition-count assert, and the emit-frontier seed check.

Division of labor worth understanding: `verify` proves the seeding is a *faithful, complete application of the tool's translation*. Translation *correctness* is pinned by the tool's own test suite against the real Kafka Streams classes — and a misclassified `type` reproduces identically on both sides of `verify`'s comparison, which is why the **post-start spot-check below is a required gate**, not a nicety.

## The cutover runbook

The shape of the whole thing — and, more importantly, where it stops being reversible. The tool never writes to a Kafka Streams topic; it reads them and writes new StoatFlow ones, which is exactly why every step before the first StoatFlow start can be abandoned for free.

```mermaid
sequenceDiagram
    participant O as Operator
    participant KS as Kafka Streams app
    participant T as Migration tool
    participant K as Kafka
    participant SF as StoatFlow app

    O->>T: 1 · plan — dry run, KS still serving
    note over T,K: the tool only ever READS the KS topics —<br/>every write lands on new StoatFlow topics
    O->>KS: 2 · quiesce — drain repartition topics,<br/>close windows, clean shutdown
    KS->>K: final commit, group EMPTY
    O->>T: 3 · plan — preflights green,<br/>consistency point captured
    O->>T: 4 · translate
    K-->>T: read the KS changelogs (read_committed)
    T->>K: write the StoatFlow store changelogs<br/>+ emit-frontier companions
    O->>T: 5 · seed-offsets — KS committed input<br/>offsets into the empty StoatFlow group
    O->>T: 6 · verify — counts, checksums, partition asserts
    note over O,SF: still fully reversible — restart KS,<br/>drop the seeded topics and group
    O->>SF: 7 · first start
    K-->>SF: forced full restore of every seeded changelog
    SF->>SF: required spot-check — queries against<br/>the still-stopped KS state
    SF->>K: first transactional commit into shared sinks
    note over SF,K: the point of no return
    O->>SF: 8 · validate side by side, switch downstream
    O->>KS: 9 · retire — deployment, group, internal topics
```

1. **Prepare (KS still running).** Port the code ([automated port](https://stoatflow.io/docs/migration/automated-port)); name all migrated stores explicitly on both sides; write the migration config; dry-run `plan`.
2. **Quiesce.** Stop input production if the SLA requires; let KS drain its repartition topics; if suppress is in play, reach a windows-closed point first so the buffer drains (stream time only advances with records — see the [caveats](https://stoatflow.io/docs/migration/with-data-migration#semantics-caveats-at-the-cutover-boundary)); cleanly stop the KS app. Under ALO, a clean shutdown is mandatory.
3. **`plan`** — all checks green; consistency point captured.
4. **`translate`** — seed all store changelogs (+ emit-frontier companions). Duration is proportional to total changelog bytes; the KS app stays down.
5. **`seed-offsets`** — carry input offsets into the target group.
6. **`verify`** — counts, checksums, partition counts green.
7. **First StoatFlow start** — a forced full restore of all seeded changelogs. Watch the `stoatflow.restoration.*` meters; `/health/ready` gates traffic until restoration completes; confirm `/offsets`. Then the **required spot-check**: interactive-query reads against the still-stopped KS state (counts match, a sampled key returns the expected value). This is the only check that catches a misclassified store type end-to-end.
8. **Validate & switch.** Bounded side-by-side output inspection where the topology allows, then point downstream at the StoatFlow outputs (or simply let it continue producing to the same sinks).
9. **Retire.** After sign-off: delete the KS deployment, its group, and its internal topics.

## The rollback line

**Before step 7, the KS side is untouched** — the tool only reads KS topics; it writes only new StoatFlow topics and the new group. Rollback = restart the KS app; delete the seeded topics and group at leisure.

**The point of no return is the first StoatFlow transactional commit into shared sinks** (step 7 onward). Rolling back to KS after that means KS resumes from its own older committed offsets and re-produces output StoatFlow already produced — downstream duplication that exactly-once cannot dedupe across two applications. Keep the KS deployment deployable until sign-off.

## Where to go next

- **[Migration carrying state](https://stoatflow.io/docs/migration/with-data-migration)** — when to translate vs reprocess, the per-store support matrix, the classification table, and the boundary caveats.
- **[AI assistants](https://stoatflow.io/docs/getting-started/ai-assistants)** — the `stoatflow-port-from-ks` skill walks the tool's commands and store classification with you.
- **[Architecture: lifecycle](https://stoatflow.io/docs/concepts/architecture#lifecycle-startup-restart-recovery)** — the per-store restore decision the first start runs.
- **[Reusing your Kafka Streams dashboards](https://stoatflow.io/docs/migration/reusing-kafka-streams-dashboards)** — watch the restore and the cutover on the dashboards you already have.
- Suppress-heavy topology or versioned stores at scale? [Get in touch](https://stoatflow.io/contact) — those are the shapes we still migrate hands-on.


# Reusing your Kafka Streams dashboards

You arrive from Kafka Streams with an observability estate — Grafana dashboards, Prometheus recording rules, alerts — all built against **Kafka Streams metric names**. By default none of it lights up against StoatFlow: every StoatFlow meter lives under the `stoatflow.*` namespace with StoatFlow-native structure. This page turns on the **opt-in Kafka Streams-compatible metrics mode**, which emits KS-named, KS-shaped, KS-unit series *derived from* StoatFlow's own meters — so your existing dashboards work with little or no editing.

::tldr-panel
- Set `runtime.metrics.naming: both` to emit **native + KS-named** series side by side — the recommended migration posture.
- Pick the **shape** that matches how you scraped KS: `micrometer-binder` (`kafka_stream_*`) or `jmx-exporter` (`kafka_streams_*`).
- Two edits cover most dashboards: `task-id` enumerates **lanes**, not partitions; there is a **single synthetic thread-id**, so `alive-stream-threads` alerts need their threshold set to `1`.
- Kafka **client** metrics (`kafka_consumer_*`, `kafka_producer_*`) already match for free — same Micrometer binder.
::

## Turn it on

The mode is a config knob under `runtime.metrics`. Zero code changes.

```yaml
runtime:
  metrics:
    enabled: true
    naming: both               # stoatflow | kafka-streams | both   (default: stoatflow)
    ks-compat:
      shape: micrometer-binder # micrometer-binder | jmx-exporter   (default: micrometer-binder)
      sample-window-ms: 30000  # KS metrics.sample.window.ms parity
      kafka-version: "4.1.0"   # value of the synthetic kafka_version tag (micrometer-binder shape)
```

The three `naming` modes:

| `naming`        | `/metrics` contains                                         | Use it when                                                                                                                                                           |
| --------------- | ----------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `stoatflow`     | Native `stoatflow.*` only (default — today's behaviour)     | You have no KS dashboards to reuse.                                                                                                                                   |
| `both`          | **Native + KS-named**                                       | **Migration default.** Your KS dashboards work *and* your StoatFlow dashboards keep working.                                                                          |
| `kafka-streams` | KS-named only; the mapped `stoatflow.*` families are hidden | A deliberate end state, or when scrape/ingest size matters. Note: StoatFlow's own reference dashboards and support runbooks key on `stoatflow_*` and won't work here. |

Start with `both`. Move to `kafka-streams` only once you're sure nothing you own reads the native names.

## Which shape does *your* Prometheus have?

"The KS metric name" in Prometheus depends on how the KS app was scraped. StoatFlow renders **either** shape:

| Shape               | Example series                                     | This is what you have if…                                                         |
| ------------------- | -------------------------------------------------- | --------------------------------------------------------------------------------- |
| `micrometer-binder` | `kafka_stream_thread_process_latency_avg`          | Your KS app used Micrometer / Spring Boot's `KafkaStreamsMetrics` binder.         |
| `jmx-exporter`      | `kafka_streams_stream_thread_metrics_process_rate` | You scraped KS JMX via the Prometheus `jmx_exporter` (Strimzi / Confluent-style). |

Look at one panel query in your existing dashboard and match the prefix. (The jmx-exporter shape targets the common default-config output; heavily customised jmx rules files may need panel tweaks.)

## The two edits that cover most dashboards

StoatFlow has **no stream threads and no tasks** — it runs one instance with virtual-thread *lanes*. The compat layer synthesises the KS identity tags, with two consequences worth knowing before you read a dashboard:

::callout{color="warning" icon="i-lucide-triangle-alert"}
**`task-id` is a lane, not a partition.** Sums and aggregations (`sum(...)`, `rate(...)`) match exactly. But a per-task drill-down enumerates **lanes** (`0_5` is lane 5, not partition 5), and the lane count is not the partition count. Panels that break down *by task* show lanes.
::

::callout{color="warning" icon="i-lucide-triangle-alert"}
**One synthetic `thread-id`.** StoatFlow collapses the whole engine into a single `<application-id>-StreamThread-1`. Per-thread breakdowns become one series, and any alert like `kafka_stream_alive_stream_threads < N` must have its threshold set to **1**.
::

### Alert-edit checklist

- `alive-stream-threads` alerts → threshold `1` (single synthetic thread).
- `failed-stream-threads` → maps from StoatFlow's engine-restart counter (a `replace_thread` restart *is* the KS thread-replacement recovery event); keep the alert, it fires on real fault storms.
- Per-`task-id` / per-`thread-id` breakdown panels → read them as per-lane / single-series.
- Thread `process-latency` → StoatFlow measures **per record**, KS per iteration-batch, so absolute values are smaller than a real KS app; keep the *shape*, re-baseline any hard thresholds.
- Ratios (`process-ratio`, `active-process-ratio`, `punctuate-ratio`) are **not** emitted (lane parallelism makes true utilization >1); remove or ignore those panels.

## What maps, and how well

The full per-metric compatibility table is the canonical mapping in the engine; the tiers are:

- **✅ full** — same meaning, same math (poll, store `range/all/flush` latency, topic consumed/produced totals, node e2e latency, …).
- **⚠️ approximate** — emitted, semantics differ, documented per row (thread `commit-latency` = barrier commit; `dropped-records` covers null-key + join + late-window drops; per-lane task metrics; store `get/put/delete` latency; RocksDB rows).
- **❌ not mappable** — architecturally absent (no rebalancing, no enforced-processing, no iterator metrics); their absence *is* parity — a Micrometer-instrumented KS app skips the same non-numeric metrics.

State-store panels key on the scope-dependent tag (`rocksdb-window-state-id`, `in-memory-session-state-id`, …); StoatFlow synthesises the correct scope key per store, so those panels populate.

## One thing to know: the native surface shifts slightly

With `naming != stoatflow` a compat filter tightens the mapped native timers' `stoatflow_*_seconds_max` to a \~30 s window (fresher than the default 2 min × 3) and adds a p0 quantile to `stoatflow.e2e.latency` (used to approximate `record-e2e-latency-min`). This is bounded and arguably an improvement, but it *is* an observable change to a few native `_max` series — expect them to be a little "twitchier".

## Sizing

`both` mode roughly adds: thread ≈ 25 series; task ≈ 8-10 × lanes (64 lanes ≈ 600 series); store ≈ 10-14 × stores (plus \~44 × stores for RocksDB when `statistics-enabled`); topic ≈ 4 × topics; cache ≈ 3 × stores. Lane count dominates. At `recording-level: debug` with many lanes this is thousands of extra series — which is remote-write / ingest cost, not just scrape payload. The recording level gates the sources (see [Metrics](https://stoatflow.io/docs/runtime/metrics)).

## Next steps

- **[Metrics](https://stoatflow.io/docs/runtime/metrics)** — the native `stoatflow.*` catalogue and recording levels.
- **[Observability](https://stoatflow.io/docs/operating/observability)** — alerting and the Grafana view.


# Reference

Fact-only lookup material. These pages enumerate the full surface — every configuration key, every REST endpoint, every Gradle DSL property and Maven `<properties>` key, the Kafka Streams DSL parity status, every Prometheus meter, and the project's vocabulary. For how-to and conceptual guidance, see [Building](https://stoatflow.io/docs/building), [Configuration](https://stoatflow.io/docs/configuration), and [Runtime](https://stoatflow.io/docs/runtime).

## Reference pages

::path-cards
  :::path-card
  ---
  icon: i-lucide-sliders-horizontal
  title: Configuration reference
  to: https://stoatflow.io/docs/reference/configuration-reference
  ---
  Every `stoatflow.*` and `runtime.*` configuration key — type, default, and description.
  :::

  :::path-card
  ---
  icon: i-lucide-webhook
  title: REST API reference
  to: https://stoatflow.io/docs/reference/rest-api-reference
  ---
  Every HTTP endpoint the runtime exposes — path, method, and response shape.
  :::

  :::path-card
  ---
  icon: i-simple-icons-gradle
  title: Gradle plugin reference
  to: https://stoatflow.io/docs/reference/gradle-plugin-reference
  ---
  The `io.stoatflow` convention plugin — the `stoatflow { }` extension DSL, tasks, and defaults.
  :::

  :::path-card
  ---
  icon: i-simple-icons-apachemaven
  title: Maven reference
  to: https://stoatflow.io/docs/reference/maven-reference
  ---
  The `stoatflow-bom`, `stoatflow-parent`, and `stoatflow-maven-plugin` — the artifacts, `<properties>` keys, and goals.
  :::

  :::path-card
  ---
  icon: i-lucide-table
  title: KS compatibility matrix
  to: https://stoatflow.io/docs/reference/ks-compatibility-matrix
  ---
  Kafka Streams DSL parity — which operators are implemented, partial, or StoatFlow extensions.
  :::

  :::path-card
  ---
  icon: i-lucide-gauge
  title: Metrics reference
  to: https://stoatflow.io/docs/reference/metrics-reference
  ---
  Every Prometheus meter on `/metrics` — name, type, tags, and what it measures, grouped by area.
  :::

  :::path-card
  ---
  icon: i-lucide-book-a
  title: Glossary
  to: https://stoatflow.io/docs/reference/glossary
  ---
  Definitions of StoatFlow terms — lanes, commit barriers, watermarks, epochs, and more.
  :::
::


# Configuration reference

Every StoatFlow application is configured from a single `application.yaml` (loaded by `StoatFlowRuntime.fromConfig(...)`), plus environment-variable and system-property overrides. This page is a **curated** reference of the keys you are most likely to set, grouped by area, each with its type, default, and a one-line description.

::callout{color="info" icon="i-lucide-file-json"}
The **authoritative, always-current** list of every key — including bound constraints, enums, and examples — is the JSON Schema shipped in the runtime artifact: `runtime/src/main/resources/schemas/stoatflow-config-schema.json`. IDEs (VS Code with the Red Hat YAML extension, IntelliJ) pick it up automatically for autocompletion and validation. This page omits a handful of advanced, fine-grained tuning knobs; the schema lists them all.
::

For the configuration model — the four override layers, env-var naming, and how `stoatflow.*` maps onto the core engine — see [Configuration model](https://stoatflow.io/docs/concepts/configuration-model). For Kafka client passthrough, see [Kafka client config](https://stoatflow.io/docs/configuration/kafka-client-config).

## How to read this page

- **Key** — the YAML path under `stoatflow:` or `runtime:`. Nest by the dots (`commit-barrier.interval-ms` lives under `stoatflow.commit-barrier.interval-ms`).
- **Type** — YAML scalar type or enum value set.
- **Default** — the value used when the key is omitted. `—` means no default (the value must be set, or it is null/unset).
- Defaults shown as expressions (e.g. `max(2, CPU cores)`) are resolved at startup from the host.

::callout{color="warning" icon="i-lucide-shield"}
Secrets (the license key, Kafka SASL passwords, Schema Registry credentials) should be supplied via environment-variable interpolation (`${STOATFLOW_LICENSE_KEY}`) — never hard-coded in committed YAML. See [License configuration](https://stoatflow.io/docs/getting-started/license-configuration).
::

## stoatflow — top level

The required and most common engine keys.

| Key                                 | Type                                   | Default                 | Description                                                                                                                                                  |
| ----------------------------------- | -------------------------------------- | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `application-id`                    | string                                 | — (required)            | Unique application identifier; used for the consumer group and transactional id.                                                                             |
| `bootstrap-servers`                 | string                                 | `localhost:9092`        | Kafka bootstrap servers (comma-separated).                                                                                                                   |
| `processing-guarantee`              | enum `EXACTLY_ONCE` \| `AT_LEAST_ONCE` | `EXACTLY_ONCE`          | Commit protocol. Exactly-once uses Kafka transactions; at-least-once uses non-transactional commit (lower latency, duplicates possible on crash recovery).   |
| `default-key-serde`                 | string (FQCN)                          | —                       | Fully qualified class name of the default key Serde when none is given explicitly.                                                                           |
| `default-value-serde`               | string (FQCN)                          | —                       | Fully qualified class name of the default value Serde when none is given explicitly.                                                                         |
| `schema-registry-url`               | string                                 | —                       | Confluent Schema Registry URL. When set, propagated to serdes and enables the Schema Registry health check.                                                  |
| `deserialization-exception-handler` | string (FQCN)                          | —                       | Handler class deciding CONTINUE vs FAIL on deserialization errors.                                                                                           |
| `production-exception-handler`      | string (FQCN)                          | —                       | Handler class for serialization and send errors.                                                                                                             |
| `processing-exception-handler`      | string (FQCN)                          | —                       | Handler class for exceptions thrown by `process()`.                                                                                                          |
| `rocks-db-config-setter`            | string (FQCN)                          | —                       | Custom `RocksDBConfigSetter` for advanced per-store RocksDB tuning.                                                                                          |
| `application-server`                | string (`host:port`)                   | `localhost:<http.port>` | Advertised host\:port for interactive-query metadata (KS `application.server`). Single instance: identifies this instance as the active host for all stores. |

See [Error handling and DLQ](https://stoatflow.io/docs/building/error-handling-dlq) for the built-in exception handler classes.

## stoatflow\.lanes

Key-affinity lane parallelism. See [Lanes and parallelism](https://stoatflow.io/docs/concepts/lanes-and-parallelism).

| Key                    | Type                         | Default             | Description                                                                                                  |
| ---------------------- | ---------------------------- | ------------------- | ------------------------------------------------------------------------------------------------------------ |
| `lanes.count`          | integer                      | `max(2, CPU cores)` | Number of key-affinity lanes (parallel processing units).                                                    |
| `lanes.queue-capacity` | integer                      | `300`               | Capacity of each lane's queue; controls backpressure.                                                        |
| `lanes.thread-type`    | enum `VIRTUAL` \| `PLATFORM` | `VIRTUAL`           | Lane thread type. Virtual (Project Loom) by default; platform threads available for the JNI RocksDB backend. |

## stoatflow\.commit-barrier

Commit-barrier cadence and the bounds the engine self-tunes within. The interval between barriers is adaptive at runtime; you set the seed and the floor/ceiling. See [Exactly-once](https://stoatflow.io/docs/concepts/exactly-once).

| Key                                       | Type    | Default          | Description                                                                                                                                                                                                                                                                                                                                                                                     |
| ----------------------------------------- | ------- | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `commit-barrier.interval-ms`              | integer | `500`            | Seed interval between commit barriers; the initial value before adaptive scheduling takes over.                                                                                                                                                                                                                                                                                                 |
| `commit-barrier.min-interval-ms`          | integer | `150`            | Lower bound on the barrier-to-barrier interval.                                                                                                                                                                                                                                                                                                                                                 |
| `commit-barrier.max-interval-ms`          | integer | `5000`           | Upper bound on the barrier-to-barrier interval.                                                                                                                                                                                                                                                                                                                                                 |
| `commit-barrier.timeout-ms`               | integer | `60000`          | Timeout for barrier completion. Bounds the commit critical path and cascades to the producer's `transaction.timeout.ms`, `max.block.ms`, and `delivery.timeout.ms`. Must be greater than `interval-ms`.                                                                                                                                                                                         |
| `commit-barrier.dedicated-commit-thread`  | boolean | `true`           | Run commits on a dedicated thread so a lane is not stalled during the Kafka transaction commit and state flush.                                                                                                                                                                                                                                                                                 |
| `commit-barrier.max-epoch-records`        | integer | — (no cap)       | Optional hard upper bound on records committed per epoch. Unset means the engine sizes epochs adaptively within the interval bounds above.                                                                                                                                                                                                                                                      |
| `commit-barrier.max-engine-restarts`      | integer | `5`              | Max fault-triggered in-place engine restarts within the rolling window below before the instance escalates to a terminal shutdown. `0` disables in-place restart recovery.                                                                                                                                                                                                                      |
| `commit-barrier.engine-restart-window-ms` | integer | `300000` (5 min) | Rolling window over which fault-triggered engine restarts are counted. Bounds a recurring fault to `max-engine-restarts` restarts per window instead of looping.                                                                                                                                                                                                                                |
| `commit-barrier.max-poison-replays`       | integer | `10`             | Max poison-epoch replays within the rolling window below before the instance shuts down with the source offsets held. A replay is an in-place engine restart that reprocesses an aborted epoch with the poison records skipped. `0` disables replay entirely. Separate from `max-engine-restarts`, and **in-memory** — exhausting it ends the process, so the pod restarts with a clean budget. |
| `commit-barrier.poison-replay-window-ms`  | integer | `60000` (1 min)  | Rolling window over which poison-epoch replays are counted. A poison that does not converge — typically one derived from accumulated state — exhausts the budget and terminates the instance rather than replaying forever.                                                                                                                                                                     |

::callout{color="info" icon="i-lucide-info"}
The schema exposes additional fine-grained commit-barrier tuning factors (smoothing and growth parameters for the adaptive epoch sizer). These rarely need changing — leave them at their defaults unless directed by support. They are listed in full in the JSON schema.
::

## stoatflow\.state

State-store storage, caching, and restoration. See [State stores](https://stoatflow.io/docs/building/state-stores) and [Migration (with data)](https://stoatflow.io/docs/migration/with-data-migration).

| Key                                   | Type                                | Default                       | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| ------------------------------------- | ----------------------------------- | ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `state.dir`                           | string                              | `${java.io.tmpdir}/stoatflow` | Directory for state-store data. Put it on fast local storage (SSD); in Kubernetes, on a persistent volume.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `state.uncommitted-max-bytes`         | integer (bytes)                     | `268435456` (256 MiB)         | Global cap on uncommitted state bytes across all stores; exceeding it triggers an early commit barrier.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `state.restoration-enabled`           | boolean                             | `true`                        | Restore state from changelog topics on startup when local state is missing.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `state.restoration-pool-size`         | integer                             | `CPU cores × 2`               | Restoration worker pool size; each worker uses its own consumer for parallel fetch.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `state.restoration-batch-size`        | integer                             | `20000`                       | Records per RocksDB write batch during restoration.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `state.sst-restoration-enabled`       | boolean                             | `false`                       | Opt-in SST-file ingestion for full cold-start KV restoration (bypasses memtable/L0).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `state.parallel-store-commits`        | boolean                             | `true`                        | Commit multiple stores in parallel at barrier time.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `state.parallel-store-commit-threads` | integer                             | `CPU cores`                   | Max platform threads for parallel store commits (used only with 2+ transactional stores).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `state.cleanup-delay-ms`              | integer                             | `600000` (10 min)             | Delay before deleting RocksDB directories for stores no longer in the topology. `0` = immediate, `-1` = never delete (detect + warn only).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `state.cleanup-dir-max-age-ms`        | integer (ms)                        | `-1` (disabled)               | Age threshold for purging local per-store state directories untouched for at least this long at startup (KIP-1259); they are rebuilt from the changelog. Set ≥ the changelog's `delete.retention.ms`. Distinct from `cleanup-delay-ms` (orphaned-store cleanup).                                                                                                                                                                                                                                                                                                                                                           |
| `state.segment-checkpoint-records`    | integer                             | `100000`                      | Periodic checkpoint cadence for segmented window/session stores, by record count. These stores run with the WAL off and become durable only on `close()`, so a checkpoint (flush dirty segments, then the offsets) bounds how much of the changelog an ungraceful restart — or a hot-standby promotion — has to re-apply. Whichever-first with `state.segment-checkpoint-ms`. Deliberately coarse to limit SST/compaction churn.                                                                                                                                                                                           |
| `state.segment-checkpoint-ms`         | integer (ms)                        | `30000` (30 s)                | The same checkpoint cadence by wall-clock time; whichever-first with `state.segment-checkpoint-records`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `state.format-downgrade`              | enum `refuse` \| `wipe-and-restore` | `refuse`                      | What to do when a store's on-disk format is newer than the topology asks for — today, a headers-aware store (KIP-1271) opened with record headers turned off. `refuse` fails startup with an actionable message and touches nothing on disk. `wipe-and-restore` acknowledges the downgrade: that store's local state is deleted and rebuilt from its changelog (one full restore). An **empty** headers keyspace is dropped in place for free either way; a store with **no recovery source** is refused either way. See [State stores](https://stoatflow.io/docs/building/state-stores#turning-record-headers-off-again). |

Advanced SST knobs (`state.sst-buffer-max-bytes`, `state.sst-flush-interval-files`) and `state.flush-min-entries-per-chunk` are in the schema.

## stoatflow\.changelog

Changelog topics that back state stores. See [State stores](https://stoatflow.io/docs/building/state-stores).

| Key                                    | Type    | Default               | Description                                                                                   |
| -------------------------------------- | ------- | --------------------- | --------------------------------------------------------------------------------------------- |
| `changelog.enabled`                    | boolean | `true`                | Write state mutations to Kafka changelog topics within the transaction.                       |
| `changelog.create-topics-if-not-exist` | boolean | `true`                | Auto-create changelog topics (with `cleanup.policy=compact`) when missing.                    |
| `changelog.replication-factor`         | integer | `-1` (broker default) | Replication factor for auto-created changelog topics.                                         |
| `changelog.num-partitions`             | integer | `1`                   | Partition count for auto-created changelog topics (single instance: 1 is usually sufficient). |

## stoatflow\.watermark

Event-time watermark generation. See [Event time and watermarks](https://stoatflow.io/docs/concepts/event-time-and-watermarks).

| Key                                 | Type    | Default          | Description                                                                         |
| ----------------------------------- | ------- | ---------------- | ----------------------------------------------------------------------------------- |
| `watermark.max-out-of-orderness-ms` | integer | `10000`          | Allowed out-of-orderness before an event is considered late.                        |
| `watermark.idleness-timeout-ms`     | integer | `300000` (5 min) | Partitions idle for this long are excluded from the global watermark. `0` disables. |
| `watermark.auto-interval-ms`        | integer | `200`            | Periodic watermark emission interval.                                               |

## stoatflow\.processing

Processing behavior. The keys below are the public, commonly-set ones; the schema additionally exposes a number of low-level dispatch/yield tuning knobs that should normally be left at defaults.

| Key                                         | Type                                   | Default | Description                                                                                                                                                             |
| ------------------------------------------- | -------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `processing.mdc-enabled`                    | boolean                                | `false` | Populate logging MDC keys (`laneId`, `sourceTopic`, `sourcePartition`, `sourceOffset`) during processing.                                                               |
| `processing.max-task-idle-ms`               | integer                                | `0`     | Wait time for empty partition buffers to preserve cross-partition timestamp order. `-1` = process immediately, `0` = wait only while broker has lag, `>0` = extra wait. |
| `processing.buffered-records-per-partition` | integer                                | `2000`  | Max records buffered per partition before pausing consumption (must be ≥ `max.poll.records`).                                                                           |
| `processing.consumer-poll-timeout-ms`       | integer                                | `20`    | Consumer `poll()` fallback timeout.                                                                                                                                     |
| `processing.dispatch-mode`                  | enum `AUTO` \| `DIRECT` \| `LANE`      | `LANE`  | Record dispatch mode. `LANE` routes via key-affinity lanes; `AUTO` may pick `DIRECT` for simple stateless topologies; `DIRECT` is experimental.                         |
| `processing.parallel-deser-mode`            | enum `AUTO` \| `ON` \| `OFF`           | `AUTO`  | Parallel deserialization. `AUTO` enables it when per-record cost is high enough; `ON`/`OFF` force it.                                                                   |
| `processing.timestamp-coordination-mode`    | enum `AUTO` \| `ENABLED` \| `DISABLED` | `AUTO`  | Event-time coordination. `AUTO` derives from topology traits; override only for debugging.                                                                              |
| `processing.timestamp-ordered-dispatch`     | boolean                                | `true`  | Dispatch records in global timestamp order across partitions.                                                                                                           |
| `processing.wall-clock-advance-interval-ms` | integer                                | `10`    | Cadence for wall-clock punctuators and scheduled sources.                                                                                                               |
| `processing.state-transition-history-size`  | integer                                | `20`    | Number of past app-state transitions retained for the `/state` endpoint.                                                                                                |

## stoatflow\.validation

Startup and single-instance safety checks. See [How StoatFlow differs from KS](https://stoatflow.io/docs/concepts/how-stoatflow-differs-from-ks).

| Key                                          | Type    | Default | Description                                                                                        |
| -------------------------------------------- | ------- | ------- | -------------------------------------------------------------------------------------------------- |
| `validation.ensure-explicit-naming`          | boolean | `true`  | Fail startup if any topology operation uses an auto-generated name (KIP-1111).                     |
| `validation.validate-complete-assignment`    | boolean | `true`  | Wait for and verify that all source-topic partitions are assigned at startup.                      |
| `validation.partition-assignment-timeout-ms` | integer | `60000` | Timeout to wait for complete partition assignment after subscribe.                                 |
| `validation.pre-flight-consumer-group-check` | boolean | `false` | Check for active consumer-group members before starting (prevents accidental duplicate instances). |

## stoatflow\.shutdown

Graceful shutdown behavior. See [Probes](https://stoatflow.io/docs/operating/probes).

| Key                                   | Type    | Default                   | Description                                                                                                            |
| ------------------------------------- | ------- | ------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| `shutdown.timeout-ms`                 | integer | `25000`                   | Timeout for graceful shutdown; lets in-flight barriers complete.                                                       |
| `shutdown.leave-group-on-close`       | boolean | `false`                   | Whether the consumer sends a LeaveGroup on close. Disabled (with static membership) gives faster restart reassignment. |
| `shutdown.hard-exit-on-shutdown-hang` | boolean | `true`                    | If graceful shutdown stalls past its budget, force a clean process exit so Kubernetes restarts the pod.                |
| `shutdown.hard-exit-budget-ms`        | integer | `2 × shutdown.timeout-ms` | Budget before the hard-exit fallback fires. Must be ≥ `shutdown.timeout-ms`.                                           |

## stoatflow\.commit-stall

Commit-pipeline freeze detection. These bounds also drive the liveness probe's stall check. See [Probes](https://stoatflow.io/docs/operating/probes).

| Key                             | Type    | Default | Description                                                                                                                                                                                |
| ------------------------------- | ------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `commit-stall.threshold-ms`     | integer | `45000` | If commit work stalls longer than this, a thread dump is captured, graceful shutdown begins, and `/health/live` returns DOWN. Must be less than `commit-barrier.timeout-ms`. `0` disables. |
| `commit-stall.poll-interval-ms` | integer | `5000`  | How often the stall check runs.                                                                                                                                                            |
| `commit-stall.dump-enabled`     | boolean | `true`  | Emit a thread dump to logs when a stall is detected.                                                                                                                                       |

## stoatflow\.rocks-db

RocksDB preset and backend. See [State stores](https://stoatflow.io/docs/building/state-stores).

| Key                | Type                                                 | Default   | Description                                                                                                                                                                                     |
| ------------------ | ---------------------------------------------------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `rocks-db.preset`  | enum `DEFAULT` \| `LOW_MEMORY` \| `HIGH_PERFORMANCE` | `DEFAULT` | Memory preset. `DEFAULT` ≈ 256 MiB, `LOW_MEMORY` ≈ 64 MiB, `HIGH_PERFORMANCE` ≈ 1 GiB.                                                                                                          |
| `rocks-db.backend` | enum `AUTO` \| `FFM` \| `JNI`                        | `AUTO`    | RocksDB binding. `AUTO` picks the faster path per runtime — FFM on the JVM, JNI under a GraalVM native image. `FFM` has lower per-call overhead on HotSpot; `JNI` uses the rocksdbjni bindings. |

## stoatflow\.caching

Downstream emission suppression for KTables. See [Aggregations](https://stoatflow.io/docs/building/aggregations).

| Key                                    | Type            | Default                  | Description                                                                                                                                                                                      |
| -------------------------------------- | --------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `caching.emission-suppression-enabled` | boolean         | `true`                   | Compact multiple writes to the same key within a barrier interval, emitting only the final value.                                                                                                |
| `caching.max-entries`                  | integer         | `2147483647` (unbounded) | Per-store cached-entry cap before an early barrier. Unbounded by default so it never bites high-cardinality workloads — bytes are the default lever; set a finite value for a count-based bound. |
| `caching.max-estimated-bytes`          | integer (bytes) | `268435456` (256 MiB)    | Max estimated cached bytes per store before an early barrier.                                                                                                                                    |

## stoatflow\.timer-backend

Timer storage backend for the Processor API. See [Processor API](https://stoatflow.io/docs/building/processor-api).

| Key                               | Type                                    | Default  | Description                                                                                                          |
| --------------------------------- | --------------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------- |
| `timer-backend.type`              | enum `HEAP` \| `PERSISTENT` \| `HYBRID` | `HYBRID` | `HEAP` = in-memory (fast, unbounded), `PERSISTENT` = RocksDB only (memory-bounded), `HYBRID` = heap cache + RocksDB. |
| `timer-backend.changelog-enabled` | boolean                                 | `true`   | Log timer mutations to a Kafka changelog for durability (applies to all types).                                      |
| `timer-backend.cache-horizon-ms`  | integer                                 | `60000`  | HYBRID only: how far ahead timers are cached in the heap. Must be > `load-interval-ms`.                              |
| `timer-backend.load-interval-ms`  | integer                                 | `10000`  | HYBRID only: how often the loader pulls upcoming timers from RocksDB. Must be < `cache-horizon-ms`.                  |

## stoatflow\.ktable / stoatflow\.fk-join / stoatflow\.dsl / stoatflow\.topology

| Key                                   | Type                            | Default   | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| ------------------------------------- | ------------------------------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ktable.auto-reuse-source-topics`     | boolean                         | `true`    | Reuse compacted KTable source topics for restoration instead of separate changelog topics.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `fk-join.use-reverse-index-cache`     | boolean                         | `true`    | Maintain an in-memory reverse index for O(1) FK-join right-side lookups (disable to save heap on very large FK tables).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `dsl.store-format`                    | enum `DEFAULT` \| `HEADERS`     | `DEFAULT` | Global default on-disk format for DSL-materialized stores (KIP-1285). `HEADERS` makes DSL aggregations persist record headers (KIP-1271). Per-store `Materialized.withRecordHeaders()` / `withoutRecordHeaders()` always wins.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `topology.validation.<rule-id>`       | enum `off` \| `warn` \| `error` | per rule  | Per-rule topology validation, raised at **topology-compile time** — so `error` fails `StoatFlow.start()` **and** `TopologyTestDriver&#x60;, with no broker, reporting every offending site in one message rather than failing on the first. Rules: &#x2A;*`unresolved-boundary-key-serde`** (default `warn&#x60;) — a sub-topology boundary resolved no key serde for the key crossing it, so the configured default is used for lane-assignment hashing; &#x2A;*`inherited-boundary-key-serde`** (default `warn&#x60;) — the boundary resolved only by inheriting a serde from before the re-key (sound for lane hashing, but only works if the key types line up). &#x2A;*`papi-key-affinity-presumed`*&#x2A; (default &#x2A;*`error`**) — a **proven** re-key feeds a Processor API node connecting a **writable** store with no boundary between them under `topology.processor-api-key-affinity: off`, so records reach it on the lane of their *old&#x2A; key and a store read-modify-write straddling a commit barrier can lose an update durably; no lock closes that window, which is why this one refuses the build rather than warning. Read-only (KIP-813) stores are excluded — they cannot be written. &#x2A;*`papi-key-affinity-presumed-chain`** (default `warn`) — the same shape where the feeder is itself a Processor API node: `process()` and `addProcessor()` declare a key change unconditionally, so the re-key may not exist at all, and Kafka Streams compiles and runs such chains without a repartition. Everything except `papi-key-affinity-presumed` defaults to `warn`, so upgrading changes nothing. An unknown rule id is rejected at startup. See [Boundary key serdes](https://stoatflow.io/docs/building/serdes#boundary-key-serdes-lane-assignment) and [Key affinity after a re-key](https://stoatflow.io/docs/building/processor-api#key-affinity-after-a-re-key). |
| `topology.sub-topology-split`         | enum `lazy` \| `eager`          | `lazy`    | Where the compiler opens sub-topology boundaries. `lazy` opens one only where a downstream operator genuinely requires the re-keyed record's lane affinity — grouped aggregations, joins, `toTable()`, plus every explicit `repartition()`; this is the Kafka Streams `repartitionRequired` model, so `describe()` matches a KS original. `eager` restores the pre-1.0.0 boundary at **every** key-changing node. Changing this renumbers sub-topology ids, which appear in thread names, the `sub_topology` metric tag and the topology endpoints. See [Lanes and parallelism](https://stoatflow.io/docs/concepts/lanes-and-parallelism#where-the-handoff-actually-happens) — in particular the skew note, if you relied on a re-key to spread work.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `topology.processor-api-key-affinity` | enum `off` \| `presumed`        | `off`     | Whether a Processor API node's key affinity is **presumed** from the adapter's static shape. `off` matches Kafka Streams, which never materialises a repartition before `process()` / `processValues()` even with connected stores. `presumed` restores the pre-1.0.0 boundary. Inert under `topology.sub-topology-split: eager`. After a **many-to-one** re-key, `off` gives up key affinity at that node — see [Key affinity after a re-key](https://stoatflow.io/docs/building/processor-api#key-affinity-after-a-re-key); the provable store-connected case is refused at build time by the `papi-key-affinity-presumed` validation rule. `off` also leaves the node in its **upstream** sub-topology, so it inherits that sub-topology's lane count instead of starting its own — re-check any deliberate per-sub-topology `numberOfLanes` override.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |

## stoatflow\.kafka

Kafka client passthrough. Values under these maps are merged on top of StoatFlow's framework defaults; forced overrides (e.g. `bootstrap.servers`, `group.id`, `enable.auto.commit=false`) cannot be changed. Full resolution order in [Kafka client config](https://stoatflow.io/docs/configuration/kafka-client-config).

| Key                          | Type                | Default | Description                                                                                                                                                                                                                        |
| ---------------------------- | ------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `kafka.consumer`             | map\<string,string> | `{}`    | Additional consumer properties (e.g. `auto.offset.reset`, `max.poll.records`). Its `security.*` / `ssl.*` / `sasl.*` entries also propagate to the producer, admin and restoration clients.                                        |
| `kafka.producer`             | map\<string,string> | `{}`    | Additional producer properties (e.g. `compression.type`, `linger.ms`).                                                                                                                                                             |
| `kafka.restoration-consumer` | map\<string,string> | `{}`    | Overrides for the restoration consumer only.                                                                                                                                                                                       |
| `kafka.admin`                | map\<string,string> | `{}`    | Overrides for the admin client (changelog topic management, broker health check, consumer-group preflight). Only needed when the admin principal differs from the consumer's — the consumer's security settings already propagate. |

## stoatflow\.license

Runtime license. See [License configuration](https://stoatflow.io/docs/getting-started/license-configuration).

| Key                      | Type    | Default                       | Description                                                                                         |
| ------------------------ | ------- | ----------------------------- | --------------------------------------------------------------------------------------------------- |
| `license.key`            | string  | —                             | License key string. Use `${STOATFLOW_LICENSE_KEY}` interpolation; do not hard-code.                 |
| `license.file`           | string  | —                             | Path to a `chmod 600` file holding the key on one line. Mutually exclusive with `key` (`key` wins). |
| `license.environment`    | string  | —                             | Deployment environment label feeding the machine fingerprint. Required for the Production tier.     |
| `license.cache-dir`      | string  | `~/.stoatflow/license-cache/` | Offline-cache directory. In Kubernetes, mount on a persistent volume so it survives restarts.       |
| `license.verbose-banner` | boolean | `false`                       | Include the customer name in the startup banner.                                                    |

::callout{color="info" icon="i-lucide-key"}
License env vars and `-D` system properties (`STOATFLOW_LICENSE_*`) take precedence over these YAML values.
::

## stoatflow\.ha

Opt-in hot-standby high availability. With `mode: off` (default) the application is a single instance. See [High availability](https://stoatflow.io/docs/operating/high-availability).

| Key                                           | Type    | Default                            | Description                                                                                                                                                                                                                                                                                                                                                      |
| --------------------------------------------- | ------- | ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ha.mode`                                     | enum    | `off`                              | `off` (single instance) or `active-standby` (hot-standby cluster: one active + one or more warm standbys).                                                                                                                                                                                                                                                       |
| `ha.pod-id`                                   | string  | `POD_NAME` → `HOSTNAME` → hostname | Stable per-pod identity distinguishing the peers.                                                                                                                                                                                                                                                                                                                |
| `ha.coordination-topic`                       | string  | `__stoatflow_<applicationId>_ha`   | Single-partition, compacted, auto-created topic the cluster coordinates through (heartbeats + the promotion token).                                                                                                                                                                                                                                              |
| `ha.desired-standbys`                         | integer | `1`                                | Caught-up standby spares the readiness gate protects. `1` is the classic active/passive pair; raise it (with `replicas`) for more redundancy. Drives the redundancy floor on rolling deploys.                                                                                                                                                                    |
| `ha.max-standbys`                             | integer | `3`                                | Ceiling on standbys, bounding changelog fan-out (each standby is one more changelog reader). Total instances ≤ `max-standbys + 1`.                                                                                                                                                                                                                               |
| `ha.failover-priority`                        | integer | `0`                                | Per-pod tie-break hint ordering candidates within the equal-lag bucket (higher wins). Never overrides lag or the hash floor.                                                                                                                                                                                                                                     |
| `ha.promotion-settle-window-ms`               | integer | `500`                              | After claiming the promotion token, how long the winner confirms it is the sole claimant before fencing (single-promoter guarantee). Bounded `1..ha.staleness-threshold-ms`.                                                                                                                                                                                     |
| `ha.heartbeat-ms`                             | integer | `1000`                             | How often each pod publishes its liveness and role.                                                                                                                                                                                                                                                                                                              |
| `ha.staleness-threshold-ms`                   | integer | `5000`                             | Gap without a peer heartbeat before it's considered stale. Must be ≥ `ha.heartbeat-ms`.                                                                                                                                                                                                                                                                          |
| `ha.staleness-misses`                         | integer | `3`                                | Consecutive missed checks before a standby is elected to promote (debounce).                                                                                                                                                                                                                                                                                     |
| `ha.acceptable-recovery-lag`                  | integer | `50000`                            | Total committed-changelog lag (records, summed across all changelog partitions) at/below which a standby is caught up (`READY_STANDBY`) and promotion-eligible without the grace fallback. A single total sum, not per-task like KS's `acceptable.recovery.lag` (10000). Correctness never depends on it (restore-before-process does); mostly a raise-me lever. |
| `ha.promotion-grace-ms`                       | integer | `5000`                             | Grace window after which the least-behind standby auto-promotes when no standby is within `acceptable-recovery-lag` (availability-first; eligibility re-checked continuously).                                                                                                                                                                                   |
| `ha.promotion-restore-no-progress-timeout-ms` | integer | `60000`                            | Progress-aware deadline for restore-before-process — fails only on a window with zero records applied (genuinely stuck), then self-terminates → K8s restart. Not the commit-path barrier timeout.                                                                                                                                                                |

## runtime.http

The built-in HTTP server. See [REST API](https://stoatflow.io/docs/runtime/rest-api).

| Key                          | Type    | Default   | Description                                                                                                                         |
| ---------------------------- | ------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `runtime.http.enabled`       | boolean | `true`    | Enable the HTTP server (health, metrics, info, topology, control endpoints).                                                        |
| `runtime.http.port`          | integer | `8080`    | Port to bind.                                                                                                                       |
| `runtime.http.host`          | string  | `0.0.0.0` | Bind address.                                                                                                                       |
| `runtime.http.backlog`       | integer | `50`      | Max pending connections in the socket backlog.                                                                                      |
| `runtime.http.debug.enabled` | boolean | `true`    | Register the `/debug/threads` and `/debug/barriers` diagnostic endpoints. Disable to reduce attack surface in hardened deployments. |

## runtime.metrics

Micrometer / Prometheus metrics. See [Metrics](https://stoatflow.io/docs/runtime/metrics).

| Key                                          | Type                                          | Default             | Description                                                                                                                                                                                                                                              |
| -------------------------------------------- | --------------------------------------------- | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `runtime.metrics.enabled`                    | boolean                                       | `true`              | Enable metrics collection and the `/metrics` Prometheus endpoint.                                                                                                                                                                                        |
| `runtime.metrics.prefix`                     | string                                        | `stoatflow`         | Prefix for all metric names.                                                                                                                                                                                                                             |
| `runtime.metrics.recording-level`            | enum `info` \| `debug` \| `trace`             | `info`              | Metric detail level. `info` is production-safe; higher levels add cardinality/overhead.                                                                                                                                                                  |
| `runtime.metrics.common-tags`                | map\<string,string>                           | `{}`                | Tags applied to all metrics (e.g. `env`, `region`).                                                                                                                                                                                                      |
| `runtime.metrics.bind-jvm-metrics`           | boolean                                       | `true`              | Bind JVM metrics (memory, GC, threads).                                                                                                                                                                                                                  |
| `runtime.metrics.naming`                     | enum `stoatflow` \| `kafka-streams` \| `both` | `stoatflow`         | Metric-name families on `/metrics`. `both` = native + KS-named (migration posture); `kafka-streams` hides the mapped native families. See [Reusing your Kafka Streams dashboards](https://stoatflow.io/docs/migration/reusing-kafka-streams-dashboards). |
| `runtime.metrics.ks-compat.shape`            | enum `micrometer-binder` \| `jmx-exporter`    | `micrometer-binder` | Prometheus shape for the KS-named series. Read only when `naming` ≠ `stoatflow`.                                                                                                                                                                         |
| `runtime.metrics.ks-compat.sample-window-ms` | integer                                       | `30000`             | Trailing window for KS `-rate`/`-avg`/`-max` twins (min `5000`).                                                                                                                                                                                         |
| `runtime.metrics.ks-compat.kafka-version`    | string                                        | `4.1.0`             | Synthetic `kafka_version` tag value (micrometer-binder shape).                                                                                                                                                                                           |
| `runtime.metrics.ks-compat.thread-id`        | string                                        | *(auto)*            | Override for the synthetic KS thread-id (default `<application-id>-StreamThread-1`).                                                                                                                                                                     |
| `runtime.metrics.ks-compat.client-id`        | string                                        | *(auto)*            | Override for the synthetic KS client-id.                                                                                                                                                                                                                 |

## runtime.endpoints

Visibility of the `/info` and `/config` endpoints. See [REST API](https://stoatflow.io/docs/runtime/rest-api).

| Key                                      | Type    | Default | Description                                                                        |
| ---------------------------------------- | ------- | ------- | ---------------------------------------------------------------------------------- |
| `runtime.endpoints.config.enabled`       | boolean | `true`  | Enable the `/config` endpoint (merged config with sensitive values masked).        |
| `runtime.endpoints.info.show-app`        | boolean | `true`  | Include application info (applicationId) in `/info`.                               |
| `runtime.endpoints.info.show-java`       | boolean | `true`  | Include Java runtime info in `/info`.                                              |
| `runtime.endpoints.info.show-kafka`      | boolean | `true`  | Include Kafka config (bootstrap servers) in `/info`.                               |
| `runtime.endpoints.info.show-uptime`     | boolean | `true`  | Include uptime in `/info`.                                                         |
| `runtime.endpoints.info.show-state`      | boolean | `true`  | Include state + transition history in `/info`.                                     |
| `runtime.endpoints.info.show-watermarks` | boolean | `true`  | Include watermark info in `/info`.                                                 |
| `runtime.endpoints.info.show-endpoints`  | boolean | `true`  | Include the list of available endpoints in `/info`.                                |
| `runtime.endpoints.info.show-offsets`    | boolean | `true`  | Include the consumer lag summary (totalLag, per-topic lags) in `/info`.            |
| `runtime.endpoints.info.show-consumer`   | boolean | `true`  | Include the consumer group summary (groupId, subscription, partitions) in `/info`. |

## runtime.health

Health-check timeouts. See [Health checks](https://stoatflow.io/docs/runtime/health-checks).

| Key                                         | Type    | Default | Description                                                                             |
| ------------------------------------------- | ------- | ------- | --------------------------------------------------------------------------------------- |
| `runtime.health.kafka-broker.timeout-ms`    | integer | `2000`  | Timeout for the Kafka broker connectivity check.                                        |
| `runtime.health.schema-registry.timeout-ms` | integer | `5000`  | Timeout for the Schema Registry check (auto-enabled when `schema-registry-url` is set). |

## logging

Per-package log levels applied at startup (overrides `logback.xml`).

| Key             | Type               | Default | Description                                                                                               |
| --------------- | ------------------ | ------- | --------------------------------------------------------------------------------------------------------- |
| `logging.level` | map\<string, enum> | `{}`    | Map of logger name → level (`TRACE`/`DEBUG`/`INFO`/`WARN`/`ERROR`/`OFF`). Use `ROOT` for the root logger. |

```yaml
logging:
  level:
    ROOT: INFO
    io.stoatflow: DEBUG
    org.apache.kafka: WARN
```

## Example

A representative `application.yaml` touching the common keys:

```yaml
stoatflow:
  application-id: order-processor
  bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVERS:-localhost:9092}
  processing-guarantee: EXACTLY_ONCE

  license:
    key: ${STOATFLOW_LICENSE_KEY}
    environment: ${DEPLOY_ENV:-dev}

  lanes:
    count: 16

  state:
    dir: /var/lib/stoatflow/state

  rocks-db:
    preset: DEFAULT

runtime:
  http:
    enabled: true
    port: ${HTTP_PORT:-8080}
  metrics:
    enabled: true
    common-tags:
      env: ${DEPLOY_ENV:-dev}
```

## Related

::path-cards
  :::path-card
  ---
  icon: i-lucide-layers
  title: Configuration model
  to: https://stoatflow.io/docs/concepts/configuration-model
  ---
  Override layers, env-var naming, and YAML-to-engine mapping.
  :::

  :::path-card
  ---
  icon: i-lucide-sliders-horizontal
  title: Kafka client config
  to: https://stoatflow.io/docs/configuration/kafka-client-config
  ---
  Consumer/producer passthrough, framework defaults, forced overrides.
  :::

  :::path-card
  ---
  icon: i-lucide-package
  title: Defaults and presets
  to: https://stoatflow.io/docs/configuration/defaults-and-presets
  ---
  What StoatFlow sets out of the box and the RocksDB presets.
  :::

  :::path-card
  ---
  icon: i-lucide-gauge
  title: Tuning
  to: https://stoatflow.io/docs/operating/tuning
  ---
  Which knobs to turn for throughput, latency, and memory.
  :::
::


# REST API reference

The `:runtime` module starts a built-in HTTP server that exposes operational, introspection, and admin endpoints. This page enumerates every endpoint: its path, method, purpose, the high-level shape of the response, and any configuration that gates it. For how-to guidance see [Runtime → REST API](https://stoatflow.io/docs/runtime/rest-api), [Health checks](https://stoatflow.io/docs/runtime/health-checks), and [Metrics](https://stoatflow.io/docs/runtime/metrics).

## Server basics

The HTTP server is configured under `runtime.http.*` (see the [Configuration reference](https://stoatflow.io/docs/reference/configuration-reference)).

| Aspect            | Value                                                                                                          |
| ----------------- | -------------------------------------------------------------------------------------------------------------- |
| Default bind      | `0.0.0.0:8080`                                                                                                 |
| Enabled by        | `runtime.http.enabled` (default `true`); the entire server — and all endpoints below — is absent when disabled |
| JSON content type | `application/json; charset=utf-8`                                                                              |
| Text content type | `text/plain; charset=utf-8`                                                                                    |
| Error body        | `{"error":"<message>"}` (JSON) for non-2xx responses                                                           |
| Wrong method      | `405 Method Not Allowed` with an `Allow` header listing the accepted method(s)                                 |

Each endpoint accepts exactly one HTTP method. A request with any other method returns `405`.

## Endpoint summary

| Path                 | Method | Purpose                                     | Gating                             |
| -------------------- | ------ | ------------------------------------------- | ---------------------------------- |
| `/health/live`       | GET    | Liveness probe                              | always                             |
| `/health/ready`      | GET    | Readiness probe                             | always                             |
| `/metrics`           | GET    | Prometheus scrape                           | `runtime.metrics.enabled`          |
| `/info`              | GET    | Application metadata + available endpoints  | always                             |
| `/license`           | GET    | Runtime license state                       | always (in-cluster exposure only)  |
| `/config`            | GET    | Merged config, sensitive values masked      | `runtime.endpoints.config.enabled` |
| `/state`             | GET    | Application state + transition history      | always                             |
| `/topology`          | GET    | StoatFlow-native topology                   | always                             |
| `/topology/ks`       | GET    | Kafka Streams-compatible topology           | always                             |
| `/topology/compiled` | GET    | Compiled (internal) topology                | always                             |
| `/watermarks`        | GET    | Global + per-partition watermarks           | always                             |
| `/offsets`           | GET    | Source + changelog offsets and lag          | always                             |
| `/consumer`          | GET    | Consumer group + partition buffer state     | always                             |
| `/pause`             | POST   | Pause processing                            | always                             |
| `/unpause`           | POST   | Resume processing                           | always                             |
| `/ha/status`         | GET    | Hot-standby cluster view                    | `ha.mode != off`                   |
| `/ha/switch`         | POST   | Swap active/standby roles                   | `ha.mode != off`                   |
| `/ha/promote`        | POST   | Promote a standby (`?pod=<id>`)             | `ha.mode != off`                   |
| `/ha/demote`         | POST   | Demote an active (`?pod=<id>`)              | `ha.mode != off`                   |
| `/debug/threads`     | GET    | Engine thread snapshot (freeze diagnostics) | `runtime.http.debug.enabled`       |
| `/debug/barriers`    | GET    | Commit-pipeline state (freeze diagnostics)  | `runtime.http.debug.enabled`       |

## Health

### `GET /health/live`

Kubernetes liveness probe. Aggregates all liveness indicators. Returns `200` with `status: "UP"` when all are healthy, otherwise `503` with `status: "DOWN"`. Liveness stays `UP` during transient states (startup, shutdown) so the orchestrator does not restart the process unnecessarily.

```json
{
  "status": "UP",
  "components": [
    { "name": "stoatflow", "status": "UP", "details": { "state": "RUNNING" } }
  ]
}
```

### `GET /health/ready`

Kubernetes readiness probe. Aggregates all readiness indicators. Returns `200`/`UP` when ready to serve, otherwise `503`/`DOWN`. Readiness reports `DOWN` during startup, state restoration, and shutdown. The response body has the same shape as `/health/live`.

The set of indicators (StoatFlow app state, Kafka broker, license, Schema Registry when configured, plus any custom indicators) is documented under [Health checks](https://stoatflow.io/docs/runtime/health-checks).

## Metrics

### `GET /metrics`

Prometheus scrape endpoint. Returns metrics in Prometheus text exposition format with content type `text/plain; version=0.0.4; charset=utf-8`. Returns `503` if metrics are not enabled.

```text
# HELP jvm_memory_used_bytes The amount of used memory
# TYPE jvm_memory_used_bytes gauge
jvm_memory_used_bytes{application="my-app",area="heap",id="G1 Eden Space"} 1.048576E7
```

Gated by `runtime.metrics.enabled`; when disabled the handler is not registered. The exported meters are catalogued under [Metrics](https://stoatflow.io/docs/runtime/metrics).

## Introspection

### `GET /info`

Application metadata: application id, JVM information, Kafka configuration, uptime, current state, and the list of registered endpoints. Optional sections (watermark, offsets, consumer, topology traits) are included when available and enabled. Always returns `200` once the server is up.

```json
{
  "app": { "applicationId": "my-app" },
  "java": { "version": "25", "vendor": "...", "runtime": "..." },
  "kafka": { "bootstrapServers": "localhost:9092", "applicationId": "my-app" },
  "uptime": { "startTime": "2026-01-19T10:00:00Z", "uptime": "PT1H30M" },
  "state": { "current": "RUNNING", "since": "2026-01-19T10:00:05Z" },
  "endpoints": ["/health/live", "/health/ready", "/info", "/metrics", "/topology"]
}
```

### `GET /license`

Current runtime license-validation state.

::callout{color="warning" icon="i-lucide-shield"}
**In-cluster exposure only.** The response contains customer-identifying fields (`licenseId`, `customerName`, machine counts). Expose only `/health/live` and `/health/ready` via internet ingress; keep `/license`, `/info`, and `/topology` on the in-cluster Service.
::

```json
{
  "status": "VALID",
  "licenseId": "L-abc12-def34",
  "tier": "production",
  "customerName": "Acme Corp - Production",
  "expiresAt": "2027-05-26T00:00:00Z",
  "daysRemaining": 365,
  "machinesUsed": 3,
  "machinesAllowed": 10,
  "lastValidatedAt": "2026-05-27T10:30:00Z",
  "cachedJwtExpiresAt": "2026-05-28T10:30:00Z",
  "heartbeatLastSuccessAt": "2026-05-27T10:30:00Z",
  "consecutiveFailures": 0,
  "graceRemainingSeconds": 604800,
  "modeOnline": true
}
```

When no license key is present, the snapshot reports a `NO_KEY` status. In the first \~5 minutes after start — before the deferred activating validation has run — it reports `PENDING` (with epoch-zero timestamps and zero counts). See [License configuration](https://stoatflow.io/docs/getting-started/license-configuration).

### `GET /config`

The merged runtime configuration with sensitive values (passwords, secrets, tokens) masked. Supports content negotiation:

| `Accept` header                                       | Response                                 |
| ----------------------------------------------------- | ---------------------------------------- |
| `application/json`                                    | JSON (`application/json; charset=utf-8`) |
| `text/yaml`, `application/yaml`, `application/x-yaml` | YAML (`text/yaml; charset=utf-8`)        |
| `text/html` (browsers) or unset                       | YAML (default — more readable)           |

```yaml
stoatflow:
  application-id: my-app
  bootstrap-servers: localhost:9092
  kafka:
    consumer:
      sasl.password: "******"
runtime:
  http:
    port: 8080
```

Gated by `runtime.endpoints.config.enabled`; when disabled the handler is not registered.

### `GET /state`

Current application state, when it was entered, and the full state-transition history (newest first; history size is bounded by `stoatflow.processing.state-transition-history-size`, default `20`). Returns `503` if the engine is not yet initialized.

```json
{
  "current": "RUNNING",
  "since": "2026-01-19T10:00:05+01:00",
  "history": [
    { "state": "RUNNING", "from": "STARTING", "at": "2026-01-19T10:00:05+01:00" },
    { "state": "STARTING", "from": "CREATED", "at": "2026-01-19T10:00:00+01:00" }
  ]
}
```

## Topology

All three topology endpoints support content negotiation: `Accept: application/json` returns JSON; otherwise a human-readable text description. Each returns `503` if the engine is not yet initialized.

### `GET /topology`

The StoatFlow-native topology with richer metadata than the Kafka Streams view — scheduled sources with their scheduling configuration, processor state-store names, and the full sub-topology structure.

```json
{ "scheduledSources": [], "subtopologies": [] }
```

### `GET /topology/ks`

The same topology rendered in a format compatible with Kafka Streams' `Topology#describe()` output, for tools that expect the standard Kafka Streams shape.

```text
Topologies:
   Sub-topology: 0
    Source: source1 (topics: [input-topic])
      --> processor1
    Processor: processor1 (stores: [])
      --> sink1
      <-- source1
```

### `GET /topology/compiled`

The compiled topology — the internal representation produced by the topology compiler, including per-source processor DAGs, sub-topology boundaries, timer-aware processors, and scheduled sources. Useful for verifying how the DSL was compiled into execution units — including where the [sub-topology boundaries](https://stoatflow.io/docs/concepts/lanes-and-parallelism#where-the-handoff-actually-happens) landed under the configured `topology.sub-topology-split` mode. Returns `503` until the application has started (the compiled form is only available after start). The exact JSON structure is an internal representation and may change between releases; treat it as a diagnostic view, not a stable contract.

## Runtime state

### `GET /watermarks`

Global and per-partition watermark state: each partition reports its watermark (epoch ms + ISO 8601), idle flag, and last event time. Returns `503` if the engine is not initialized or no partitions are registered yet.

```json
{
  "globalMs": 1706889600000,
  "globalIso": "2026-02-02T12:00:00+01:00",
  "partitions": {
    "input-topic:0": {
      "watermarkMs": 1706889600000,
      "watermarkIso": "2026-02-02T12:00:00+01:00",
      "idle": false,
      "lastEventTimeMs": 1706889650000,
      "lastEventTimeIso": "2026-02-02T12:00:50+01:00"
    }
  }
}
```

### `GET /offsets`

Per-partition committed offsets and lag for source topics, plus committed offsets for state-store changelog topics. All values are read from locally cached state (no Kafka RPCs). Returns `503` if the engine is not initialized.

```json
{
  "totalLag": 79,
  "sources": [
    {
      "topic": "orders",
      "totalLag": 79,
      "partitions": [
        { "partition": 0, "committedOffset": 12345, "logEndOffset": 12400, "lag": 55 }
      ]
    }
  ],
  "changelogs": [
    {
      "topic": "my-app-order-counts-changelog",
      "store": "order-counts",
      "partitions": [ { "partition": 0, "committedOffset": 5432 } ]
    }
  ]
}
```

### `GET /consumer`

Consumer group metadata, source subscription, last commit time, and per-partition buffer utilization with pause state. Returns `503` if the engine is not initialized or the consumer has not yet been assigned partitions.

```json
{
  "group": {
    "groupId": "my-app",
    "memberId": "consumer-my-app-1-abc123",
    "generationId": 3,
    "groupInstanceId": null
  },
  "subscription": ["orders", "payments"],
  "lastCommitMs": 1706889700000,
  "lastCommitIso": "2026-02-02T12:01:40+01:00",
  "assignment": {
    "orders:0": { "buffered": 42, "bufferCapacity": 10000, "paused": false },
    "payments:0": { "buffered": 8500, "bufferCapacity": 10000, "paused": true }
  }
}
```

## Admin

### `POST /pause`

Pauses processing: transitions the application toward `PAUSED`, draining in-flight work first. Returns `503` if the engine is not initialized, or `400` if the current state cannot be paused (the error body explains why).

```json
{ "status": "pausing", "state": "DRAINING" }
```

### `POST /unpause`

Resumes processing: transitions from `PAUSED` / `DRAINING` back to `RUNNING`. Returns `503` if the engine is not initialized, or `400` if the current state cannot be unpaused.

```json
{ "status": "resumed", "state": "RUNNING" }
```

See [Pause / unpause](https://stoatflow.io/docs/runtime/pause-unpause) for the operational guide.

## High availability

These endpoints are registered only when `ha.mode != off`. They return `404` when HA is disabled and `503` before the engine is initialized. See [High availability](https://stoatflow.io/docs/operating/high-availability) for the operational guide.

### `GET /ha/status`

This pod's view of the hot-standby cluster: its own role, replication lag, coordinator-tail freshness, ready-standby count vs `ha.desired-standbys`, promotion-token epoch/holder, restart-required reason (or `null`), whether it is restarting its engine in place, and the peers it observes.

```json
{
  "selfPodId": "my-stream-app-0",
  "selfRole": "ACTIVE",
  "selfState": "ACTIVE",
  "selfRestarting": false,
  "replicationLagRecords": 0,
  "replicationLagMs": 0,
  "tailFreshnessMs": 120,
  "tailCaughtUp": true,
  "readyStandbyCount": 1,
  "desiredStandbys": 1,
  "redundancyBelowDesired": false,
  "tokenEpoch": 7,
  "tokenHolder": "my-stream-app-0",
  "restartRequired": null,
  "peers": [
    { "podId": "my-stream-app-1", "role": "STANDBY", "state": "READY_STANDBY",
      "replicationLagRecords": 1240, "replicationLagMs": 85, "generation": 7,
      "freshnessMs": 420, "failoverPriority": 0, "restarting": false }
  ]
}
```

`selfRestarting` (and a peer's `restarting`) is `true` for the duration of an [in-place engine restart](https://stoatflow.io/docs/operating/high-availability#application-faults-under-hot-standby) — the pod is absorbing a processing fault by rebuilding its engine. It stays `ACTIVE` throughout, so `selfRole` does not move; this field is what distinguishes "absorbing a fault" from "healthy", and it is the field to poll after a `409` from a command endpoint. The same signal is exported as the `stoatflow.ha.restarting` gauge.

::callout{color="info" icon="i-lucide-info"}
On a **mixed-version** cluster — mid rolling upgrade — a `role` or `state` published by a newer build that this pod does not recognise is reported as `UNKNOWN`. The peer still counts as alive, but takes part in no promotion decision on this pod. Seeing `UNKNOWN` means finish the upgrade, not that anything is wrong.
::

### `POST /ha/switch`

Swap roles — the active drains and the peer promotes. Publishes a command and returns `202 Accepted` with the command's offset as an idempotency token.

```json
{ "command": "SWITCH", "target": null, "force": false, "commandOffset": 42 }
```

All three command endpoints share **two independent `409 Conflict` causes**, and only the first is overridable:

**1. Readiness gate — overridable.** Rejected unless a caught-up target exists (a node in `READY_STANDBY`) — handing off to a not-ready peer is a self-inflicted outage. Override with `?force=true`; the override travels with the command and is re-checked on the acting pod, but a forced promotion still fully restores state before processing.

**2. In-place restart — NOT overridable.** Rejected while the pod the command would move is restarting its engine in place. A role change cannot be serialised against an engine swap, so the command would land as a silent no-op behind a success response. `?force=true` does not override this: force exists to overrule the *readiness judgement*, not to make a structurally impossible operation succeed. The restart is bounded — it completes in seconds, or exhausts `commit-barrier.max-engine-restarts` and hands off on its own — so poll `GET /ha/status` until `selfRestarting` (or the peer's `restarting`) is `false`, then re-issue.

Because every command moves the **active**, the restart check covers the active for all three verbs, plus the named `?pod=` target for `promote`/`demote`.

### `POST /ha/promote`

Ask a specific standby pod to promote. Requires a `?pod=<id>` target (`400` if missing); `409` unless that pod is `READY_STANDBY` (override `?force=true`), or if either it or the current active is restarting in place (not overridable). Returns `202` with the command offset.

### `POST /ha/demote`

Ask a specific active pod to demote. Requires a `?pod=<id>` target (`400` if missing); `409` unless some node is `READY_STANDBY` to take over (override `?force=true`), or if the target is restarting in place (not overridable). Returns `202` with the command offset.

## Debug

The two `/debug/*` endpoints are diagnostic aids for investigating a stalled commit pipeline. They are gated by `runtime.http.debug.enabled` (default `true`); when disabled, neither handler is registered and both paths return `404`. They are free at rest and invaluable during a freeze; disable only for hardened deployments that minimise attack surface. When enabled, both return `503` until the engine is initialized.

### `GET /debug/threads`

A JSON snapshot of the engine's threads (and the JVM's platform threads), with each thread's state and a truncated stack trace — for answering "what is the engine stuck on?" during a freeze.

Optional query parameters:

| Param    | Default | Meaning                                                                              |
| -------- | ------- | ------------------------------------------------------------------------------------ |
| `depth`  | `40`    | Maximum stack frames per thread                                                      |
| `filter` | `all`   | `all`, `stuck` (only threads not `RUNNABLE`), or `known` (only engine-owned threads) |

### `GET /debug/barriers`

A JSON snapshot of commit-pipeline state, designed to answer "which commit is stuck, at which phase, and what is blocking it?" from a single call. The response includes a derived `phase` field summarising the pipeline state at a glance (for example `IN_TX_COMMIT`, `AWAITING_LANE_ACKS`, or `IDLE`).

## Related

- [Runtime → REST API](https://stoatflow.io/docs/runtime/rest-api) — narrative guide to the endpoints with curl examples.
- [Health checks](https://stoatflow.io/docs/runtime/health-checks) — the health-indicator system behind `/health/*`.
- [Metrics](https://stoatflow.io/docs/runtime/metrics) — the meters exposed at `/metrics`.
- [Configuration reference](https://stoatflow.io/docs/reference/configuration-reference) — the `runtime.http.*`, `runtime.metrics.*`, and `runtime.endpoints.*` keys.
- [Kubernetes](https://stoatflow.io/docs/operating/kubernetes) and [Probes](https://stoatflow.io/docs/operating/probes) — wiring the probes and choosing which endpoints to expose.


# Gradle plugin reference

The `io.stoatflow` convention plugin encapsulates StoatFlow's build conventions: the JDK 25 toolchain, the required preview + native-access JVM flags, a runnable fat JAR, and opt-in Docker (Jib) and GraalVM native-image builds. Configure it through the typed `stoatflow { }` extension (Kotlin DSL). For the Maven equivalent — the same conventions via a parent POM — see the [Maven reference](https://stoatflow.io/docs/reference/maven-reference).

Apply it after registering the plugin-marker repository — see [Project setup](https://stoatflow.io/docs/getting-started/project-setup) and [Installation](https://stoatflow.io/docs/getting-started/installation#_3-configure-the-jvm-toolchain).

```kotlin
// build.gradle.kts
plugins {
    kotlin("jvm") version "2.4.0"   // omit for a Java-only project
    id("io.stoatflow") version "<stoatflow-version>"
}

stoatflow {
    mainClass.set("com.example.MainKt")
}
```

The current release is :stoatflow-version — substitute it for the `<stoatflow-version>` placeholder.

## Extension DSL

The plugin registers a `stoatflow { }` extension. The full surface, with every default:

```kotlin
stoatflow {
    mainClass.set("com.example.MainKt")             // default: unset

    docker {
        enabled.set(true)                           // default: false
        imageName.set("my-app")                     // default: project.name
        port.set(8080)                              // default: 8080
        user.set("1000:1000")                       // default: 1000:1000
        baseImage.set("eclipse-temurin:25-jre-noble") // default shown (JVM/Jib only)
        stateDir.set("/data/state")                 // default: /data/state
        rocksdbMb.set(256)                          // default: auto-detect (see below)
        reproducibleBuild.set(true)                 // default: true
        jvmFlags.set(listOf("-XX:+UseG1GC"))        // overridable; preview/native-access always added
    }

    nativeImage {
        enabled.set(true)                           // default: false
        gc.set("G1")                                // default: absent (Serial GC)
        runtimeTarget.set("runtime")                // default: runtime; or "runtime-lean"
        additionalInitializeAtRunTime.add("com.example.MyClass")
        removeInitializeAtRunTime.add("org.rocksdb.RocksDB")
    }
}
```

### `stoatflow { }`

| Property          | Type               | Default   | Description                                                                                                                                                                                                                                         |
| ----------------- | ------------------ | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `mainClass`       | `Property<String>` | *(unset)* | Application entry point. Wired as a lazy provider into the `application` plugin's `mainClass`, the shadow-jar `Main-Class` manifest attribute, and (when enabled) the Docker and native-image main class. Required for a runnable JAR or any image. |
| `docker { }`      | nested             | —         | Opt-in Jib Docker image build. See below.                                                                                                                                                                                                           |
| `nativeImage { }` | nested             | —         | Opt-in GraalVM native-image build. See below.                                                                                                                                                                                                       |

### `docker { }`

| Property            | Type                   | Default                        | Description                                                                                                                                                                                           |
| ------------------- | ---------------------- | ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `enabled`           | `Property<Boolean>`    | `false`                        | When `true`, applies the Jib plugin and the StoatFlow image conventions.                                                                                                                              |
| `imageName`         | `Property<String>`     | `project.name`                 | Target image name. Tagged `latest` and `<project.version>`.                                                                                                                                           |
| `port`              | `Property<Int>`        | `8080`                         | Exposed container port (the runtime's HTTP admin / metrics server).                                                                                                                                   |
| `user`              | `Property<String>`     | `1000:1000`                    | Non-root run-as user (`uid:gid`).                                                                                                                                                                     |
| `baseImage`         | `Property<String>`     | `eclipse-temurin:25-jre-noble` | JVM base image (JVM/Jib path only; the native runtime base stays distroless).                                                                                                                         |
| `stateDir`          | `Property<String>`     | `/data/state`                  | RocksDB state directory; set as `STOATFLOW_STATE_DIR`. Mount a volume here.                                                                                                                           |
| `rocksdbMb`         | `Property<Int>`        | *auto-detect*                  | Off-heap RocksDB reservation (MiB) → `STOATFLOW_ROCKSDB_MB`. Unset = **256 if RocksDB is on the runtime classpath, else the env var is omitted**; `0` always omits it; an explicit value always wins. |
| `reproducibleBuild` | `Property<Boolean>`    | `true`                         | Pins the image `creationTime` to the HEAD commit timestamp. `false` = wall-clock (`USE_CURRENT_TIMESTAMP`).                                                                                           |
| `jvmFlags`          | `ListProperty<String>` | `["-XX:+UseG1GC"]`             | Container JVM flags. **Overridable** — but `--enable-preview` and `--enable-native-access=ALL-UNNAMED` are always added on top.                                                                       |

### `nativeImage { }`

| Property                        | Type                   | Default   | Description                                                                                                                                                                                                                |
| ------------------------------- | ---------------------- | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `enabled`                       | `Property<Boolean>`    | `false`   | When `true`, applies the GraalVM native-build plugin and registers the `nativeDockerBuild` task.                                                                                                                           |
| `gc`                            | `Property<String>`     | *absent*  | GC algorithm, passed as `--gc=<value>`. Absent = the native image default (Serial GC). Set `"G1"` for stateful (RocksDB) workloads — &#x2A;*G1 needs Oracle GraalVM.**                                                     |
| `runtimeTarget`                 | `Property<String>`     | `runtime` | Docker target stage for `nativeDockerBuild` (`--target`): `runtime` (busybox distroless + container-aware heap entrypoint) or `runtime-lean` (minimal distroless, no heap entrypoint — SubstrateVM sizes the heap itself). |
| `additionalInitializeAtRunTime` | `ListProperty<String>` | `[]`      | Extra fully-qualified class names **added** to the curated `--initialize-at-run-time` base.                                                                                                                                |
| `removeInitializeAtRunTime`     | `ListProperty<String>` | `[]`      | Curated `--initialize-at-run-time` entries to **drop** — lets a consumer remove a problematic entry without forking the plugin.                                                                                            |

## What the plugin applies

Applied unconditionally on every project the plugin is added to:

| Concern                      | Effect                                                                                                                                                                       |
| ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `application` plugin         | Applied. `mainClass` wired from the extension; `applicationDefaultJvmArgs` set to the preview JVM args (below).                                                              |
| `com.gradleup.shadow` plugin | Applied. Configures `ShadowJar` with `archiveClassifier = "all"`, `mergeServiceFiles()`, and a `Main-Class` manifest attribute (from `mainClass`).                           |
| Java toolchain               | `JavaLanguageVersion.of(25)`.                                                                                                                                                |
| Kotlin toolchain             | `jvmToolchain(25)` — applied (reflectively) **only** when the `org.jetbrains.kotlin.jvm` plugin is present. The Kotlin Gradle plugin is **not** bundled; you bring your own. |
| `JavaCompile` tasks          | `--enable-preview` added to compiler args on every `JavaCompile` task.                                                                                                       |
| `test` task                  | JUnit Platform with the `IntegrationTest` tag excluded; the test JVM args (below) applied.                                                                                   |
| `integrationTest` task       | Registered (when the `java` plugin is present); runs only `@Tag("IntegrationTest")` tests, after `test`.                                                                     |

### JVM args

| Context                                                            | Flags                                                                                                         |
| ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------- |
| Application run (`applicationDefaultJvmArgs`) and Docker container | `--enable-preview`, `--enable-native-access=ALL-UNNAMED`, plus `docker { jvmFlags }` (default `-XX:+UseG1GC`) |
| Test tasks (`test`, `integrationTest`)                             | `--enable-preview`, `--enable-native-access=ALL-UNNAMED`, `-Dnet.bytebuddy.experimental=true`                 |

`--enable-preview` is required because StoatFlow builds on JDK preview features; `--enable-native-access=ALL-UNNAMED` is required because RocksDB's state store uses the Foreign Function & Memory API. See [Installation](https://stoatflow.io/docs/getting-started/installation#_3-configure-the-jvm-toolchain) for the rationale behind `-XX:+UseG1GC`.

## Tasks the plugin contributes

| Task                                   | Group        | Available when                              | Purpose                                                                                                                                                                                                                   |
| -------------------------------------- | ------------ | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `run`                                  | application  | always (from `application`)                 | Runs the app with the preview JVM args.                                                                                                                                                                                   |
| `shadowJar`                            | shadow       | always (from `shadow`)                      | Builds the runnable fat JAR (`-all` classifier, merged service files, `Main-Class` manifest).                                                                                                                             |
| `integrationTest`                      | verification | the `java` plugin is present                | Runs JUnit Platform tests tagged `IntegrationTest` (Docker / Testcontainers). Runs after `test`.                                                                                                                          |
| `copyStoatFlowEntrypoint`              | *(none)*     | `docker.enabled = true`                     | Stages the container-aware heap entrypoint into the Jib extra directory. All `jib*` tasks depend on it.                                                                                                                   |
| `jib`, `jibDockerBuild`, `jibBuildTar` | —            | `docker.enabled = true` (from Jib)          | Build the image. `jibDockerBuild` targets the local Docker daemon.                                                                                                                                                        |
| `nativeDockerBuild`                    | build        | `nativeImage.enabled = true`                | Builds a native image **inside Docker** from the prebuilt fat JAR, against the bundled generic `Dockerfile.native` (staged into `build/stoatflow-native/`). Tags `<imageName>:native` and `<imageName>:<version>-native`. |
| `nativeCompile`, `nativeRun`           | —            | `nativeImage.enabled = true` (from GraalVM) | Build / run the native image directly on the host GraalVM toolchain.                                                                                                                                                      |

::callout{color="info" icon="i-lucide-info"}
`docker.enabled` and `nativeImage.enabled` are read in a single `afterEvaluate` (the only deferral the plugin needs), because they're set in the build-script body — the Jib and GraalVM plugins are applied only when the corresponding flag is `true`. `mainClass` and the ShadowJar `Main-Class` are wired as lazy providers, not in `afterEvaluate`.
::

## Docker image defaults

When `docker.enabled = true`, the Jib build is configured with:

| Aspect                      | Value                                                                                            |
| --------------------------- | ------------------------------------------------------------------------------------------------ |
| Base image                  | `baseImage` (default `eclipse-temurin:25-jre-noble`)                                             |
| Platform                    | `linux/arm64` on `aarch64` hosts, otherwise `linux/amd64`                                        |
| Target image                | `imageName` (or `project.name`), tagged `latest` and `project.version`                           |
| `creationTime`              | HEAD commit timestamp when `reproducibleBuild = true` (default), else wall-clock                 |
| Entrypoint                  | `/app/stoatflow-entrypoint.sh` (bundled container-aware heap script)                             |
| Working directory           | `/app`                                                                                           |
| Exposed port                | `port` (default `8080`)                                                                          |
| User                        | `user` (default `1000:1000`)                                                                     |
| JVM flags                   | `--enable-preview`, `--enable-native-access=ALL-UNNAMED`, + `docker { jvmFlags }`                |
| Env: `STOATFLOW_STATE_DIR`  | `stateDir` (default `/data/state`)                                                               |
| Env: `STOATFLOW_ROCKSDB_MB` | `rocksdbMb`, or `256` when RocksDB is auto-detected on the runtime classpath (omitted otherwise) |

::callout{color="warning" icon="i-lucide-triangle-alert"}
**Configuration cache.** The plugin is configuration-cache clean for the plain and native paths. `jibDockerBuild` is **not** CC-compatible — an upstream jib-gradle-plugin limitation, not StoatFlow's. Run docker builds without `--configuration-cache`.
::

See [Docker](https://stoatflow.io/docs/runtime/docker) for the full container guide.

## Native image defaults

When `nativeImage.enabled = true`, the GraalVM build is configured from the bundled `native-image.args` (the single source of truth, identical for Gradle, Maven, and the containerized build), plus the dynamic `gc` / add / remove deltas:

| Aspect                     | Value                                                                                                                                                                                                                                                                                    |
| -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Toolchain detection        | enabled                                                                                                                                                                                                                                                                                  |
| Static build args          | `-H:+JNI`, `-H:+ReportExceptionStackTraces`, `-H:+AddAllCharsets`, the Kotlin resource includes (`*.kotlin_builtins`, `META-INF/*.kotlin_module`), `--enable-preview`, `--no-fallback`                                                                                                   |
| `--initialize-at-run-time` | `org.rocksdb`, `org.rocksdb.RocksDB`, `org.rocksdb.util.Environment`, `io.stoatflow.core.state.rocksdb.backend.ffm.FfmRocksDbNative`, `org.apache.kafka.common.security.authenticator.SaslClientAuthenticator` — plus `additionalInitializeAtRunTime`, minus `removeInitializeAtRunTime` |
| `--gc`                     | added only when `nativeImage { gc }` is set                                                                                                                                                                                                                                              |
| `SOURCE_DATE_EPOCH`        | HEAD commit timestamp when `reproducibleBuild = true`                                                                                                                                                                                                                                    |
| Runtime stage              | `--target` from `runtimeTarget` (default `runtime`)                                                                                                                                                                                                                                      |

The `nativeDockerBuild` task tags the resulting image `<imageName>:native` and `<imageName>:<version>-native`. See [Native image](https://stoatflow.io/docs/runtime/native-image) for the full guide.

## Version compatibility

The plugin pins the build tools it bundles. As of :stoatflow-version the pins are:

| Tool                                             | Version                                                  |
| ------------------------------------------------ | -------------------------------------------------------- |
| Gradle                                           | 9.5+ (developed and tested on 9.6.1)                     |
| Shadow (`com.gradleup.shadow`)                   | 9.4.3                                                    |
| Jib (`com.google.cloud.tools.jib`)               | 3.5.2                                                    |
| GraalVM Native (`org.graalvm.buildtools.native`) | 1.1.3                                                    |
| Kotlin Gradle plugin                             | 2.4.0 (you bring your own; the plugin doesn't bundle it) |

## Related

- [Project setup](https://stoatflow.io/docs/getting-started/project-setup) — applying the conventions in a fresh project (Gradle or Maven).
- [Maven reference](https://stoatflow.io/docs/reference/maven-reference) — the same conventions for Maven projects.
- [Docker](https://stoatflow.io/docs/runtime/docker) — building and running the container image.
- [Native image](https://stoatflow.io/docs/runtime/native-image) — ahead-of-time compilation with GraalVM.


# Maven reference

StoatFlow gives Maven projects the same build conventions as the [Gradle plugin](https://stoatflow.io/docs/reference/gradle-plugin-reference), through three published artifacts under `io.stoatflow`. This page enumerates them; for the setup walkthrough see [Project setup](https://stoatflow.io/docs/getting-started/project-setup).

## The three artifacts

| Artifact                                  | What it is                                                                                                                                                                                                                                                    |
| ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **`io.stoatflow:stoatflow-bom`**          | A Maven BOM (`<dependencyManagement>` import) that aligns the StoatFlow modules with their tested kafka-clients / rocksdbjni / micrometer versions and imports the JUnit BOM.                                                                                 |
| **`io.stoatflow:stoatflow-parent`**       | The convention parent POM — JDK 25 + `--enable-preview`, kotlin-maven jvmTarget 25, the shade fat JAR, the surefire/failsafe `IntegrationTest` tag-split, `exec:exec` run, and the `stoatflow-docker` / `stoatflow-native` profiles. Imports the BOM for you. |
| **`io.stoatflow:stoatflow-maven-plugin`** | Four goals (prefix `stoatflow`) for the things XML can't express: RocksDB auto-detection, the staged native `docker build`, and the forked preview-flag run.                                                                                                  |

## Set up

Inherit from the parent and declare your dependency (its version comes from the BOM the parent imports — so you omit it):

```xml
<!-- pom.xml -->
<parent>
  <groupId>io.stoatflow</groupId>
  <artifactId>stoatflow-parent</artifactId>
  <version>REPLACE-WITH-CURRENT-RELEASE</version>   <!-- literal — Maven can't resolve a property in <parent> -->
  <relativePath/>
</parent>

<properties>
  <stoatflow.mainClass>com.example.Main</stoatflow.mainClass>
</properties>

<dependencies>
  <dependency>
    <groupId>io.stoatflow</groupId>
    <artifactId>stoatflow-runtime</artifactId>
    <!-- version managed by the imported stoatflow-bom -->
  </dependency>
  <dependency>
    <groupId>io.stoatflow</groupId>
    <artifactId>stoatflow-test-utils</artifactId>
    <scope>test</scope>
  </dependency>
</dependencies>
```

The current release is :stoatflow-version — substitute it for the `REPLACE-WITH-CURRENT-RELEASE` placeholder (the `<parent>` version must be a **literal**; Maven does not interpolate properties in parent coordinates). For a Kotlin project, point Maven at `src/main/kotlin`, add the `kotlin-stdlib` (version `${kotlin.version}`, from the parent), and use the `…Kt` main class — the parent already configures `kotlin-maven-plugin` (jvmTarget 25) and binds its `compile`/`test-compile` phases.

::callout{color="info" icon="i-lucide-terminal"}
**Short goal prefix (`mvn stoatflow:…`).** To call the plugin's standalone goals by their short prefix, add `io.stoatflow` to `<pluginGroups>` in your `~/.m2/settings.xml`:

```xml
<pluginGroups>
  <pluginGroup>io.stoatflow</pluginGroup>
</pluginGroups>
```

Without it, use full coordinates — e.g. `mvn io.stoatflow:stoatflow-maven-plugin:run`.
::

## What the parent configures

| Convention            | Detail                                                                                                                                           |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Compiler**          | `--release 25` + `--enable-preview` (javac).                                                                                                     |
| **Kotlin**            | `kotlin-maven-plugin` jvmTarget 25 (no preview flag — that's javac-only).                                                                        |
| **Fat JAR**           | `maven-shade-plugin` produces a `<finalName>-all.jar` with the `Main-Class` manifest + merged service files.                                     |
| **Unit tests**        | `maven-surefire-plugin` on JUnit Platform with the preview/native-access JVM args; **excludes** `@Tag("IntegrationTest")`.                       |
| **Integration tests** | `maven-failsafe-plugin` runs **only** `@Tag("IntegrationTest")`, bound to `integration-test` / `verify`.                                         |
| **Run**               | `exec-maven-plugin` configured for `exec:exec` — a **forked** JVM so `--enable-preview` applies (`exec:java` runs in Maven's own JVM and can't). |
| **Profiles**          | `stoatflow-docker` (Jib image at `package`), `stoatflow-native` (native compile at `package`), and host-arch selection.                          |

## Properties reference

Set these in your `<properties>`. Defaults come from the parent.

| Property                                          | Default                        | Description                                                                                                                                |
| ------------------------------------------------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `stoatflow.mainClass`                             | *(required)*                   | Application entry point. `stoatflow:configure` fails the build if unset.                                                                   |
| `stoatflow.docker.imageName`                      | `${project.artifactId}`        | Image name. Tagged `latest` + `<version>`.                                                                                                 |
| `stoatflow.docker.port`                           | `8080`                         | Exposed container port.                                                                                                                    |
| `stoatflow.docker.user`                           | `1000:1000`                    | Non-root run-as user.                                                                                                                      |
| `stoatflow.docker.baseImage`                      | `eclipse-temurin:25-jre-noble` | JVM base image.                                                                                                                            |
| `stoatflow.docker.stateDir`                       | `/data/state`                  | RocksDB state dir → `STOATFLOW_STATE_DIR`.                                                                                                 |
| `stoatflow.docker.rocksdbMb`                      | *empty = auto-detect*          | Off-heap RocksDB MiB → `STOATFLOW_ROCKSDB_MB`. Empty = **256 if RocksDB on the classpath, else omitted**; set a value to force; `0` omits. |
| `stoatflow.docker.reproducibleBuild`              | `true`                         | Pin `creationTime` to the HEAD commit.                                                                                                     |
| `stoatflow.docker.jvmFlags`                       | `-XX:+UseG1GC`                 | Container GC flag (preview/native-access always added).                                                                                    |
| `stoatflow.nativeImage.gc`                        | *empty*                        | `--gc` value for the native **docker** build (e.g. `G1`; needs Oracle GraalVM).                                                            |
| `stoatflow.nativeImage.runtimeTarget`             | `runtime`                      | Docker target stage for `native-docker-build` (`runtime` or `runtime-lean`).                                                               |
| `stoatflow.nativeImage.removeInitializeAtRunTime` | *empty*                        | Comma-separated `--initialize-at-run-time` entries to drop from the curated base.                                                          |

::callout{color="info" icon="i-lucide-info"}
**`docker.rocksdbMb`, `creationTime`, and the docker executable carry no `<properties>` default.** Maven's model interpolator bakes `<properties>` into the plugin `<configuration>` *before* any Mojo runs, which would override the auto-detection — so these stay undefined and resolve live from `stoatflow:configure`. To **add** native flags (the counterpart of Gradle's `additionalInitializeAtRunTime`), extend the `native-maven-plugin` `<buildArgs>` in your POM, or ship reachability metadata with your value classes (see [Native image → Custom serde metadata](https://stoatflow.io/docs/runtime/native-image#custom-serde-metadata)).
::

## Goals

| Goal                            | Bound to                           | Purpose                                                                                                                                                                                        |
| ------------------------------- | ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `stoatflow:configure`           | `initialize` (both profiles)       | Auto-detects RocksDB → `STOATFLOW_ROCKSDB_MB`, probes the docker executable (macOS), computes the reproducible `creationTime`, stages the effective native argfile, and validates `mainClass`. |
| `stoatflow:copy-entrypoint`     | `prepare-package` (docker profile) | Stages the container-aware heap entrypoint into `target/jib-extra/app/`.                                                                                                                       |
| `stoatflow:native-docker-build` | *standalone*                       | Stages the native build context (prebuilt JAR + argfile + `Dockerfile.native` + entrypoint) and runs the host-arch `docker build`.                                                             |
| `stoatflow:run`                 | *standalone*                       | Forks a JVM with the preview flags + your runtime classpath + `mainClass`.                                                                                                                     |

## Build commands

| Command                             | Result                                                                                      |
| ----------------------------------- | ------------------------------------------------------------------------------------------- |
| `mvn package`                       | Compiles + the fat JAR `target/<name>-<version>-all.jar`.                                   |
| `mvn -Pstoatflow-docker package`    | Fat JAR **and** a Jib image to your local Docker daemon — one command.                      |
| `mvn -Pstoatflow-native package`    | A GraalVM native binary via `native-maven-plugin` (host toolchain).                         |
| `mvn stoatflow:native-docker-build` | A native **image** built inside Docker from the prebuilt fat JAR (run `mvn package` first). |
| `mvn stoatflow:run`                 | The app in a forked JVM with `--enable-preview`.                                            |

::callout{color="warning" icon="i-lucide-triangle-alert"}
**Profiles are activated with `-P`, not POM `<properties>`.** Maven `<activation>` reads system/user properties, not your project's `<properties>` — so a `<stoatflow.docker.enabled>` flag does **not** turn the build on. Activate with `mvn -Pstoatflow-docker …` (or `-Dstoatflow.docker.enabled=true`). The `stoatflow-docker` profile binds the Jib `dockerBuild` goal to `package`, which is why `mvn -Pstoatflow-docker package` is the whole image build.
::

## Using just the BOM

If you don't want the parent POM (you already have a corporate parent, say), import the BOM directly for version alignment and configure the toolchain + flags yourself — the DIY path is in [Installation → Option B](https://stoatflow.io/docs/getting-started/installation#option-b-configure-it-yourself):

```xml
<properties>
  <stoatflow.version>REPLACE-WITH-CURRENT-RELEASE</stoatflow.version>
</properties>

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>io.stoatflow</groupId>
      <artifactId>stoatflow-bom</artifactId>
      <version>${stoatflow.version}</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>
```

## Version compatibility

The parent pins the build plugins it uses. As of :stoatflow-version the pins are:

| Plugin              | Version                  |
| ------------------- | ------------------------ |
| Maven               | 3.8+ (3.9.x recommended) |
| jib-maven-plugin    | 3.5.1                    |
| native-maven-plugin | 1.1.0                    |
| maven-shade-plugin  | 3.6.0                    |
| kotlin-maven-plugin | 2.4.0                    |
| surefire / failsafe | 3.5.2                    |

::callout{color="info" icon="i-lucide-info"}
jib-**maven**-plugin (3.5.1) trails the jib-**gradle**-plugin (3.5.2) — that's their independent release cadence, not a defect; both produce equivalent images.
::

## Related

- [Project setup](https://stoatflow.io/docs/getting-started/project-setup) — the setup walkthrough (Gradle and Maven).
- [Gradle plugin reference](https://stoatflow.io/docs/reference/gradle-plugin-reference) — the same conventions for Gradle projects.
- [Docker](https://stoatflow.io/docs/runtime/docker) · [Native image](https://stoatflow.io/docs/runtime/native-image) — the container and GraalVM build guides.


# Kafka Streams compatibility matrix

The engineering view of Kafka Streams DSL parity in StoatFlow: every operator, helper class, and store method, with its current status and notes where behaviour differs. Targets **Kafka Streams 4.3.x**: StoatFlow reimplements the Kafka Streams DSL surface under its own package, so most KS applications port by swapping the dependency and rewriting imports — see [Migrating from Kafka Streams](https://stoatflow.io/docs/migration), and let the [automated port](https://stoatflow.io/docs/migration/automated-port) recipe do it for you.

For the high-level, feature-by-feature comparison against Kafka Streams and Flink, see [Comparison matrix](https://stoatflow.io/product/comparison-matrix). For how-to guidance on the operators below, see [Building](https://stoatflow.io/docs/building).

::callout{color="info" icon="i-lucide-git-compare"}
This matrix is transcribed from the in-repository compatibility reference and curated for the public docs. Status reflects the current release ( :stoatflow-version ). A small number of rows differ in signature from Kafka Streams (marked ⚠️) or are intentionally absent (marked 🛑) — read the notes.
::

## Legend

| Symbol | Meaning                                                                                             |
| ------ | --------------------------------------------------------------------------------------------------- |
| ✅      | Implemented                                                                                         |
| ⚠️     | Partial — stub, limited, or differing signature                                                     |
| ❌      | Not implemented yet                                                                                 |
| 🛑     | Incompatible, or won't be implemented (not needed in the single-instance model)                     |
| ↘️     | Deprecated in Kafka Streams                                                                         |
| 🚀     | StoatFlow extension — not in Kafka Streams                                                          |
| 🆕     | StoatFlow extension on an otherwise KS-compatible interface                                         |
| 🟡     | Recognised no-op — accepted and warned about, never an error (configuration keys only)              |
| 🧩     | Client pass-through — forwarded to the Kafka clients the key is valid for (configuration keys only) |

## StreamsBuilder

| Method                                                          | Status | Notes                                                                                                                                                                                                                           |
| --------------------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| StreamsBuilder()                                                | ✅      | No-arg                                                                                                                                                                                                                          |
| StreamsBuilder(TopologyConfig)                                  | ✅      | KSC-77 — seeds build-time defaults (serdes, DSL store format) &#x2A;*and the KIP-1112 `ProcessorWrapper`** from a `TopologyConfig` projection of `StreamsConfig`. Named-topology / per-topology overrides N/A (single-instance) |
| stream(topic, consumed?)                                        | ✅      | Single topic subscription; single-arg form Java-callable                                                                                                                                                                        |
| stream(Collection\<String>)                                     | ✅      | Multiple topics subscription                                                                                                                                                                                                    |
| stream(Pattern)                                                 | ✅      | Pattern-based subscription                                                                                                                                                                                                      |
| table(topic, consumed?)                                         | ✅      |                                                                                                                                                                                                                                 |
| table(topic, materialized)                                      | ✅      | Explicit skip-Consumed overload (KS parity)                                                                                                                                                                                     |
| table(topic, consumed?, materialized)                           | ✅      | With state store                                                                                                                                                                                                                |
| globalTable(topic, consumed?)                                   | ✅      | Wraps KTable                                                                                                                                                                                                                    |
| globalTable(topic, materialized)                                | ✅      | Without Consumed parameter                                                                                                                                                                                                      |
| build()                                                         | ✅      | Returns Topology                                                                                                                                                                                                                |
| build(Properties)                                               | ⚠️     | The `Properties` are **ignored** — logs a warning and delegates to `build()`. Config carries over via `StreamsConfig.fromProperties(...)` / the KS-shaped `StreamsConfig(Properties)` ctor instead (ADR-124)                    |
| addStateStore(storeBuilder)                                     | ✅      | Register standalone store                                                                                                                                                                                                       |
| addStateStore(storeBuilder, keySerde, valueSerde)               | ✅      | With changelog Serdes                                                                                                                                                                                                           |
| addStateStore(storeSupplier, keySerde, valueSerde)              | ✅      | Convenience overload                                                                                                                                                                                                            |
| addGlobalStore(storeBuilder, topic, consumed, processor)        | ✅      | Custom update processor                                                                                                                                                                                                         |
| addGlobalStore(storeBuilder, topic, consumed)                   | ✅      | Default put/delete semantics                                                                                                                                                                                                    |
| addReadOnlyStateStore(storeBuilder, topic, consumed, processor) | ✅      | KIP-813: custom processor                                                                                                                                                                                                       |
| addReadOnlyStateStore(storeBuilder, topic, consumed)            | ✅      | KIP-813: default put/delete semantics                                                                                                                                                                                           |
| scheduled(interval, type, emitter)                              | 🚀     | Interval-based periodic record source                                                                                                                                                                                           |
| scheduled(interval, type, named, emitter)                       | 🚀     | With explicit naming                                                                                                                                                                                                            |
| scheduled(interval, type, keySerde, emitter)                    | 🚀     | With explicit key serde for lane assignment                                                                                                                                                                                     |
| scheduled(cron, emitter)                                        | 🚀     | Cron-based calendar scheduling                                                                                                                                                                                                  |
| scheduled(cron, named, emitter)                                 | 🚀     | Cron with explicit naming                                                                                                                                                                                                       |
| scheduled(cron, keySerde, emitter)                              | 🚀     | Cron with explicit key serde for lane assignment                                                                                                                                                                                |

## Topology (Processor API)

Both the high-level DSL (`StreamsBuilder`) and the low-level `Topology` builder are supported.

| Method                                   | Status | Notes                                                                                                                                                                                                                                              |
| ---------------------------------------- | ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| describe()                               | ✅      | Returns the Kafka Streams `TopologyDescription` (nested `Source`/`Processor`/`Sink`/`Subtopology`/`GlobalStore` node graph). &#x2A;*Breaking vs older StoatFlow:** previously returned the native model — that view moved to `describeStoatFlow()` |
| describeStoatFlow()                      | 🚀     | StoatFlow-native topology model (the former `describe()` shape)                                                                                                                                                                                    |
| addSource(name, topics...)               | ✅      | 14 overloads: offset reset, timestamp extractor, deserializers, pattern                                                                                                                                                                            |
| addProcessor(name, supplier, parents...) | ✅      | User-defined processor with ProcessorSupplier                                                                                                                                                                                                      |
| addSink(name, topic, parents...)         | ✅      | 8 overloads: serializers, partitioner, TopicNameExtractor                                                                                                                                                                                          |
| addStateStore(builder, processors...)    | ✅      | Register store and optionally connect to processors                                                                                                                                                                                                |
| addGlobalStore(...)                      | ✅      | 2 variants: with/without TimestampExtractor                                                                                                                                                                                                        |
| addReadOnlyStateStore(...)               | ✅      | 2 variants (KIP-813): source topic is changelog                                                                                                                                                                                                    |
| connectProcessorAndStateStores(...)      | ✅      | Connect existing store to processor                                                                                                                                                                                                                |

### TopologyDescription

`describe()` returns the Kafka Streams `TopologyDescription` — a graph of nested node interfaces.

| Member                                      | Status | Notes                                 |
| ------------------------------------------- | ------ | ------------------------------------- |
| subtopologies() / globalStores()            | ✅      | Sets of `Subtopology` / `GlobalStore` |
| Subtopology.id() / nodes()                  | ✅      |                                       |
| Node.name() / predecessors() / successors() | ✅      |                                       |
| Source.topicSet() / topicPattern()          | ✅      |                                       |
| Processor.stores()                          | ✅      | Connected store names                 |
| Sink.topic() / topicNameExtractor()         | ✅      | Static or dynamic routing             |
| GlobalStore.id() / source() / processor()   | ✅      |                                       |

## KStream — stateless transformations

| Method                            | Status | Notes                                                                                                                                              |
| --------------------------------- | ------ | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| filter(predicate)                 | ✅      |                                                                                                                                                    |
| filterNot(predicate)              | ✅      |                                                                                                                                                    |
| map(mapper)                       | ✅      | Key-changing; the sub-topology boundary opens downstream, where an operator needs the new key's affinity (KS `repartitionRequired` model, ADR-138) |
| mapValues(ValueMapper)            | ✅      |                                                                                                                                                    |
| mapValues(ValueMapperWithKey)     | ✅      | Key-aware value mapper variant                                                                                                                     |
| flatMap(mapper)                   | ✅      | Key-changing; the sub-topology boundary opens downstream, where an operator needs the new key's affinity (KS `repartitionRequired` model, ADR-138) |
| flatMapValues(ValueMapper)        | ✅      |                                                                                                                                                    |
| flatMapValues(ValueMapperWithKey) | ✅      | Key-aware value mapper variant                                                                                                                     |
| selectKey(keySelector)            | ✅      | Key-changing; the sub-topology boundary opens downstream, where an operator needs the new key's affinity (KS `repartitionRequired` model, ADR-138) |
| peek(action)                      | ✅      | Java BiConsumer overload                                                                                                                           |
| merge(other)                      | ✅      |                                                                                                                                                    |
| split()                           | ✅      | Returns BranchedKStream                                                                                                                            |
| repartition()                     | ✅      | In-memory only                                                                                                                                     |
| repartition(Repartitioned)        | ✅      | Custom partitioner, naming                                                                                                                         |
| print()                           | ✅      | Configurable via Printed                                                                                                                           |

## KStream — terminal operations

| Method                            | Status | Notes                           |
| --------------------------------- | ------ | ------------------------------- |
| forEach(action)                   | ✅      | Java BiConsumer overload        |
| to(topic, produced?)              | ✅      | Static topic routing            |
| to(TopicNameExtractor, produced?) | ✅      | Dynamic topic routing           |
| toTable()                         | ✅      | Auto-generates store name       |
| toTable(named?, materialized?)    | ✅      | With naming and materialization |

## KStream — Processor API

| Method                                                  | Status | Notes                                                                                                                                                                                          |
| ------------------------------------------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| process(processorSupplier, named?, storeNames...)       | ✅      | Key-changing; `ProcessorSupplier`. Connects the named stores (and any declared via `supplier.stores()`). Like KS, it does **not** open a sub-topology boundary of its own — see the note below |
| processValues(processorSupplier, named?, storeNames...) | ✅      | Non-key-changing; `FixedKeyProcessorSupplier` + named/declared stores. Like KS, it does **not** open a sub-topology boundary of its own — see the note below                                   |
| transform()                                             | 🛑     | Removed from KStream in KS 4.x; use process()                                                                                                                                                  |
| transformValues()                                       | 🛑     | Removed from KStream in KS 4.x; use processValues()                                                                                                                                            |

::callout{color="info" icon="i-lucide-info"}
**Key affinity at a Processor API node.** Kafka Streams never materialises a repartition before `process()` / `processValues()`, even with connected stores, and since 1.0.0 neither does StoatFlow (`topology.processor-api-key-affinity: off` — `presumed` restores the pre-1.0.0 boundary). After a **many-to-one** re-key that means per-key state at such a node is fragmented across lanes, exactly as it is across tasks in KS. StoatFlow adds a guard KS does not have: it **refuses to compile** the provable case — a proven re-key into a node with a writable store (`topology.validation.papi-key-affinity-presumed`, default `error`) — and warns where the feeder is itself a Processor API node, whose key change is only a conservative default (`papi-key-affinity-presumed-chain`, default `warn`). See [Key affinity after a re-key](https://stoatflow.io/docs/building/processor-api#key-affinity-after-a-re-key).
::

## KStream — grouping

| Method                        | Status | Notes                                                                                                                                              |
| ----------------------------- | ------ | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| groupByKey()                  | ✅      | Returns KGroupedStream                                                                                                                             |
| groupByKey(Grouped)           | ✅      | With Serdes configuration                                                                                                                          |
| groupBy(keySelector)          | ✅      | Key-changing; the sub-topology boundary opens downstream, where an operator needs the new key's affinity (KS `repartitionRequired` model, ADR-138) |
| groupBy(keySelector, Grouped) | ✅      | With Serdes configuration                                                                                                                          |

## KStream — joins

| Method                                                                      | Status | Notes                          |
| --------------------------------------------------------------------------- | ------ | ------------------------------ |
| join(KStream, ValueJoiner, windows)                                         | ✅      | Stream-stream windowed join    |
| join(KStream, ValueJoiner, windows, streamJoined)                           | ✅      | With StreamJoined config       |
| join(KStream, ValueJoinerWithKey, windows)                                  | ✅      | Key-aware joiner variant       |
| join(KStream, ValueJoinerWithKey, windows, streamJoined)                    | ✅      | Key-aware joiner with config   |
| leftJoin(KStream, ValueJoiner, windows)                                     | ✅      | Stream-stream left join        |
| leftJoin(KStream, ValueJoiner, windows, streamJoined)                       | ✅      | With StreamJoined config       |
| leftJoin(KStream, ValueJoinerWithKey, windows)                              | ✅      | Key-aware joiner variant       |
| leftJoin(KStream, ValueJoinerWithKey, windows, streamJoined)                | ✅      | Key-aware joiner with config   |
| outerJoin(KStream, ValueJoiner, windows)                                    | ✅      | Stream-stream outer join       |
| outerJoin(KStream, ValueJoiner, windows, streamJoined)                      | ✅      | With StreamJoined config       |
| outerJoin(KStream, ValueJoinerWithKey, windows)                             | ✅      | Key-aware joiner variant       |
| outerJoin(KStream, ValueJoinerWithKey, windows, streamJoined)               | ✅      | Key-aware joiner with config   |
| join(KTable, ValueJoiner)                                                   | ✅      | Stream-table inner join        |
| join(KTable, ValueJoinerWithKey)                                            | ✅      | Key-aware joiner variant       |
| leftJoin(KTable, ValueJoiner)                                               | ✅      | Stream-table left join         |
| leftJoin(KTable, ValueJoinerWithKey)                                        | ✅      | Key-aware joiner variant       |
| join(GlobalKTable, keySelector, ValueJoiner)                                | ✅      | Stream-GlobalKTable inner join |
| join(GlobalKTable, keySelector, ValueJoinerWithKey)                         | ✅      | Key-aware joiner variant       |
| join(GlobalKTable, keySelector, ValueJoiner\|ValueJoinerWithKey, Named)     | ✅      | `Named` overloads              |
| leftJoin(GlobalKTable, keySelector, ValueJoiner)                            | ✅      | Stream-GlobalKTable left join  |
| leftJoin(GlobalKTable, keySelector, ValueJoinerWithKey)                     | ✅      | Key-aware joiner variant       |
| leftJoin(GlobalKTable, keySelector, ValueJoiner\|ValueJoinerWithKey, Named) | ✅      | `Named` overloads              |

::callout{color="info" icon="i-lucide-info"}
**Temporal join semantics (KIP-914).** When the backing table uses a versioned store (KIP-889), stream-table joins perform a temporal lookup against the stream record's timestamp — returning the table value that was valid at the stream event time, so results stay correct when records arrive out of order. Stream-table joins are otherwise point-in-time lookups; a table update does not re-trigger the join.
::

## KStream — branching (BranchedKStream)

| Method                      | Status | Notes                |
| --------------------------- | ------ | -------------------- |
| branch(predicate)           | ✅      |                      |
| branch(predicate, branched) | ✅      | With Branched config |
| defaultBranch()             | ✅      |                      |
| defaultBranch(branched)     | ✅      | With Branched config |
| noDefaultBranch()           | ✅      |                      |

## KGroupedStream

| Method                             | Status | Notes                                                                                                                             |
| ---------------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------- |
| count()                            | ✅      | With Materialized overloads                                                                                                       |
| reduce(reducer)                    | ✅      | With Materialized overloads                                                                                                       |
| aggregate(initializer, aggregator) | ✅      | With Materialized overloads                                                                                                       |
| windowedBy(TimeWindows)            | ✅      | Tumbling/hopping windows → `TimeWindowedKStream`                                                                                  |
| windowedBy(SlidingWindows)         | ✅      | Sliding windows (KIP-450) → returns `TimeWindowedKStream`, exactly like Kafka Streams (no separate `SlidingWindowedKStream` type) |
| windowedBy(SessionWindows)         | ✅      | Session windows                                                                                                                   |
| cogroup()                          | ✅      | Returns CogroupedKStream                                                                                                          |

## Windowed grouped streams

### TimeWindowedKStream

Reached by both `windowedBy(TimeWindows)` (tumbling/hopping) and `windowedBy(SlidingWindows)` (KIP-450
sliding) — sliding windows flow through this type, exactly as in Kafka Streams. All overloads take a
plain-key `Materialized<K, V, …>`.

| Method                             | Status | Notes                                   |
| ---------------------------------- | ------ | --------------------------------------- |
| count()                            | ✅      | With Materialized overloads (plain key) |
| reduce(reducer)                    | ✅      | With Materialized overloads             |
| aggregate(initializer, aggregator) | ✅      | With Materialized overloads             |
| emitStrategy()                     | ✅      | OnWindowUpdate (default), OnWindowClose |

### SessionWindowedKStream

| Method                                            | Status | Notes                                   |
| ------------------------------------------------- | ------ | --------------------------------------- |
| count()                                           | ✅      | With session merger                     |
| reduce(reducer)                                   | ✅      | Reducer doubles as merger               |
| aggregate(initializer, aggregator, sessionMerger) | ✅      | With session merger                     |
| emitStrategy()                                    | ✅      | OnWindowUpdate (default), OnWindowClose |

## Cogrouped streams

### CogroupedKStream

| Method                              | Status | Notes                                                            |
| ----------------------------------- | ------ | ---------------------------------------------------------------- |
| cogroup(KGroupedStream, aggregator) | ✅      | Add another stream to the cogroup                                |
| aggregate(initializer)              | ✅      | With Materialized overloads                                      |
| windowedBy(TimeWindows)             | ✅      | Returns TimeWindowedCogroupedKStream                             |
| windowedBy(SlidingWindows)          | ✅      | Sliding windows (KIP-450) → returns TimeWindowedCogroupedKStream |
| windowedBy(SessionWindows)          | ✅      | Returns SessionWindowedCogroupedKStream                          |

### TimeWindowedCogroupedKStream

| Method                 | Status | Notes                                   |
| ---------------------- | ------ | --------------------------------------- |
| aggregate(initializer) | ✅      | With Materialized overloads             |
| emitStrategy()         | ✅      | OnWindowUpdate (default), OnWindowClose |

### SessionWindowedCogroupedKStream

| Method                                | Status | Notes                                                                                            |
| ------------------------------------- | ------ | ------------------------------------------------------------------------------------------------ |
| aggregate(initializer, sessionMerger) | ✅      | With Materialized overloads; `(initializer, sessionMerger)` argument order matches Kafka Streams |
| emitStrategy()                        | ✅      | OnWindowUpdate (default), OnWindowClose                                                          |

## KTable — transformations

| Method                                                                                 | Status | Notes                                                                                                                                                                                                                                                         |
| -------------------------------------------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| filter(predicate)                                                                      | ✅      | Derived table (no state store)                                                                                                                                                                                                                                |
| filter(predicate, Named)                                                               | ✅      | With naming                                                                                                                                                                                                                                                   |
| filter(predicate, Materialized)                                                        | ✅      | With materialization                                                                                                                                                                                                                                          |
| filter(predicate, Named, Materialized)                                                 | ✅      | With naming and materialization                                                                                                                                                                                                                               |
| filterNot(predicate)                                                                   | ✅      | Inverse of filter                                                                                                                                                                                                                                             |
| filterNot(predicate, Named)                                                            | ✅      | With naming                                                                                                                                                                                                                                                   |
| filterNot(predicate, Materialized)                                                     | ✅      | With materialization                                                                                                                                                                                                                                          |
| filterNot(predicate, Named, Materialized)                                              | ✅      | With naming and materialization                                                                                                                                                                                                                               |
| mapValues(ValueMapper)                                                                 | ✅      | Derived table (no state store)                                                                                                                                                                                                                                |
| mapValues(ValueMapper, Named)                                                          | ✅      | With naming                                                                                                                                                                                                                                                   |
| mapValues(ValueMapper, Materialized)                                                   | ✅      | With materialization                                                                                                                                                                                                                                          |
| mapValues(ValueMapper, Named, Materialized)                                            | ✅      | With naming and materialization                                                                                                                                                                                                                               |
| mapValues(ValueMapperWithKey)                                                          | ✅      | Key-aware value mapper variant                                                                                                                                                                                                                                |
| mapValues(ValueMapperWithKey, Named)                                                   | ✅      | With naming                                                                                                                                                                                                                                                   |
| mapValues(ValueMapperWithKey, Materialized)                                            | ✅      | With materialization                                                                                                                                                                                                                                          |
| mapValues(ValueMapperWithKey, Named, Materialized)                                     | ✅      | With naming and materialization                                                                                                                                                                                                                               |
| toStream()                                                                             | ✅      | Same key type                                                                                                                                                                                                                                                 |
| toStream(KeyValueMapper)                                                               | ✅      | Key-changing variant; the sub-topology boundary opens downstream, where an operator needs the new key's affinity (ADR-138)                                                                                                                                    |
| queryableStoreName()                                                                   | ✅      | Returns store name if materialized                                                                                                                                                                                                                            |
| groupBy(keyValueSelector)                                                              | ✅      | Selector returns KeyValue\<KR, VR>; returns KGroupedTable\<KR, VR>                                                                                                                                                                                            |
| groupBy(keyValueSelector, Grouped)                                                     | ✅      | With Serdes configuration                                                                                                                                                                                                                                     |
| suppress(Suppressed)                                                                   | ✅      | untilWindowCloses, untilTimeLimit                                                                                                                                                                                                                             |
| transformValues(ValueTransformerWithKeySupplier, named?, storeNames...)                | ✅      | Stateful per-key value transform; returns `KTable<K, VR>`                                                                                                                                                                                                     |
| transformValues(ValueTransformerWithKeySupplier, Materialized, [Named,] storeNames...) | ✅      | Materializes the result into a queryable store (tombstone-aware: a `null` transform deletes). Supplier parameter carries Kafka Streams' `? super K, ? super V, ? extends VR` wildcards, so a supplier producing a subtype of the result value type still fits |

## KTable — primary-key joins

Both tables share key type `K`. No co-partitioning required (global state model).

| Method                                         | Status | Notes                                 |
| ---------------------------------------------- | ------ | ------------------------------------- |
| join(KTable, joiner)                           | ✅      | Inner join; both must be materialized |
| join(KTable, joiner, Materialized)             | ✅      | With materialization of join result   |
| join(KTable, joiner, Named, Materialized)      | ✅      | With naming and materialization       |
| leftJoin(KTable, joiner)                       | ✅      | Left table always emits               |
| leftJoin(KTable, joiner, Materialized)         | ✅      | With materialization of join result   |
| leftJoin(KTable, joiner, Named, Materialized)  | ✅      | With naming and materialization       |
| outerJoin(KTable, joiner)                      | ✅      | Full outer join                       |
| outerJoin(KTable, joiner, Materialized)        | ✅      | With materialization of join result   |
| outerJoin(KTable, joiner, Named, Materialized) | ✅      | With naming and materialization       |

::callout{color="info" icon="i-lucide-info"}
**KIP-914 semantics.** When both tables are backed by versioned stores, out-of-order records do not trigger join evaluation, and temporal lookups return the correct value from the other table — preventing inconsistent results under out-of-order arrival.
::

## KTable — foreign-key joins

Foreign-key extractor: a Java `Function<V, KO?>` (or `BiFunction<K, V, KO?>` for key-aware extraction) — or a Kotlin `(V) -> KO?` / `(K, V) -> KO?` lambda.

| Method                                                           | Status | Notes                               |
| ---------------------------------------------------------------- | ------ | ----------------------------------- |
| join(KTable, fkExtractor, joiner)                                | ✅      | Inner FK join                       |
| join(KTable, fkExtractor, joiner, Materialized)                  | ✅      | With materialization of join result |
| join(KTable, fkExtractor, joiner, TableJoined, Materialized)     | ✅      | With naming and materialization     |
| leftJoin(KTable, fkExtractor, joiner)                            | ✅      | Left FK join                        |
| leftJoin(KTable, fkExtractor, joiner, Materialized)              | ✅      | With materialization of join result |
| leftJoin(KTable, fkExtractor, joiner, TableJoined, Materialized) | ✅      | With naming and materialization     |
| outerJoin(KTable, fkExtractor, joiner)                           | ⚠️     | Not supported in KS either          |

## KGroupedTable

`KGroupedTable` is the result of `KTable.groupBy()` and uses changelog semantics — moving a key from group A to group B decrements A and increments B. The source `KTable` must be materialized for `groupBy()` to track old values.

| Method                             | Status | Notes                                                                                                                    |
| ---------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------ |
| count()                            | ✅      | Changelog semantics                                                                                                      |
| reduce(adder, subtractor)          | ✅      | Kafka Streams compatible; both adder and subtractor are `Reducer<V>`                                                     |
| aggregate(init, adder, subtractor) | ✅      | Kafka Streams compatible; both adder **and** subtractor are `Aggregator<K, V, VR>` — no StoatFlow-only `Subtractor` type |

## GlobalKTable

| Method               | Status | Notes                                 |
| -------------------- | ------ | ------------------------------------- |
| queryableStoreName() | ✅      | Returns underlying table's store name |
| asKTable()           | 🚀     | StoatFlow extension to unwrap         |

::callout{color="info" icon="i-lucide-info"}
GlobalKTable wraps KTable internally. Because all state is global, its behaviour is identical to KTable.
::

## Processor API

### ProcessorContext

| Method                                                | Status | Notes                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| ----------------------------------------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| forward(record)                                       | ✅      | Forward to downstream processors                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| forward(record, childName)                            | ✅      | Forward to specific child                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| getStateStore(name)                                   | ✅      | Access connected state stores — declared via `ProcessorSupplier.stores()` or the `process(supplier, "name")` varargs. An **undeclared** store throws `StoreNotConnectedException` (KS parity). Global stores (`addGlobalStore`, `globalTable`) are exempt and returned **read-only** unless the node declares them, in which case they are writable (a StoatFlow extension — KS makes declaring a global store a `TopologyException`). The read-only view covers every store family (plain, timestamped, versioned, headers-aware) and keeps the store's declared type, so the caller's `getStateStore<TimestampedKeyValueStore<..>>` cast still succeeds; read-only stores (`addReadOnlyStateStore`) are exempt too — a deliberate divergence, KS exempts only globals |
| schedule(interval, type, punctuator)                  | ✅      | KS-compatible punctuator                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| schedule(interval, type, mode, punctuator)            | 🚀     | StoatFlow extension with BLOCKING/NON\_BLOCKING mode                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| schedule(interval, startTime, type, punctuator)       | ✅      | KIP-1146: anchored punctuation with grid-aligned fire times                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| schedule(interval, startTime, type, mode, punctuator) | ✅      | KIP-1146: anchored punctuation with BLOCKING/NON\_BLOCKING mode                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| recordMetadata()                                      | ✅      | Source partition, offset, topic                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| currentSystemTimeMs()                                 | ✅      | Wall-clock time via `System.currentTimeMillis()`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| currentStreamTimeMs()                                 | ✅      | Delegates to `currentWatermarkMs()` (global watermark)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| currentWatermarkMs()                                  | 🆕     | Global watermark in epoch millis                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| timerService()                                        | 🚀     | Key-based timer access (Flink-inspired)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |

### Processor interfaces

| Type                                                          | Status | Notes                                                                                                                                                                                                                                                                                 |
| ------------------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Processor\<KIn,VIn,KOut,VOut>                                 | ✅      | Key-changing processor interface                                                                                                                                                                                                                                                      |
| FixedKeyProcessor\<KIn,VIn,VOut>                              | ✅      | Key-preserving processor interface                                                                                                                                                                                                                                                    |
| ContextualProcessor\<KIn,VIn,KOut,VOut>                       | ✅      | Abstract base managing ProcessorContext                                                                                                                                                                                                                                               |
| ContextualFixedKeyProcessor\<KIn,VIn,VOut>                    | ✅      | Abstract base managing FixedKeyProcessorContext                                                                                                                                                                                                                                       |
| ProcessorSupplier\<KIn,VIn,KOut,VOut>                         | ✅      | Factory for Processor instances                                                                                                                                                                                                                                                       |
| ProcessorSupplier.stores()                                    | ✅      | KS 4.x pattern for declaring stores inline                                                                                                                                                                                                                                            |
| FixedKeyProcessorSupplier\<KIn,VIn,VOut>                      | ✅      | Factory for FixedKeyProcessor instances                                                                                                                                                                                                                                               |
| ValueTransformerWithKey\<K,V,VR>                              | ✅      | Consumed by `KTable.transformValues`; `init` binds StoatFlow's `FixedKeyProcessorContext`                                                                                                                                                                                             |
| ValueTransformerWithKeySupplier\<K,V,VR>                      | ✅      | Factory for `ValueTransformerWithKey`; declares stores via `stores()`. As a `KTable.transformValues` parameter it carries Kafka Streams' `? super K, ? super V, ? extends VR` wildcards                                                                                               |
| ValueTransformer / Transformer (+ suppliers)                  | ↘️     | Deprecated in KS and removed from `KStream`; kept as nameable types for source compatibility                                                                                                                                                                                          |
| `init()` / `close()` lifecycle                                | ✅      | Once per task — `init()` before any records, `close()` once at shutdown (KS contract), across all processor paths incl. `processValues` / `transformValues`                                                                                                                           |
| `FixedKeyProcessor.onTimer` + `schedule()` / `timerService()` | ✅      | Full `Processor` timer parity for `FixedKeyProcessor` (`processValues`): periodic punctuators via `schedule()` and keyed timers via `timerService()` → `onTimer(...)` (key-preserving `FixedKeyTimerContext`, full state access, re-registration). Record timestamp is the event time |

### Punctuators and timers

| Type / Method                                            | Status | Notes                                                 |
| -------------------------------------------------------- | ------ | ----------------------------------------------------- |
| PunctuationType.STREAM\_TIME                             | ✅      | Fires on event-time watermark advancement             |
| PunctuationType.WALL\_CLOCK\_TIME                        | ✅      | Fires on system clock                                 |
| PunctuatorMode.BLOCKING                                  | 🚀     | Barriers wait for punctuator completion before commit |
| PunctuatorMode.NON\_BLOCKING                             | 🚀     | Barriers proceed independently (default)              |
| Punctuator.punctuate(timestamp)                          | ✅      | Callback invoked at scheduled time                    |
| Cancellable.cancel()                                     | ✅      | Cancel scheduled punctuator or timer                  |
| TimerService.registerEventTimeTimer(key, timestamp)      | 🚀     | Fire when watermark ≥ timestamp                       |
| TimerService.registerProcessingTimeTimer(key, timestamp) | 🚀     | Fire at wall-clock time                               |
| TimerService.deleteEventTimeTimer(key, timestamp)        | 🚀     | Cancel event-time timer                               |
| TimerService.deleteProcessingTimeTimer(key, timestamp)   | 🚀     | Cancel processing-time timer                          |
| TimerService.currentWatermark()                          | 🚀     | Current event-time watermark                          |
| TimerService.currentProcessingTime()                     | 🚀     | Current wall-clock time                               |

::callout{color="info" icon="i-lucide-info"}
Kafka Streams punctuators are partition-scoped with writable state and always non-blocking. StoatFlow punctuators run globally (topology-wide state view) on a dedicated punctuation lane per sub-topology, with full read/write state access; `PunctuatorMode` and the key-based `TimerService` (which fires in the key's affinity lane, also with full read/write state access) are StoatFlow extensions.
::

## Scheduled sources (StoatFlow extension)

A `ScheduledEmitter` — registered via `StreamsBuilder.scheduled(...)` — periodically emits records (interval- or cron-based) using the `ScheduledEmitterContext` below, without consuming from Kafka.

### ScheduledEmitterContext

| Method                         | Status | Notes                                                                                                                                                                                                                                                                          |
| ------------------------------ | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| forward(key, value)            | 🚀     | Emit record with current wall-clock time                                                                                                                                                                                                                                       |
| forward(key, value, timestamp) | 🚀     | Emit record with explicit timestamp                                                                                                                                                                                                                                            |
| forward(record)                | 🚀     | Emit record with key, value, timestamp, headers                                                                                                                                                                                                                                |
| getStateStore(name)            | 🚀     | Access a state store **read-only** — writes throw `UnsupportedOperationException`. Every family is covered (plain, timestamped, versioned, headers-aware) and the view keeps the store's declared type, so a `getStateStore<TimestampedKeyValueStore<..>>` cast still succeeds |
| currentWatermarkMs()           | 🆕     | Current watermark (global event-time progress)                                                                                                                                                                                                                                 |
| currentStreamTimeMs()          | ✅      | Delegates to `currentWatermarkMs()` (KS compat)                                                                                                                                                                                                                                |
| currentSystemTimeMs()          | ✅      | Wall-clock time via `System.currentTimeMillis()` (KS compat)                                                                                                                                                                                                                   |
| currentWallClockTime()         | 🚀     | Current wall-clock time                                                                                                                                                                                                                                                        |

### CronExpression

| Method                                  | Status | Notes                                      |
| --------------------------------------- | ------ | ------------------------------------------ |
| unix(expression)                        | 🚀     | Unix cron (5 fields: min hour dom mon dow) |
| quartz(expression)                      | 🚀     | Quartz cron (6 fields with seconds)        |
| spring(expression)                      | 🚀     | Spring cron (6 fields with seconds)        |
| parse(expression, cronType)             | 🚀     | Parse with explicit type                   |
| nextExecutionTime(afterMs, zoneId?)     | 🚀     | Calculate next fire time                   |
| timeUntilNextExecution(fromMs, zoneId?) | 🚀     | Duration until next fire                   |

::callout{color="info" icon="i-lucide-info"}
Cron-based scheduling always uses wall-clock time.
::

## Functional interfaces

KS-compatible functional interfaces backing the Java-friendly DSL overloads.

| Interface                           | Status | Used in                                                                                                                                                                                           |
| ----------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `KeyValueMapper<K, V, VR>`          | ✅      | `map`, `flatMap`, `groupBy`, FK joins                                                                                                                                                             |
| `ValueMapper<V, VR>`                | ✅      | `KTable.mapValues`, FK join extractors                                                                                                                                                            |
| `ValueMapperWithKey<K, V, VR>`      | ✅      | `mapValues` with key access                                                                                                                                                                       |
| `Predicate<K, V>`                   | ✅      | `filter`, `filterNot`                                                                                                                                                                             |
| `ForeachAction<K, V>`               | ✅      | `peek`, `forEach`                                                                                                                                                                                 |
| `Initializer<VA>`                   | ✅      | All `aggregate()`                                                                                                                                                                                 |
| `Aggregator<K, V, VA>`              | ✅      | All `aggregate()` — incl. both the adder and subtractor of `KGroupedTable.aggregate()` (KS uses `Aggregator` for both; no `Subtractor` type)                                                      |
| `Reducer<V>`                        | ✅      | All `reduce()`                                                                                                                                                                                    |
| `Merger<K, V>`                      | ✅      | Session window `aggregate()`                                                                                                                                                                      |
| `ValueJoiner<V1, V2, VR>`           | ✅      | All joins                                                                                                                                                                                         |
| `ValueJoinerWithKey<K, V1, V2, VR>` | ✅      | Key-aware joins                                                                                                                                                                                   |
| `StreamPartitioner<K, V>`           | ✅      | KIP-837 `partitions(...): Optional<Set<Integer>>` (sole method). Sink multicast matches KS; the in-memory repartition path throws on multicast (single-instance). Lives in the `topology` package |

## Configuration helper classes

### Consumed

| Method                                                      | Status | Notes                                                                                                                                                                                                                          |
| ----------------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| with(keySerde, valueSerde)                                  | ✅      |                                                                                                                                                                                                                                |
| with(keySerde, valueSerde, timestampExtractor, resetPolicy) | ✅      | Extractor adapted to a stream-time watermark strategy                                                                                                                                                                          |
| keySerde(serde)                                             | ✅      |                                                                                                                                                                                                                                |
| valueSerde(serde)                                           | ✅      |                                                                                                                                                                                                                                |
| as(name)                                                    | ✅      |                                                                                                                                                                                                                                |
| withWatermarkStrategy(strategy)                             | 🚀     | StoatFlow extension (mutually exclusive with withTimestampExtractor)                                                                                                                                                           |
| withTimestampExtractor()                                    | ✅      | KS `TimestampExtractor` adapted to a per-source watermark strategy that assigns event time + emits stream-time watermarks (windowing/suppress close as in KS); use withWatermarkStrategy for bounded-out-of-orderness/idleness |
| withOffsetResetPolicy()                                     | ✅      | Per-source offset reset. `AutoOffsetReset`: `Earliest` / `Latest` / `None` / `byDuration(Duration)` (KIP-1106 — seek to the first offset at/after `now − duration`)                                                            |
| withNumberOfLanes(int)                                      | 🆕     | Sets this source chain's lane count (parallelism)                                                                                                                                                                              |
| withLaneQueueCapacity(int)                                  | 🆕     | Sets this source chain's per-lane queue capacity                                                                                                                                                                               |

### WatermarkStrategy (Flink-inspired)

Event-time progress is driven by a `WatermarkStrategy`. A KS `Consumed.withTimestampExtractor(...)` is supported and adapted to one under the hood; `Consumed.withWatermarkStrategy(...)` is the richer StoatFlow-native form (bounded out-of-orderness, idleness, alignment).

| Method                                                      | Status | Notes                                                                                    |
| ----------------------------------------------------------- | ------ | ---------------------------------------------------------------------------------------- |
| forBoundedOutOfOrderness(maxOutOfOrderness)                 | 🚀     | Bounded out-of-orderness                                                                 |
| forMonotonousTimestamps()                                   | 🚀     | Strictly increasing timestamps                                                           |
| noWatermarks()                                              | 🚀     | Processing-time only                                                                     |
| withTimestampAssigner(assigner)                             | 🚀     | Custom event-time assignment                                                             |
| withIdleness(duration)                                      | 🚀     | Idle-source handling                                                                     |
| withWatermarkAlignment(group, maxDrift [, updateInterval] ) | 🚀     | Flink FLIP-217 alignment; pauses a partition whose watermark drifts beyond the group cap |

The `TimestampExtractor` SAM is present with the KS-exact, non-generic signature `long extract(ConsumerRecord<Object, Object>, long)`, and `Consumed.withTimestampExtractor(...)` wires it to event-time + stream-time watermarks.

KS-optional builder arguments (serdes, grace, partitioner, offset-reset, name) across `Consumed`/`Grouped`/`Produced`/`Repartitioned`/`Joined`/`StreamJoined`/`Materialized`/`Branched` accept `null` (= "use the default"), matching Kafka Streams; `Named` itself still requires a non-null name.

### Grouped

| Method                           | Status | Notes                                                   |
| -------------------------------- | ------ | ------------------------------------------------------- |
| as(name)                         | ✅      |                                                         |
| keySerde(serde)                  | ✅      |                                                         |
| valueSerde(serde)                | ✅      |                                                         |
| with(keySerde, valueSerde)       | ✅      |                                                         |
| with(name, keySerde, valueSerde) | ✅      |                                                         |
| withName(name)                   | ✅      |                                                         |
| withKeySerde(serde)              | ✅      |                                                         |
| withValueSerde(serde)            | ✅      |                                                         |
| withNumberOfLanes(int)           | 🆕     | Sets the grouped sub-topology's lane count              |
| withLaneQueueCapacity(int)       | 🆕     | Sets the grouped sub-topology's per-lane queue capacity |

### Produced

| Method                         | Status | Notes |
| ------------------------------ | ------ | ----- |
| with(keySerde, valueSerde)     | ✅      |       |
| keySerde(serde)                | ✅      |       |
| valueSerde(serde)              | ✅      |       |
| as(name)                       | ✅      |       |
| streamPartitioner(partitioner) | ✅      |       |

### Materialized

| Method                           | Status | Notes                                                                               |
| -------------------------------- | ------ | ----------------------------------------------------------------------------------- |
| as(storeName)                    | ✅      |                                                                                     |
| as(storeSupplier)                | ✅      |                                                                                     |
| as(DslStoreSuppliers)            | ✅      | Store type supplier                                                                 |
| with(keySerde, valueSerde)       | ✅      |                                                                                     |
| keySerde(serde)                  | ✅      |                                                                                     |
| valueSerde(serde)                | ✅      |                                                                                     |
| withCachingEnabled()             | ✅      | Suppress downstream emissions (default enabled)                                     |
| withCachingDisabled()            | ✅      | Emit all intermediate updates downstream                                            |
| withLoggingEnabled()             | ✅      | Enable changelog with optional topic config                                         |
| withLoggingDisabled()            | ✅      | Disable changelog for this store                                                    |
| withRetention()                  | ✅      | Override computed retention for windowed stores                                     |
| withStoreType(DslStoreSuppliers) | ✅      | Store type selection                                                                |
| withRecordHeaders()              | 🆕     | Persist the record's headers alongside key/value in the store (KIP-1271 / KIP-1285) |
| withoutRecordHeaders()           | 🆕     | Disable header persistence (default)                                                |

### Branched

| Method                 | Status | Notes                  |
| ---------------------- | ------ | ---------------------- |
| as(name)               | ✅      |                        |
| withName(name)         | ✅      |                        |
| withFunction(chain)    | ✅      |                        |
| withConsumer(consumer) | ✅      | Java Consumer overload |

### Named

`Named` is `open`/subclassable, so KSML's name-validation subclasses work.

| Method         | Status | Notes                    |
| -------------- | ------ | ------------------------ |
| as(name)       | ✅      |                          |
| withName(name) | ✅      | Fluent builder form      |
| name()         | ✅      | Read the configured name |

### Printed

| Method                     | Status | Notes                                                        |
| -------------------------- | ------ | ------------------------------------------------------------ |
| toSysOut()                 | ✅      | Default stdout                                               |
| toFile(filePath)           | ✅      | File output                                                  |
| withLabel(label)           | ✅      | Prefix label                                                 |
| withKeyValueMapper(mapper) | ✅      | Custom formatting; takes a KS `KeyValueMapper<K, V, String>` |
| withName(processorName)    | ✅      | Set processor name                                           |

### Joined

`Joined<K, VLeft, VRight>` is the Kafka Streams generic form. Its serdes and grace period are **advisory** — stored and round-tripped for source compatibility, but StoatFlow resolves the effective serdes/grace from the upstream `Consumed`/`Grouped` configuration.

| Method                                        | Status | Notes                                 |
| --------------------------------------------- | ------ | ------------------------------------- |
| as(name)                                      | ✅      |                                       |
| create()                                      | ✅      | Create with defaults                  |
| with(keySerde, valueSerde, otherValueSerde)   | ✅      | Static factory; serdes advisory       |
| withName(name)                                | ✅      |                                       |
| withGracePeriod(duration)                     | ✅      | Advisory grace period                 |
| withKeySerde(serde)                           | ✅      | Advisory                              |
| withValueSerde(serde)                         | ✅      | Advisory                              |
| withOtherValueSerde(serde)                    | ✅      | Advisory                              |
| name()                                        | ✅      | Read the configured name              |
| gracePeriod()                                 | ✅      | Round-trips the advisory grace period |
| keySerde() / valueSerde() / otherValueSerde() | ✅      | Round-trip the advisory serdes        |

### TableJoined

| Method                              | Status | Notes                                                     |
| ----------------------------------- | ------ | --------------------------------------------------------- |
| as(name)                            | ✅      |                                                           |
| with()                              | ✅      | Defaults                                                  |
| with(partitioner, otherPartitioner) | ✅      | Accepted no-op stub (partitioning is N/A single-instance) |
| withName(name)                      | ✅      |                                                           |
| withPartitioner()                   | ✅      | Accepted no-op stub for KS source compatibility           |
| withOtherPartitioner()              | ✅      | Accepted no-op stub for KS source compatibility           |

### JoinWindows

| Method                                          | Status | Notes                                       |
| ----------------------------------------------- | ------ | ------------------------------------------- |
| of(timeDifference)                              | ↘️     | Deprecated; use ofTimeDifferenceWithNoGrace |
| ofTimeDifferenceWithNoGrace(timeDifference)     | ✅      | Symmetric window, no grace                  |
| ofTimeDifferenceAndGrace(timeDifference, grace) | ✅      | Symmetric window with grace                 |
| before(timeDifference)                          | ✅      | Asymmetric window (look back)               |
| after(timeDifference)                           | ✅      | Asymmetric window (look forward)            |
| grace(duration)                                 | ↘️     | Deprecated; use ofTimeDifferenceAndGrace    |
| size()                                          | ✅      | Window size in ms                           |
| gracePeriodMs()                                 | ✅      | Grace period in ms                          |

::callout{color="info" icon="i-lucide-info"}
`beforeMs` / `afterMs` are exposed as read-only properties (`getBeforeMs()` / `getAfterMs()` from Java; `before` / `after` return `Duration`). A KS port that reads the public `jw.beforeMs` field replaces it with the accessor.
::

### StreamJoined

| Method                                          | Status | Notes                                      |
| ----------------------------------------------- | ------ | ------------------------------------------ |
| as(name)                                        | ✅      | Create with processor name                 |
| with(keySerde, leftValueSerde, rightValueSerde) | ✅      | Create with Serdes                         |
| create()                                        | ✅      | Create with defaults                       |
| withName(name)                                  | ✅      | Set processor name                         |
| withKeySerde(serde)                             | ✅      | Set key Serde                              |
| withLeftValueSerde(serde)                       | ✅      | Set left stream value Serde                |
| withRightValueSerde(serde)                      | ✅      | Set right stream value Serde               |
| withThisStoreSupplier(storeSupplier)            | ✅      | Custom left window store                   |
| withOtherStoreSupplier(storeSupplier)           | ✅      | Custom right window store                  |
| withLoggingEnabled()                            | ✅      | Enable changelog logging                   |
| withLoggingEnabled(config)                      | ✅      | Enable changelog logging with topic config |
| withLoggingDisabled()                           | ✅      | Disable changelog logging                  |
| withDslStoreSuppliers(suppliers)                | ✅      | Both join stores use same type             |
| withStoreName(name)                             | ✅      | Set base store name                        |

### Repartitioned

| Method                             | Status | Notes                                                        |
| ---------------------------------- | ------ | ------------------------------------------------------------ |
| as(name)                           | ✅      |                                                              |
| streamPartitioner(partitioner)     | ✅      | numPartitions = numLanes                                     |
| withStreamPartitioner(partitioner) | ✅      |                                                              |
| withName(name)                     | ✅      |                                                              |
| with(keySerde, valueSerde)         | ✅      | keySerde used for lane assignment hashing                    |
| withKeySerde()                     | ✅      | Used for lane assignment hashing at sub-topology boundary    |
| withValueSerde()                   | ✅      | Reserved for future use                                      |
| withNumberOfLanes(int)             | 🆕     | Sets the repartition sub-topology's lane count               |
| numberOfLanes(int)                 | 🆕     | Static factory; sets the lane count                          |
| withLaneQueueCapacity(int)         | 🆕     | Sets the per-lane queue capacity                             |
| withNumberOfPartitions()           | 🆕     | KS-compat alias of `withNumberOfLanes` (sets the lane count) |
| numberOfPartitions(int)            | 🆕     | KS-compat static-factory alias of `numberOfLanes`            |

### EmitStrategy

| Method           | Status | Notes                                                                   |
| ---------------- | ------ | ----------------------------------------------------------------------- |
| onWindowUpdate() | ✅      | Default; emit on every update                                           |
| onWindowClose()  | ✅      | Emit only when window closes; driven by the global event-time watermark |

::callout{color="info" icon="i-lucide-info"}
`OnWindowClose` emission is **watermark-driven**: a window's final result is emitted when the global watermark passes the window end plus any configured grace period. See [Event time and watermarks](https://stoatflow.io/docs/concepts/event-time-and-watermarks).
::

### Suppressed

| Method                                 | Status | Notes                                                                                                              |
| -------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------ |
| untilWindowCloses(bufferConfig)        | ✅      | For windowed KTables; requires a `StrictBufferConfig` (KS parity); flush driven by the global event-time watermark |
| untilTimeLimit(duration, bufferConfig) | ✅      | Rate-limits per-key updates; flush driven by the global event-time watermark                                       |
| withName(name)                         | ✅      | KS `NamedOperation`; names the suppression node                                                                    |

::callout{color="info" icon="i-lucide-info"}
Suppression buffers are flushed on **watermark advancement** — `untilWindowCloses` releases a key's final value once the window closes, and `untilTimeLimit` releases the latest buffered value once the per-key time limit elapses in event time. See [Event time and watermarks](https://stoatflow.io/docs/concepts/event-time-and-watermarks). Suppress buffers (and versioned stores, which the migration tool supports as experimental) are the two store families with [state-carrying migration caveats](https://stoatflow.io/docs/migration/with-data-migration#semantics-caveats-at-the-cutover-boundary).
::

::callout{color="info" icon="i-lucide-info"}
**KIP-914:** `suppress()` throws a validation exception for tables backed by versioned stores — their temporal semantics conflict with suppression.
::

### BufferConfig

| Method                                             | Status | Notes                                                                                                   |
| -------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------- |
| unbounded()                                        | ✅      | No limits (use with caution)                                                                            |
| maxRecords(count)                                  | ✅      | Bounded by record count                                                                                 |
| maxBytes(bytes)                                    | ⚠️     | Accepted so KS code compiles, but **throws** at build time — in-memory object buffers have no byte size |
| withMaxRecords(count)                              | ✅      | Fluent record bound                                                                                     |
| withMaxBytes(bytes)                                | ⚠️     | Same as `maxBytes` — throws                                                                             |
| withNoBound()                                      | ✅      | Remove bounds                                                                                           |
| emitEarlyWhenFull()                                | ✅      | Emit instead of evict on overflow                                                                       |
| shutDownWhenFull()                                 | ✅      | Shut down on buffer overflow                                                                            |
| withLoggingEnabled(config) / withLoggingDisabled() | ✅      | Changelog control for the suppress buffer                                                               |

### WindowedSerdes

| Method                                 | Status | Notes                                                          |
| -------------------------------------- | ------ | -------------------------------------------------------------- |
| timeWindowedSerdeFrom(...)             | ⚠️     | Takes `Serde<T>` instead of `Class<T>` (no serde registry)     |
| timeWindowedSerdeFrom(..., windowSize) | ⚠️     | Takes `Serde<T>` only; window size encoded in serialized bytes |
| sessionWindowedSerdeFrom(...)          | ⚠️     | Takes `Serde<T>` instead of `Class<T>`                         |
| TimeWindowedSerde(inner)               | ✅      | `WindowedSerdes.TimeWindowedSerde(serde)`                      |
| TimeWindowedSerde(inner, windowSize)   | ⚠️     | Window size not required (encoded in serialized bytes)         |
| SessionWindowedSerde(inner)            | ✅      | `WindowedSerdes.SessionWindowedSerde(serde)`                   |

::callout{color="warning" icon="i-lucide-triangle-alert"}
StoatFlow's factory methods take `Serde<T>` rather than KS's `Class<T>` (there is no serde registry), and omit the window-size parameter (window bounds are encoded in the serialized bytes). `TimeWindowedSerde.forChangelog()` is not needed — StoatFlow uses identical encoding for client and changelog.
::

## State stores

### Record types & iterators

| Type                                   | Status | Notes                                                        |
| -------------------------------------- | ------ | ------------------------------------------------------------ |
| Windowed\<K>                           | ✅      | Windowed-key wrapper (key + `Window`)                        |
| ValueAndTimestamp\<V>                  | ✅      | Timestamped-store value wrapper (KIP-258)                    |
| VersionedRecord\<V>                    | ✅      | Versioned-store record; `validTo()` returns `Optional<Long>` |
| KeyValueIterator / WindowStoreIterator | ✅      | Closeable iterators for range/scan queries                   |

### State store metrics

| Metric              | Status | Notes                                                    |
| ------------------- | ------ | -------------------------------------------------------- |
| num-keys (KIP-1250) | ✅      | Entry-count gauge for in-memory KV/window/session stores |

### KeyValueStore

| Method                               | Status | Notes                                                                                                                                                                                             |
| ------------------------------------ | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| get(key)                             | ✅      |                                                                                                                                                                                                   |
| put(key, value)                      | ✅      |                                                                                                                                                                                                   |
| putIfAbsent(key, value)              | ✅      |                                                                                                                                                                                                   |
| putAll(entries)                      | ✅      |                                                                                                                                                                                                   |
| delete(key)                          | ✅      |                                                                                                                                                                                                   |
| compute(key, remappingFunction)      | 🚀     | Atomic read-compute-write; Kotlin lambda + Java BiFunction overloads                                                                                                                              |
| merge(key, value, remappingFunction) | 🚀     | Atomic merge; Kotlin lambda + Java BiFunction overloads                                                                                                                                           |
| all()                                | ✅      | Read-your-writes semantics                                                                                                                                                                        |
| range(from, to)                      | ✅      | Read-your-writes semantics. Null bounds = open-ended (KIP-763, KSC-89); from > to returns empty + WARN (KS-lenient)                                                                               |
| reverseAll()                         | ✅      | Read-your-writes semantics                                                                                                                                                                        |
| reverseRange(from, to)               | ✅      | Read-your-writes semantics. Null bounds = open-ended (KIP-763, KSC-89); from > to returns empty + WARN (KS-lenient); (low, high) bounds order, KS-exact (KSC-90 — was (high, low) pre-2026-07-31) |
| approximateNumEntries()              | ✅      |                                                                                                                                                                                                   |
| prefixScan()                         | ✅      | KIP-614; read-your-writes semantics                                                                                                                                                               |

### ReadOnlyKeyValueStore

| Method                         | Status | Notes                                                                                                                                                                 |
| ------------------------------ | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| get(key)                       | ✅      |                                                                                                                                                                       |
| containsKey(key)               | 🚀     | StoatFlow extension; bloom filter optimization for RocksDB                                                                                                            |
| all()                          | ✅      |                                                                                                                                                                       |
| range(from, to)                | ✅      | Null bounds = open-ended (KIP-763, KSC-89); from > to returns empty + WARN (KS-lenient)                                                                               |
| reverseAll()                   | ✅      |                                                                                                                                                                       |
| reverseRange(from, to)         | ✅      | Null bounds = open-ended (KIP-763, KSC-89); from > to returns empty + WARN (KS-lenient); (low, high) bounds order, KS-exact (KSC-90 — was (high, low) pre-2026-07-31) |
| approximateNumEntries()        | ✅      |                                                                                                                                                                       |
| prefixScan(prefix, serializer) | ✅      | KIP-614                                                                                                                                                               |

### WindowStore

Key-range bounds are compared as serialized bytes (unsigned lexicographic); null bounds are open-ended (KIP-763, KSC-91). All Instant variants delegate to the Long variants.

| Method                                          | Status | Notes                                                                                                                                                |
| ----------------------------------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| put(key, value, windowStartTime)                | ✅      |                                                                                                                                                      |
| fetch(key, windowStartTime)                     | ✅      | Point lookup                                                                                                                                         |
| fetch(key, timeFrom, timeTo)                    | ✅      | Single key time range; Long and Instant variants                                                                                                     |
| backwardFetch(key, timeFrom, timeTo)            | ✅      | Reverse iteration; Long and Instant variants                                                                                                         |
| fetch(keyFrom, keyTo, timeFrom, timeTo)         | ✅      | Key range + time range; Long and Instant variants. Null key bounds = open-ended (KIP-763, KSC-91); keyFrom > keyTo returns empty + WARN (KS-lenient) |
| backwardFetch(keyFrom, keyTo, timeFrom, timeTo) | ✅      | Reverse key range; bounds stay (low, high) like fetch (KS-exact); Long and Instant variants. Null key bounds = open-ended (KIP-763, KSC-91)          |
| fetchAll(timeFrom, timeTo)                      | ✅      | All keys time range; Long and Instant variants                                                                                                       |
| backwardFetchAll(timeFrom, timeTo)              | ✅      | Reverse all keys; Long and Instant variants                                                                                                          |
| all()                                           | ✅      | All entries                                                                                                                                          |
| backwardAll()                                   | ✅      | Reverse iteration                                                                                                                                    |
| windowSizeMs                                    | 🚀     | StoatFlow extension for window size access                                                                                                           |
| retentionMs                                     | 🚀     | StoatFlow extension for retention access                                                                                                             |
| expireWindows(watermark)                        | 🚀     | StoatFlow extension for manual expiration                                                                                                            |
| approximateNumEntries()                         | 🚀     | StoatFlow extension for entry count                                                                                                                  |

### ReadOnlyWindowStore

| Method                             | Status | Notes                                          |
| ---------------------------------- | ------ | ---------------------------------------------- |
| All query methods from WindowStore | ✅      | 16 query methods total                         |
| containsKey(key, windowStartTime)  | 🚀     | StoatFlow extension; Long and Instant variants |

### SessionStore

| Method                                                                 | Status | Notes                                                                                                                                   |
| ---------------------------------------------------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------- |
| put(sessionKey, aggregate)                                             | ✅      | Windowed key                                                                                                                            |
| remove(sessionKey)                                                     | ✅      | Windowed key                                                                                                                            |
| fetchSession(key, startTime, endTime)                                  | ✅      | Long and Instant variants                                                                                                               |
| findSessions(key, earliestEndTime, latestStartTime)                    | ✅      | Long and Instant variants                                                                                                               |
| backwardFindSessions(key, earliestEndTime, latestStartTime)            | ✅      | Long and Instant variants                                                                                                               |
| findSessions(keyFrom, keyTo, earliestEndTime, latestStartTime)         | ✅      | Key range; Long and Instant variants. Null key bounds = open-ended (KIP-763, KSC-91); keyFrom > keyTo returns empty + WARN (KS-lenient) |
| backwardFindSessions(keyFrom, keyTo, earliestEndTime, latestStartTime) | ✅      | Key range; Long and Instant variants. Null key bounds = open-ended (KIP-763, KSC-91)                                                    |
| fetch(key)                                                             | ✅      | All sessions for single key                                                                                                             |
| backwardFetch(key)                                                     | ✅      | Reverse order                                                                                                                           |
| fetch(keyFrom, keyTo)                                                  | ✅      | Key range queries. Null key bounds = open-ended (KIP-763, KSC-91); keyFrom > keyTo returns empty + WARN (KS-lenient)                    |
| backwardFetch(keyFrom, keyTo)                                          | ✅      | Reverse order; bounds stay (low, high) (KS-exact). Null key bounds = open-ended (KIP-763, KSC-91)                                       |
| findAll()                                                              | 🚀     | StoatFlow extension                                                                                                                     |
| expireSessions(watermark)                                              | 🚀     | StoatFlow extension                                                                                                                     |
| approximateNumEntries()                                                | 🚀     | StoatFlow extension                                                                                                                     |

### ReadOnlySessionStore

| Method                                   | Status | Notes                                          |
| ---------------------------------------- | ------ | ---------------------------------------------- |
| All query methods from SessionStore      | ✅      | Query methods for IQ access                    |
| containsSession(key, startTime, endTime) | 🚀     | StoatFlow extension; Long and Instant variants |

### Stores factory

StoatFlow's suppliers implement the KS `*BytesStoreSupplier` interfaces and the `*StoreBuilder(...)` factories return a KS `StoreBuilder<T>`. Some signatures take a `Serde<T>` where KS takes `Class<T>` (no serde registry); `StoreBuilder.withCaching*`/`withLogging*` are honored, and `build()` throws (StoatFlow builds stores via the registry).

| Method                                                                              | Status | Notes                                                                                      |
| ----------------------------------------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------ |
| persistentKeyValueStore(name)                                                       | ✅      | RocksDB key-value store                                                                    |
| inMemoryKeyValueStore(name)                                                         | ✅      | In-memory key-value store                                                                  |
| persistentWindowStore(name, retention, windowSize, retainDuplicates)                | ✅      | RocksDB window store                                                                       |
| inMemoryWindowStore(name, retention, windowSize, retainDuplicates)                  | ✅      | In-memory window store                                                                     |
| persistentSessionStore(name, retentionPeriod)                                       | ✅      | RocksDB session store                                                                      |
| inMemorySessionStore(name, retentionPeriod)                                         | ✅      | In-memory session store                                                                    |
| persistentTimestampedKeyValueStore(name)                                            | ✅      | Timestamped KV store (KIP-258)                                                             |
| persistentTimestampedWindowStore(name, retention, windowSize [, retainDuplicates] ) | ✅      | Timestamped window store (KIP-258); 4-arg `retainDuplicates` overload supported            |
| inMemoryTimestampedKeyValueStore(name)                                              | ✅      | In-memory timestamped KV store (KIP-258)                                                   |
| inMemoryTimestampedWindowStore(name, ...)                                           | ✅      | In-memory timestamped window store (KIP-258)                                               |
| persistentVersionedKeyValueStore(name, historyRetention)                            | ✅      | Versioned store — RocksDB (KIP-889)                                                        |
| persistentVersionedKeyValueStore(name, historyRetention, segmentInterval)           | ✅      | 3-arg KS overload; `segmentInterval` accepted but ignored (not segmented) — logs a warning |
| inMemoryVersionedKeyValueStore(name, historyRetention)                              | ✅      | Versioned store — in-memory (KIP-889)                                                      |
| lruMap(name, maxCacheSize)                                                          | ✅      | LRU cache store with bounded entries                                                       |
| keyValueStoreBuilder(supplier, keySerde, valueSerde)                                | ✅      | Returns a KS `StoreBuilder`                                                                |
| timestampedKeyValueStoreBuilder(supplier, keySerde, valueSerde)                     | ✅      | KIP-258                                                                                    |
| windowStoreBuilder(supplier, keySerde, valueSerde)                                  | ✅      | Returns a KS `StoreBuilder`                                                                |
| timestampedWindowStoreBuilder(supplier, keySerde, valueSerde)                       | ✅      | KIP-258                                                                                    |
| sessionStoreBuilder(supplier, keySerde, aggSerde)                                   | ✅      | Returns a KS `StoreBuilder`                                                                |
| versionedKeyValueStoreBuilder(supplier, keySerde, valueSerde)                       | ✅      | KIP-889                                                                                    |
| timestamped{KeyValue,Window} / sessionStoreWithHeadersBuilder(...)                  | ✅      | Headers-aware builders (KIP-1271); pass a `*WithHeaders` supplier                          |
| KeyValue/Window/Session/VersionedBytesStoreSupplier, StoreBuilder                   | ✅      | KS supplier/builder interfaces are present                                                 |

### KeyLockManager (StoatFlow extension)

Striped lock utility for multi-key atomic sections in user-defined Processors. Not in Kafka Streams.

| Method                 | Status | Notes                                            |
| ---------------------- | ------ | ------------------------------------------------ |
| withLock(key, block)   | 🚀     | Single-key lock scope                            |
| withLocks(keys, block) | 🚀     | Multi-key lock scope with deadlock-free ordering |

## Record headers in state stores (KIP-1271 / KIP-1285)

Opt-in persistence of a record's `Headers` alongside key/value, across the timestamped KV, versioned, timestamped-window, and session store families.

| Surface                                                         | Status         | Notes                                                                                                                                                                        |
| --------------------------------------------------------------- | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Materialized.withRecordHeaders()                                | 🆕             | Per-store opt-in (DSL)                                                                                                                                                       |
| dsl.store.format = HEADERS                                      | ✅              | Global default to headers-aware stores                                                                                                                                       |
| Stores.\*WithHeaders(...) suppliers + \*WithHeadersBuilder(...) | ✅              | Processor-API headers stores (all four families)                                                                                                                             |
| QueryableStoreTypes.\*WithHeaders()                             | ✅              | Read stored headers back via IQ                                                                                                                                              |
| In-place upgrade (existing store → headers-aware)               | ✅              | Flip on, restart — no wipe, no changelog replay                                                                                                                              |
| Downgrade refusal (headers-aware → plain)                       | ✅              | Like KS, a downgrade is refused with a diagnosed, actionable message rather than a raw RocksDB error — it is a declared format change, not corruption                        |
| `state.format.downgrade = wipe-and-restore`                     | 🚀             | StoatFlow-only: acknowledge the downgrade in config and it rebuilds that store from the changelog. KS offers no equivalent — its only remedy is deleting local state by hand |
| Empty-keyspace downgrade (nothing written in headers mode)      | 🚀             | Dropped in place, free. KS refuses this case too                                                                                                                             |
| StreamJoined headers                                            | ❌ (matches KS) | KS does not persist headers on join buffers                                                                                                                                  |

## Interactive Queries

### StoatFlow (entry point) methods

| Method                                      | Status | Notes                                                                                                                                                                                                                                                                                                                                         |
| ------------------------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| store(StoreQueryParameters)                 | ✅      | Returns read-only store view                                                                                                                                                                                                                                                                                                                  |
| storeNames()                                | ✅      | List available store names                                                                                                                                                                                                                                                                                                                    |
| queryMetadataForKey(store, key, serializer) | ✅      | Returns `KeyQueryMetadata` — single-instance: the local host, no standbys, the key's faithful Kafka partition                                                                                                                                                                                                                                 |
| streamsMetadataForStore(store)              | ✅      | The single local host materializing the store                                                                                                                                                                                                                                                                                                 |
| metadataForAllStreamsClients()              | ✅      | The one local instance                                                                                                                                                                                                                                                                                                                        |
| setUncaughtExceptionHandler(handler)        | ✅      | KS `StreamsUncaughtExceptionHandler`; `REPLACE_THREAD` triggers an in-place engine restart (fault-storm-guarded — repeated faults escalate to shutdown), under hot-standby HA as well as without it: the fault is absorbed in place and the pod keeps the active role; `SHUTDOWN_CLIENT`/`SHUTDOWN_APPLICATION` shut the single instance down |
| pause()                                     | 🚀     | Pause processing with drain semantics                                                                                                                                                                                                                                                                                                         |
| unpause()                                   | 🚀     | Resume processing after pause                                                                                                                                                                                                                                                                                                                 |
| awaitState(state, timeout)                  | 🚀     | Wait for target state with timeout                                                                                                                                                                                                                                                                                                            |
| stateTransitionHistory()                    | 🚀     | Recent state transitions with timestamps                                                                                                                                                                                                                                                                                                      |

### StoreQueryParameters

| Method                      | Status | Notes                          |
| --------------------------- | ------ | ------------------------------ |
| fromNameAndType(name, type) | ✅      | Factory method                 |
| enableStaleStores()         | ✅      | Allow queries during RESTORING |
| withPartition(partition)    | 🛑     | N/A for single-instance        |

### QueryableStoreTypes

| Method                     | Status | Notes                                                              |
| -------------------------- | ------ | ------------------------------------------------------------------ |
| keyValueStore()            | ✅      | Returns ReadOnlyKeyValueStore                                      |
| windowStore()              | ✅      | Returns ReadOnlyWindowStore                                        |
| sessionStore()             | ✅      | Returns ReadOnlySessionStore                                       |
| timestampedKeyValueStore() | ✅      | Returns ReadOnlyKeyValueStore\<K, ValueAndTimestamp\<V>> (KIP-258) |
| timestampedWindowStore()   | ✅      | Returns ReadOnlyWindowStore\<K, ValueAndTimestamp\<V>> (KIP-258)   |
| versionedKeyValueStore()   | ✅      | Returns ReadOnlyVersionedKeyValueStore\<K, V> (KIP-889)            |

::callout{color="info" icon="i-lucide-info"}
The global state model eliminates partition routing — all state is locally accessible from a single process, so `withPartition()` and inter-instance RPC are unnecessary.
::

### IQ metadata (single-instance: always local)

Kafka Streams' host/partition-distribution metadata types exist so cross-instance "find the host, then query" code compiles **and runs**. Under StoatFlow's single-instance model the answer is always "this host, locally."

| Type                 | Status | Notes                                                                                        |
| -------------------- | ------ | -------------------------------------------------------------------------------------------- |
| HostInfo(host, port) | ✅      | The local host; sourced from `application.server` (synthetic `localhost:0` if unset)         |
| StreamsMetadata      | ✅      | `hostInfo()`, `stateStoreNames()`, `topicPartitions()`; no standbys                          |
| KeyQueryMetadata     | ✅      | `activeHost()` = local, no standby hosts, `partition()` = the key's faithful Kafka partition |

## Exception handlers

| Handler                         | Status | Notes                                                                                                          |
| ------------------------------- | ------ | -------------------------------------------------------------------------------------------------------------- |
| DeserializationExceptionHandler | ✅      | KS-ordered `handle(ErrorHandlerContext, ConsumerRecord, Exception)`; LogAndFail (default), LogAndContinue, DLQ |
| ProcessingExceptionHandler      | ✅      | KS-ordered `handle(ErrorHandlerContext, Record, Exception)`; LogAndFail (default), LogAndContinue, DLQ         |
| ProductionExceptionHandler      | ✅      | KS-ordered `handle(ErrorHandlerContext, ProducerRecord, Exception)`; intelligent retry                         |
| ErrorHandlerContext             | ✅      | KS interface: `topic()`/`partition()`/`offset()`/`timestamp()`/`headers()`/`processorNodeId()`/`taskId()`      |
| StreamsException                | ✅      | Extends `KafkaException`; StoatFlow's fatal failures extend it, so `catch (StreamsException)` works            |

## Lifecycle

| Feature                         | Status | Notes                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| ------------------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| State enum                      | ✅      | StoatFlow keeps a **richer** set than KS's 7 — `CREATED`, `STARTING`, `VALIDATING_STATE`, `RESTORING`, `RUNNING`, `DRAINING`, `PAUSED`, `STOPPING`, `STOPPED`, `ERROR`. For a KS-shaped `KafkaStreams.State`, use the `KafkaStreamsState` mirror + `State.toKafkaStreamsState()` / `kafkaStreamsState()` (KSC-81)                                                                                                                                                             |
| StateListener                   | ✅      | KS-exact single `setStateListener(StateListener)` — a Java method reference / lambda resolves with no cast (KSC-82); an existing `BiConsumer<State,State>` migrates via `StateListener.of(...)`                                                                                                                                                                                                                                                                               |
| StateRestoreListener            | ✅      | Full KS-compatible: onRestoreStart / onBatchRestored / onRestoreEnd                                                                                                                                                                                                                                                                                                                                                                                                           |
| StreamsUncaughtExceptionHandler | ✅      | `setUncaughtExceptionHandler()`; `REPLACE_THREAD` = in-place engine restart (fault-storm-guarded; works under hot-standby HA too — absorbed in place, no failover until the budget is spent), `SHUTDOWN_*` = single-instance shutdown                                                                                                                                                                                                                                         |
| cleanUp()                       | ✅      | Deletes all local state for the application id; valid only in CREATED/STOPPED; rebuilt from changelog on next start                                                                                                                                                                                                                                                                                                                                                           |
| close()                         | ✅      | Graceful shutdown; blocks until terminal. Every concurrent close caller now blocks until completion (a losing second `close()` no longer returns early mid-shutdown) — KSC-83                                                                                                                                                                                                                                                                                                 |
| close(Duration)                 | ✅      | The timeout bounds only the **caller's wait**, never the shutdown work (KS-exact). `Duration.ZERO` = async signal-and-return; negative → `IllegalArgumentException`; shutdown continues after a `false` return. Returns `true` on **any** terminal state incl. `ERROR` (divergence — KS only `NOT_RUNNING`) — KSC-83                                                                                                                                                          |
| close(CloseOptions)             | ✅      | Mirrors the **top-level KIP-1153** `io.stoatflow.core.CloseOptions` (KS-exact defaults + mutate-and-return-`this` fluents + `@JvmStatic` factories). `GroupMembershipOperation` is a per-call override that **always wins** over the `leaveGroupOnClose` config (incl. its implicit `REMAIN_IN_GROUP` default); no-op under HA `assign()`. The deprecated nested KIP-812 `KafkaStreams.CloseOptions` is **not** mirrored — a port rewrites one line to the factories — KSC-83 |

## Configuration

| Aspect                                                  | Status | Notes                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| ------------------------------------------------------- | ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| StreamsConfig builder                                   | ✅      | Type-safe builder pattern (`StreamsConfig.builder(appId, bootstrap).…build()`) — the primary config model                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `Properties` / `*_CONFIG` constants                     | ✅      | **Canonical KS adapter (ADR-124).** `StreamsConfig.fromProperties(props)` / `fromMap(map)` adapt a KS-keyed `Properties`/`Map` onto the typed builder with **complete coverage of the KS-4.3.0 overlap surface** — all 74 keys, listed one by one in [KS config-key mapping](https://stoatflow.io/#ks-config-key-mapping) below: typed mappings, class-name pluggables (serdes, the three exception handlers, `default.client.supplier`, `rocksdb.config.setter`), client prefixes (`consumer.`/`main.consumer.`/`restore.consumer.`/`producer.`/`admin.`), **unprefixed client keys routed by per-client validity exactly as Kafka Streams does** (`security.protocol` reaches consumer + producer + admin, `acks` only the producer; a prefixed spelling wins, and StoatFlow's forced overrides win over both — `main.consumer.` is a main-consumer-only *scope*, as in KS, not an alias for `consumer.`), and a **lenient 3-tier unknown-key policy** (mapped / recognised-no-op / unknown) + opt-in `stoatflow.config.strict`, which throws on truly-unknown keys instead of forwarding them to every client as custom props. The typed builder/YAML stays the recommended model; the adapter is the canonical KS-compat surface, not a stepping stone to a Map-backed swap (ADR-124). The KS `*_CONFIG` symbol constants and prefix helpers (`consumerPrefix(...)` … `adminClientPrefix(...)`) are also shipped for symbol-level source compatibility (KSC-80). |
| `StreamsConfig(Map)` / `StreamsConfig(Properties)` ctor | ✅      | **KSC-79.** The KS-faithful `new StreamsConfig(props)` / `new StreamsConfig(map)` bootstrap idiom compiles after the import-swap (`Properties` binds the single `Map<*,*>` ctor); it delegates to the same `fromMap(...).build()` adapter as `fromProperties`. A missing `application.id` throws at construction (`IllegalArgumentException`, where KS throws `ConfigException`). The typed `builder(...)` / `fromProperties(...)` paths stay recommended.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `toProperties()`                                        | 🆕     | `StreamsConfig.toProperties()` emits the config back to KS-keyed `Properties` (resolved-view, masked) — for `/config`, diffing, inspect/reconstruct                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `default.timestamp.extractor`                           | ✅      | `StreamsConfig.defaultTimestampExtractor` — a global fallback `TimestampExtractor` for sources with no per-`Consumed` `WatermarkStrategy` (event-time basis only; the watermark default is unaffected)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| Admin client config                                     | ✅      | `StreamsConfig.adminConfig` / `admin.*` prefix — passthrough merged at all AdminClient sites so a secured cluster authenticates                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| KafkaClientSupplier                                     | ✅      | Custom client factory for instrumentation/testing                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| DefaultKafkaClientSupplier                              | ✅      | Standard KafkaConsumer/KafkaProducer/Admin factory                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| RocksDB config                                          | ✅      | Bounded memory by default                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| Serdes                                                  | ✅      | Compatible                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| Default key/value Serdes                                | ✅      | `defaultKeySerde`/`defaultValueSerde` propagate through the DSL                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| Consumer/Producer config                                | ✅      | Pass-through maps                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| application.server                                      | ✅      | `host:port` for the local host reported by interactive-query metadata; defaults from the runtime HTTP host\:port when unset                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| state.cleanup.dir.max.age.ms (KIP-1259)                 | ✅      | Age-gated purge of stale local state at startup (rebuilt from changelog); default disabled                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| state.cleanup.delay.ms                                  | ✅      | Orphaned-store cleanup interval                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |

### KS config-key mapping

Full Kafka Streams 4.3.0 `StreamsConfig` `ConfigDef` coverage — all **74 keys** — as accepted by `StreamsConfig.fromProperties` / `fromMap` (ADR-124). A build-time drift test pins this set against the real Kafka Streams `ConfigDef`, so a key added upstream cannot go unnoticed. Some rows below cover a family of keys.

Status in this table: ✅ mapped · 🆕 added (ADR-124) · 🟡 recognised-no-op (warns, never throws — even under `stoatflow.config.strict`) · 🧩 client pass-through (routed to every client whose own `configNames()` contains it, exactly as Kafka Streams does; a `consumer.` / `producer.` / `admin.` prefix still wins, and StoatFlow's forced overrides win over both).

A port that references the *symbol* rather than the raw string is covered too (KSC-80): `StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG`, `StreamsConfig.METRIC_REPORTER_CLASSES_CONFIG`, the six client-prefix constants (`CONSUMER_PREFIX` / `MAIN_CONSUMER_PREFIX` / `RESTORE_CONSUMER_PREFIX` / `GLOBAL_CONSUMER_PREFIX` / `PRODUCER_PREFIX` / `ADMIN_CLIENT_PREFIX`) and the prefix helpers (`StreamsConfig.consumerPrefix(prop)` … `adminClientPrefix(prop)`) all ship, and the keys they build route or no-op exactly as the raw strings do. `TOPIC_PREFIX` and the topology-optimization *value* constants are not shipped.

| KS key                                                                                                                                                                                              | Status | StoatFlow mapping / reason                                                                                                                                                                                                                                                                                                                                                                 |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `application.id`                                                                                                                                                                                    | ✅      | `applicationId`                                                                                                                                                                                                                                                                                                                                                                            |
| `bootstrap.servers`                                                                                                                                                                                 | ✅      | `bootstrapServers`                                                                                                                                                                                                                                                                                                                                                                         |
| `state.dir`                                                                                                                                                                                         | ✅      | `stateDir` (StoatFlow default `/var/stoatflow/state`)                                                                                                                                                                                                                                                                                                                                      |
| `default.key.serde`                                                                                                                                                                                 | ✅      | `defaultKeySerde` (Class or class-name)                                                                                                                                                                                                                                                                                                                                                    |
| `default.value.serde`                                                                                                                                                                               | ✅      | `defaultValueSerde` (Class or class-name)                                                                                                                                                                                                                                                                                                                                                  |
| `num.stream.threads`                                                                                                                                                                                | ✅      | → `numLanes` (closest single-instance analogue)                                                                                                                                                                                                                                                                                                                                            |
| `processing.guarantee`                                                                                                                                                                              | ✅      | `processingGuarantee` (plus the `exactly_once_beta` alias; StoatFlow defaults to `EXACTLY_ONCE` where KS defaults to `at_least_once` — KSC-84)                                                                                                                                                                                                                                             |
| `application.server`                                                                                                                                                                                | ✅      | `applicationServer` (interactive-query metadata `host:port` — KSC-47)                                                                                                                                                                                                                                                                                                                      |
| `deserialization.exception.handler` (+ deprecated `default.deserialization.exception.handler`)                                                                                                      | 🆕     | `deserializationExceptionHandler` (class-name)                                                                                                                                                                                                                                                                                                                                             |
| `production.exception.handler` (+ deprecated `default.production.exception.handler`)                                                                                                                | 🆕     | `productionExceptionHandler` (class-name)                                                                                                                                                                                                                                                                                                                                                  |
| `processing.exception.handler`                                                                                                                                                                      | 🆕     | `processingExceptionHandler` (class-name)                                                                                                                                                                                                                                                                                                                                                  |
| `default.timestamp.extractor`                                                                                                                                                                       | 🆕     | `defaultTimestampExtractor` (global fallback, event-time basis)                                                                                                                                                                                                                                                                                                                            |
| `default.client.supplier`                                                                                                                                                                           | 🆕     | `kafkaClientSupplier` (class-name)                                                                                                                                                                                                                                                                                                                                                         |
| `rocksdb.config.setter`                                                                                                                                                                             | 🆕     | `rocksDbConfigSetter` (class-name)                                                                                                                                                                                                                                                                                                                                                         |
| `dsl.store.format`                                                                                                                                                                                  | 🆕     | `dslStoreFormat` (`DslStoreFormat.fromConfigString`)                                                                                                                                                                                                                                                                                                                                       |
| `commit.interval.ms`                                                                                                                                                                                | 🆕     | → `barrierIntervalMs` (a semantic seed; the commit cadence is adaptive)                                                                                                                                                                                                                                                                                                                    |
| `buffered.records.per.partition`                                                                                                                                                                    | 🆕     | `bufferedRecordsPerPartition` (keep it `>= max.poll.records`)                                                                                                                                                                                                                                                                                                                              |
| `max.task.idle.ms`                                                                                                                                                                                  | 🆕     | `maxTaskIdleMs` (1:1; both allow `-1`)                                                                                                                                                                                                                                                                                                                                                     |
| `replication.factor`                                                                                                                                                                                | 🆕     | `changelogReplicationFactor`                                                                                                                                                                                                                                                                                                                                                               |
| `state.cleanup.delay.ms` / `state.cleanup.dir.max.age.ms`                                                                                                                                           | 🆕     | `stateCleanupDelayMs` / `stateCleanupDirMaxAgeMs` (KIP-1259)                                                                                                                                                                                                                                                                                                                               |
| `ensure.explicit.internal.resource.naming`                                                                                                                                                          | 🆕     | `ensureExplicitNaming` (StoatFlow defaults to `true`, KS to `false`; the mapping applies only when the key is present)                                                                                                                                                                                                                                                                     |
| `processor.wrapper.class`                                                                                                                                                                           | ⚠️     | KIP-1112 — **supported**. One `ProcessorWrapper` wraps every node, DSL and Processor API alike, at topology-compile time. Full KS fidelity for Processor API nodes; DSL-internal nodes get the record path only, and replacing an internal node is rejected at build time. A wrapper set on the runtime config alone *is* honoured, where KS ignores that case as too late                 |
| `num.standby.replicas`                                                                                                                                                                              | 🟡     | Single-instance: no standby replicas                                                                                                                                                                                                                                                                                                                                                       |
| `acceptable.recovery.lag` / `max.warmup.replicas` / `task.assignor.class` / `probing.rebalance.interval.ms`                                                                                         | 🟡     | Single-instance: no standby, warmup, assignor or probing                                                                                                                                                                                                                                                                                                                                   |
| `rack.aware.assignment.*` / `upgrade.from`                                                                                                                                                          | 🟡     | Single-instance: no rack-aware assignment, no rolling-upgrade protocol                                                                                                                                                                                                                                                                                                                     |
| `group.protocol`                                                                                                                                                                                    | 🟡     | StoatFlow manages consumer-group membership internally                                                                                                                                                                                                                                                                                                                                     |
| `statestore.cache.max.bytes` / `cache.max.bytes.buffering`                                                                                                                                          | 🟡     | StoatFlow's cache caps are **per-store** and **commit-triggering** (`caching.max-entries` / `caching.max-estimated-bytes`), not a global evict budget; global memory is bounded by `state.uncommitted-max-bytes`                                                                                                                                                                           |
| `client.id`                                                                                                                                                                                         | 🟡     | Client ids are derived from `application.id`; override with `consumer.client.id` / `producer.client.id`                                                                                                                                                                                                                                                                                    |
| `task.timeout.ms`                                                                                                                                                                                   | 🟡     | Commit-path waits are bounded by `barrierTimeoutMs` instead                                                                                                                                                                                                                                                                                                                                |
| `topology.optimization`                                                                                                                                                                             | 🟡     | Re-keying is in-memory; KTable source reuse is `autoReuseKTableSourceTopics`                                                                                                                                                                                                                                                                                                               |
| `repartition.purge.interval.ms`                                                                                                                                                                     | 🟡     | In-memory re-keying: there are no repartition topics to purge                                                                                                                                                                                                                                                                                                                              |
| `processing.exception.handler.global.enabled`                                                                                                                                                       | 🟡     | The processing handler is already applied globally                                                                                                                                                                                                                                                                                                                                         |
| `errors.dead.letter.queue.topic.name`                                                                                                                                                               | 🟡     | The DLQ is configured per handler (`DeadLetterQueue*ExceptionHandler`)                                                                                                                                                                                                                                                                                                                     |
| `default.dsl.store` / `dsl.store.suppliers.class`                                                                                                                                                   | 🟡     | Choose the store with `Materialized.withStoreType`                                                                                                                                                                                                                                                                                                                                         |
| `default.list.{key,value}.serde.{inner,type}`                                                                                                                                                       | 🟡     | No `ListSerde` default-type config; use explicit serdes                                                                                                                                                                                                                                                                                                                                    |
| `windowstore.changelog.additional.retention.ms`                                                                                                                                                     | 🟡     | Window changelogs use compaction; no extra retention                                                                                                                                                                                                                                                                                                                                       |
| `window.size.ms` / `windowed.inner.class.serde`                                                                                                                                                     | 🟡     | Deprecated in KS; use the typed `WindowedSerdes`                                                                                                                                                                                                                                                                                                                                           |
| `built.in.metrics.version` / `metric.reporters` / `metrics.*` / `enable.metrics.push`                                                                                                               | 🟡     | Metrics go through Micrometer, not the Kafka metrics subsystem                                                                                                                                                                                                                                                                                                                             |
| `config.providers`                                                                                                                                                                                  | 🟡     | Resolve externalised config before constructing `StreamsConfig`                                                                                                                                                                                                                                                                                                                            |
| `allow.os.group.write.access`                                                                                                                                                                       | 🟡     | StoatFlow does not relax state-dir permissions via config                                                                                                                                                                                                                                                                                                                                  |
| `log.summary.interval.ms`                                                                                                                                                                           | 🟡     | No periodic summary-log toggle                                                                                                                                                                                                                                                                                                                                                             |
| `poll.ms`                                                                                                                                                                                           | 🟡     | Use `consumerPollTimeoutMs`                                                                                                                                                                                                                                                                                                                                                                |
| `security.protocol` / `connections.max.idle.ms` / `metadata.max.age.ms` / `{receive,send}.buffer.bytes` / `request.timeout.ms` / `reconnect.backoff.*` / `retry.backoff.ms` / `metadata.recovery.*` | 🧩     | Client properties, **routed unprefixed to every client they are valid for** (KSC-92, KS parity): `security.protocol` reaches consumer, producer and admin; the rest go by their own `ConfigDef`. A `consumer.` / `main.consumer.` / `producer.` / `admin.` prefix overrides the bare form. The same routing applies to bare `ssl.*` and `sasl.*`, and to producer-only keys such as `acks` |

## Key architectural differences

The runtime-model deltas — single instance vs. rebalancing cluster, lanes vs. partition-bound tasks, in-memory re-keying vs. repartition topics, barrier-based exactly-once, global vs. partition-scoped state — are summarized in the canonical [deltas-at-a-glance table](https://stoatflow.io/docs/concepts/how-stoatflow-differs-from-ks#the-deltas-at-a-glance), with the conceptual treatment on the same page. The rows below are the operational and time-semantics differences that surface while reading this matrix:

| Aspect             | Kafka Streams                     | StoatFlow                                                      |
| ------------------ | --------------------------------- | -------------------------------------------------------------- |
| RocksDB memory     | Unbounded default                 | Bounded default (256 MiB)                                      |
| Window closure     | Stream-time based                 | Watermark-based                                                |
| Grace period       | Retention after close             | Extends acceptance window                                      |
| Watermark strategy | N/A                               | Flink-style WatermarkStrategy                                  |
| Punctuators        | Task-scoped, on the stream thread | Dedicated punctuation lane per sub-topology, full state access |
| Key-based timers   | N/A                               | Flink-inspired TimerService                                    |
| Scheduled sources  | N/A                               | Interval & cron-based emission                                 |
| Runtime control    | N/A                               | `pause()`/`unpause()`                                          |

## Test utilities

`TopologyTestDriver` drives a topology in-memory (no broker) for unit tests. All state stores — including RocksDB-`Materialized` ones — run in-memory in the test driver; real-engine + RocksDB end-to-end coverage uses integration tests with an embedded broker.

### TopologyTestDriver

| Member                                                                                                                    | Status | Notes                                                                                                                       |
| ------------------------------------------------------------------------------------------------------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------- |
| `new TopologyTestDriver(topology)` / `(topology, Properties)` / `(topology, Instant)` / `(topology, Properties, Instant)` | ✅      | KS-shaped constructors; `Properties` via `StreamsConfig.fromProperties`                                                     |
| `createInputTopic(name, Serde/Serializer, …)`                                                                             | ✅      | `Serde` and `Serializer` overloads + 5-arg `(…, Instant, Duration)` start-time/auto-advance form (serdes accepted, ignored) |
| `createOutputTopic(name, Serde/Deserializer, …)`                                                                          | ✅      | `Serde` and `Deserializer` overloads                                                                                        |
| `getAllStateStores()` / `producedTopicNames()`                                                                            | ✅      | All stores by name; the set of produced topics                                                                              |
| `advanceWallClockTime(Duration)` / `get*Store(name)` / `close()`                                                          | ✅      |                                                                                                                             |

### TestRecord

KS positional order `(key, value, headers, timestamp)`; ctors `(K,V)`, `(K,V,Instant)`, `(K,V,Headers[,Instant])`, value-only `(V)`, `(ConsumerRecord)`, `(ProducerRecord)`; bean getters `getKey()/getValue()/getHeaders()/getRecordTime()` + KS accessors `key()/value()/headers()/timestamp()`.

### TestInputTopic

| Method                                            | Status | Notes                                                                                                                    |
| ------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------ |
| pipeInput(key, value [, timestamp] )              | ✅      | `timestamp` as `long` or `Instant`                                                                                       |
| pipeInput(value) / pipeInput(value, Instant)      | ✅      | Value-only (null key) — for value-centric / Avro tests                                                                   |
| pipeKeyValueList / pipeValueList / pipeRecordList | ✅      | Bulk piping; key-value lists use `KeyValue`; timed `(…, Instant, Duration)` overloads; `pipeRecordList` is `out`-variant |
| advanceTime(Duration)                             | ✅      | Advance the input topic's record clock                                                                                   |

### TestOutputTopic

| Method                                 | Status | Notes                                                   |
| -------------------------------------- | ------ | ------------------------------------------------------- |
| readKeyValue() / readKeyValuesToList() | ✅      | Return `KeyValue<K, V>` (`.key` / `.value`), not `Pair` |
| readKeyValuesToMap()                   | ✅      |                                                         |
| readValue() / readRecord()             | ✅      |                                                         |

`KeyValue` is StoatFlow's KS-shaped key-value type (the same one the DSL uses); a KS port rewrites the `KeyValue` import to match.

## High availability (hot-standby)

Not a KS-parity surface — Kafka Streams scales out with standby tasks; StoatFlow's opt-in hot-standby pairs one active with one or more warm standbys. The surface, and the two deliberate gaps:

| Surface                                        | Status              | Notes                                                                                                                                                |
| ---------------------------------------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| `stoatflow.ha.mode` = `off` / `active-standby` | 🚀                  | Deploy-time opt-in; `off` (default) is byte-identical to single-instance                                                                             |
| Active/passive failover                        | 🚀                  | Crash failover in seconds; graceful deploys / `/ha/switch` hand off cleanly                                                                          |
| Split-brain fencing                            | 🚀                  | EOS: broker-enforced producer-epoch fence; ALO: metadata-topic lease                                                                                 |
| `/ha/{status,switch,promote,demote}` endpoints | 🚀                  | Operator shims; SWITCH/PROMOTE/DEMOTE return `409` unless a caught-up target exists (`?force=true` override)                                         |
| `ha.acceptable-recovery-lag`                   | 🚀                  | A single **total** lag sum (default 50000) — not per-task like KS `acceptable.recovery.lag` (10000)                                                  |
| Continuous changelog replication               | 🚀                  | The standby streams committed changelog deltas into local state in real time — like KS standby tasks' continuous restore, for the single-active pair |
| Restore-before-process on promotion            | 🚀                  | A promoting pod restores local state to the committed changelog end **before** processing                                                            |
| Queryable standby (KIP-535 IQ from standby)    | ❌                   | Standbys are non-queryable; store queries on a standby are rejected                                                                                  |
| KS standby tasks / `num.standby.replicas`      | ❌ (different model) | Single-active with warm standbys — not N-instance standby tasks                                                                                      |

See [High availability](https://stoatflow.io/docs/operating/high-availability) for the operational guide.

## Related

- [Comparison matrix](https://stoatflow.io/product/comparison-matrix) — feature-by-feature against Kafka Streams and self-hosted / managed Flink
- [Building](https://stoatflow.io/docs/building) — how-to guidance for the operators above
- [KStream and KTable](https://stoatflow.io/docs/building/kstream-ktable) · [Aggregations](https://stoatflow.io/docs/building/aggregations) · [Windowing](https://stoatflow.io/docs/building/windowing) · [Joins](https://stoatflow.io/docs/building/joins) · [Processor API](https://stoatflow.io/docs/building/processor-api)
- [State stores](https://stoatflow.io/docs/building/state-stores) · [Scheduled sources](https://stoatflow.io/docs/building/scheduled-sources)


# Metrics reference

The `:runtime` module exports metrics on `GET /metrics` in Prometheus text format, backed by a Micrometer `PrometheusMeterRegistry`. This page is the meter catalogue: every StoatFlow meter the current release registers, grouped by area, with its type, tags, and a one-line description. For how to enable, scrape, and configure metrics — recording levels, naming rewrites, PromQL examples — see [Metrics](https://stoatflow.io/docs/runtime/metrics). For what to alert on, see [Observability](https://stoatflow.io/docs/operating/observability).

::callout{color="info" icon="i-lucide-file-check"}
**Contract.** This catalogue lists the stable meter set. The live `/metrics` scrape is authoritative for the exact tag sets and any point-release additions — when in doubt, trust the scrape.
::

## How to read this page

- **Meter** — the canonical dotted metric ID. Prometheus rewrites `.` to `_` and appends the type suffix (`_total` for counters; `_seconds_count` / `_seconds_sum` / `_seconds_max` for timers), so `stoatflow.barrier.commit.latency` is scraped as `stoatflow_barrier_commit_latency_seconds_*`.
- **Tags** — per-meter tags beyond the common ones. Every StoatFlow meter carries `application_id`; the registry-level `application` tag plus anything under `runtime.metrics.common-tags` is applied to every meter on the endpoint, including JVM and Kafka-client meters.
- **Recording level*&#x2A; — meters marked &#x2A;(debug)* in the description only appear with `runtime.metrics.recording-level: debug`; everything else is exported at the default `info` level.
- The commit-path latency timers publish client-side P50/P95/P99 quantiles as `{quantile="…"}` series on the base `_seconds` name — there are no `_bucket` histograms.

## Application and engine

| Meter                                       | Type    | Tags                               | What it measures                                                                                                                                                        |
| ------------------------------------------- | ------- | ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `stoatflow.client.state`                    | Gauge   | —                                  | Application state ordinal: `CREATED=0`, `STARTING=1`, `VALIDATING_STATE=2`, `RESTORING=3`, `RUNNING=4`, `DRAINING=5`, `PAUSED=6`, `STOPPING=7`, `STOPPED=8`, `ERROR=9`. |
| `stoatflow.client.lane.alive`               | Gauge   | —                                  | Currently alive/healthy lanes.                                                                                                                                          |
| `stoatflow.client.uptime.seconds`           | Gauge   | —                                  | Seconds since the application started.                                                                                                                                  |
| `stoatflow.client.start.time.epoch.seconds` | Gauge   | —                                  | Application start time (epoch seconds).                                                                                                                                 |
| `stoatflow.lanes`                           | Gauge   | `sub_topology`, `chain`            | Configured key-affinity lanes per sub-topology; `sum(stoatflow_lanes)` gives the cross-topology total.                                                                  |
| `stoatflow.engine.restart.total`            | Counter | `disposition`, `target`, `trigger` | Successful in-place engine restarts (fault recovery, programmatic, HA demote).                                                                                          |
| `stoatflow.topology.stateless`              | Gauge   | —                                  | 1 if the topology has no state stores, 0 otherwise.                                                                                                                     |
| `stoatflow.topology.event.time.tracking`    | Gauge   | —                                  | 1 if watermark tracking is active, 0 if bypassed.                                                                                                                       |
| `stoatflow.topology.dispatch.mode`          | Gauge   | —                                  | 0 for timestamp-ordered dispatch, 1 for FIFO.                                                                                                                           |

## Lanes and throughput

| Meter                                    | Type    | Tags                    | What it measures                                                                                             |
| ---------------------------------------- | ------- | ----------------------- | ------------------------------------------------------------------------------------------------------------ |
| `stoatflow.lane.queue.size`              | Gauge   | `lane_id`, `chain`      | Current lane queue depth (backpressure signal).                                                              |
| `stoatflow.lane.queue.capacity`          | Gauge   | `sub_topology`, `chain` | Configured lane queue capacity per sub-topology.                                                             |
| `stoatflow.lane.queue.full.total`        | Counter | `lane_id`               | Times a lane queue was full (backpressure events).                                                           |
| `stoatflow.lane.records.processed.total` | Counter | `lane_id`               | Records processed by a lane.                                                                                 |
| `stoatflow.lane.bytes.processed.total`   | Counter | `lane_id`               | Bytes processed by a lane.                                                                                   |
| `stoatflow.lane.process.latency`         | Timer   | `lane_id`               | Per-record processing latency.                                                                               |
| `stoatflow.e2e.latency`                  | Timer   | —                       | Per-record end-to-end latency, record timestamp to processing completion (KS `record-e2e-latency` analogue). |

Lane IDs follow the `{subtopology}_{lane}` convention — e.g. `0_0`, `1_7`. Aggregate with a regex matcher: `sum(stoatflow_lane_queue_size{lane_id=~"1_.*"})`.

## Commit barriers and transactions

| Meter                                             | Type                | Tags             | What it measures                                                                                                                                                           |
| ------------------------------------------------- | ------------------- | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `stoatflow.barrier.initiated.total`               | Counter             | —                | Barriers initiated.                                                                                                                                                        |
| `stoatflow.barrier.completed.total`               | Counter             | —                | Barriers successfully completed.                                                                                                                                           |
| `stoatflow.barrier.failed.total`                  | Counter             | `error_type`     | Barrier failures, by error type.                                                                                                                                           |
| `stoatflow.barrier.in.flight`                     | Gauge               | —                | Currently pending barriers.                                                                                                                                                |
| `stoatflow.barrier.latency`                       | Timer               | —                | Barrier initiation to completion.                                                                                                                                          |
| `stoatflow.barrier.alignment.latency`             | Timer               | —                | Time for all lanes to receive the barrier.                                                                                                                                 |
| `stoatflow.lane.holdback.records`                 | DistributionSummary | `sub_topology`   | Records held back and replayed per barrier at a sub-topology boundary — receiver-side epoch hold-back that keeps the commit a consistent cut across the cascade (ADR-135). |
| `stoatflow.lane.holdback.drains.total`            | Counter             | `sub_topology`   | Non-empty hold-back drains — a receiving lane replayed stashed ahead-of-epoch records after crossing its barrier (ADR-135).                                                |
| `stoatflow.barrier.commit.latency`                | Timer               | —                | Commit latency: producer drain + Kafka TX + state flush (KS `commit-latency` analogue).                                                                                    |
| `stoatflow.barrier.records.committed.total`       | Counter             | —                | Records committed across barriers.                                                                                                                                         |
| `stoatflow.barrier.bytes.committed.total`         | Counter             | —                | Bytes committed across barriers.                                                                                                                                           |
| `stoatflow.barrier.subtopology.received.total`    | Counter             | `subtopology_id` | Barriers received per sub-topology. &#x2A;(debug)*                                                                                                                         |
| `stoatflow.barrier.subtopology.pending.lanes`     | Gauge               | `subtopology_id` | Pending lane completions per sub-topology. &#x2A;(debug)*                                                                                                                  |
| `stoatflow.barrier.interval.target.ms`            | Gauge               | —                | Current target barrier-to-barrier interval (ms).                                                                                                                           |
| `stoatflow.barrier.commit.duration.estimate.ms`   | Gauge               | —                | Smoothed estimate of commit duration (ms).                                                                                                                                 |
| `stoatflow.barrier.epoch.max.records`             | Gauge               | —                | Current dynamic epoch record limit.                                                                                                                                        |
| `stoatflow.barrier.epoch.actual.records`          | Gauge               | —                | Records dispatched in the last epoch.                                                                                                                                      |
| `stoatflow.barrier.epoch.memory.cap`              | Gauge               | —                | Memory-derived epoch record cap; `-1` = uncapped.                                                                                                                          |
| `stoatflow.barrier.epoch.bytes.per.record`        | Gauge               | —                | Smoothed uncommitted state bytes per record (feeds the memory cap).                                                                                                        |
| `stoatflow.s1.epoch.records.smoothed`             | Gauge               | —                | Smoothed epoch-records average used by the epoch-size controller.                                                                                                          |
| `stoatflow.s1.cap.grow.events.total`              | Counter             | `reason`         | Epoch-cap grow events by binding signal (`record_count`, `memory_pressure`, `stall`, `soft_binding`, `time_grow`).                                                         |
| `stoatflow.s1.cap.decay.events.total`             | Counter             | —                | Epoch-cap decay events.                                                                                                                                                    |
| `stoatflow.barrier.trigger.total`                 | Counter             | `trigger`        | Barrier creation count by trigger: `TIME`, `RECORD_COUNT`, `MEMORY_PRESSURE`, `CACHE_PRESSURE`.                                                                            |
| `stoatflow.dispatcher.stall.latency`              | Timer               | —                | Dispatcher stall time waiting for an active commit after hitting the epoch record limit.                                                                                   |
| `stoatflow.barrier.commit.changelog.latency`      | Timer               | —                | Aggregate changelog serialization+send latency per commit epoch.                                                                                                           |
| `stoatflow.barrier.commit.state.flush.latency`    | Timer               | —                | Aggregate state-store flush latency per commit epoch.                                                                                                                      |
| `stoatflow.barrier.commit.producer.flush.latency` | Timer               | —                | Pre-commit producer flush latency.                                                                                                                                         |
| `stoatflow.barrier.commit.send.offsets.latency`   | Timer               | —                | `sendOffsetsToTransaction` RPC latency.                                                                                                                                    |
| `stoatflow.barrier.commit.queue.wait.latency`     | Timer               | —                | Barrier wait in the commit queue between alignment and commit-thread pickup.                                                                                               |
| `stoatflow.barrier.commit.post.tx.latency`        | Timer               | —                | Post-transaction phase latency (guard wait + epoch prepare + send-buffer drain).                                                                                           |
| `stoatflow.barrier.commit.window.duration`        | Timer               | —                | Duration of the commit-in-progress blocking window. &#x2A;(debug)*                                                                                                         |
| `stoatflow.commit.state.flush.guard.wait`         | Timer               | —                | Time the next transaction waited on the previous epoch's async state flush.                                                                                                |
| `stoatflow.commit.state.flush.async.latency`      | Timer               | —                | Async (off-commit-path) state flush latency.                                                                                                                               |
| `stoatflow.commit.state.flush.overlap`            | Counter             | —                | Commits that overlapped a still-running previous flush.                                                                                                                    |
| `stoatflow.commit.stall.detected.total`           | Counter             | —                | Commit-pipeline stalls detected by the in-process watchdog.                                                                                                                |
| `stoatflow.commit.stall.duration.ms`              | Gauge               | —                | Age of the most recently observed commit stall (ms).                                                                                                                       |
| `stoatflow.transaction.begin.total`               | Counter             | —                | Kafka transactions begun.                                                                                                                                                  |
| `stoatflow.transaction.commit.total`              | Counter             | —                | Kafka transactions committed.                                                                                                                                              |
| `stoatflow.transaction.abort.total`               | Counter             | —                | Kafka transactions aborted.                                                                                                                                                |
| `stoatflow.transaction.commit.latency`            | Timer               | —                | Kafka transaction commit latency.                                                                                                                                          |
| `stoatflow.alo.commit.delegation.wait`            | Timer               | —                | Commit delegation wait (at-least-once mode only).                                                                                                                          |
| `stoatflow.alo.commit.sync.latency`               | Timer               | —                | `commitSync` latency (at-least-once mode only).                                                                                                                            |
| `stoatflow.commit.barrier.idle.throttled`         | Gauge               | —                | `1` while the barrier scheduler is parked on the stretched warm-idle cadence, else `0` (ADR-134).                                                                          |

## State stores and caching

Most store meters are per-store (`store_name`) and require `recording-level: debug`; the changelog counters and `store.num.keys` are exported at `info`.

| Meter                                             | Type                | Tags                       | What it measures                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| ------------------------------------------------- | ------------------- | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `stoatflow.store.get.total`                       | Counter             | `store_name`               | Get operations. &#x2A;(debug)*                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `stoatflow.store.put.total`                       | Counter             | `store_name`               | Put operations. &#x2A;(debug)*                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `stoatflow.store.delete.total`                    | Counter             | `store_name`               | Delete operations. &#x2A;(debug)*                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `stoatflow.store.get.latency`                     | Timer               | `store_name`               | Point-read latency. &#x2A;(debug)*                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `stoatflow.store.put.latency`                     | Timer               | `store_name`               | Write latency. &#x2A;(debug)*                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `stoatflow.store.delete.latency`                  | Timer               | `store_name`               | Delete latency. &#x2A;(debug)*                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `stoatflow.store.range.latency`                   | Timer               | `store_name`               | Range query latency. &#x2A;(debug)*                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `stoatflow.store.all.latency`                     | Timer               | `store_name`               | Full-iteration latency. &#x2A;(debug)*                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `stoatflow.store.flush.latency`                   | Timer               | `store_name`               | Store flush latency. &#x2A;(debug)*                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `stoatflow.store.cache.hit.total`                 | Counter             | `store_name`               | Store cache hits. &#x2A;(debug)*                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `stoatflow.store.cache.miss.total`                | Counter             | `store_name`               | Store cache misses. &#x2A;(debug)*                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `stoatflow.store.changelog.records.written.total` | Counter             | `store_name`               | Changelog records written.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `stoatflow.store.changelog.bytes.written.total`   | Counter             | `store_name`               | Changelog bytes written.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `stoatflow.store.commit.changelog.latency`        | Timer               | `store_name`               | Per-store changelog serialization+send latency during the commit epoch. &#x2A;(debug)*                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `stoatflow.store.commit.flush.latency`            | Timer               | `store_name`               | Per-store flush latency during the commit epoch. &#x2A;(debug)*                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `stoatflow.store.flush.entries`                   | DistributionSummary | `store_name`               | Entry count per epoch flush. &#x2A;(debug)*                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `stoatflow.store.flush.sub.batches`               | DistributionSummary | `store_name`               | Sub-batch count per flush (1 = sequential, N = parallel). &#x2A;(debug)*                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `stoatflow.store.num.keys`                        | Gauge               | `store_name`, `store_type` | Committed key count of an **in-memory** store (KIP-1250 parity); silent for RocksDB-backed stores.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `stoatflow.store.expired.records`                 | Counter             | `store_name`, `store_type` | Records event-time-expired (pruned past retention) from an in-memory window or session store; `0` for plain key-value stores (ADR-133).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `stoatflow.store.format.downgrade.total`          | Counter             | `outcome`                  | State stores whose on-disk format was newer than the topology asked for — record headers (KIP-1271) turned off over a store that carries them. Reported once per affected store at startup, before it opens. `outcome` is `dropped-empty` (the headers keyspace was empty and was dropped in place — free, no restore), `wiped` (acknowledged via `stoatflow.state.format-downgrade: wipe-and-restore` in YAML, or the `state.format.downgrade` Properties key; local state deleted and rebuilt from the changelog) or `refused` (startup failed). &#x2A;*Do not alert on `refused`:** the process exits moments later, so a scrape almost never lands in the window — treat a refusal as a log/crash-loop signal, and use this counter for the `dropped-empty` and `wiped` outcomes, which run on a process that goes on to serve. See [state stores](https://stoatflow.io/docs/building/state-stores). |
| `stoatflow.fk.enumeration.behind.scans`           | Counter             | `store_name`               | Foreign-key join evaluations that enumerated the subscription store as-of an earlier committed epoch because the reader was behind (ADR-136).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `stoatflow.suppress.buffer.count`                 | Gauge               | `buffer_name`              | Records currently held in a suppression buffer.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `stoatflow.suppress.buffer.bytes`                 | Gauge               | `buffer_name`              | Bytes currently held in a suppression buffer.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `stoatflow.suppress.emit.total`                   | Counter             | `buffer_name`              | Records released from a suppression buffer.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |

Suppression buffers never drop records, so there is no drop counter here: a late record is dropped *upstream* by the windowed-aggregation guards and counted as `stoatflow.dropped.records.total` with `reason=late_window` or `late_session`. On a full buffer, `EAGER` early-emits the oldest entry (a forward, not a drop) while `STRICT` and `maxBytes` throw.

## RocksDB

Opt-in, and off by default — set `stoatflow.rocks-db.metrics.enabled: true` for the property and cache gauges (`info`, cheap), and additionally `statistics-enabled: true` for the ticker, ratio and histogram meters (`debug`; enabling RocksDB statistics costs roughly 5–10% on the write path, and takes effect at store open, so it needs a restart). A recorder thread samples every `interval-ms` (default 60 s).

This section elides related meters into one row — `compaction.time.{avg,min,max}.ms` is three meters, not one. That is a deliberate exception to the rest of this page, which spells every name out: the family is \~43 series *per store*, and elision keeps it readable. All meters carry `store_name` except the three global shared-cache gauges.

Two caveats worth knowing before you build a dashboard on these. `open.files` is **not** published — it is derived from a RocksDB ticker (`NO_FILE_CLOSES`) that no longer exists in the bundled RocksJava. And `*.min.ms` is **JNI-only**: the default FFM backend reads statistics from a text dump that carries no MIN token, so those gauges stay silent unless you set `stoatflow.rocks-db.backend: JNI`. Averages, maxima and totals are unaffected.

| Meter                                                                                             | Type    | Tags         | What it measures                                                                                                                                                                                          |
| ------------------------------------------------------------------------------------------------- | ------- | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `stoatflow.rocksdb.{bytes.written,bytes.read}.total`                                              | Counter | `store_name` | Bytes written / read by RocksDB. &#x2A;(debug)*                                                                                                                                                           |
| `stoatflow.rocksdb.memtable.bytes.flushed.total`                                                  | Counter | `store_name` | Bytes written to disk by memtable flushes. &#x2A;(debug)*                                                                                                                                                 |
| `stoatflow.rocksdb.compaction.bytes.{read,written}.total`                                         | Counter | `store_name` | Bytes read / written by compaction — the write-amplification signal. &#x2A;(debug)*                                                                                                                       |
| `stoatflow.rocksdb.file.errors.total`                                                             | Counter | `store_name` | RocksDB file errors. &#x2A;(debug)*                                                                                                                                                                       |
| `stoatflow.rocksdb.memtable.hit.ratio`                                                            | Gauge   | `store_name` | Memtable hit ratio over the last sample interval. &#x2A;(debug)*                                                                                                                                          |
| `stoatflow.rocksdb.block.cache.{,data.,index.,filter.}hit.ratio`                                  | Gauge   | `store_name` | Block-cache hit ratio over the last sample interval — overall, then per block role. A falling data-block ratio with a healthy index ratio usually means the working set outgrew the cache. &#x2A;(debug)* |
| `stoatflow.rocksdb.compaction.time.{avg,min,max}.ms`                                              | Gauge   | `store_name` | Compaction time. &#x2A;(debug)*                                                                                                                                                                           |
| `stoatflow.rocksdb.compaction.time.total.ms`                                                      | Counter | `store_name` | Cumulative compaction time. &#x2A;(debug)*                                                                                                                                                                |
| `stoatflow.rocksdb.flush.time.{avg,min,max}.ms`                                                   | Gauge   | `store_name` | Memtable-flush time. &#x2A;(debug)*                                                                                                                                                                       |
| `stoatflow.rocksdb.flush.time.total.ms`                                                           | Counter | `store_name` | Cumulative memtable-flush time. &#x2A;(debug)*                                                                                                                                                            |
| `stoatflow.rocksdb.write.stall.duration.avg.ms`                                                   | Gauge   | `store_name` | Write-stall duration. Any sustained value is RocksDB backpressuring your commit path. &#x2A;(debug)*                                                                                                      |
| `stoatflow.rocksdb.write.stall.duration.total.ms`                                                 | Counter | `store_name` | Cumulative time writes were stalled. &#x2A;(debug)*                                                                                                                                                       |
| `stoatflow.rocksdb.num.immutable.mem.table`                                                       | Gauge   | `store_name` | Immutable memtables not yet flushed.                                                                                                                                                                      |
| `stoatflow.rocksdb.{cur.size.active.mem.table,cur.size.all.mem.tables,size.all.mem.tables}.bytes` | Gauge   | `store_name` | Memtable sizes: active, active plus unflushed, and including pinned.                                                                                                                                      |
| `stoatflow.rocksdb.num.entries.{active.mem.table,imm.mem.tables}`                                 | Gauge   | `store_name` | Entries in the active / unflushed immutable memtables.                                                                                                                                                    |
| `stoatflow.rocksdb.num.deletes.{active.mem.table,imm.mem.tables}`                                 | Gauge   | `store_name` | Delete entries in the active / unflushed immutable memtables.                                                                                                                                             |
| `stoatflow.rocksdb.{mem.table.flush.pending,compaction.pending}`                                  | Gauge   | `store_name` | 1 while a flush / compaction is pending, else 0.                                                                                                                                                          |
| `stoatflow.rocksdb.num.running.{flushes,compactions}`                                             | Gauge   | `store_name` | Currently running flushes / compactions.                                                                                                                                                                  |
| `stoatflow.rocksdb.estimate.pending.compaction.bytes`                                             | Gauge   | `store_name` | Bytes compaction must rewrite to bring the LSM tree back to target size. A climbing value means compaction is losing ground.                                                                              |
| `stoatflow.rocksdb.{total.sst.files.size,live.sst.files.size}.bytes`                              | Gauge   | `store_name` | SST bytes on disk: all files, and those in the latest version.                                                                                                                                            |
| `stoatflow.rocksdb.num.live.versions`                                                             | Gauge   | `store_name` | Live versions — more than one means an open iterator is pinning old files.                                                                                                                                |
| `stoatflow.rocksdb.estimate.num.keys`                                                             | Gauge   | `store_name` | Estimated live keys. This is the RocksDB counterpart of `store.num.keys`, which covers in-memory stores only.                                                                                             |
| `stoatflow.rocksdb.estimate.table.readers.mem.bytes`                                              | Gauge   | `store_name` | Memory held by table readers (indexes and filters), outside the block cache.                                                                                                                              |
| `stoatflow.rocksdb.background.errors`                                                             | Gauge   | `store_name` | Accumulated background errors. Alert on any non-zero value.                                                                                                                                               |
| `stoatflow.rocksdb.block.cache.{capacity,usage,pinned.usage}.bytes`                               | Gauge   | `store_name` | The shared block cache, repeated per store so Kafka Streams dashboards port unchanged.                                                                                                                    |
| `stoatflow.rocksdb.shared.block.cache.{capacity,usage,pinned.usage}.bytes`                        | Gauge   | —            | The shared block cache, reported once. StoatFlow runs **one** LRU cache across all stores, so these are the real numbers — prefer them over the per-store duplicates above.                               |

## State restoration

| Meter                                          | Type    | Tags         | What it measures                                                                                                                                                                       |
| ---------------------------------------------- | ------- | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `stoatflow.restoration.in.progress`            | Gauge   | —            | 1 while restoration is running, 0 otherwise.                                                                                                                                           |
| `stoatflow.restoration.stores.total`           | Gauge   | —            | Total stores to restore.                                                                                                                                                               |
| `stoatflow.restoration.stores.completed`       | Gauge   | —            | Stores that have finished restoring.                                                                                                                                                   |
| `stoatflow.restoration.records.restored.total` | Counter | `store_name` | Records restored, per store.                                                                                                                                                           |
| `stoatflow.restoration.bytes.restored.total`   | Counter | `store_name` | Changelog bytes restored, per store — serialized key plus value as read off the changelog, excluding record and batch overhead. Compare against changelog volume to size a cold start. |
| `stoatflow.restoration.duration.seconds`       | Timer   | `store_name` | Per-store restoration duration, recorded once when the store finishes. A store's changelog partitions restore in parallel, so this is the slowest partition, not their sum.            |

## Consumer, producer, and buffers

| Meter                                         | Type                | Tags                           | What it measures                                                    |
| --------------------------------------------- | ------------------- | ------------------------------ | ------------------------------------------------------------------- |
| `stoatflow.consumer.poll.latency`             | Timer               | —                              | Consumer poll latency.                                              |
| `stoatflow.consumer.poll.total`               | Counter             | —                              | Poll invocations.                                                   |
| `stoatflow.consumer.poll.records`             | DistributionSummary | —                              | Records per poll.                                                   |
| `stoatflow.consumer.records.consumed.total`   | Counter             | `topic`, `partition`           | Records consumed.                                                   |
| `stoatflow.consumer.bytes.consumed.total`     | Counter             | `topic`, `partition`           | Bytes consumed.                                                     |
| `stoatflow.consumer.lag.records`              | Gauge               | `topic`, `partition`           | Consumer lag, in records.                                           |
| `stoatflow.consumer.assigned.partitions`      | Gauge               | —                              | Number of assigned partitions.                                      |
| `stoatflow.consumer.partitions.paused`        | Gauge               | —                              | Currently paused partitions.                                        |
| `stoatflow.consumer.partitions.delay.pending` | Gauge               | —                              | Partitions with a pending resume delay.                             |
| `stoatflow.consumer.partition.paused.total`   | Counter             | `topic`, `partition`, `reason` | Partition pause events, by reason.                                  |
| `stoatflow.consumer.partition.resumed.total`  | Counter             | `topic`, `partition`, `reason` | Partition resume events, by reason.                                 |
| `stoatflow.producer.records.produced.total`   | Counter             | `topic`                        | Records produced.                                                   |
| `stoatflow.producer.bytes.produced.total`     | Counter             | `topic`                        | Bytes produced.                                                     |
| `stoatflow.producer.send.latency`             | Timer               | `topic`                        | Producer send latency. &#x2A;(debug)*                               |
| `stoatflow.buffer.partition.size`             | Gauge               | `topic`, `partition`           | Current partition buffer depth. &#x2A;(debug)*                      |
| `stoatflow.buffer.partition.capacity`         | Gauge               | `topic`, `partition`           | Partition buffer capacity. &#x2A;(debug)*                           |
| `stoatflow.buffer.offer.total`                | Gauge               | `topic`, `partition`           | Cumulative records offered to the partition buffer. &#x2A;(debug)*  |
| `stoatflow.buffer.poll.total`                 | Gauge               | `topic`, `partition`           | Cumulative records polled from the partition buffer. &#x2A;(debug)* |
| `stoatflow.buffer.oldest.age.ms`              | Gauge               | `topic`, `partition`           | Age of the oldest buffered record. &#x2A;(debug)*                   |
| `stoatflow.buffer.scheduled.source.size`      | Gauge               | —                              | Scheduled-source dispatch buffer depth.                             |
| `stoatflow.buffer.timer.event.size`           | Gauge               | —                              | Event-time timer buffer depth.                                      |
| `stoatflow.buffer.timer.processing.size`      | Gauge               | —                              | Processing-time timer buffer depth.                                 |

## Dispatcher internals

| Meter                                      | Type                | Tags         | What it measures                                                        |
| ------------------------------------------ | ------------------- | ------------ | ----------------------------------------------------------------------- |
| `stoatflow.dispatcher.batch.size`          | DistributionSummary | —            | Records per dispatch batch. &#x2A;(debug)*                              |
| `stoatflow.dispatcher.batch.held.total`    | Counter             | —            | Batches held because a target lane lacked capacity. &#x2A;(debug)*      |
| `stoatflow.dispatcher.dispatch.latency`    | Timer               | —            | Per-batch dispatch latency.                                             |
| `stoatflow.parallel.deser.activated.total` | Counter             | —            | Times parallel deserialization was activated.                           |
| `stoatflow.queue.offer.failure.total`      | Counter             | `queue_type` | Internal queue offer rejections (`lane`, `partition_buffer`, `commit`). |
| `stoatflow.coordinator.select.latency`     | Timer               | —            | Time spent selecting the next timestamp-ordered batch. &#x2A;(debug)*   |
| `stoatflow.coordinator.partition.switches` | DistributionSummary | —            | Partition switches per selected batch. &#x2A;(debug)*                   |

## Watermarks and event time

| Meter                                             | Type    | Tags                          | What it measures                                                                  |
| ------------------------------------------------- | ------- | ----------------------------- | --------------------------------------------------------------------------------- |
| `stoatflow.watermark.current`                     | Gauge   | —                             | Current global watermark (epoch ms).                                              |
| `stoatflow.watermark.lag.ms`                      | Gauge   | —                             | Wall-clock time minus watermark (event-time lag).                                 |
| `stoatflow.watermark.advance.total`               | Counter | —                             | Watermark advancement events.                                                     |
| `stoatflow.watermark.partition.current`           | Gauge   | `topic`, `partition`          | Per-partition watermark. &#x2A;(debug)*                                           |
| `stoatflow.watermark.late.records.total`          | Counter | `topic`, `partition`          | Records arriving after the watermark. &#x2A;(debug)*                              |
| `stoatflow.watermark.out.of.orderness.max`        | Gauge   | —                             | Maximum observed out-of-orderness (ms). &#x2A;(debug)*                            |
| `stoatflow.watermark.alignment.drift.ms`          | Gauge   | `topic`, `partition`, `group` | Per-partition drift above the alignment group's minimum watermark. &#x2A;(debug)* |
| `stoatflow.watermark.alignment.group.min`         | Gauge   | `group`                       | Minimum watermark across the alignment group (epoch ms). &#x2A;(debug)*           |
| `stoatflow.watermark.alignment.partitions.paused` | Gauge   | `group`                       | Partitions currently paused for watermark alignment.                              |

## Timers, punctuators, and scheduled sources

| Meter                                              | Type    | Tags                            | What it measures                                             |
| -------------------------------------------------- | ------- | ------------------------------- | ------------------------------------------------------------ |
| `stoatflow.timer.registered.total`                 | Counter | `type`                          | Keyed timers registered (`event_time`, `processing_time`).   |
| `stoatflow.timer.fired.total`                      | Counter | `type`                          | Timers fired.                                                |
| `stoatflow.timer.deleted.total`                    | Counter | `type`                          | Timers explicitly deleted.                                   |
| `stoatflow.timer.queue.size`                       | Gauge   | `type`                          | Timer dispatch-buffer depth, by timer type.                  |
| `stoatflow.timer.fire.latency`                     | Timer   | `type`                          | Delay between scheduled and actual fire time. &#x2A;(debug)* |
| `stoatflow.timer.fire.late.total`                  | Counter | `type`                          | Timers that fired late. &#x2A;(debug)*                       |
| `stoatflow.punctuator.invocation.total`            | Counter | `punctuator_id`, `type`, `mode` | Punctuator invocations.                                      |
| `stoatflow.punctuator.skip.total`                  | Counter | `punctuator_id`                 | Punctuator invocations skipped (previous run still active).  |
| `stoatflow.punctuator.active`                      | Gauge   | —                               | Currently registered punctuators.                            |
| `stoatflow.punctuator.execution.latency`           | Timer   | `punctuator_id`, `type`         | Punctuator execution time. &#x2A;(debug)*                    |
| `stoatflow.scheduled.source.trigger.total`         | Counter | `source_name`, `type`           | Scheduled source triggers.                                   |
| `stoatflow.scheduled.source.records.emitted.total` | Counter | `source_name`                   | Records emitted by scheduled sources.                        |

## Error handling

| Meter                                                | Type    | Tags                     | What it measures                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| ---------------------------------------------------- | ------- | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `stoatflow.error.total`                              | Counter | `error_type`, `topic`    | Errors by type (`topic` omitted when unknown).                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `stoatflow.error.typed.total`                        | Counter | `category`, `error_type` | Categorised errors (e.g. production failures), by category and type.                                                                                                                                                                                                                                                                                                                                                                                            |
| `stoatflow.dropped.records.total`                    | Counter | `reason`                 | Records dropped before processing (KS `dropped-records` parity). The `reason` tag is the semantics — six values: `null_key` (a record whose key is null, or serializes to null, reaching an aggregation), `late_window` / `late_session` (a record arriving after its window or session already closed), `join_null_key` / `join_null_value` (a join input the join cannot match on), `table_source_null_key` (a null-keyed record on a `KTable` source topic). |
| `stoatflow.sink.records.dropped.total`               | Counter | `sink_name`, `topic`     | Records dropped because a custom partitioner returned no partition (KIP-837).                                                                                                                                                                                                                                                                                                                                                                                   |
| `stoatflow.sink.records.multicast.total`             | Counter | `sink_name`, `topic`     | Output records produced via multicast — a custom partitioner returned multiple partitions (KIP-837).                                                                                                                                                                                                                                                                                                                                                            |
| `stoatflow.dlq.poison.quarantined.total`             | Counter | —                        | Source records quarantined as poison after an asynchronous broker-side production failure returned `continue`. The epoch's source offsets are held and the epoch is replayed with exactly these records skipped, so the epoch's other records survive — but a quarantined record loses **all** of its outputs, across every sink it fans out to. See [Error-handling model](https://stoatflow.io/docs/concepts/error-handling-model).                           |
| `stoatflow.dlq.poison.replays.total`                 | Counter | —                        | Poison-epoch replays: in-place engine restarts that reprocess an aborted epoch with the quarantined records skipped. Bounded by `stoatflow.commit-barrier.max-poison-replays` within `poison-replay-window-ms`. That budget is **in-memory** — exhausting it ends the process and the pod restarts with a clean budget — so `continue` is bounded per process, never end to end. Alert on this meter.                                                           |
| `stoatflow.dlq.poison.replay.budget.exhausted.total` | Counter | —                        | The replay budget ran out: `max-poison-replays` replays were already consumed within `poison-replay-window-ms`, so the instance shut down with the source offsets held rather than replaying again. &#x2A;*Terminal — any non-zero value is an incident.** Alert on it alongside `…replays.total`, which counts replays and therefore *stops* incrementing at the moment a non-converging poison starts crash-looping the pod.                                  |
| `stoatflow.dlq.poison.quarantine.size`               | Gauge   | —                        | Source offsets currently quarantined and skipped by a replay. `0` in steady state; entries retire as soon as a commit moves permanently past them.                                                                                                                                                                                                                                                                                                              |

## High availability

Registered only when [hot standby](https://stoatflow.io/docs/operating/high-availability) is enabled (`ha.mode != off`).

| Meter                                              | Type    | Tags      | What it measures                                                                                                                                                                                                            |
| -------------------------------------------------- | ------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `stoatflow.ha.role`                                | Gauge   | —         | `1` when this pod is the active, `0` when standby.                                                                                                                                                                          |
| `stoatflow.ha.standby.replication.lag.records`     | Gauge   | —         | Total committed-changelog lag in records across all changelog partitions; `0` when active or caught up.                                                                                                                     |
| `stoatflow.ha.standby.replication.applied.records` | Counter | —         | Cumulative changelog records the standby applier has applied; `rate(...)` = apply throughput.                                                                                                                               |
| `stoatflow.ha.standby.replication.lag.ms`          | Gauge   | —         | Standby replication lag in milliseconds.                                                                                                                                                                                    |
| `stoatflow.ha.peer.freshness.ms`                   | Gauge   | —         | Time since the freshest observed peer's last heartbeat.                                                                                                                                                                     |
| `stoatflow.ha.peers`                               | Gauge   | —         | Number of observed HA peers.                                                                                                                                                                                                |
| `stoatflow.ha.restart.required`                    | Gauge   | —         | `1` when this pod needs a restart (e.g. after a source-topic partition-count change).                                                                                                                                       |
| `stoatflow.ha.restarting`                          | Gauge   | —         | `1` while this pod is restarting its engine in place to absorb a processing fault. The pod stays `ACTIVE` throughout, so `stoatflow.ha.role` does not move — this is what distinguishes "absorbing a fault" from "healthy". |
| `stoatflow.ha.redundancy.ready.standbys`           | Gauge   | —         | Caught-up (`READY_STANDBY`) fresh standby count, including self.                                                                                                                                                            |
| `stoatflow.ha.redundancy.below.desired`            | Gauge   | —         | `1` when the active has fewer ready standby spares than `ha.desired-standbys`.                                                                                                                                              |
| `stoatflow.ha.token.epoch`                         | Gauge   | —         | Current promotion-token epoch on the metadata topic.                                                                                                                                                                        |
| `stoatflow.ha.election.outcome`                    | Counter | `outcome` | Cumulative election outcomes: `won`, `reclaimed`, `stood_down_fresh`, `stood_down_lost`.                                                                                                                                    |
| `stoatflow.ha.election.latency.ms`                 | Gauge   | —         | Last claim-to-ACTIVE promotion latency (ms); `-1` until the first promotion.                                                                                                                                                |

## License

Registered once at startup; exported at every recording level. See [License configuration](https://stoatflow.io/docs/getting-started/license-configuration).

| Meter                                                        | Type    | Tags     | What it measures                                                       |
| ------------------------------------------------------------ | ------- | -------- | ---------------------------------------------------------------------- |
| `stoatflow.license.valid`                                    | Gauge   | `tier`   | `1` under a VALID or GRACE\_PERIOD license, else `0`.                  |
| `stoatflow.license.expiry_timestamp_seconds`                 | Gauge   | —        | Unix epoch seconds of the license entitlement deadline.                |
| `stoatflow.license.days_remaining`                           | Gauge   | —        | Whole days until expiry; negative within the post-expiry grace window. |
| `stoatflow.license.grace_remaining_seconds`                  | Gauge   | —        | Seconds left in the offline-grace budget; floored at 0.                |
| `stoatflow.license.heartbeat_consecutive_failures`           | Gauge   | —        | Heartbeat ticks failed since the last success.                         |
| `stoatflow.license.heartbeat_last_success_timestamp_seconds` | Gauge   | —        | Unix epoch seconds of the last successful heartbeat.                   |
| `stoatflow.license.validations_total`                        | Counter | `result` | Cold-start validation outcomes, by result.                             |
| `stoatflow.license.heartbeats_total`                         | Counter | `result` | Heartbeat tick outcomes, by result.                                    |

## JVM, system, and Kafka-client meters

Beyond the `stoatflow.*` meters above, the same endpoint carries:

- **JVM and system meters** (`jvm_*`, `system_*`, `process_*`) — bound by default (`runtime.metrics.bind-jvm-metrics: true`): heap/non-heap memory, GC pauses, thread counts, classloading, CPU usage.
- **Native Kafka client meters** (`kafka_consumer_*`, `kafka_producer_*`) — the Kafka clients' own JMX metrics surfaced through Micrometer.

These carry the registry common tags (`application` plus your `common-tags`) but not `application_id`. See [Metrics](https://stoatflow.io/docs/runtime/metrics) for details.

## Next steps

- **[Metrics](https://stoatflow.io/docs/runtime/metrics)** — enabling, scraping, recording levels, naming and tag conventions, PromQL examples.
- **[Observability](https://stoatflow.io/docs/operating/observability)** — wiring `/metrics` into a monitoring stack and what to alert on.
- **[Configuration reference](https://stoatflow.io/docs/reference/configuration-reference)** — the `runtime.metrics.*` keys.


# Glossary

Definitions of the vocabulary used throughout the StoatFlow documentation. Each entry links to the page that covers the concept in depth.

## Concepts

### At-least-once (ALO)

A delivery guarantee under which every record is processed at least once, but a crash may cause some records to be processed again and re-emitted downstream. StoatFlow's at-least-once mode drops the Kafka transaction: the runtime uses a non-transactional producer and commits consumer offsets directly, on a faster cadence. Trades duplicate-tolerance for a lower latency floor. See [Exactly-once and at-least-once](https://stoatflow.io/docs/concepts/exactly-once).

### Changelog

A compacted Kafka topic that records every write to a state store. Because the topic is compacted by key, the latest value for every key is preserved indefinitely without unbounded growth. On restart, the runtime rebuilds local state by reading the changelog. See [State stores](https://stoatflow.io/docs/building/state-stores) and the [Architecture](https://stoatflow.io/docs/concepts/architecture) page.

### Commit barrier

A marker — not a data record — that the runtime periodically injects into the processing lanes. As records flow through the topology the barrier flows with them, cascading across each sub-topology boundary in turn; when it has reached every lane, the runtime executes one Kafka transaction that atomically commits the epoch's state writes, output records, and input offsets. Across a sub-topology boundary the epoch a record belongs to travels with it, and the receiving lane briefly holds back records that run ahead of the barrier, so the committed cut stays exact across the boundary. The mechanism belongs to the Chandy-Lamport family of distributed-snapshot algorithms (the same lineage as Flink's checkpoint barriers), scoped to a single process. See [Exactly-once and at-least-once](https://stoatflow.io/docs/concepts/exactly-once).

### Dead-letter queue (DLQ)

A Kafka topic that receives records the pipeline could not handle — deserialization failures, processing exceptions, or production errors — when the corresponding exception handler is configured with one. The failed record is preserved together with error-context headers (the `__stoatflow.errors.*` namespace) for offline inspection or replay, while the topology keeps processing. See [Error handling and DLQ](https://stoatflow.io/docs/building/error-handling-dlq).

### Epoch

The unit of work bounded by two consecutive commit barriers — all the records processed, state written, and output produced between one commit and the next. Either an epoch commits atomically or, on a crash, its partial work is aborted and reprocessed. Epoch size and commit cadence self-tune within configured bounds; see [Tuning](https://stoatflow.io/docs/operating/tuning).

### Event time

The time at which an event actually occurred, carried on the record itself — the Kafka record timestamp by default, or whatever a custom `TimestampExtractor` returns. Stateful operators that reason about time (windowed aggregations, session windows, time-bounded joins) use event time, not arrival time. Contrast with **processing time**. See [Event time and watermarks](https://stoatflow.io/docs/concepts/event-time-and-watermarks).

### Exactly-once (EOS)

A guarantee that each input record affects state and produces output exactly once, even across crashes. StoatFlow achieves it with the commit barrier: state writes, sink output, and consumer offsets commit together in a single Kafka transaction, or not at all. Downstream consumers reading with `read_committed` isolation never see duplicates. Contrast with **at-least-once**. See [Exactly-once and at-least-once](https://stoatflow.io/docs/concepts/exactly-once).

### Grace period

A configurable interval, set on a window specification (for example `TimeWindows.ofSizeAndGrace(...)`), during which late records are still folded into a window after the watermark has passed the window's end. Once the grace period elapses the window closes and later records that would have belonged to it are dropped. See [Windowing](https://stoatflow.io/docs/building/windowing).

### Hot standby

An opt-in high-availability mode (`stoatflow.ha.mode: active-standby`) that runs the application as a cluster of one active instance plus one or more passive warm standbys: the active processes and commits while each standby continuously follows the changelog and stays warm; on failover a lag-aware election promotes the freshest, ready in seconds. It adds redundancy without changing the single-active-instance model. See [High availability](https://stoatflow.io/docs/operating/high-availability).

### Key-affinity lane

A unit of concurrent processing inside the single JVM. Each lane runs the full topology independently against the shared state stores. Records are routed to lanes by **key affinity** — the same key always goes to the same lane — which guarantees per-key ordering while letting different keys process in parallel. Lane count is configured independently of Kafka partition count and scales with CPU cores. See [Lanes and parallelism](https://stoatflow.io/docs/concepts/lanes-and-parallelism).

### Keyed timer (TimerService)

A one-shot, per-key callback registered from a custom `Processor` through `TimerService` — a Flink-inspired extension. A timer fires `onTimer(...)` for a specific key at a specific instant, in event time (when the watermark passes it) or processing time, in the same per-key serialized context as `process(...)` and with full state access. With the persistent (RocksDB) timer backend, pending timers survive restarts. See [Processor API → Timers](https://stoatflow.io/docs/building/processor-api#timers).

### Late record

A record whose event time is older than the current watermark. While its window remains within its grace period, a late record still updates the window's result; after the window has closed it is dropped. See [Event time and watermarks](https://stoatflow.io/docs/concepts/event-time-and-watermarks).

### Materialized

The configuration object (`Materialized`) that tells a stateful DSL operator to back its result with a named, queryable state store — controlling the store name, store type, serdes, caching, and changelog. Operators like `count`, `reduce`, `aggregate`, and table transforms accept a `Materialized` to expose their result as a store. See [State stores](https://stoatflow.io/docs/building/state-stores).

### Processing time

The wall-clock time at which a record is processed by the runtime, independent of when the underlying event occurred. Used by processing-time timers and as the default clock when no event-time semantics are configured. Contrast with **event time**. See [Event time and watermarks](https://stoatflow.io/docs/concepts/event-time-and-watermarks).

### Promotion token

A single record on the [hot-standby](https://stoatflow.io/#hot-standby) coordination topic, claimed by compare-and-set, that authorizes exactly one standby to promote during a failover election. A standby that loses the claim stands down and remains a standby — no fence, no restart. Under exactly-once the token is the election layer and Kafka's producer-epoch fence remains the safety backstop; under at-least-once the token itself is the fence. See [High availability](https://stoatflow.io/docs/operating/high-availability).

### Punctuator

A periodic callback scheduled by a processor via `context.schedule(...)`, firing on stream time (watermark advance) or wall-clock time — useful for heartbeats, metrics, or flushing buffered state. Punctuators run on a dedicated punctuation lane per sub-topology with full state access, concurrently with record processing; for per-key work in the record-serialized context, use a **keyed timer** instead. See [Processor API → Punctuators](https://stoatflow.io/docs/building/processor-api#punctuators).

### Repartition

Re-routing a record to a different lane because an operation changed its key — `selectKey`, `groupBy`, or a key-changing `map`/`flatMap`/join. In standard Kafka Streams this requires writing to and re-reading from an internal repartition topic; in StoatFlow it happens in-memory between lanes, with no broker round-trip and no extra serialization. See [Lanes and parallelism](https://stoatflow.io/docs/concepts/lanes-and-parallelism).

### Scheduled source

A topology-level source that emits records on a clock instead of consuming from Kafka — `StreamsBuilder.scheduled(...)` returns a normal `KStream` for downstream processing. Supports fixed intervals (stream-time or wall-clock) and cron expressions (`CronExpression.unix/quartz/spring`, always wall-clock). A StoatFlow extension with no Kafka Streams equivalent. See [Scheduled sources](https://stoatflow.io/docs/building/scheduled-sources).

### Serde

A serializer/deserializer pair that converts between typed keys/values and the byte form Kafka stores. Source topics, sink topics, and state stores each need serdes for their key and value types; the DSL supplies defaults and per-operator overrides (`Consumed`, `Produced`, `Materialized`, `Grouped`). See [Serdes](https://stoatflow.io/docs/building/serdes).

### State store

A keyed store that stateful operators read and write. StoatFlow ships several types — key-value, window, session, versioned (timestamped lookups), and timer — each in a RocksDB-backed (persistent) or in-memory variant. State is global: every store lives in the one JVM and any lane can read or write any key, with correctness guaranteed by key affinity. Durability comes from the changelog. See [State stores](https://stoatflow.io/docs/building/state-stores) and [State and thread safety](https://stoatflow.io/docs/concepts/state-and-thread-safety).

### Sub-topology

A segment of a topology bounded by an in-memory re-partitioning handoff. StoatFlow opens a boundary where a record that has been re-keyed reaches an operator that needs the *new* key's lane affinity — a grouped aggregation, a join, `toTable()` — or at an explicit `repartition()`, which always forces one. A re-key on its own does not: `selectKey → mapValues → to()` is a single sub-topology, because nothing downstream cares which lane the record is on. This mirrors Kafka Streams' rule for materialising a repartition topic, so `describe()` reports the same structure as an equivalent KS application.

A **Processor API node** (`process()` / `processValues()`) is deliberately not on that list either — Kafka Streams never repartitions before one, and since 1.0.0 neither does StoatFlow; see [Key affinity after a re-key](https://stoatflow.io/docs/building/processor-api#key-affinity-after-a-re-key) for what that costs after a many-to-one re-key, and `stoatflow.topology.processor-api-key-affinity: presumed` to restore it.

Set `stoatflow.topology.sub-topology-split: eager` to restore StoatFlow's pre-1.0.0 rule, which opened a boundary at *every* key-changing operator. See [Lanes and parallelism](https://stoatflow.io/docs/concepts/lanes-and-parallelism#where-the-handoff-actually-happens) and [Aggregations](https://stoatflow.io/docs/building/aggregations) for where the boundary appears in practice.

### Suppression

Holding a windowed aggregation's intermediate updates and emitting only the final result — declaratively via `EmitStrategy.onWindowClose()`, or with explicit buffer control via `KTable.suppress(Suppressed.untilWindowCloses(...))` / `untilTimeLimit(...)`. Suppressed results flush when the watermark closes the window. See [Windowing → Suppression](https://stoatflow.io/docs/building/windowing#suppression).

### Topology

The directed graph of source nodes, processors, and sink nodes that defines a stream-processing application — built with `StreamsBuilder` (high-level DSL) or `Topology` directly (Processor API). The runtime executes the same topology independently in every lane. See [StreamsBuilder](https://stoatflow.io/docs/building/streams-builder) and [Architecture](https://stoatflow.io/docs/concepts/architecture).

### Watermark

A claim made by the runtime that no further records earlier than a given time are expected: "I do not expect any record older than *T*." Watermarks are tracked per source partition and combined into a single global watermark for the application. When the global watermark passes a window's end (plus its grace period), the window closes. The watermark advances together with the commit barrier so windowed results commit alongside the watermark progress that produced them. See [Event time and watermarks](https://stoatflow.io/docs/concepts/event-time-and-watermarks).

## See also

- [Architecture](https://stoatflow.io/docs/concepts/architecture) — how these concepts fit together in the single-instance engine
- [How StoatFlow differs from Kafka Streams](https://stoatflow.io/docs/concepts/how-stoatflow-differs-from-ks) — where StoatFlow's model diverges from the terms' Kafka Streams meaning
- [KS compatibility matrix](https://stoatflow.io/docs/reference/ks-compatibility-matrix) — DSL operator parity


# StoatFlow: Kafka Streams compatible engine built to scale up — not out

> **TL;DR**
>
> - **What:** Single-replica JVM stream processor with the Kafka Streams DSL — JDK 25, Project Loom virtual threads.
> - **Kafka Streams DSL compatibility:** Drop-in — existing topology code ports with a dependency swap.
> - **Why:** up to 3.4× less CPU and 7.8× less container memory than Kafka Streams on the same hardware; up to 13.6× lower P99 latency on stateful workloads.
> - **The catch:** one instance per app, so no horizontal scale-out. A single 8-vCPU machine saturates around 200–300 MB/s of uncompressed throughput — well above most stream-processing workloads, but a real ceiling.
> - **Access:** Private alpha — [reach out](https://stoatflow.io/contact) for early access.

Most stream-processing workloads fit on a single machine. Kafka Streams and Flink scale them out anyway — and you pay the architectural cost of distribution whether your workload needs it or not.

StoatFlow is the alternative for the workloads that don't. Same Kafka Streams DSL, one replica per app, built on JDK 25 virtual threads: your existing topology code compiles against StoatFlow, and your operators stop paying the *distribution tax*.

For the *how*, head to [Getting Started](https://stoatflow.io/docs/getting-started).

## What we set out to fix

Stream processing on the JVM gives you two well-known choices: Kafka Streams or Apache Flink. Both are remarkable. Both also scale horizontally by default — which is where most of their architectural and operational complexity comes from.

Three problems compound:

- **Hard to build, harder to run.** Stateful joins, exactly-once, watermarks on out-of-order streams — each is a deep practice. Production then layers on rebalance storms, restart loops, checkpoint failures, and state migrations that miss SLAs.
- **Every layer is a decision.** Which Kafka client knobs to tune? What about RocksDB? StatefulSets, persistent volumes, static group membership, standby replicas? Deploy on Kubernetes without downtime? You answer all of it before your first event flows.
- **Most workloads don't need to scale out.** A workload that comfortably fits on one modern machine pays the *distribution tax* for capacity it will never use.

## A different approach

StoatFlow runs as **exactly one instance per application**. No consumer-group rebalancing — there's no group. No state migration — state lives on the instance that owns it. No repartition topics — key-changing operations route through in-memory queues to other lanes inside the same process.

![StoatFlow high-level architecture: Kafka consumer (all partitions) → lane dispatcher → N virtual-thread lanes with global state → transactional producer.](https://stoatflow.io/assets/docs/architecture/StoatFlow_high-level_architecture_detailed_20260517.png){.rounded-lg}

The single-replica bet rests on two recent shifts:

- **Modern JVM concurrency.** Virtual threads (GA in JDK 21) give you thousands of concurrent lanes without platform-thread overhead. StoatFlow targets JDK 25 — virtual threads, structured concurrency, and the Foreign Function & Memory API all in.
- **Modern hardware.** 16-vCPU compute-optimised instances, 10+ Gbps networking, NVMe storage — off-the-shelf on every major cloud.

From there, the model is a few moving parts that fit together:

- Records dispatch to **key-affinity lanes** by consistent hashing — same key, same lane, ordered. Lane count is decoupled from Kafka partition count, so parallelism scales with cores, not partitions.
- Key-changing operations stay **in-process** — they route to another lane through an in-memory queue, with no repartition-topic round-trip through the broker.
- State (RocksDB or in-memory) is **globally accessible** to any virtual thread, by any key — layered with epoch buffers for read-your-writes consistency.
- Exactly-once flows through **commit barriers** that sweep every lane and align with a single Kafka transaction.

## Your topologies port unchanged

The DSL is the one Kafka Streams users already know: KStream, KTable, joins (primary-key, foreign-key, windowed), count / reduce / aggregate / cogroup, tumbling / hopping / session windows with grace and suppression, versioned state stores, the Processor API, and interactive queries — all there.

**Existing topologies port with a dependency swap and a config cleanup.** Your `StreamsBuilder`, your operations, your `Materialized` definitions all compile against StoatFlow. What you delete is the multi-instance scaffolding: standby replicas, stream-thread counts, partition-aware tuning.

And **on top of the DSL**, StoatFlow ships primitives Kafka Streams doesn't:

- Flink-style event-time and processing-time **timers** from any `Processor`
- Flink-style **watermarks** with idleness alignment
- **Scheduled sources** — topology-level emitters on an interval or cron
- Atomic store operations (`compute`, `merge`)
- **KeyLockManager** — atomic sections across multiple keys and stores

For the full surface, see [Features](https://stoatflow.io/product/features).

## The numbers

Benchmarked against Kafka Streams 4.3.1 on a Hetzner 8-vCPU machine — identical topologies, throughput parity:

:headline-numbers

The gains concentrate where the work is: the stateful-join workload, with ten-plus RocksDB stores and Avro over Schema Registry, is where a single instance stops paying for coordination. On the simplest stateless transform the two engines are level, and StoatFlow uses more memory — the figures beside each bar say so. State restoration is quoted here at its default WriteBatch setting; direct SST ingestion reaches **1.65× faster** at a higher CPU cost.

The [full benchmark report](https://stoatflow.io/product/benchmarks){rel="&#x22;nofollow&#x22;"} walks every scenario — topology, infrastructure stack, serdes (String / Avro / Protobuf), load rates, and event-size distributions — so you can compare against the workloads you actually run.

## Where StoatFlow fits — and where it doesn't

One instance per app is a deliberate trade, and it has edges worth stating plainly.

- **There's a single-machine ceiling, and we name it.** On that Hetzner 8-core VM, benchmarks measure a **200–300 MB/s uncompressed network-bandwidth ceiling** — roughly \~124K events/sec on a 1KB stateless transform, up to \~2.1M events/sec output on word-count-style aggregation. High-end hardware (96+ cores, faster NICs) hasn't been benchmarked yet.
- **No horizontal scale-out — by design.** If a workload genuinely needs to span machines, that's not a StoatFlow workload; Kafka Streams and Flink remain the right answer.
- **State migration is a reprocess, not a restore.** The recommended path onto StoatFlow is to reprocess your input topics — direct restoration from existing Kafka Streams changelog topics isn't supported. If your input retention rules that out, get in touch.
- **This post doesn't cover everything.** Failover behaviour, cold-start times, and how representative the benchmark scenarios are for your workload are all fair questions — and the docs are still being written.

All of this is where the 1.0.0 alpha stands today, not where it's headed. The single-replica design is deliberate and here to stay — but how far one replica goes is exactly what we keep pushing. Benchmarking higher-end hardware to lift the throughput ceiling, shortening cold starts, widening the scenarios we measure, and ongoing refactoring and tuning for more performance on the same hardware are all in flight; the numbers above are a starting point, not a finish line. Scaling *up* harder is the point — expect these edges to move.

## Get early access

StoatFlow is in private alpha — distribution is invite-only while we work directly with each early-access team.

[Reach out](https://stoatflow.io/contact) to request alpha or beta access — especially if you're running stateful Kafka Streams in production today. For release news and updates, [follow on LinkedIn](https://www.linkedin.com/in/hartmut-co-uk/){rel="&#x22;nofollow&#x22;"}.


# llms.txt for StoatFlow: docs your AI agent can fetch

> **TL;DR**
>
> - **What:** the StoatFlow documentation in machine-readable form — [`/llms.txt`](https://stoatflow.io/llms.txt){rel="&#x22;nofollow&#x22;"} (index), [`/llms-full.txt`](https://stoatflow.io/llms-full.txt){rel="&#x22;nofollow&#x22;"} (one bundle), and any page as raw markdown at `/raw/<path>.md`.
> - **Install:** nothing. Point your agent, or your retrieval pipeline, at the URL.
> - **Scope:** the eight documentation sections, published blog posts, and the changelog. Marketing pages stay out.
> - **Freshness:** regenerated on every docs deploy, so it cannot lag the site.
> - **The catch:** the full bundle is around 1.2 MB — roughly 300,000 tokens. Sized for retrieval, not for pasting into a 200K-token context.

The StoatFlow documentation is now published in a form an AI agent can fetch directly. [`stoatflow.io/llms.txt`](https://stoatflow.io/llms.txt){rel="&#x22;nofollow&#x22;"} is a curated index of every documentation page; [`stoatflow.io/llms-full.txt`](https://stoatflow.io/llms-full.txt){rel="&#x22;nofollow&#x22;"} is the same documentation concatenated into a single markdown file; and every indexed page is also served as raw markdown at `/raw/<path>.md`. There is nothing to install and no key to configure.

If the convention is new to you, Chrome's documentation puts it plainly:

> The `llms.txt` file is an [emerging convention](https://llmstxt.org/){rel="&#x22;nofollow&#x22;"} used to provide a machine-readable summary of a website's content, specifically designed for LLMs and AI agents. Without this file, agents may spend more time crawling the site to understand its high-level structure and primary content.
>
> — [Chrome for Developers, *llms.txt*](https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt){rel="&#x22;nofollow&#x22;"}

That last sentence is the whole value proposition. An agent that has to crawl a documentation site burns turns discovering structure before it can answer anything; one that fetches an index gets the map in a single request.

This is the other half of a pair. The [AI Assistant Skills pack](https://stoatflow.io/blog/ai-assistant-skills) is a set of skills and editor rule files you install so your assistant writes correct StoatFlow instead of hallucinated Kafka Streams. That channel **pushes** — the rules land in the assistant's context before it writes a line. `llms.txt` **pulls** — the agent goes and gets the answer when it needs one.

## Two channels, not one

Both exist because they fail differently.

The pack only works where it's installed, and only for the assistants it targets. Where it applies it is the stronger intervention: the rules are in context before the model starts generating, which is where they shift behaviour most.

`llms.txt` reaches everything else — an agent nobody configured, a retrieval pipeline building an internal knowledge base, a coding tool that follows URLs. It's weaker per interaction, because the model has to decide to look. It has no setup step to skip.

An assistant with both gets the rules that stop it reaching for `org.apache.kafka.streams.*`, *and* the page that says what `barrier.max-interval-ms` actually does.

## What's published

| URL              | Contents                                                                                                      |
| ---------------- | ------------------------------------------------------------------------------------------------------------- |
| `/llms.txt`      | Curated index — every documentation page with its title and description, grouped by section. 65 links, 17 KB. |
| `/llms-full.txt` | The same pages concatenated into one markdown document. About 1.2 MB.                                         |
| `/raw/<path>.md` | Any indexed page as raw markdown — for example `/raw/docs/concepts/exactly-once.md`.                          |

Scope is the eight documentation sections — getting started, concepts, building topologies, configuration, running in production, operating, migration, reference — plus published blog posts and the changelog. Product and pricing pages stay out: they are marketing surfaces, and structured page content serialises poorly to markdown.

The format follows the [llms.txt convention](https://llmstxt.org/){rel="&#x22;nofollow&#x22;"}, which a growing number of documentation sites now publish. Nothing here is StoatFlow-specific: if your tooling already understands `llms.txt` from another vendor, it understands ours.

## How it stays correct

Documentation that has drifted from the code is worse than none — it is wrong with authority, and an agent repeats it without the hesitation a human reader might feel. The skills pack fights that with drift checks and a version that matches the release. This side avoids it structurally instead: both files are generated from the same markdown that renders the site, in the same build, on every deploy. There is no second copy to maintain and no hand-written index to rot.

The weight moves to configuration, which is worth one caution if you are publishing your own. Every section we expose is declared explicitly, because a generator like this never executes the page components that enforce a site's own rules. It is also worth knowing that a filter which *looks* like it excludes drafts can quietly exclude everything else instead, depending on how an unset field is stored: in SQL, unset is not "false". We check by reading the generated files, not by trusting a green build.

## What it doesn't do

The bundle is around 1.2 MB, roughly 300,000 tokens. That suits chunked retrieval and long-context models; it does not suit pasting into a 200K-token window. Use the index, or fetch the two or three `/raw/` pages you actually need.

It is a pull channel. The agent has to decide to fetch it, and many won't unless told. If you want the rules in front of the model unconditionally, install the pack.

It covers the public API and the concepts, and nothing beneath them. StoatFlow's engine is obfuscated in the shipped jar; the documentation holds that same line, so the bundle explains what the API does and how the architecture works — never how the engine is implemented inside.

And it amplifies whatever the documentation says. A page that is wrong is now wrong at machine scale, republished on every deploy. That raises the bar on the docs themselves, which is our problem to stay on top of rather than one to discover later.

## Where to go next

- The index: [`stoatflow.io/llms.txt`](https://stoatflow.io/llms.txt){rel="&#x22;nofollow&#x22;"} — start here, then fetch what you need.
- The bundle: [`stoatflow.io/llms-full.txt`](https://stoatflow.io/llms-full.txt){rel="&#x22;nofollow&#x22;"} — for retrieval pipelines.
- The push-side channel and its per-editor install: [AI assistants](https://stoatflow.io/docs/getting-started/ai-assistants).
- Something the docs get wrong? [Reach out](https://stoatflow.io/contact) — a concrete miss is the most useful thing you can send, and it now propagates to every agent on the next deploy.

For the running commentary on how StoatFlow gets built, [follow along on LinkedIn](https://www.linkedin.com/in/hartmut-co-uk/){rel="&#x22;nofollow&#x22;"}.


# Three gates, one transaction: error handling in StoatFlow

> **TL;DR**
>
> - **Three gates:** a record can fail to **deserialize** on the way in, a **processor** can throw mid-topology, or **production** (serialize + send) can fail on the way out. The handler surface is Kafka Streams' KIP-1033/1034 shape, so the mental model ports.
> - **Three verdicts, plus retry:** log-and-continue, log-and-fail (the default for deserialization and processing), or dead-letter. Production alone adds *retry* — transient send errors back off exponentially before giving up.
> - **The rule:** every failure resolves inside the transaction. A dead-lettered record rides the same transactional producer and commits on the same barrier as the epoch's output — never lost, never duplicated under `read_committed`. A FAIL aborts the in-flight epoch and replays from the last committed barrier.
> - **The catch:** a broker-side rejection of an already-sent record (think `max.message.bytes`) poisons the transaction. With a DLQ configured — which is the default the moment you set a DLQ topic — the epoch's innocent outputs are dropped: WARN-logged, counted on a dedicated metric, bounded by epoch size, and the source position advances past the poison so the application keeps making progress. Kafka Streams is materially worse on this exact scenario: it drops the same records, never advances, and aborts its own dead-letter record along with the transaction.
> - **Above the gates:** a fatal error reaches the KS-compatible `StreamsUncaughtExceptionHandler`; `REPLACE_THREAD` maps to an in-place engine restart, budgeted at 5 per 5 minutes. A failed commit is not configurable at all — abort, exit, restart from the last barrier.

Error handling is where exactly-once pipelines usually lose their guarantee — not in the transaction protocol, but in the catch block: a dead-letter record published through a second, non-transactional producer, or a skipped record whose offset advances with no trace left behind. Both look harmless in review. Both break the guarantee the rest of the system worked so hard for.

StoatFlow closes that gap with one rule. Every failure is resolved inside the transaction model: either a handler settles it within the current epoch — and anything the handler emits, dead-letter records included, commits on the same barrier as the epoch's output — or the failure is fatal and the epoch dies whole, replayed from the last committed barrier after recovery. There is no third state where the application limps on with a weakened guarantee.

This post walks the full model: the three places a record can fail, the verdicts you can configure at each, what a dead-letter queue means when it lives inside the transaction, and the escalation ladder above the handlers — from an in-place engine restart to the process exit that hands recovery to Kubernetes.

## Three gates, three verdicts

A record can fail in three places, and the classes are worth keeping apart because each fails with different evidence in hand.

- **Deserialization, on the way in.** The record is still raw bytes, and the configured serde rejects the key or the value. The topology never saw it, so there is nothing to roll back — and the original bytes are still available, untouched, which matters for what a dead-letter record can carry.
- **Processing, inside the topology.** The record deserialized, entered the DAG, and a processor threw — a `NullPointerException` in a `mapValues`, a failed enrichment call, a bug. This happens on a processing lane, mid-flight, possibly after state was touched. It sounds like the dangerous one, but the same commit barrier that governs everything else governs those writes too: whatever a failed record did to state belongs to the current epoch, and the epoch commits as a whole or not at all. A skipped record cannot leak half-applied state into the committed snapshot.
- **Production, on the way out.** The topology emitted a record and the runtime could not publish it. This class splits in two, and the split drives the policy: **serialization** failures are deterministic — the same value fails the same way every time, so StoatFlow never retries them — while **send** failures may be transient: a timeout, a briefly unreachable broker. Production is therefore the only class where *retry* is a meaningful verdict; the default production handler retries transient sends with exponential backoff before giving up.

Each class has its own exception handler, configured independently — `deserialization.exception.handler`, `processing.exception.handler`, `production.exception.handler`, or the typed builder equivalents. The interfaces follow Kafka Streams deliberately: the [KIP-1033](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1033%3A+Add+Kafka+Streams+exception+handler+for+exceptions+occurring+during+processing){rel="&#x22;nofollow&#x22;"} processing handler, the [KIP-1034](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1034%3A+Dead+letter+queue+in+Kafka+Streams){rel="&#x22;nofollow&#x22;"} dead-letter design, and [KIP-1065](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=311627309){rel="&#x22;nofollow&#x22;"}'s retry option, with the same `handle(context, record, exception)` signature Kafka Streams 4.x uses. Those KIPs got the shape right, and keeping to it means an existing application's error-handling setup ports without relearning anything.

Every handler chooses between the same verdicts: **log and continue** (skip the record and move on), **log and fail** (stop the application), or **dead-letter** (route the record and its error context to a topic you own, then continue). The defaults are strict on purpose — deserialization and processing both default to log-and-fail. Skipping records should be a decision you make explicitly, not a behaviour you inherit.

The whole surface fits in one picture:

![A record travelling left to right past three gates — deserialize before the topology, process mid-DAG on a lane, and produce, which splits into serialize (never retried) and send (may be transient). Each gate offers the same log-and-continue, log-and-fail and dead-letter verdicts and carries its own default; produce adds a fourth, retry. Every dead-letter verdict feeds one Kafka transaction that commits the output records, the DLQ records and the state plus source offsets together on the same barrier.](https://stoatflow.io/assets/docs/concepts/failure-gates_20260727.svg)

Look at the right edge. Every dead-letter arrow, from every gate, lands in the same place: one Kafka transaction holding the epoch's output records, its dead-letter records, and its state changes plus source offsets. That box is the model — the rest of this post is what happens inside it, and what happens when something cannot get into it.

## The dead-letter record rides the transaction

A DLQ handler in StoatFlow does not publish to the dead-letter topic itself. It returns the failed record — wrapped as a `ProducerRecord` targeting your DLQ topic — attached to its verdict, and the engine sends it through the same transactional producer as the epoch's normal output. (There is no separate dead-letter verdict in the response enum; dead-lettering is a continue-or-fail decision that carries records, the same shape KIP-1034 chose.) The DLQ record commits on the same barrier as everything else, which buys two properties a hand-rolled dead-letter path cannot offer:

- **No loss.** If the epoch commits, the DLQ record is on the topic. If the epoch aborts — a crash mid-commit — the DLQ record is discarded with everything else, the source offset never advanced, and the record is re-read and re-handled after restart.
- **No duplicates.** The record appears on the DLQ topic exactly once for a `read_committed` consumer — the same guarantee as your real output.

Compare that with the pattern most teams build by hand: a second producer in the catch block. It has exactly two failure modes. Either the DLQ write lands and the offset commit does not, and after restart the record is dead-lettered again — duplicates on the DLQ. Or the offset commits and the DLQ write did not, and the record is gone with nothing to show for it. Across enough restarts you will meet both, and both are impossible when the DLQ record and the offset advance are one atomic commit.

Kafka Streams gained the same response-attached DLQ shape with KIP-1034, so the interface is shared; the context differs. Exactly-once is StoatFlow's default processing guarantee, so DLQ-on-the-barrier is the out-of-the-box behaviour rather than a property of a mode you remembered to enable.

What lands on the topic depends on the gate. Deserialization DLQ records preserve the original raw key and value bytes verbatim — nothing ever deserialized, so the untouched payload is exactly what you need to diagnose or replay. And every DLQ record carries error context in headers under a `__stoatflow.errors.*` namespace: the exception class and message, the source topic, partition and offset, the failing component, optionally the stack trace. The namespace is deliberately not Kafka Streams' `__streams.errors.*` — tooling should be able to tell which engine produced a dead letter. The full header table is in the [error-handling guide](https://stoatflow.io/docs/building/error-handling-dlq).

Wiring it is two builder calls. This captures bad input and undeliverable output while keeping processing on its fail-fast default:

```kotlin
streamsConfigOverrides {
    // capture undeserialisable input — original bytes preserved on the DLQ record
    deserializationExceptionHandler(
        DeadLetterQueueDeserializationExceptionHandler(dlqTopic = "orders.deserialization.dlq"),
    )
    // retry transient sends; dead-letter what cannot be delivered
    productionExceptionHandler(
        DefaultProductionExceptionHandler(dlqTopic = "orders.production.dlq"),
    )
    // processing stays log-and-fail: a bug should stop the app, not drain into a topic
}
```

The dead-letter topics are ordinary topics you create, own and retain; the runtime only produces to them and never reads them back, and there is no auto-naming. One boundary worth drawing: a record that parses fine but fails your business rules is not an exception — route those explicitly with a branch and a sink in the topology, where you control the payload and the serde. The [guide](https://stoatflow.io/docs/building/error-handling-dlq) shows both patterns side by side.

## Fail means the epoch dies

The other half of the model is what *fail* actually does. A FAIL verdict — the default handler's, or your custom handler deciding an exception is not survivable — does not try to unwind one record. It ends the epoch. The in-flight transaction aborts at the broker, and with it everything the epoch had done: output records, changelog writes, state store changes, offset advances, all discarded together. Recovery resumes from the last committed barrier and re-processes everything after it. A `read_committed` consumer never saw the aborted work.

One path is exempt, and it is the subject of [the honest part](https://stoatflow.io/#the-honest-part-what-this-model-costs) below: on an asynchronous broker-side production failure the offsets have already advanced by the time the verdict is read, so that epoch is *not* replayed — and *fail* loses exactly what *continue* loses.

FAIL is a controlled crash with crash-recovery semantics, in other words — and that is the feature. There is no degraded mode in which the application keeps running with the guarantee suspended.

![A commit barrier cascading downstream through three sub-topologies against a wall-clock axis. The span between two commits is one epoch, and transaction N commits state, output and offsets together — all three or none of them. A crash partway through epoch N plus 1 aborts transaction N plus 1, discarding its state, output and offsets together; that output was never visible to a read\_committed consumer. On restart, state rebuilds to the last committed barrier and the consumer resumes from transaction N's offsets, re-processing everything after it into a fresh epoch.](https://stoatflow.io/assets/docs/concepts/commit-barrier-epochs_20260727.svg)

The diagram is the same one our docs use to explain exactly-once, and that is no accident: failure recovery *is* the exactly-once mechanism, pointed at a different trigger. An epoch dies the same way whether a processor threw or the process crashed.

This is also where the strict defaults earn their keep. Under exactly-once, a FAIL costs replay time and nothing else — no duplicates, no partial state, no reconciliation. Failing fast is cheap, so it can be the default. Under an at-least-once default — which is what Kafka Streams ships — a crash means duplicates, and that price pushes teams towards continue-and-hope handlers. Same handler API, different default guarantee, different economics.

## Above the gates: replace the engine or exit the process

A FAIL verdict stops the topology. What happens next follows a ladder, and its top rung is the one Kafka Streams users already know: the fatal error is offered to a `StreamsUncaughtExceptionHandler`, the KS-compatible interface with the familiar three answers — `REPLACE_THREAD`, `SHUTDOWN_CLIENT`, `SHUTDOWN_APPLICATION`.

There is no stream thread to replace in StoatFlow, so `REPLACE_THREAD` maps to the strongest recovery the architecture allows: an **in-place engine restart**. The processing engine is torn down inside the live process — the faulted epoch aborted, exactly as above — and a fresh engine is built and resumes from the last committed barrier. State stores stay open and the JVM stays warm, so recovery costs engine-rebuild time rather than pod-reschedule time. The mechanics have [their own post](https://stoatflow.io/blog/in-place-restart-multi-standby).

Two constraints keep that honest. Only processing and production failures are restartable — a record that cannot deserialize will not deserialize for a rebuilt engine either, so a deserialization FAIL goes straight to shutdown, as do punctuator failures and commit stalls. And restarts are budgeted — `commit-barrier.max-engine-restarts` (default 5) within `commit-barrier.engine-restart-window-ms` (default 5 minutes) — so a recurring fault escalates to a terminal shutdown instead of looping forever.

::callout{color="info" icon="i-lucide-info"}
**A deserialization FAIL on Kubernetes *is* a restart — and that is the point.** The pod exits, the kubelet starts it again, the same record is still there, and it fails again. What you get is not recovery, it is a loud `CrashLoopBackOff` and an alert. Choose that deliberately: it is the right answer when a malformed record means something upstream is broken and you want processing halted until a human looks. If you would rather keep running, handle the poison record — log-and-continue, or route it to a dead-letter topic.
::

Hot standby does not change any of that. Turn it on and a restartable fault is still absorbed in place — the pod keeps the active role, the standby stays a standby, and no failover happens. The role moves only when the budget above is spent, and then it is a graceful hand-off: one failover after a bounded number of local attempts, rather than a role transfer per fault. (An earlier version of this post said the opposite, and said the handler was not consulted under HA. Both were wrong; the [high-availability page](https://stoatflow.io/docs/operating/high-availability) has the current behaviour.)

Everything that is not restartable ends the same way: a graceful shutdown that aborts the in-flight transaction and exits the process, backstopped by a hard-exit timer so that a shutdown which hangs still becomes a clean `System.exit(1)` rather than a zombie pod. From Kubernetes' point of view that is an ordinary container restart. From the data's point of view, it is a resume from the last committed barrier.

At the bottom of the ladder sits the one failure no handler is consulted about: the commit itself failing — a transaction timeout, a broker rejection, a fenced producer. There is no policy hook because there is no safe alternative to abort-and-restart. Kafka Streams has one more move here: it can migrate the fenced task to another instance. StoatFlow does not, and the trade is deliberate. Task migration is part of the *distribution tax* — a rebalance protocol, standby-task placement, and a state-transfer path, all of which you operate and debug — and what replaces it is a shorter contract. The single instance restarts, and the restart is cheap: state stores are node-local and carry their own committed offsets, so recovery [delta-restores the aborted epoch](https://stoatflow.io/blog/kip-1035-state-store-managed-offsets) rather than rebuilding from the changelog, in about the time it takes the container to come back, largely independent of how much state you hold. Every failure, at every rung of the ladder, resolves to the same known-good place: the last committed barrier.

(If you run [hot-standby HA](https://stoatflow.io/docs/operating/high-availability), there *is* another instance — a warm standby that takes the role over. That is an availability layer bolted onto the same contract, not the distribution model coming back: still exactly one instance processing, still no partition-level task migration.)

## The honest part: what this model costs

**One production path used to drop records that did nothing wrong. It no longer does — and this section
said otherwise when the post first went out.** Everything above describes synchronous failures, caught
before or during the send. A broker-side rejection is different: the record was already handed to the
producer and rejected asynchronously — the classic case is a record exceeding the topic's
`max.message.bytes`. That rejection poisons the whole in-flight transaction; nothing in it can commit any
more.

The original version of this post described what StoatFlow then did: abort the epoch, commit a small
secondary transaction carrying only the poison's dead letter **and the epoch's offset advance**, and move
on — losing every innocent output in that epoch, on *fail* exactly as on *continue*. That was accurate,
and it was the wrong design. It rested on the claim that dropping the epoch is the only terminating
semantic for a deterministic poison under batched exactly-once, which is true only *without replay
machinery*. StoatFlow has that machinery: the in-place engine restart that absorbs a restartable fault
higher up this same ladder.

So the semantics changed. The offsets are now **held** on every verdict, and the epoch is **replayed**
with the poison skipped: the secondary transaction carries the dead letter alone, the poison's
`(topic, partition, offset)` goes into a quarantine that outlives the engine, the engine restarts in
place, and the epoch is reprocessed with exactly that record dropped. Its epoch-mates commit normally.
*fail* now means what it says — stop before further harm, with the offsets held so an operator restart
replays the epoch — and it is no longer eligible for the in-place restart rung, which used to make it
quietly equivalent to *continue* for anyone running a `REPLACE_THREAD` handler.

![A wall-clock timeline of one epoch. Source records are read while innocent output records and state writes accumulate, and one output record is sent that the broker rejects asynchronously. At the commit barrier the whole transaction aborts, but the source offsets are held rather than advanced: a secondary transaction commits only the poison's dead-letter record, the engine restarts in place, and the epoch is replayed with the poison record skipped so the innocent records commit normally.](https://stoatflow.io/assets/docs/concepts/poison-epoch-abort_20260813.svg)

**What it still costs, because "no data is lost" would be the same kind of overstatement.** Quarantining a
source record skips *all* of its outputs, so a record that fans out to several sinks loses the good sends
with the bad one — read the guarantee as *loss is bounded to the poison record's own outputs*. And a poison
derived from **accumulated state**, an aggregate that outgrew `max.message.bytes`, does not converge:
skipping the triggering record does not shrink the accumulator, so the next record on that key reproduces
it. Each replay costs a full engine restart and spends a budget that is deliberately in-memory, so
exhausting it ends the process, the pod restarts, and the budget comes back clean — *continue* is bounded
**per process**, never end to end. Alert on `stoatflow.dlq.poison.replays.total`. The thing that prevents
the case entirely is still sizing the target topic's `max.message.bytes` for your largest output.

Two cases cannot be replayed at all and stop the instance immediately, offsets held: a *fail* verdict, and
a poison emitted by something with no source record behind it — a punctuator, a window close, a suppression
flush, a timer, a scheduled source. Those fire on their own schedule and would re-poison every attempt, so
the runtime names the plane and stops rather than burning the budget discovering it.

**A poisoned exactly-once transaction has no gentle exit anywhere, but the exits differ.** Kafka Streams
4.3.1 honours *continue* locally and then loses on three counts. Its offsets ride
`sendOffsetsToTransaction`, which throws once the producer is in `ABORTABLE_ERROR`, so nothing reaches
`__consumer_offsets` and the restart replays straight back onto the same poison — the
[open bug](https://issues.apache.org/jira/browse/KAFKA-15259){rel="&#x22;nofollow&#x22;"} for exactly this. Its own KIP-1034
dead-letter record is appended to the transaction the rejection already doomed, so it is aborted with
everything else and no `read_committed` consumer ever sees it. And the commit throws, the stream thread
dies, and the default `SHUTDOWN_CLIENT` stops the client — under a supervisor, a restart loop onto the same
record with no evidence written anywhere. StoatFlow now also stops advancing, deliberately; the difference
is what survives it. We keep a Testcontainers probe against real Kafka Streams 4.3.1 so that comparison
stays true; on the run behind this post it observed a dead-letter topic with no dead letter for the poison,
a committed offset still pointing at the poison, and a client in `ERROR`.

**Log-and-continue is data loss with a log line.** Without a DLQ, a skipped record leaves a log entry and a metric tick, nothing else. If there is any chance you will want the record back, dead-letter it instead.

**Processing dead letters are not the original bytes.** Only the deserialization DLQ preserves the raw payload. A processing failure happens after deserialization, so its DLQ record carries the key and value rendered to strings — good for diagnosis, not a byte-faithful replay source.

**The DLQ topics are yours.** You create them, size their retention, and monitor them; nothing is auto-created. And because the DLQ handlers take the topic as a constructor argument, they cannot be configured from YAML, which can only instantiate no-arg handlers — wire them in code, or write a small no-arg subclass that hard-codes the topic.

**An exhausted restart budget is downtime.** Five faults in five minutes ends the process, and unless you run the opt-in [hot standby](https://stoatflow.io/docs/operating/high-availability), recovery is a cold start with state restoration ahead of it.

## What you see when it breaks

Every verdict leaves a signal, and the metric names are worth knowing before you need them: `stoatflow.error.total` counts errors by type and topic, `stoatflow.dropped.records.total` is the Kafka Streams-parity dropped-records counter, `stoatflow.dlq.poison.replays.total` counts the epoch replays above (with `stoatflow.dlq.poison.quarantined.total` and `stoatflow.dlq.poison.quarantine.size` showing what is being skipped), `stoatflow.engine.restart.total` tags each in-place restart with its trigger, and `stoatflow.barrier.failed.total` plus `stoatflow.commit.stall.detected.total` cover the commit path. All are exposed on `/metrics` in Prometheus form.

If you set one alert on day one, make it this one:

```promql
rate(stoatflow_engine_restart_total{trigger="replace_thread"}[10m]) > 0
```

An in-place restart is self-healing, but a fault that recurs is still a fault — and the budget means five of them in five minutes will end the process. Alerting on the first buys you the investigation window.

Alert on `rate(stoatflow_dlq_poison_replays_total[10m]) > 0` for the same reason, and with more urgency: a poison replay is self-healing exactly once per poison, its budget is in-memory, and a poison that does not converge will spend that budget and crash-loop the pod.

The commit path also watches itself: a commit-pipeline watchdog (45-second stall threshold by default) and bounded waits on every commit-critical call turn a silent freeze into a loud failure — a stall exception with a thread dump attached, then the restart path above. In practice that is the difference between a consumer-lag graph climbing while the process looks healthy, and a process that tells you what it was stuck on before recovering.

And when a record lands on a DLQ topic, the headers make it self-describing:

```bash
kafka-console-consumer.sh --bootstrap-server localhost:9092 \
  --topic orders.deserialization.dlq --from-beginning \
  --property print.headers=true --property print.key=true
```

`__stoatflow.errors.type` names the gate that fired, `__stoatflow.errors.topic` / `.partition` / `.offset` point at the exact source record, and `__stoatflow.errors.exception` and `.message` carry the why — enough to triage from the console before any tooling gets involved.

A restart, throughout all of this, is the recovery path rather than an incident: the readiness probe stays down while a restarted instance restores state, so traffic waits until it has caught up. The design assumes restarts happen and makes them boring.

That is the whole model. Three gates a record can fail at, the same verdicts at each, a dead-letter path that lives inside the exactly-once transaction instead of beside it — and above the handlers, a ladder of engine restart, process exit and orchestrator restart where every rung ends at the last committed barrier. Failures are handled inside the epoch, or the epoch dies. Nothing in between.

**Read on:**

- [The error-handling model](https://stoatflow.io/docs/concepts/error-handling-model) — the behavioural reference this post narrates.
- [Error handling and DLQ](https://stoatflow.io/docs/building/error-handling-dlq) — handler classes, config keys, the full header table, and custom handlers in Kotlin and Java.
- [Exactly-once](https://stoatflow.io/docs/concepts/exactly-once) — the commit barrier the whole model hangs off.
- [In-place engine restart: the primitive behind multi-standby HA](https://stoatflow.io/blog/in-place-restart-multi-standby) — what `REPLACE_THREAD` actually does.
- [KIP-1033: Add Kafka Streams exception handler for exceptions occurring during processing](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1033%3A+Add+Kafka+Streams+exception+handler+for+exceptions+occurring+during+processing){rel="&#x22;nofollow&#x22;"}
- [KIP-1034: Dead letter queue in Kafka Streams](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1034%3A+Dead+letter+queue+in+Kafka+Streams){rel="&#x22;nofollow&#x22;"}
- [KIP-1065: Add "retry" return-option to ProductionExceptionHandler](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=311627309){rel="&#x22;nofollow&#x22;"}


# From first alpha to release candidate: the road to StoatFlow 1.0.0

> **TL;DR**
>
> - **What:** StoatFlow **1.0.0-rc.1** is cut — the twenty-seventh release on the 1.0.0 line, and the one that freezes it: from here to general availability, bug fixes only.
> - **The journey:** twelve weeks and 26 releases since the [first alpha](https://stoatflow.io/blog/introducing-stoatflow). Hot standby grew into multi-standby HA with in-place recovery, state stores became bounded, metrics became Kafka Streams–compatible, emission became internally consistent, a rejected output record stopped taking its epoch down with it, and porting gained a codemod, a state-migration tool, and an AI skills pack.
> - **Compatibility:** [KSML](https://axual.github.io/ksml/1.3.0/){rel="&#x22;nofollow&#x22;"} — an entire framework built on Kafka Streams — now has an experimental StoatFlow integration. A framework exercises what no single application does: every public class, constructor, join variant, handler, and config key. Working through its findings — and our own audits behind them — grew the [compatibility matrix](https://stoatflow.io/docs/reference/ks-compatibility-matrix) from 417 tracked entries to 619, almost all of it around the DSL, not in it. Credit where it is due: [Jeroen van Disseldorp](https://www.linkedin.com/in/dizzl){rel="&#x22;nofollow&#x22;"}.
> - **The honest part:** before freezing, we ran three full-codebase review rounds. They surfaced 4 critical and 36 high-severity findings; every one is fixed, and the later rounds re-audited the earlier fixes rather than taking them on trust.
> - **GA:** no date. The candidate proves out against real workloads first — what remains is proof, not features.

StoatFlow 1.0.0-rc.1 is cut, and with it the 1.0 feature set is locked. From here to general availability the rule is strict: bug fixes only — no new features, no API changes, no configuration changes. The eight beta releases carried the last of it: a correctness campaign for the most part, and in the final two a handful of behaviour changes we would rather make before a freeze than after one. What separates the candidate from 1.0.0 is proof, not features.

This post is the account of the twelve weeks between the [first alpha](https://stoatflow.io/blog/introducing-stoatflow) and this candidate: the journey in three acts, the framework integration that sharpened what "Kafka Streams–compatible" means here, the bugs we found because we went looking for them — and the ledger of limitations the launch post named, settled item by item.

## Twelve weeks, three acts

**Act one — make it port (alpha.1–alpha.13).** Compatibility was the premise, not the project: at alpha.1 the [compatibility matrix](https://stoatflow.io/docs/reference/ks-compatibility-matrix) already tracked the DSL, the topology API and the state stores method by method, with nothing marked missing. The early alphas worked the level beneath that: the accessors, store suppliers, `describe()` output, interactive-query metadata, and exact overloads that real Kafka Streams code actually calls. Two milestones anchor the act. The in-memory test driver became a drop-in — constructed the Kafka Streams way, so that, as the alpha.11 notes put it, "a Kafka Streams test suite ports with an import swap". And configuration followed in alpha.13: build StoatFlow's configuration from a standard Kafka-Streams-keyed `Properties` or `Map` — "a config that works with Kafka Streams now configures StoatFlow with no changes". The same act shipped [record headers in state stores](https://stoatflow.io/blog/kip-1271-record-headers-in-state-stores) with an in-place format upgrade, first-class Maven builds alongside Gradle, Flink-style watermark alignment, and [GraalVM native-image support](https://stoatflow.io/blog/native-image-g1-pgo-jni-vs-ffm).

**Act two — make it operable (alpha.14–alpha.18).** With porting credible, the middle alphas turned to running the thing. [Hot-standby high availability](https://stoatflow.io/blog/hot-standby-high-availability) arrived as an opt-in active/passive pair, then hardened into [multi-standby clusters with in-place recovery](https://stoatflow.io/blog/in-place-restart-multi-standby) — an engine that rebuilds inside the live process instead of bouncing the pod. Windowed, session, sliding, cogroup, and join stores became bounded; the alpha.17 notes name the defect plainly — "these stores previously grew without limit" — and that is the act in one line: say what was wrong, fix it, move on. Idle exactly-once applications stopped churning empty commits (roughly 200x fewer idle transactions when quiet). RocksDB internals became observable, and an opt-in metrics mode publishes under the Kafka Streams metric names, so existing Grafana dashboards and alerts keep working. The act closed with alpha.18: [barrier-consistent emission](https://stoatflow.io/blog/internal-consistency), a published OpenRewrite recipe that automates the code port, and licence validation that defers itself out of the way of CI runs.

**Act three — make it correct (beta.1–beta.8).** The betas were a different kind of work. The beta.1 note says it plainly: "a top-to-bottom internal review of the entire codebase drove a full pass of fixes, hardening exactly-once behaviour and Kafka Streams compatibility ahead of general availability." beta.2 is the correctness release — eleven named fixes plus a sweep of smaller ones, which the review section below returns to. beta.3 shipped the two halves of an assisted migration: the [AI assistant skills pack](https://stoatflow.io/blog/ai-assistant-skills) and a [data-migration tool](https://stoatflow.io/docs/migration/with-data-migration) that carries an existing application's state across the cutover. beta.4 added `processor.wrapper.class` — decorate every node in a topology, DSL-built and Processor API alike, through the same interface Kafka Streams uses — and published the documentation machine-readably as [`llms.txt`](https://stoatflow.io/blog/llms-txt-machine-readable-docs). beta.5 turned an undiagnosed RocksDB startup failure into a supported, explained downgrade path for header-format stores — one Kafka Streams refuses outright — and closed a metrics audit: every meter documented, every documented meter real. And beta.6 closed the range-query surface: open-ended (`null`) bounds now behave across every store type the way Kafka Streams treats them, and `reverseRange` takes its bounds low-first the way Kafka Streams does — passed the other way round it had been quietly returning the wrong answer rather than complaining. The same release stopped the changelog topics behind headers-enabled window, session and versioned stores growing without limit; deployments already carrying one need a manual `kafka-configs --alter` to recover the space.

Then two more releases, and neither was polish. beta.7 stopped splitting a topology at every key change and started splitting it where Kafka Streams would — `selectKey → mapValues → to()` is one sub-topology now, not two — and made unprefixed Kafka client properties actually apply: a ported application's `security.protocol` and `sasl.*` had been recognised, warned about, and then ignored, so it connected unauthenticated and unencrypted. It also closed a race between closing a RocksDB store and flushing it that took the whole JVM down with a `SIGSEGV` — on the standby, which meant a promotion could kill the instance meant to replace the active one. beta.8 was the largest release on the line, and its centre is the poison-epoch rewrite: a record the broker rejects poisons the transaction it sits in, and StoatFlow used to abort that transaction and advance past it, losing every innocent record that happened to share the epoch. Now the source offsets are held, the epoch is replayed with the one bad record quarantined, and its epoch-mates commit normally. Alongside it, the dead letter for an oversized record became deliverable at all: it had been assembled larger than the record the broker had just refused, so a DLQ topic sized like the output topic refused it too — nothing was ever written, in precisely the case a DLQ exists for. Two more from that release deserve naming, because neither was visible from outside: an engine restart arriving mid-commit could break exactly-once, silently; and an at-least-once topology with no sink committed nothing at all — no offsets, no changelog — so every restart replayed its input from the beginning. The one deliberate tightening in the set: a Processor API processor may now only reach the stores it declared, where before it could quietly read someone else's.

The numbers underneath the acts, for the record:

| The number                                                        | What it measures                                                                                                   |
| ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| 27 releases on the 1.0.0 line (18 alphas, 8 betas, one candidate) | a release roughly every three days, for twelve weeks                                                               |
| 1,086 commits, 94 merged pull requests                            | the volume of change between alpha.1 and the candidate                                                             |
| 212 → 586 test files, 3,136 → 5,721 tests                         | 4,353 unit, 840 end-to-end through the in-memory test driver, 528 integration against real brokers and real builds |
| 417 → 619 tracked entries in the compatibility matrix             | the portability surface, pinned method by method — next section                                                    |
| 4 critical + 36 high review findings, all fixed                   | three full-codebase review rounds — two sections down                                                              |

One caution about reading that table: volume is evidence of effort, not of correctness. Nothing in a commit count says the engine is right. That claim belongs to the two sections that follow — the integration that tried to break compatibility from the outside, and the reviews that tried to break the engine from the inside.

## The integration that kept us honest

The most useful thing that happened to StoatFlow in these twelve weeks was another engineer's test suite.

Early in the alpha, [Jeroen van Disseldorp](https://www.linkedin.com/in/dizzl){rel="&#x22;nofollow&#x22;"} — CTO of [Axual](https://axual.com/){rel="&#x22;nofollow&#x22;"} and creator of [KSML](https://axual.github.io/ksml/1.3.0/){rel="&#x22;nofollow&#x22;"} — began integrating StoatFlow as an alternative engine underneath KSML. KSML is an open-source framework that builds Kafka Streams applications from YAML and Python, without writing Java. That is not porting *an application*: it means swapping the engine underneath an entire framework and running the framework's own test suite against the replacement. The integration is experimental — an honest effort to support StoatFlow as a second engine, and to test how far Kafka Streams portability really goes.

It is also the hardest compatibility exercise we could have asked for. A framework touches essentially the whole public surface: all the operators, not one topology's worth; every constructor and overload, because a framework cannot pick the convenient one; the exception handlers, the lifecycle state machine, the configuration surface down to individual keys, the serde boundaries, and the test harness its own suite runs on. KSML's dynamic data model reaches cases hand-written Java never produces — a key that is not null but *serialises* to null bytes, serdes chosen at runtime, topic names from a custom extractor. The findings arrived in rounds through June; our own porting audits kept pulling on the same threads well after.

The measure of that work is the public [compatibility matrix](https://stoatflow.io/docs/reference/ks-compatibility-matrix). At alpha.1 it tracked 417 methods and behaviours across 62 API surfaces, and none was marked "not implemented" — the operator core was compatible on day one; that was the premise of the product. Twelve weeks later it pins 619 entries across 72 surfaces, and the growth sits exactly where a framework lives and an application port rarely looks: all 74 `StreamsConfig` keys of Kafka Streams 4.3 accounted for — mapped, passed through to the Kafka clients, or documented as moot on a single instance; the test harness pinned method by method, all green; topology description, interactive-query metadata, exception handlers, lifecycle, each from present to pinned. The early waves were about API shape — does Kafka Streams–idiom code compile? The later ones were about execution fidelity — does it behave, configure, observe, and test the same?

Some of what came out you have already met: the drop-in test driver and the Kafka Streams config model in act one both trace back to these rounds. The rest reshaped the surfaces around the DSL:

- **Runtime behaviour.** `StreamsUncaughtExceptionHandler.REPLACE_THREAD` stopped being a documented downgrade and became a real recovery — the [in-place engine restart](https://stoatflow.io/blog/in-place-restart-multi-standby). Null keys are dropped before aggregation exactly as Kafka Streams drops them, across all nineteen aggregation operators.
- **Handlers and interop.** The [error-handling surface](https://stoatflow.io/blog/error-handling-three-gates-one-transaction) follows the Kafka Streams handler shape; a custom `KafkaClientSupplier` uses the exact Kafka Streams signature; `describe()` prints the Kafka Streams topology description, character for character where tooling depends on it.
- **Tooling.** The port path itself got tooled: the OpenRewrite recipe rewrites the code, the migration tool carries the state, and the skills pack keeps AI assistants writing StoatFlow rather than half-remembered Kafka Streams.

The integration also exposed a process weakness we did not like: divergences were being discovered by a person, not by our CI. That is now guarded — build gates compile Kafka Streams–idiom Java against the artefacts we actually publish, so the next divergence has to get past a machine before it can reach anyone's port.

## We went looking

Compatibility is one axis of confidence. The other is whether the engine is right — and for that we did not wait for reports.

Before freezing the surface we ran three full-codebase review rounds, each one reading the engine end to end against a fixed set of lenses: correctness, exactly-once semantics, thread safety, crash windows. Between them they surfaced **four critical and thirty-six high-severity findings. Every one is fixed** — and because a fix you have not re-verified is just a claim, the later rounds re-audited the earlier rounds' fixes rather than taking them on trust.

The beta.2 release notes are what that looked like from the outside: eleven named fixes plus a sweep. Three are worth retelling for what they teach:

- **KTable-KTable join retractions.** A non-materialised join did not forward the `(key, null)` retraction when one side was deleted — a downstream store or compacted topic kept the stale pre-delete join result forever. Correct-looking output, permanently wrong at rest.
- **Versioned-store reads during commits.** A versioned store read could return a stale older version for the whole commit window. The rewrite that fixed it also made the read path measurably faster — the fix and the optimisation were the same change.
- **Foreign-key join consistency under exactly-once.** When the two sides of a foreign-key join process concurrently, an intermediate update could — around a crash — go permanently missing from the output stream. Final values always converged, which is precisely why nobody would have noticed.

None of the three arrived as a field report. All of them came out of reviews that went looking — which is the point. A release candidate is not a claim that no bugs remain; it is the evidence of having gone looking, three times over, with the findings written down and closed.

## The ledger from May

The launch post made claims, and — deliberately — named its own limitations. A release candidate should answer for both.

**"State migration is a reprocess, not a restore" — closed.** In May, the recommended path onto StoatFlow was to reprocess your input topics; if retention ruled that out, we asked you to get in touch. Now the [migration tool](https://stoatflow.io/docs/migration/with-data-migration) translates a Kafka Streams application's changelog topics into StoatFlow's format and carries the input offsets across, so a stateful cutover resumes exactly where Kafka Streams stopped — reading the old application's topics, never modifying them.

**"Failover behaviour is a fair question" — answered, with measurements.** The launch post admitted it had no failover story. It now has one with numbers attached: [hot standby](https://stoatflow.io/blog/hot-standby-high-availability), grown into multi-standby clusters with a lag-aware election and [in-place recovery](https://stoatflow.io/blog/in-place-restart-multi-standby), and [measured scenario by scenario from the pod logs](https://stoatflow.io/blog/ha-failover-testing). "The docs are still being written" no longer applies either — the documentation now runs from getting started through operations and reference.

**The single-machine ceiling — stands.** On the benchmarked 8-vCPU machine it is 200–300 MB/s of uncompressed throughput; higher-end hardware remains unbenchmarked. That number moved nowhere in twelve weeks because we spent them elsewhere. It is still the honest boundary of the design.

**No horizontal scale-out — stands, by design.** Twenty-six releases changed a great deal, and none of it touched the bet the product is built on: one instance per application, and no *distribution tax* for scale-out your workload does not need. Standbys are redundancy, not throughput. To go faster, scale up.

Two closed, two standing — and the two that stand are the product, not the gap.

## What a release candidate means here — and what's next

Feature-locked is a discipline, not a mood. Between rc.1 and 1.0.0 the only changes are bug fixes; the API, the configuration surface, and the wire formats hold still. There is no GA date to announce, deliberately: the candidate now proves out against real workloads — ports, soak runs, failure drills — and 1.0.0 follows the evidence, not a calendar.

Locking 1.0 also frees the thinking past it. Three items lead the 1.1 line: **Flink-style side outputs**, so a processor can route to multiple named outputs instead of contorting a topology around one; an **async I/O processor**, for calling slow external services at high parallelism without giving up per-key ordering; and **interactive queries over REST**, reading state stores over HTTP without writing a server. Behind those sits a bench that is designed and reviewed but not scheduled: multi-table joins, CEP pattern recognition, read-only queries served from hot standbys. Directions, not commitments — the only commitment right now is the freeze.

## Credits

A warm special thanks to [Jeroen van Disseldorp](https://www.linkedin.com/in/dizzl){rel="&#x22;nofollow&#x22;"}, CTO of [Axual](https://axual.com/){rel="&#x22;nofollow&#x22;"}, who built the experimental KSML integration and filed findings against us round after round. The compatibility half of this story owes its precision to those reports — the matrix is as sharp as it is because of them. If you want to see what a framework on top of Kafka Streams looks like, [KSML is worth your time](https://axual.github.io/ksml/1.3.0/){rel="&#x22;nofollow&#x22;"}.

Thanks as well to everyone who tested StoatFlow, put it through an evaluation, or reviewed our work and told us what was wrong. rc.1 is better for it.

**Read on:**

- [StoatFlow: Kafka Streams compatible engine built to scale up — not out](https://stoatflow.io/blog/introducing-stoatflow) — the first alpha, and the ledger this post settles.
- [Hot standby](https://stoatflow.io/blog/hot-standby-high-availability), [measuring failover](https://stoatflow.io/blog/ha-failover-testing), and [in-place restart + multi-standby](https://stoatflow.io/blog/in-place-restart-multi-standby) — the availability arc.
- [Internal consistency on Kafka](https://stoatflow.io/blog/internal-consistency) — barrier-consistent emission, measured to zero.
- [Three gates, one transaction](https://stoatflow.io/blog/error-handling-three-gates-one-transaction) — the error-handling model.
- [Migration](https://stoatflow.io/docs/migration) — the tooled port path, code and state.
- [Getting started](https://stoatflow.io/docs/getting-started).

For what happens between here and 1.0.0, [reach out](https://stoatflow.io/contact) — and for the running commentary on how StoatFlow gets built, [follow along on LinkedIn](https://www.linkedin.com/in/hartmut-co-uk/){rel="&#x22;nofollow&#x22;"}.


# KIP-1035: Why Kafka Streams 4.3 lets state stores own their offsets

> **TL;DR**
>
> - **What changed:** Kafka Streams 4.3 (22 May 2026) ships KIP-1035 — state stores now manage their own changelog offsets, persisted atomically with the data, instead of in a separate `.checkpoint` file.
> - **Why it mattered:** offsets and state lived in different places with no atomicity. On an unclean shutdown they could disagree — and under exactly-once that meant discarding local state and restoring the whole changelog.
> - **What it unlocks:** consistent offset-and-data recovery, no full re-restore after a crash, and — the real prize — the hard prerequisite for transactional state stores (KIP-892).
> - **Who:** designed and contributed by Nick Telford (Meltwater), split out of his KIP-892 work. His Kafka Summit London 2024 talk is the best primer.
> - **In StoatFlow:** the same design from day one — changelog offsets live in a RocksDB `_offsets` column family, flushed atomically with the data, with no checkpoint file to retire and no opt-in. Commit barriers plus a strict data-before-offsets ordering reach the KIP-892 prize from the other direction: delta restore in under a second, whatever the state size.

The Kafka 4.3 release notes describe KIP-1035 as "an internal runtime change, and only relevant for custom StateStore implementations." That's accurate, and it undersells it. The change is small in surface area and large in consequence: it fixes a structural weakness that has shaped how Kafka Streams recovers state for years, and it removes the last roadblock in front of transactional state stores.

## The checkpoint file, and what it was for

Before 4.3, a Kafka Streams instance tracked changelog offsets in a small file on disk — one `.checkpoint` file per task directory, maintained by the Streams engine itself, independently of whatever StateStore you were running.

The file's job is simple. When a task commits, Streams records the changelog offset that corresponds to the local state it has written: "the state for this store is caught up to changelog offset N." That record is read in two situations:

- **At restore.** On startup, Streams compares the checkpointed offset to the end of the changelog. Match, and the local state is current — no restore needed. Behind, and it replays the changelog from the checkpoint forward.
- **At rebalance.** The offsets are read to work out how far behind each task's local state is, so the assignor can hand a task to whichever instance holds the warmest copy.

![Before KIP-1035: a Kafka Streams task directory holding the RocksDB state store next to a separate .checkpoint file. RocksDB flushes on its own schedule, the Streams engine writes the file on commit, and nothing makes the two writes atomic — two writers, two moments, no shared commit.](https://stoatflow.io/assets/blog/kip-1035_before-separate-checkpoint.svg)

## Where it broke

The weakness is in that last line: two write paths, no shared commit. The state data and the offset that describes it are persisted separately, and nothing makes those two writes atomic.

On a clean shutdown that's fine — everything flushes in order. On an unclean shutdown it isn't. RocksDB buffers writes in memtables and may not have flushed them to disk when the process dies; the checkpoint file, written on its own path, can end up ahead of the data it points at. After the crash, the offset and the state disagree.

Kafka Streams handles that disagreement conservatively. Under exactly-once, the on-disk state can be inconsistent with the checkpoint, so on restart it discards the local state and restores the entire changelog from scratch. For a small store that's seconds. For a large one it is not.

How not-small? Nick Telford put real numbers on it at Kafka Summit London 2024, from Meltwater's production Streams deployment: around 45 state stores, changelogs totalling roughly 8 TiB, restorations spiking to about 10 million records in flight, and restore time scaling roughly linearly with total changelog size. When recovery time is proportional to how much state you hold rather than how much you were mid-processing, growing your state grows your worst-case downtime with it.

There's a second, quieter cost. To keep the on-disk state in step with the checkpoint, Streams force-flushes RocksDB memtables on commit — under at-least-once, once at least 10,000 records have been processed. That flush schedule is opaque and not configurable; it overrides whatever you set through `RocksDBConfigSetter`, so the engine and your tuning pull in different directions.

And then the one that blocks everything else: you cannot build transactional state stores on top of this. The KIP says it plainly:

> this KIP is a hard-dependency of KIP-892, as transactional behaviour requires that we can atomically sync the changelog offsets with their corresponding state, which is not possible while those offsets are tracked in the separate `.checkpoint` file.

If offsets live in a file the store doesn't control, the store can't commit its data and its offset as one unit — and atomicity is the whole point of a transaction.

![After an unclean shutdown the .checkpoint file still names changelog offset 4711, while the store's SST files only reach 4620 and the memtable holding the rest never got flushed. The offset is 91 records ahead of the data it describes — which forces a full changelog restore under exactly-once, and blocks transactional state stores.](https://stoatflow.io/assets/blog/kip-1035_unclean-shutdown-gap.svg)

## KIP-892 and the missing piece

The transactional-store work that needs this is also Nick's: KIP-892, Transactional StateStores. Its idea is to stop writing straight into RocksDB as records arrive, and instead buffer a task's writes — in a RocksDB `WriteBatchWithIndex` — applying them only when the Kafka transaction commits. Reads during the transaction are served from the buffer, so you still see your own writes, and interactive queries gain an isolation level: `READ_UNCOMMITTED` to see in-flight data, `READ_COMMITTED` to see only what's durable.

Done right, this means local state never gets ahead of the changelog. Uncommitted writes sit in memory and are simply dropped if the process dies, because they were never committed. Recovery stops depending on total state size.

But all of it rests on one requirement: committing the buffered data and its changelog offset together, atomically. With offsets stranded in a separate checkpoint file, you can't. So that requirement was lifted out of KIP-892 into its own proposal — "broken out to provide more focus and make it easier to contribute and review." That proposal is KIP-1035, and it had to land first.

## KIP-1035: the state store owns its offsets

KIP-1035's move is to make the StateStore responsible for its own changelog offsets, and to persist them atomically with the data. The checkpoint file stops being the source of truth.

It adds three methods to the `StateStore` interface, each with a default so existing stores keep compiling:

```java
// Does this store manage its own offsets? New stores return true.
@Deprecated
default boolean managesOffsets() {
    return false;
}

// Commit written records. If managesOffsets() is true, the given
// offsets are persisted to disk atomically with those records.
default void commit(final Map<TopicPartition, Long> changelogOffsets) {
    flush();
}

// The offset of the most recently committed changelog record for a
// partition — read back from the store itself.
default Long committedOffset(final TopicPartition partition) {
    return null;
}
```

`managesOffsets()` is deprecated the moment it ships. It exists only so the ecosystem can migrate; eventually every store is expected to manage its own offsets and the method goes away.

The built-in `RocksDBStore` opts in. It keeps its offsets in a dedicated RocksDB column family next to the data, and opens the database with atomic flush enabled — so when RocksDB flushes memtables to disk, the data and the offsets flush together, or not at all. The offset can no longer get ahead of the data it describes, because they're flushed in the same operation.

![After KIP-1035: one RocksDB database with a default column family holding the store's data and an offsets column family holding the changelog offset and the interactive-query position. Both are made durable by a single atomic flush — they reach disk together or neither does — and the .checkpoint and .position side files retire.](https://stoatflow.io/assets/blog/kip-1035_offsets-column-family.svg)

Two consequences fall out of that:

- **Flushing goes back to RocksDB.** Because offsets ride along with the data automatically, Streams no longer forces a flush on commit to keep a separate file in sync. Flush timing is dictated by your RocksDB configuration again — the way `RocksDBConfigSetter` always implied it should be.
- **The side files retire.** The `.checkpoint` file is replaced; the `.position` file used by interactive queries (KIP-796) moves into the same column family. Both migrate automatically on first start under 4.3. `flush()` is deprecated in favour of `commit()`, and the `flush-*` metrics give way to `commit-*` ones.

For custom StateStores, nothing breaks. The defaults keep the legacy checkpoint-file behaviour until you opt in by implementing the three methods and returning `true` from `managesOffsets()`.

## What it unlocks

The headline is consistency: after KIP-1035 the recorded offset can never be ahead of the data it names, on any kind of shutdown. That single guarantee changes recovery.

- **No full re-restore after a crash.** With offsets and data consistent, a restart only replays the gap between the store's committed offset and the end of the changelog — the uncommitted tail — not the whole topic. Recovery time tracks how much you were mid-processing, not how much state you hold.
- **RocksDB tuning means what it says.** Flush scheduling is yours again; the engine isn't forcing flushes behind your back to keep a file in step.
- **Transactional state stores become buildable.** This is the real prize. With atomic offset-and-data commits in place, KIP-892 can keep local state across restarts instead of rebuilding it, and the isolation levels for interactive queries have a consistent store to read from.

The release notes were right that it's an internal change. They were modest about what it makes possible.

## StoatFlow was built on this from the ground up

StoatFlow took this as a founding decision rather than a migration. Its state engine never had a checkpoint file to retire — changelog offsets have always lived inside RocksDB, in a dedicated `_offsets` column family, flushed atomically with the data through the same atomic-flush mechanism KIP-1035 standardises for Kafka Streams.

Building greenfield, and building single-instance, meant we carried none of the legacy. There's no `managesOffsets()` defaulting to `false`, no opt-in for custom stores, no automatic migration from `.checkpoint` and `.position` files — store-managed offsets are the only mode. And without a consumer group or rebalancing, the offset story is simpler still: one instance, one set of offsets, fixated at each commit barrier.

We push the same idea a step further in two places. Commits flow through Flink-style commit barriers rather than per-task Kafka transactions, and we enforce a strict data-before-offsets ordering — an epoch's offset is written only once that epoch's data has landed, so a timeout can leave the offset behind the data but never ahead of it. The payoff is the one KIP-892 is chasing: delta restore in under a second regardless of total state size — reached from a different starting point.

None of this is our idea. It's Nick Telford's design, adopted early because it was the right one.

## Credit where it's due

KIP-1035 — and the transactional state stores it makes possible — are the work of [Nick Telford](https://www.linkedin.com/in/nicholastelford/){rel="&#x22;nofollow&#x22;"}, who proposed, designed, and contributed both KIPs while running Kafka Streams at scale at Meltwater. It is a substantial piece of engineering: a change to a foundational interface, carried through with the backward-compatibility and migration care that a system as widely deployed as Kafka Streams demands.

If you want the full story from the source, his Kafka Summit London 2024 talk, [*Improving Streams Scalability with Transactional StateStores (KIP-892)*](https://www.confluent.io/events/kafka-summit-london-2024/improving-streams-scalability-with-transactional-statestores-kip-892/){rel="&#x22;nofollow&#x22;"}, is the best primer — it's where the Meltwater numbers above come from, and it walks the problem and the design far better than a release note can.

**Read on:**

- [KIP-1035: StateStore managed changelog offsets](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1035%3A+StateStore+managed+changelog+offsets){rel="&#x22;nofollow&#x22;"}
- [KIP-892: Transactional StateStores](https://cwiki.apache.org/confluence/display/KAFKA/KIP-892%3A+Transactional+Semantics+for+StateStores){rel="&#x22;nofollow&#x22;"}
- [Nick Telford — Improving Streams Scalability with Transactional StateStores (Kafka Summit London 2024)](https://www.confluent.io/events/kafka-summit-london-2024/improving-streams-scalability-with-transactional-statestores-kip-892/){rel="&#x22;nofollow&#x22;"}
- [Apache Kafka 4.3.0 release announcement](https://kafka.apache.org/blog/2026/05/22/apache-kafka-4.3.0-release-announcement/){rel="&#x22;nofollow&#x22;"}


# Compiling StoatFlow to a GraalVM native image: G1, PGO, and why JNI beats FFM under AOT

> **TL;DR**
>
> - **Native image is viable today** — but only with **Oracle GraalVM + G1 + PGO**. GraalVM Community Edition ships Serial GC only, and Serial GC can't carry an allocation-heavy stateful workload.
> - **We blamed FFM; it was the GC.** A stateful app that "couldn't keep up" under native looked like a RocksDB FFM problem. Switching Serial → G1 fixed it outright, to JVM parity. The bottleneck was the garbage collector, not the foreign-function path.
> - **JNI beats FFM under AOT** — the reverse of the JVM. On HotSpot the JIT intrinsifies FFM's `critical(false)` downcalls; SubstrateVM has no JIT, so a plain JNI call ends up cheaper. This drove a new `AUTO` backend default: &#x2A;*FFM on the JVM, JNI under native image.**
> - **The cost, across all four apps:*&#x2A; CPU within &#x2A;*−4% to +10%** of the JVM (the Avro stateful app is *cheaper*), latency equal-or-better everywhere, after PGO. Down from +42% / +75% on the Serial floor.
> - **Memory is tunable, not free.** Serial native is tiny (78 MB RSS for a stateless app vs 496 MB on the JVM). G1 sizes its heap from `-Xmx` and sprawls unless you pin it; the stateful app still lands 23% *below* the JVM.
> - **The serde / security surface is still being widened** — Avro + Schema Registry, SASL, OAuth, mTLS, and non-zstd codecs all need native metadata we haven't fully mapped yet.

We've been building and running StoatFlow applications as GraalVM native images. For a throughput-first stream processor the interesting question isn't the one native image is usually sold on — cold-start and footprint — it's whether steady-state CPU and tail latency hold up under sustained load. They do: with Oracle GraalVM, G1, and Profile-Guided Optimisation, the four apps we measured run within **−4% to +10% CPU** of the JVM, latency equal-or-better everywhere — after one wrong turn worth the telling.

## Where we started

A StoatFlow app doesn't begin from a blank GraalVM configuration. The framework already bundles native-image support, discovered automatically the moment an app depends on it:

- **`:core`** ships the metadata for everything the engine touches — the Kafka serdes, assignors and partitioners, and the RocksDB API behind both backends: the JNI bindings *and* the FFM foreign downcalls to RocksDB's C API, plus JCTools and zstd. GraalVM finds it by classpath (`META-INF/native-image/io.stoatflow/core/`), so any dependent app inherits it.
- **`:runtime`** adds the configuration layer — the Hoplite YAML binding for the runtime config classes, Logback, Micrometer/HdrHistogram, and the rest of the runtime stack.
- **The `io.stoatflow` Gradle plugin** turns it on in one line: `nativeImage { enabled = true }` applies GraalVM's build plugin, the `--initialize-at-run-time` flags that RocksDB and the Kafka SASL client need, and a `nativeDockerBuild` task that drives a multi-stage `Dockerfile.native`.

So compiling a StoatFlow app to a native image is a one-line opt-in: the framework covers its own reflective, JNI and FFM surface, and the only per-app work is registering the metadata for your own serde value classes. That gets you a binary that runs. This post is about the next question — does it run *well*? The honest answer runs through a wrong turn, a reversal, and a memory result you have to tune for.

We benchmarked four apps; the deep dive that follows uses two representatives:

- **`stateless-simple`** — a stateless 1 KB transform at 20K msg/s. No state stores.
- **`word-count`** — stateful, RocksDB-backed, \~300K records/s processed (7.5K msg/s in, fanned out by aggregation).

Both ran under exactly-once semantics, four configurations each, 10-minute warmup + 10-minute measurement on a Hetzner `ccx33` worker (8 dedicated vCPU); the full four-app matrix closes the post. What follows is the honest version, wrong turns included — because the wrong turns are where the interesting engineering lives.

## Act 1 — The floor: GraalVM CE and Serial GC

GraalVM Community Edition only ships **Serial GC** for native image. G1 and the parallel collector are Oracle-GraalVM-only. So the first build we measured had no choice in collector — Serial it was.

The stateless app worked, but at a price: **+42% CPU** vs the JVM (1.7 cores vs 1.2). Tolerable, if not free.

The stateful app didn't work at all. It couldn't keep up:

| Metric           |       JVM |   Serial native |
| ---------------- | --------: | --------------: |
| True E2E P50     |  120.6 ms | **13,504.7 ms** |
| Max consumer lag |       100 |      **17,498** |
| Avg CPU          | 2.0 cores |       3.5 cores |

That's a **112× regression** in end-to-end latency and a **175× regression** in lag. The app was falling behind the input and never catching up.

The obvious suspect was the RocksDB **FFM** (Foreign Function & Memory API) backend. StoatFlow talks to RocksDB's C API through FFM downcalls, and native image has to do special gymnastics to make those work (more on that below). It's the most exotic part of the native build, and when the workload that touches state explodes, you blame the exotic thing.

**We were wrong.**

## Act 2 — The reversal: Oracle GraalVM and G1

Oracle GraalVM is now free for production use under the GFTC licence, and it unlocks two things CE doesn't: the **G1 garbage collector** and **PGO** (Profile-Guided Optimisation).

We rebuilt the stateful app with nothing changed but `--gc=G1`:

| Metric           |      JVM |      Serial |       **G1** |
| ---------------- | -------: | ----------: | -----------: |
| True E2E P50     | 120.6 ms | 13,504.7 ms | **128.1 ms** |
| Max consumer lag |      100 |      17,498 |       **91** |
| Avg CPU          |      2.0 |         3.5 |          2.9 |

G1 fixed it **outright**. 128 ms P50, 91 lag — that's JVM parity. The 112× latency blowup was gone with a one-flag change to the garbage collector.

So the "FFM is the bottleneck" hypothesis was simply false. &#x2A;*The stateful failure was GC-bound, not FFM-bound.** Serial GC, single-threaded and stop-the-world, was choking on the allocation churn of the stateful path (RocksDB read/write buffers, aggregation state, serialisation). G1's concurrent and parallel collection kept up where Serial stalled. The FFM path had been carrying the heat for the GC the whole time.

This is the beat worth slowing down on: the exotic component is rarely the bottleneck. The boring component — the collector that runs on every allocation — was. We reached for the interesting explanation and missed the mundane one. The benchmark matrix is what caught it, not intuition.

One caveat we name in the report: this comparison mixes two variables. Serial was GraalVM CE; G1 was Oracle GraalVM, which also applies optimisations CE lacks. So "CE-Serial → Oracle-G1" is edition *and* collector. The actionable comparison is JVM vs the best shippable native build — and that's what every headline number here uses.

## Act 3 — The CPU lever: PGO

G1 brought latency and throughput to parity, but CPU was still high: &#x2A;*+33% (stateless)*&#x2A; and &#x2A;*+45% (stateful)** vs the JVM. The JVM's advantage here is the JIT — it profiles the running code and recompiles hot paths with real branch and type information. A native image is compiled ahead of time with static heuristics; it never sees the workload.

**PGO closes that gap.** It's a two-pass build:

1. Compile an **instrumented** binary (`--pgo-instrument`).
2. Run it under representative load to collect a `.iprof` profile.
3. Rebuild with the profile (`--pgo=<app>.iprof`) so the AOT compiler gets the hot-path inlining and code layout the static heuristics miss.

PGO is purely a CPU lever — it doesn't touch latency or throughput, which were already at parity. It cut CPU:

| App       | G1 (no PGO) |  **G1 + PGO** |
| --------- | ----------: | ------------: |
| stateless |   1.6 cores | **1.4 cores** |
| stateful  |   2.9 cores | **2.4 cores** |

That's −13% and −17% on top of G1. The residual gap to the JVM lands at &#x2A;*+17% (stateless)*&#x2A; and &#x2A;*+20% (stateful)** — down from the Serial floor of +42% / +75%. That residual is the irreducible AOT-vs-JIT peak-code difference, plus the FFM downcall path under SubstrateVM, which is the next act.

## Act 4 — The counterintuitive finding: JNI beats FFM under AOT

Here's the part that inverts conventional wisdom.

**On the HotSpot JVM, StoatFlow's FFM RocksDB backend is faster than the JNI one.** The reason is `Linker.Option.critical(false)` — it marks a downcall as not needing a full safepoint transition, and the JIT *intrinsifies* it, compiling the foreign call down to nearly a direct C call. FFM on a warm JVM is genuinely excellent.

**Under SubstrateVM (native image), there is no JIT.** That intrinsification never happens — there's no profiling compiler to perform it. So the `critical(false)` optimisation that makes FFM win on the JVM simply doesn't fire. And once you strip it away, an FFM downcall ends up *slower* than a plain JNI call. (Both pin the virtual thread for the duration of the native call anyway, so there's no Loom advantage either way.)

We measured it on word-count, G1+PGO, swapping only the RocksDB backend:

| Metric           | FFM (native) | **JNI (native)** |
| ---------------- | -----------: | ---------------: |
| Avg CPU          |    2.4 cores |    **2.2 cores** |
| Avg RSS          |       606 MB |           601 MB |
| True E2E P99     |     226.9 ms |     **211.8 ms** |
| Produced rec/s   |      304,725 |          307,418 |
| Max consumer lag |          113 |              102 |

JNI uses **\~8% less CPU** (2.2 vs 2.4 cores) with a **better P99** (211.8 vs 226.9 ms, −6.7%) and marginally higher throughput. &#x2A;*The optimisation that's correct on one runtime is wrong on the other.**

So the fix is to pick per-runtime. We added a new `RocksDbBackendType.AUTO` default that resolves to:

- **FFM** on the JVM — where the JIT intrinsifies it and it wins.
- **JNI** under a native image — where there's no JIT and JNI wins.

Each runtime gets its faster path automatically. This also has a pleasant side effect on the native build's fragility, which the snares section gets into: the FFM-under-native machinery is genuinely brittle, and `AUTO` takes it off the default path for native images entirely.

## Memory: real, but GC-dependent

Native image's headline memory win is real — and more nuanced than the marketing suggests.

**Under Serial GC, native is tiny.** The stateless app's RSS was **78 MB** vs the JVM's 496 MB — an 84% reduction. Serial keeps a compact heap and reclaims aggressively. That is a genuine, large win.

**Under G1, it depends on the workload's live footprint**, because G1 &#x2A;sizes its heap reservation from `-Xmx`* — and the container entrypoint hands `-Xmx` almost the whole container (≈15.6 GB). So G1 reserves generously:

- **Stateless app** (live heap \~40 MB): G1's reservation dominates RSS, which jumps to **587 MB — above the JVM's 496 MB**, and far above Serial's 78 MB. The Serial memory win is gone.
- **Stateful app** (larger live footprint): G1's reservation is comparable-to-lower than the JVM's heap + metaspace + JIT code cache, so RSS lands at **606 MB — 23% below the JVM's 791 MB**.

The takeaway: &#x2A;*G1 native wants a tight `-Xmx` to keep the memory win.** The stateless app needs \~40 MB of heap; giving G1 15 GB to reserve from is the whole problem. That's a tuning follow-up — an env-var override, no rebuild — and we haven't done it yet. So the honest statement is: native memory is a win you tune for under G1, not one you get for free.

## The snares

These are the discoveries — the things that cost an afternoon each and that you only hit because native image moves failures from compile time to first-record time. Closed-world compilation means *anything* reflective, JNI, or FFM must be declared up front; miss one and it surfaces as a runtime error the first time that code path executes, never at build.

### glibc version pinning

Oracle's default `native-image:25` builder image is built on Oracle Linux 10 (glibc 2.39). The binary it produces then *requires* `GLIBC_2.38` at runtime. But the distroless `cc-debian12` runtime image ships glibc **2.36** — so the binary crashloops on startup with `version GLIBC_2.38 not found`.

**Fix:** pin the builder to the `:25-ol9` tag (Oracle Linux 9, glibc 2.34). 2.34 ≤ 2.36, so the binary runs. The rule: **a native binary requires the build host's glibc version or newer at runtime** — build on the oldest glibc you can, run anywhere newer.

### reachability-metadata.json silently shadows the legacy reflection-config.json

GraalVM has two metadata formats: the older `reflection-config.json` and the newer `reachability-metadata.json`. When **both** are present, the newer file wins — and silently drops the old file's registrations, including the `allDeclaredConstructors: true` on our runtime config data classes. The symptom: a runtime `MissingReflectionRegistrationError` on a config class that *was* registered, in the file that *used* to win.

**Fix:** consolidate the registrations into the reachability-metadata file. Don't run the two formats side by side and expect them to merge — they don't.

### Everything reflective must be declared, and you find out at first contact

Each of these surfaced only when its code path first executed, never at build time:

- **JCTools** lane-queue field-holder classes (accessed via `Unsafe`) → `NoSuchFieldException`
- **HdrHistogram** `ConcurrentHistogram` → `NoSuchFieldException`
- **zstd-jni** native field handles → `NoSuchFieldError`
- **Kafka serialisers** instantiated by class name → `NoSuchMethodException`

There's no shortcut here other than discipline: a closed world means the failure budget moves entirely from compile time to first-record time. Drive every code path before you trust the binary.

### FFM under native image needs hand-rolled symbol resolution

This is the brittle one. `SymbolLookup.loaderLookup()` — the normal way to find a native function — &#x2A;*can't see the symbols of a JNI library loaded at runtime under SubstrateVM.** `librocksdbjni` is loaded by RocksDB's JNI bootstrap at runtime; the FFM backend then can't resolve `rocksdb_*` symbols against it.

The workaround:

1. Detect native image via the `org.graalvm.nativeimage.imagecode` system property.
2. Read the library's mapped path out of `/proc/self/maps`, then `libraryLookup` it explicitly.
3. `--initialize-at-run-time` for the FFM holder class.
4. Register **every** foreign downcall descriptor — and the leaf type includes the implicit function-pointer first argument, which is trivially easy to miss on one of forty descriptors.

It works, but it's fragile and **Linux-only** (the `/proc/self/maps` trick has no portable equivalent). The good news: the `AUTO` backend change (JNI under native) means this whole apparatus is **off the default path now**. It's there if you force FFM on native, but you have to ask for it.

### Avro + Schema Registry is the worst offender

Confluent's Avro stack is reflection-heavy in three compounding ways, and native image trips on all of them:

- **The serde config instantiates classes by name from its own defaults**, even properties you never set — e.g. `NullContextNameStrategy`. Symptom: `ConfigException: … could not be found` at startup, for a class you've never heard of.
- **The schema-id serialisers and the registry REST entities** (Jackson-deserialised from the registry's HTTP responses) all need registering.
- **Native image disables URL protocols by default**, so the HTTP schema fetch fails until you add `--enable-url-protocols=http,https`.

Out of this fell a clean ownership pattern: &#x2A;*keep serde reachability metadata in the app module, not in the engine.** The app is the closed-world boundary. StoatFlow ships metadata for everything *it* touches — the engine, Kafka client, RocksDB (both backends), JCTools, Micrometer/HdrHistogram, the config layer. Customers add only their own value classes. The tracing agent catches the config-default classes in one pass and is the surer route for anything Avro. (Full recipe in the [custom-serde native-image guide](https://stoatflow.io/docs/runtime/native-image#custom-serde-metadata).)

### Extracting a PGO profile from a server that never exits

PGO's instrumented binary only writes its `.iprof` profile **on clean shutdown**. There's no on-demand dump API. A streaming app runs forever and never exits on its own, so getting the profile out takes some plumbing.

The trick that's race-free:

- An entrypoint that runs the binary as a **child process** (not `exec`), and **forwards SIGTERM** to it — so `kubectl exec -- kill -TERM 1` drains and dumps the profile.
- After the child exits, the entrypoint runs `sleep infinity` so the pod stays alive (and the liveness probe's eventual restart doesn't race the extraction).
- Dump to an &#x2A;*`emptyDir`** so the file survives that restart, then `kubectl cp` it out.

Drive \~4 minutes of representative load, `kill -TERM 1`, copy the `.iprof`, rebuild with it. Done.

## Results

The full four-configuration matrix, both apps, JVM as the baseline. Every number traces to the [benchmark report](https://stoatflow.io/product/benchmarks){rel="&#x22;nofollow&#x22;"}; deltas are vs the JVM.

### stateless-simple (20K msg/s, stateless)

| Metric                 |    JVM | Serial |     G1 | **G1+PGO** | Δ vs JVM |
| ---------------------- | -----: | -----: | -----: | ---------: | -------- |
| Produced rec/s         | 19,962 | 19,942 | 19,954 |     19,947 | ≈ 0%     |
| Max consumer lag       |    326 |    353 |    323 |        339 | ≈        |
| True E2E P50 (ms)      |  128.9 |  131.7 |  129.1 |      128.8 | ≈ 0%     |
| True E2E P99 (ms)      |    252 |  233.7 |  250.2 |      229.6 | **−9%**  |
| Avg CPU (cores)        |    1.2 |    1.7 |    1.6 |    **1.4** | **+17%** |
| Avg container RSS (MB) |    496 | **78** |    596 |        587 | **+18%** |

Throughput and latency parity (P99 actually *beats* the JVM by 9%) at +17% CPU. The memory catch is real: G1's heap reservation pushes RSS 18% above the JVM. Serial native's 78 MB is the memory-optimal build here; G1+PGO is the CPU/latency-optimal one.

### word-count (7.5K msg/s in, \~300K/s processed, stateful — RocksDB)

| Metric                 |     JVM |       Serial |      G1 | **G1+PGO (FFM)** | Δ vs JVM |
| ---------------------- | ------: | -----------: | ------: | ---------------: | -------- |
| Produced rec/s         | 303,770 |      308,180 | 296,387 |          304,725 | ≈ 0%     |
| Max consumer lag       |     100 |   **17,498** |      91 |              113 | ≈        |
| True E2E P50 (ms)      |   120.6 | **13,504.7** |   128.1 |            124.8 | +3%      |
| True E2E P99 (ms)      |   211.3 |     17,233.9 |   248.8 |            226.9 | +7%      |
| Avg CPU (cores)        |     2.0 |          3.5 |     2.9 |          **2.4** | **+20%** |
| Avg container RSS (MB) |     791 |          142 |     779 |          **606** | **−23%** |

Serial couldn't keep up (112× E2E latency). G1 fixed it to parity; PGO trimmed CPU from +75% to +20%. And because this app's live footprint is larger, G1's reservation lands *below* the JVM's total — so the stateful native image keeps a 23% memory advantage **and** keeps up.

### FFM vs JNI backend, under native image (word-count, G1+PGO)

| Metric                 | FFM (native) | **JNI (native)** | Δ (JNI vs FFM) |
| ---------------------- | -----------: | ---------------: | -------------- |
| Avg CPU (cores)        |          2.4 |          **2.2** | **−8%**        |
| Avg container RSS (MB) |          606 |              601 | ≈              |
| True E2E P99 (ms)      |        226.9 |        **211.8** | **−7%**        |
| Produced rec/s         |      304,725 |          307,418 | ≈              |
| Max consumer lag       |          113 |              102 | ≈              |

JNI is the better native backend on every axis that moves — which is exactly the inversion of the JVM, and exactly why `AUTO` resolves to JNI under native image.

### All four apps under G1+PGO (with the AUTO backend)

We then re-benchmarked all four apps on the best config — G1 + PGO + `AUTO` (so the two stateful apps run on JNI). The two serde-heavy apps ran as native images for the first time; Protobuf and Avro + Schema Registry both processed cleanly under load. Numbers are native vs the JVM baseline:

| App                | Serde    |      Avg CPU (nat / jvm) |         RSS MB (nat / jvm) |    E2E P99 ms (nat / jvm) | Produced rec/s |
| ------------------ | -------- | -----------------------: | -------------------------: | ------------------------: | -------------: |
| stateless-simple   | String   |          1.3 / 1.2 (+8%) |           577 / 496 (+16%) | 216 / 252 (&#x2A;*−14%**) |         19,966 |
| word-count         | String   |         2.2 / 2.0 (+10%) |  600 / 791 (&#x2A;*−24%**) |           210 / 211 (≈0%) |        303,424 |
| stateless-advanced | Protobuf |  1.2 / 1.2 (&#x2A;*0%**) |            537 / 522 (+3%) | 206 / 244 (&#x2A;*−16%**) |         13,532 |
| stateful-joins     | Avro     | 2.7 / 2.8 (&#x2A;*−4%**) | 920 / 1151 (&#x2A;*−20%**) |  251 / 273 (&#x2A;*−8%**) |         20,213 |

That is the bottom line of this whole exercise. CPU lands between &#x2A;*−4% and +10%** of the JVM; **latency is equal-or-better on every app** (P99 −8% to −16% on three, parity on the fourth); and **memory beats the JVM on both stateful apps** (−20%, −24%), where the RocksDB working set is large enough that G1's reservation comes in under the JVM's heap + metaspace + code cache. The Protobuf app is at outright **parity** on CPU. Only the two tiny-heap stateless apps give back a little memory — the same G1 `-Xmx` sprawl, tunable. Throughput and lag are at parity throughout.

The standout: the Avro stateful-joins app — the most complex of the four, with windowed joins over five Avro streams — runs &#x2A;*leaner than the JVM on CPU, memory, and tail latency at once.** A long way from "native can't do stateful."

### Smaller on disk — and what the build costs

There's a footprint win too, the one native image is actually known for. word-count's native image is **147 MB** on disk against **222 MB** for the JVM build — a **34% reduction**. The other three apps land within a few MB of that same split (the native binaries are near-identical in size, \~147–155 MB; the JVM images \~222–233 MB), so one number stands for all four. Most of the native image is the GraalVM binary plus the distroless base; the JVM image carries a JRE and the shadow JAR.

What you pay for it is build time. A native PGO build is two passes — an instrumented compile, a few minutes of profiling load, then the optimised compile — each running minutes on a dedicated amd64 worker, where the JVM image is a shadow JAR plus a Jib layer in seconds. Call it a 10–50× build-time multiplier, on Oracle GraalVM with a buildx-on-Kubernetes builder to keep alive. It's a release-time cost rather than a per-change one — but it's real, and it's the honest other half of the footprint win.

### Can FFM be fixed on native?

The honest answer, after digging into it: not from our code, not today. The slowness is GraalVM's, and Oracle says so — their FFM tracking issue ([oracle/graal #8113](https://github.com/oracle/graal/issues/8113){rel="&#x22;nofollow&#x22;"&#x7D;) lists &#x2A;"improve downcall performance (currently always unoptimized)"* as an **open goal**, undelivered in GraalVM 25, with no committed date, and they explicitly intend to keep JNI and the `@CFunction` C-interface as the performance-grade native call paths. SubstrateVM emits a generic downcall stub and has no JIT to intrinsify the `critical()` hint the way HotSpot's C2 does.

We tried the one lever that lives in application code: `critical(true)` (allow-heap-access), which lets the hot path pass the on-heap key directly instead of copying it off-heap first. It doesn't recover the intrinsification — it only saves the copy — and a RocksDB key is a handful of bytes, so we didn't expect much. It turned out to be worse than "not much": under native image `critical(true)` changes the downcall's leaf types, so it needs a whole new set of foreign-metadata registrations and crashes at startup without them. It's not a flag, it's a rebuild-per-missing-entry slog — and it leaves the actual cost, the unoptimised stub, untouched. The other option, a `@CFunction` no-transition binding, *would* match HotSpot's fast path, but it's native-image-only: a third call path, not "FFM everywhere."

So the foreign-function path doesn't have a cheap fix on native, and — this is the part that matters — it doesn't need one *to ship today*. JNI already gets the stateful apps to JVM-parity-or-better. `AUTO` (JNI on native, FFM on the JVM) is the pragmatic default that routes each runtime to its faster path without anyone having to know any of this.

But `AUTO` is a stopgap, not the destination. The goal is the single most efficient, most resource-frugal path on each runtime — and on the JVM that's FFM, so what we actually want is FFM *everywhere*, not a runtime-dependent split. The GraalVM team has named the route: the slow stub is what **bound** downcall handles (the jextract pattern we use today) compile to; an **unbound** handle — the function pointer passed as a call argument, the holder class initialised at build time — is reported to reach near-`@CFunction` speed under native image. There's no flag for it yet, and the `registerForDirectDowncall` API that would formalise it is proposed but uncommitted ([oracle/graal #12219 / GR-75754](https://github.com/oracle/graal/issues/12219){rel="&#x22;nofollow&#x22;"}). So what's next is concrete and not ours to ship: rewrite the RocksDB FFM backend to unbound handles, validate it on GraalVM 25.1+, and the day native FFM matches JNI, `AUTO` collapses to FFM everywhere — one path, the most efficient one. Until that API lands, `AUTO` routes each runtime to its current best — eyes open that "current best" isn't yet "best possible."

## What's next

Native image is viable **today** for the surface we've covered: G1 + PGO + the `AUTO` backend get you JVM-parity-or-better latency and throughput, CPU within −4% to +10%, and memory that beats the JVM on the stateful apps. The work from here is widening that covered surface, and being honest that it isn't complete:

- **Kafka client security isn't covered yet.** SASL mechanisms (SCRAM, GSSAPI/Kerberos), OAuth bearer (`OAUTHBEARER`), and mTLS each pull in JAAS, login-module, and JCA reflection that will need native metadata. None of it is mapped yet.
- **Compression codecs beyond zstd.** snappy, lz4, and gzip each have their own JNI/native or reflective surface to register. Only zstd is covered today.
- **More as real workloads exercise more of the Kafka client and serde surface.** Closed-world compilation guarantees that every code path a customer hits for the first time is a potential `MissingReflectionRegistrationError` until we (or they) declare it.
- **The G1 `-Xmx` tuning follow-up.** A tight `-Xmx` should recover the stateless memory win that G1's default container-fill reservation currently gives away — cheap to do, not yet done.

The shape of the thing is clear: native image earns its keep for the covered workloads, the wrong turns taught us where the real costs live (the collector, not the foreign-function path), and the `AUTO` backend means you don't have to know any of this to get the faster path on each runtime. The surface we widen next is set by the workloads that hit it first — which serde and security paths a real deployment exercises is exactly what tells us what to map.


# KIP-1271: record headers in state stores, and the cost of a value format

> **TL;DR**
>
> - **What:** [KIP-1271](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1271%3A+Allow+to+Store+Record+Headers+in+State+Stores){rel="&#x22;nofollow&#x22;"} (storage) and [KIP-1285](https://cwiki.apache.org/confluence/x/4ow8G){rel="&#x22;nofollow&#x22;"} (DSL opt-in) let a state store persist a record's `Headers` alongside its value. Shipped in **Apache Kafka 4.3.0** (22 May 2026).
> - **Why:** a schema id, a `traceparent`, a tenant tag, or lineage that rides on the input record now survives aggregation — readable via Interactive Queries and on the changelog.
> - **Kafka Streams:** the core landed in 4.3.0 (timestamped key-value, windowed and session stores). Versioned stores, lazy header parsing, and some interactive-query verification are **still open pull requests**.
> - **StoatFlow 1.0.0:** the full set — **all four store families, versioned included** — plus lazy header parsing, on the **same on-disk and changelog bytes** as Kafka Streams. Level on the core; a step ahead on the pieces Kafka has not merged yet.
> - **The engineering lesson:** the on-disk change is one varint. Implementing it faithfully is an engine-wide change — because a value type is load-bearing.

A record's headers — the small key/value pairs that ride beside the key and value — carry the metadata a payload shouldn't: the schema id a Schema-Registry serialiser prepends, a `traceparent` for distributed tracing, a tenant or routing tag, a change-data-capture operation type. Through a stream processor they flow from input to processor to output. But at the **state store** boundary they have always stopped: aggregate a stream into a table, and the headers on the records that built each row are gone.

KIP-1271 and KIP-1285 close that gap, and they shipped in Apache Kafka 4.3.0. This post is three things in order: **what the KIP is**, **where Kafka Streams has got to** with it (and what is still in flight), and **how StoatFlow compares** — implementing the same feature, on the same bytes. The through-line is an engineering one: the on-disk change is a single varint, and implementing it faithfully is, unavoidably, an engine-wide change.

## The feature: headers that survive the state store

*Kafka Streams and StoatFlow alike — this section is the KIP itself.*

There are two KIPs. **KIP-1271** is the storage half: a state store can persist a record's `Headers` with the value, and you can read them back — through the Processor API, through Interactive Queries, and on the changelog. **KIP-1285** is the DSL half: you opt in per store, or set an application-wide default, and the framework attaches the *processing record's* headers for you, with no headers API in your topology code.

It is **opt-in by design** — a store changes format only when you ask for it. That is a deliberate choice, not a hedge: most stores don't need headers, and the ones that do, say so.

Persisting headers enables a few things that were awkward or impossible before:

- **Schema-id-aware deserialisation of stored state.** If the value was written by a Schema-Registry serialiser, the schema id lives in the headers. Persist them, and the value can be deserialised correctly on the way out — including from the changelog, by a different consumer.
- **Trace and lineage surviving an aggregation.** The `traceparent` of the record that last updated a row is right there on the row.
- **Routing or tenant metadata surviving a windowed aggregate**, readable later through an Interactive Query without re-deriving it.

## The format is small

*Universal — this is the KIP's on-disk layout.*

The format is the easy part. A stored value, with headers enabled, is laid out like this:

![An unsigned varint giving the length of the headers block, the block itself as one length-prefixed key and value per header, and then the payload the store already held — unchanged](https://stoatflow.io/assets/blog/kip-1271_value-format.svg)

The payload is exactly what the store held before — a timestamp and the value for timestamped stores, just the value for the others. There is no version byte; the store's location on disk is what tells the engine which format to expect. Empty headers cost one byte. That is the whole of it. Read the KIP and you would budget a day.

## Why it isn't small: a value type is load-bearing

*Universal — true of any KIP-1271 implementation, Kafka Streams included.*

The cost is not in the format. It is in how many independent code paths hold, move, or reconstruct a value — each of which must now carry headers too, correctly, on the commit-critical path.

- **There are four store families, not one.** Key-value, versioned, windowed, and session stores each have their own typed wrapper, caching overlay, changelog decorator, and backends (in-memory, single-RocksDB, segmented-RocksDB). And they are not copies of each other: versioned stores keep history rather than overwriting; session stores have no per-record timestamp; windowed and session stores are split across RocksDB segments by time. The same idea has to land four different ways.
- **The value serde signature changes.** Reading a schema id out of the headers at store-access time means the value must be deserialised *with* its headers in hand — the three-argument, header-aware serde. That is a new boundary on every read and write.
- **The changelog dictates where you pack.** The changelog carries the headers *natively* — as Kafka record headers, beside an unchanged value — not buried in the value bytes. So the in-memory cache holds headers separately and folds them into the on-disk layout only when it writes to RocksDB, never on the changelog path. Get that boundary wrong and it's invisible until another consumer — or another engine — reads the changelog and finds the headers in the wrong place.
- **Restoration is the inverse.** State rebuilds from the changelog (unpacked value + native headers), so restoration has to re-pack each record into the on-disk format as it writes it.
- **The DSL has to attach the headers for you.** The useful end state isn't "a header-aware store exists"; it's that an ordinary aggregation, opted in, persists the headers of the record being processed — with no headers API in the topology. That needs the current record's headers captured per processing context and threaded into the store behind an adapter.

How load-bearing? Kafka Streams implemented this across **more than forty pull requests**, spanning some two dozen Jira sub-tasks, over roughly five months: a foundation layer (the byte format, the header-aware serdes), then a multi-PR series per store family, then the DSL wiring. The format is one varint; the change is the engine.

## Kafka Streams vs StoatFlow

The core of KIP-1271/1285 shipped in Kafka **4.3.0**. A handful of pieces are still open pull requests on Kafka's side — versioned stores, lazy header parsing, and some interactive-query verification. StoatFlow implements the full set on the same bytes, which leaves it level on the core and a step ahead on the still-open pieces.

| Capability                                                                    | Kafka Streams 4.3.0 (22 May 2026)                                                                                                                                                                   | StoatFlow 1.0.0                                      |
| ----------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- |
| On-disk byte format `[headers_size][headers][payload]`                        | ✅                                                                                                                                                                                                   | ✅ byte-identical                                     |
| Changelog: unpacked value + native headers                                    | ✅                                                                                                                                                                                                   | ✅ byte-identical                                     |
| Header-aware (3-arg) value serde                                              | ✅                                                                                                                                                                                                   | ✅                                                    |
| Timestamped key-value / windowed / session headers stores                     | ✅                                                                                                                                                                                                   | ✅                                                    |
| Versioned headers store + versioned KTable in the DSL                         | 🚧 open PRs ([KAFKA-20399](https://issues.apache.org/jira/browse/KAFKA-20399){rel="&#x22;nofollow&#x22;"} / [20400](https://issues.apache.org/jira/browse/KAFKA-20400){rel="&#x22;nofollow&#x22;"}) | ✅                                                    |
| DSL opt-in (`dsl.store.format = HEADERS`, materialised aggregations + tables) | ✅                                                                                                                                                                                                   | ✅ (+ `Materialized.withRecordHeaders()` convenience) |
| Interactive-query read-back of headers                                        | ✅                                                                                                                                                                                                   | ✅                                                    |
| In-place upgrade of existing state (lazy dual column family)                  | ✅ (the shipped families)                                                                                                                                                                            | ✅ (all four families)                                |
| Lazy header parsing on read                                                   | 🚧 open PR ([KAFKA-20155](https://issues.apache.org/jira/browse/KAFKA-20155){rel="&#x22;nofollow&#x22;"})                                                                                           | ✅                                                    |
| `StreamJoined` / join-buffer headers                                          | — out of scope                                                                                                                                                                                      | — out of scope (matches KS)                          |
| Non-timestamped key-value / windowed headers stores                           | — out of scope                                                                                                                                                                                      | — out of scope (matches KS)                          |
| Headers-specific metrics                                                      | — none                                                                                                                                                                                              | — none (matches KS)                                  |

Two rows are where StoatFlow is currently ahead, and both are simply pieces Kafka's contributors have in flight rather than design disagreements:

- **The versioned store family** — both the store itself and versioned KTables in the DSL. On the same lazy dual-column-family migration as the rest.
- **Lazy header parsing.** A `Headers` wrapper that defers parsing the header block until something actually reads it. A header-agnostic value serde — or a DSL read that only wants the value and strips the headers — never pays for the parse; a Schema-Registry serde that reaches for the schema id triggers it on first access. It is the biggest win for versioned stores, which have no caching layer in front of them. (Kafka's own version of this is [KAFKA-20155](https://issues.apache.org/jira/browse/KAFKA-20155){rel="&#x22;nofollow&#x22;"}, still open.)

Everywhere else the two line up — including where they both stop. Headers on join buffers, non-timestamped header-aware stores, and headers-specific metrics are all outside both KIPs, and StoatFlow draws those lines in the same place Kafka does.

## StoatFlow's take: an upgrade that costs nothing

*StoatFlow-specific.*

Here is the operational question that outranks the rest for anyone running a stateful application: turning headers on changes the value format, so what happens to the gigabytes of state already on disk in the *old* format?

There are two honest answers. **Eager migration** rewrites every entry on the first boot with headers enabled — simple, but a migration pause and an I/O spike proportional to your state size, before the app processes a record. **Lazy migration** rewrites nothing up front: keep the old data where it is, write new data in the new format, reconcile the two on read — using two RocksDB **column families**, independent keyspaces inside one database. Kafka Streams takes the lazy path; so does StoatFlow.

![One RocksDB database with two column families: the legacy one holding everything already on disk, the new one taking every write, a read falling back from the new family to the legacy one, and the legacy family draining as keys are rewritten instead of being migrated up front](https://stoatflow.io/assets/blog/kip-1271_lazy-migration-two-cfs.svg)

You flip the flag and restart, and not a byte of existing state is rewritten. A read checks both keyspaces and converts an old-format hit on the way out — old entries read back with empty headers, which is exactly right, because they predate the feature. A write lands in the new keyspace and clears the old copy, so each key upgrades itself the first time it's touched; the legacy keyspace drains to nothing over time. With the flag off again before any write has landed in the new keyspace, the engine drops the empty keyspace and the store is exactly what it was. Once data has migrated, turning the flag off is still supported, but it is a declared operation: the engine refuses with instructions until you acknowledge a one-time rebuild from the changelog — and persisted headers are shed, because the old format cannot hold them. That rebuild is unavailable for a store with no recovery source (logging disabled, or the changelog globally off), where the refusal is final.

There is an honest nuance worth stating, because it cuts against our own grain. Kafka Streams went lazy for a specific reason: in a horizontally-scaled deployment, an eager migration would block a partition's recovery during a rebalance, and a rebalance is exactly when you cannot afford a long pause. StoatFlow has no rebalances — [it runs as a single instance by design](https://stoatflow.io/blog/introducing-stoatflow) — and opens each store once at startup. That argument for lazy doesn't apply to us; eager would have worked. We chose lazy anyway, because a frictionless opt-in — flip a flag, restart, no pause, no I/O spike, reversible — free before data migrates, one acknowledged changelog rebuild after — is worth more than the column family it saves, and because it keeps us byte-for-byte on Kafka's on-disk layout. Choosing the more involved implementation to preserve that compatibility is the trade we keep making.

It covers **all four families**. Key-value and versioned stores carry the two column families directly; windowed and session stores are split into time segments, so each segment gets its own pair and upgrades independently as records land in it. The versioned store is the one variation worth a footnote: it keeps every version of a key rather than overwriting, so old versions are legitimate history surfaced through the merge, not stale copies to be drained.

The lazy path has sharp edges you only meet once you commit to it. A delete must remove the key from *both* keyspaces — miss the legacy copy and the old value resurrects through the read-merge, a tombstone that didn't take. And you cannot delete from the legacy keyspace any data the current transaction hasn't committed, or an aborted transaction takes real data with it. We learned the restoration corner from a real-Kafka integration test: a unit test restored happily because it exercised the typed wrapper, but the end-to-end test — write, wipe local disk, restore from an actual changelog — came back with malformed values, because the re-pack step was keyed off the wrong object on the restore path. Only restoring against a real broker surfaced it. These are the price of not rewriting your state up front, and they're exactly the kind of thing a faithful implementation has to get right out of sight.

## Scope, and what it costs

*The scope lines are universal (they're the KIP's); the costs are universal effects.*

The feature draws two deliberate lines, and StoatFlow draws them where Kafka does:

- **Timestamped stores only.** There are no non-timestamped header-aware stores — the key-value-with-headers store is always timestamped. That is the KIP's scope, not a missing feature.
- **Joins are out of scope.** A KStream-KStream join buffers records in a plain window store, and neither KIP wires headers through it. Headers on join state would be a step *beyond* the KIPs, with its own design questions.

A few behaviours the KIP explicitly leaves for later — making stored headers visible to user functions, attaching them to DSL *results* — are deferred in both Kafka Streams and StoatFlow. Downgrade is the one place we went further: Kafka Streams refuses it and leaves you to delete local state by hand, which we match, but StoatFlow will also do the rebuild for you when you acknowledge it in config.

And it has costs, which are the expected effects of doing what it says, not drawbacks:

- **Headers take space.** Every stored value grows by the header block (empty headers: one byte). That is the storage cost of keeping them — on disk, in the cache, and on the changelog.
- **A transient read-merge.** While a store is mid-upgrade, a read that misses the new keyspace does a second lookup in the legacy one and converts on the fly. It drains to a single keyspace as keys are rewritten; steady state is a single lookup.

## The lesson

*Universal.*

The size of a change is set by how load-bearing the thing you change is, not by the size of its specification. A serialised value is about as load-bearing as it gets in a stateful stream processor: it lives in the cache, on the changelog, in the restoration path, behind the query API, and in the on-disk format — every one of them on the commit-critical path. So a one-varint value-format change is, unavoidably, a state-layer change. Kafka Streams measured it in dozens of pull requests; StoatFlow's effort was comparably large.

And the part a user actually experiences is the smallest slice of that work: a flag that turns the feature on and lets state already on disk keep working, untouched, until it upgrades itself one key at a time. The reason that flag is cheap to use is the same reason the rest was expensive — staying faithful to a format down to the byte, in every path that touches a value.

This sits directly on top of an earlier foundation: [KIP-1035, which moved changelog offsets into a RocksDB column family of their own](https://stoatflow.io/blog/kip-1035-state-store-managed-offsets). The column-family mechanism that makes lazy header migration possible is the same one that holds those offsets. The on-disk layout keeps earning its keep.

**Read on:**

- [KIP-1271: Allow to Store Record Headers in State Stores](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1271%3A+Allow+to+Store+Record+Headers+in+State+Stores){rel="&#x22;nofollow&#x22;"}
- [KIP-1285: DSL support for headers in state stores](https://cwiki.apache.org/confluence/x/4ow8G){rel="&#x22;nofollow&#x22;"}
- [KIP-1035: why Kafka Streams 4.3 lets state stores own their offsets](https://stoatflow.io/blog/kip-1035-state-store-managed-offsets)
- [Apache Kafka 4.3.0 release announcement](https://kafka.apache.org/blog/2026/05/22/apache-kafka-4.3.0-release-announcement/){rel="&#x22;nofollow&#x22;"}


# Hot standby for StoatFlow: failover in seconds, not a cold restore

A StoatFlow application recovers by restarting: the process dies, the orchestrator brings it back, and it resumes from the last committed barrier — no data loss, no duplicate output. Where local state survives the restart, on a persistent volume, that costs a sub-second delta at worst. Where it doesn't, the store rebuilds from the changelog, and then recovery time scales with how much state there is. At tens of gigabytes, a cold restore can run into minutes.

Hot standby removes that scaling. It is an opt-in active/passive pair where a warm standby takes over in seconds, independent of state size — and under exactly-once, with no duplicates across the handoff. The full operator guide is on [High availability](https://stoatflow.io/docs/operating/high-availability); here is why it exists and what it gives you.

> **TL;DR**
>
> - **What:** Hot standby — an opt-in active/passive pair. Off by default; the single-instance runtime is byte-for-byte unchanged when you don't enable it.
> - **Why:** Recovery has always been fast restart, but a cold rebuild from the changelog scales with state size. For large state or a tight recovery-time objective, that window is too long.
> - **How:** A passive standby continuously follows the changelog and stays warm. Promotion is seconds, independent of state size — no cold restore on the critical path.
> - **Guarantee:** Under exactly-once, broker-enforced transactional fencing makes split-brain impossible — at most one instance ever commits. Under at-least-once, failover is bounded-duplicate.
> - **The catch:** It is redundancy, not scale. Still one active instance, roughly double the footprint. To go faster, scale up, not out.
> - **Docs:** [High availability](https://stoatflow.io/docs/operating/high-availability).

## In this post

- [Why this exists](https://stoatflow.io/#why-this-exists)
- [What hot standby gives you](https://stoatflow.io/#what-hot-standby-gives-you)
- [Fast restart vs hot standby](https://stoatflow.io/#fast-restart-vs-hot-standby)
- [How it works](https://stoatflow.io/#how-it-works)
- [Exactly-once across failover](https://stoatflow.io/#exactly-once-across-failover)
- [Tradeoffs and limits](https://stoatflow.io/#tradeoffs-and-limits)
- [Running it](https://stoatflow.io/#running-it)
- [When to turn it on](https://stoatflow.io/#when-to-turn-it-on)

## Why this exists

StoatFlow runs as exactly one instance per application. That is the whole point: no consumer-group rebalancing, no state migration, no repartition topics — you stop paying the *distribution tax* for scale-out you don't need. Availability in that model comes from fast restart, and fast restart is a credible story because there is no rebalancing to wait out: the only thing a restart pays for is whatever state has to be rebuilt.

On a persistent volume, that is usually little or nothing. Changelog offsets live inside the store and are flushed atomically with the data, so a clean shutdown leaves local state already consistent with the changelog — nothing to restore, and the pod is processing again after app init. An unclean shutdown costs a delta restore of the uncommitted tail, which completes in under a second whatever the store's total size.

The expensive case is a full restore: a fresh pod, an ephemeral volume, or a local store that fails validation. Then the engine replays the changelog from the beginning, and *that* scales with state size — the [benchmarks](https://stoatflow.io/product/benchmarks) measure cold-start across representative workloads. For a workload carrying tens of gigabytes of state with a tight recovery-time objective, minutes of recovery on an unplanned restart is the line between acceptable and not.

When we shipped the [first alpha](https://stoatflow.io/blog/introducing-stoatflow), failover behaviour was a fair question we said we hadn't answered yet. This is the answer: an availability tier for the workloads that can't absorb a cold-restore window — without giving up the single-instance model, and without reintroducing the distribution tax. There is still exactly one active instance. There is still no rebalancing and no horizontal scale. There is now a warm spare.

## What hot standby gives you

- **Failover in seconds, independent of state size.** The standby is already warm, so taking over is not a restore — it is a handoff. The cold-start window that scales with your state is exactly what this removes.
- **Near-zero-downtime rolling deploys.** A version roll costs about one graceful handoff: the active drains, commits, and hands over to the standby. No rebalance, no cluster-wide reconvergence.
- **Exactly-once preserved across the handoff.** The recovery anchor is the last committed barrier whether you restart cold or fail over to a standby. Under exactly-once there are no duplicates downstream for `read_committed` consumers.
- **An order-independent rollout, with no custom controller.** Readiness gates the roll: a standby reports ready only once it has caught up, and a catching-up standby is not killed mid-catch-up. Kubernetes never advances the roll onto an instance that isn't ready — whichever pod rolls first.
- **Opt-in, with the default untouched.** Hot standby is off unless you turn it on. Leave it off and you get exactly today's single-instance behaviour, with none of the extra moving parts.
- **Observable and operable.** With the `:runtime` module you get it wired up: a `/ha/status` endpoint reports each pod's role, replication lag, and the peers it sees; `/ha/switch`, `/ha/promote`, and `/ha/demote` drive a controlled handoff; and the pair exports `stoatflow.ha.*` metrics for dashboards and alerts. Embedding `:core` directly, the same information is public API — `haStatus()` and `publishHaCommand()` on `StoatFlow` — and you expose it through your own endpoints and metrics.

## Fast restart vs hot standby

Neither tier involves two *active* instances — that remains the rule. You choose per application.

|                     | Fast restart (default)           | Hot standby (opt-in)                            |
| ------------------- | -------------------------------- | ----------------------------------------------- |
| Model               | One instance                     | Active/passive pair                             |
| Recovery on failure | Restart + rebuild from changelog | Promote the warm standby                        |
| Failover time       | Scales with state size           | Seconds, independent of state size              |
| Cost                | One instance                     | \~2× — a second always-on instance + its volume |
| Choose it for       | Most workloads                   | Large state or a tight recovery-time objective  |

## How it works

At any moment one instance is the **active** — it owns the source partitions, processes records, and commits — and the other is a **passive standby**. The standby consumes the same changelog the active writes, keeping its own local state warm and a step behind; it does not process source records or produce output. The two coordinate through a compacted internal Kafka topic, so each knows the other is alive and which role it holds.

Promotion happens two ways. A **graceful handoff** — a rolling deploy or an explicit `/ha/switch` — drains the active, commits, and hands the role over. A **failover** — the active crashes — is detected when the standby stops seeing the active's heartbeats, and the standby promotes itself. Because it is already warm, taking over does not wait on a cold restore.

The safety property underneath is Kafka's own. Under exactly-once, a transactional producer is fenced by its epoch: if a network partition ever left two instances briefly believing they are active, the broker lets only one commit and fences the other, which then shuts itself down. Correctness does not depend on the failure detector being perfect — it depends on the fence, which the broker enforces.

## Exactly-once across failover

How clean a failover is depends on the processing guarantee the application runs under — and the two behave differently enough to call out.

Under **exactly-once** (the default), failover is split-brain-proof. The transactional fence guarantees at most one instance can ever commit a given transaction, so the handoff produces no duplicates and loses no committed work. There is nothing extra to do downstream.

Under **at-least-once**, failover is *bounded-duplicate*. The pair still coordinates promotion safely, but without transactional fencing an ungraceful failover can re-process a small, bounded window of records, so a few duplicate output records are possible. If you run hot standby under at-least-once, make downstream consumers idempotent or duplicate-tolerant. The guarantee model is covered on [Exactly-once](https://stoatflow.io/docs/concepts/exactly-once).

## Tradeoffs and limits

Hot standby is a deliberate trade, and it has edges worth stating plainly.

- **It is redundancy, not scale.** A warm spare does not add throughput. There is still exactly one active instance — to go faster, give it more cores and memory.
- **Roughly double the infrastructure footprint.** A second always-on instance and its own persistent volume — though not a second licence: the pair shares one machine fingerprint and counts as a single seat.
- **At-least-once is bounded-duplicate**, not split-brain-proof. Exactly-once is the guarantee that makes a handoff clean.
- **Crash failover is not zero-downtime.** It is bounded by detection plus a near-instant promotion — short, but not zero. A graceful handoff (deploy or `/ha/switch`) is about one handoff.
- **Single region.** Both instances must reach the same Kafka cluster; this is not a multi-region or active-active mechanism.
- **The default is often the right answer.** If a cold-restore window of seconds-to-minutes is acceptable, fast restart is simpler and cheaper. Hot standby exists to remove that window, not because the default is unsafe.

## Running it

You deploy the pair as a two-replica `StatefulSet` — the same single-replica manifest you already use, with `replicas: 2`, a per-pod persistent volume, and a rolling update. The split `/health/live` and `/health/ready` probes are what make the roll order-independent, and a termination grace period long enough for the active to finish its handoff keeps deploys clean.

Turning it on is one configuration knob: `stoatflow.ha.mode: active-standby`. The rest — the coordination topic, the per-pod identity, the staleness and lag thresholds — has sensible defaults; on Kubernetes a typical deployment sets only the mode. [High availability](https://stoatflow.io/docs/operating/high-availability) has the manifest, the configuration reference, and the operator endpoints; [Running on Kubernetes](https://stoatflow.io/docs/operating/kubernetes) covers the deployment shape it extends.

## When to turn it on

Reach for hot standby when a cold-restore window is the thing you can't afford — large state, a tight recovery-time objective, or a deploy cadence where even a fast restart's gap shows up downstream. For everything else, the single-instance default remains the simpler, cheaper choice, and it is unchanged whether hot standby exists or not.

This is the first cut of the failover story, and it is built to get smarter: promotion selection becomes lag-aware so the freshest standby wins, and the readiness threshold becomes a tunable rather than a binary. The model — one active, a warm passive spare, the fence as the backstop — is the part that stays. Start with the [High availability guide](https://stoatflow.io/docs/operating/high-availability), and if you're running large-state stateful Kafka Streams in production today, that is exactly the workload this was built for.


# Measuring StoatFlow failover: four scenarios, from the logs

The [hot-standby](https://stoatflow.io/blog/hot-standby-high-availability) post made a claim: failover in seconds, independent of state size. A claim like that earns the right to be measured. So I put a live active/passive pair under load and triggered every way a StoatFlow active can lose its role — and the first thing the test surfaced is that the obvious way to test failover does not test failover at all. `kubectl delete pod` is a graceful SIGTERM handoff, not a crash.

This is the measured failover story: four distinct scenarios, the millisecond timing taken from pod logs — because Prometheus cannot resolve a one-second event — and the things the numbers contradicted, including a single crash that recovers in place without ever failing over. Measured on StoatFlow :stoatflow-version, an active/passive word-count pair on Kubernetes at \~1,000 input records/s (≈ 44K output records/s), exactly-once.

> **TL;DR**
>
> - **`kubectl delete pod` is not a crash.** Kubernetes sends SIGTERM first; the JVM shutdown hook drains and hands off gracefully. A real crash needs a SIGKILL of the JVM's host PID *from the node*.
> - **Four scenarios, not one:** graceful switch, SIGTERM/rolling deploy, JVM crash with in-place restart, and node loss with a standby takeover. They behave — and time — very differently.
> - **Graceful is sub-second and lossless:** \~0.4–0.9 s of no-processing, the in-flight epoch committed before handover, zero reprocessing under exactly-once.
> - **A single JVM crash recovers in place (\~3.6 s) and never fails over** — the container restarts faster than the standby's detection window. The standby takes over only when the active is genuinely gone (\~8.5 s, detection-dominated).
> - **That is the *hard-crash* tier specifically.** A fatal the engine catches behaves differently again: a *restartable* one (a processing or production failure, with a `REPLACE_THREAD` handler) is absorbed in place and never fails over either; anything else drains and hands off, so it *does* — see [application faults](https://stoatflow.io/docs/operating/high-availability#application-faults-under-hot-standby).
> - **Grafana shows the shape; logs show the timing.** A 10-second scrape and a one-minute rate window cannot measure a one-second failover.

## In this post

- [The obvious test isn't a crash](https://stoatflow.io/#the-obvious-test-isnt-a-crash)
- [Four ways an active loses its role](https://stoatflow.io/#four-ways-an-active-loses-its-role)
- [Expected versus measured](https://stoatflow.io/#expected-versus-measured)
- [What the numbers revealed](https://stoatflow.io/#what-the-numbers-revealed)
- [Tradeoffs and limits](https://stoatflow.io/#tradeoffs-and-limits)
- [Reproduce it](https://stoatflow.io/#reproduce-it)
- [What's coming next](https://stoatflow.io/#whats-coming-next)

## The obvious test isn't a crash

The instinct, testing failover on Kubernetes, is `kubectl delete pod`. It feels like pulling the plug. It isn't. Kubernetes deletes a pod gracefully: it sends the container **SIGTERM**, waits out the termination grace period, and only then SIGKILLs. StoatFlow installs a JVM shutdown hook on SIGTERM, so a `kubectl delete` runs the *graceful* path — the active drains its in-flight epoch, commits it, publishes a `DRAINING` marker, and the standby promotes on that marker. The logs of the "crashed" pod give it away: `Graceful shutdown completed in 21ms`, then `Stopped … in 99 ms`. That is a clean handoff, not a crash.

To actually crash the process you have to SIGKILL the JVM — and not from inside the container. The JVM runs as PID 1 there, and the kernel protects a PID-namespace init process: an in-container `kill -9 1` is silently dropped. The kill has to come from the node, against the JVM's *host* PID:

```bash
# on the node hosting the active pod
crictl inspect --output go-template --template '{{.info.pid}}' "$CONTAINER_ID"  # -> host PID
kill -9 "$HOST_PID"
```

The second trap is measurement. The temptation is to read failover time off the Grafana dashboard. Don't. The benchmark ServiceMonitor scrapes every 10 seconds, so the `stoatflow_ha_role` gauge only flips at a 10-second boundary — a ±10-second quantisation on a one-second event. Throughput is a `rate(…[1m])` that smooths a sub-second dip into nothing, and the end-to-end latency panel is a sticky high-water gauge that shows the blip's *size* but not *when*. Grafana is the right tool for the *shape* of a failover — the role line flipping, the brief throughput dip, the latency spike. For the *numbers*, the source of truth is the pod logs: millisecond timestamps on a single NTP-synced cluster clock.

![StoatFlow HA role panel across a /ha/switch: the active role hands from one pod to the other in a single step — exactly one pod at role = 1 at any moment, never two. The flip lands between two 10-second scrapes, which is why the dashboard can show the handover but not time it.](https://stoatflow.io/assets/blog/ha-failover-role-flip.png)

## Four ways an active loses its role

There are four, and conflating them is how you get a wrong number.

1. **Graceful switch** — an operator `POST /ha/switch`. The active drains, commits, drops the producer fence, and hands the role to the standby on the `DRAINING` signal.
2. **SIGTERM / pod-delete / rolling deploy** — the same graceful handoff, but triggered by the JVM shutdown hook instead of an operator command. This is what `kubectl delete pod` and a rolling deploy do.
3. **JVM crash, in-place restart** — a true SIGKILL of the JVM. Kubernetes restarts the *same* container, so its node-local state is still there: the store delta-restores the aborted epoch rather than rebuilding from the changelog, and the pod re-promotes itself. A warm restart, not a cold start. The standby is never involved.
4. **Node loss, standby failover** — the active is genuinely gone and cannot come back. The standby stops seeing its heartbeats and promotes itself via staleness detection.

Scenarios 1 and 2 are graceful: the active drops the fence deliberately and the standby promotes on an explicit signal. Scenarios 3 and 4 are ungraceful: the active dies without warning. The distinction between 3 and 4 is the one that surprised me, and it gets its own section below.

## Expected versus measured

The design — [ADR-125](https://stoatflow.io/docs/operating/high-availability) — sets the expectations. Promotion is *fence-and-assign in parallel, then resume against already-warm state&#x2A;, which the design measured at **\~383 ms, independent of state size**. Ungraceful failover adds a detection cost on top: the standby has to notice the active is gone before it can promote.

Here is what a live pair under load actually did, every figure taken from the logs.

| Scenario                     | Trigger                              | Detection              | Promotion critical path           | Stop-the-world  | Reprocessing |
| ---------------------------- | ------------------------------------ | ---------------------- | --------------------------------- | --------------- | ------------ |
| **Graceful switch**          | `POST /ha/switch`                    | immediate (`DRAINING`) | \~0.5–0.8 s                       | **\~0.4–0.9 s** | none         |
| **SIGTERM / rolling deploy** | `kubectl delete` / `rollout restart` | immediate (`DRAINING`) | \~0.35–0.4 s                      | **\~0.4 s**     | none         |
| **JVM crash, in-place**      | SIGKILL the JVM                      | — (same pod recovers)  | n/a (warm restart, self-promotes) | **\~3.6 s**     | one epoch    |
| **Node loss, failover**      | node gone                            | \~7 s (staleness)      | \~0.8 s                           | **\~8.5 s**     | one epoch    |

Two definitions hold the table together. The **promotion critical path** is the work to bring a warm standby to *serving* once it has decided to promote — fence-and-assign, catch the last changelog records, start the engine threads. Measured at 350–820 ms across runs, it brackets the design's \~383 ms; the spread is the changelog catch-up and thread start under load. **Stop-the-world** is the window in which *neither* instance is processing — bounded by the old active stopping and the new active resuming. That is the number that matters for availability, and it is much smaller than it looks from the dashboard, for a reason worth its own paragraph below.

The graceful paths (switch, SIGTERM) confirm the design cleanly: sub-second, and the in-flight epoch is *committed before* the handover — `Phase 2: Final barrier N committed successfully`, then a clean `Phase 4b: Commit thread stopped`, in about 21 ms — so the new active resumes from a committed boundary with nothing to reprocess. The ungraceful node-loss path matches the design's other published figure: detection dominates, at roughly seven seconds, and promotion is the fast part.

## What the numbers revealed

Three things the measurement contradicted or sharpened.

**A single JVM crash recovers in place — it does not fail over.** Kill the active's JVM and the standby does *nothing&#x2A;. Kubernetes restarts the crashed container in \~1.4 s; StoatFlow restores its node-local state — a warm restart that delta-restores only the aborted epoch's records, not a rebuild from the changelog — and re-promotes itself in \~2.2 s, back to serving in **\~3.6 s total**. That is well inside the standby's \~7-second detection window, so by the time the standby could have concluded the active was gone, the active is already back. The warm spare earns its keep on *node* loss, not on a process crash that the orchestrator can paper over faster than the failure detector can fire. The cost is one reprocessed epoch — the in-flight transaction was aborted, never committed, never visible to `read_committed` consumers — not a duplicate, not a loss.

Read "crash" literally, though: this is a SIGKILL, where the JVM dies without running anything. A fatal the engine *catches* takes a different path, and which one depends on whether a rebuilt engine could survive it. A **restartable** fault — a processing or production failure, with a `REPLACE_THREAD` handler registered — is absorbed in place: the engine is rebuilt inside the live JVM and the pod keeps the active role, so there is no failover at all until the restart budget is spent. Anything else drains, publishes `DRAINING` on its way out, and the standby promotes off that signal via the same fast path as a graceful switch — for that case the table's graceful rows are the ones to read, not this one.

There is a sharp edge here: this is the *first-crash* figure. A pod that keeps crashing trips Kubernetes' **CrashLoopBackOff**, an exponential restart delay that grows to tens of seconds. The StoatFlow recovery stays \~2 s; the rest is the kubelet holding the container down. If you benchmark crash recovery, reset the restart count first — or you are measuring the kubelet's patience, not the engine's.

**Stop-the-world is not the latency blip.** The dashboard's end-to-end latency spikes to \~2 s on a graceful switch; the measured stop-the-world is \~0.9 s. Both are correct — they measure different things. Stop-the-world is the control-plane gap: no engine running. The latency blip is longer because, the moment processing resumes, the new active has to drain the input that piled up *during* the gap — at 44K records/s, even a sub-second pause leaves a backlog to chew through. The felt recovery is stop-the-world plus the catch-up; the availability gap is stop-the-world alone. Reporting one as the other overstates the outage by 2–3×. The blip is largest on a node-loss failover, where the gap itself is seconds — the panel below catches one: p95 and p99 climb to \~8 s while the standby takes over, then snap back.

![End-to-end latency during a node-loss failover (p50 / p95 / p99 / max): p95 and p99 jump to \~8 s the moment the active goes silent and stay there until the standby promotes and works through the backlog, then return to baseline. The max line is a sticky high-water gauge, so it stays pinned for a while after the event.](https://stoatflow.io/assets/blog/ha-failover-latency-blip.png)

**Detection is the whole cost of an ungraceful failover.** A clean node-loss measurement — the active cordoned off, SIGKILLed, and force-deleted so it cannot restart — promotes the standby in \~8.5 s of stop-the-world. Of that, \~7 s is detection (`staleness-threshold-ms` plus the missed-heartbeat debounce) and only \~0.8 s is the promotion itself. If your recovery-time objective is tight, the lever is the staleness window, not the promotion path — the promotion is already as fast as a warm standby allows.

## Tradeoffs and limits

- **Ungraceful failover is detection-dominated.** The \~7-second window is a deliberate default that trades failover speed for tolerance of transient network jitter. Tightening it speeds up node-loss failover and raises the risk of a spurious promotion on a blip. It is a knob; set it against your recovery-time objective, not by instinct.
- **A process crash is not a failover.** In-place restart (\~3.6 s) is the common case for a crash, and it is faster than the standby would be. The standby is insurance against losing the node, not against the JVM dying.
- **Honest crash testing needs node access.** You cannot crash a StoatFlow active faithfully with `kubectl` alone — a real SIGKILL comes from the node. If your test harness can only reach the API server, it is testing the graceful path and calling it a crash.
- **Two is the supported topology.** The pair is one active and one standby. Scaling to three and back leaves a stale peer record in the coordination topic that pushes the survivor onto an unsupported multi-standby code path, where it can decline to promote. Clean it up by recreating the coordination topic, and stay at two.
- **Exactly-once holds across every path.** Graceful commits the in-flight epoch; crash aborts it and reprocesses one epoch. Either way the broker-enforced transactional fence guarantees at most one instance ever commits — no duplicates, no lost committed work, for `read_committed` consumers.

## Reproduce it

The four scenarios are automated in a single drill that prints the log-based timing — promotion critical path and stop-the-world — rather than the inflated wall-clock:

```bash
ha-failover-drill.sh --mode switch     # graceful handoff
ha-failover-drill.sh --mode sigterm    # SIGTERM / pod-delete (graceful, not a crash)
ha-failover-drill.sh --mode crash      # true SIGKILL of the JVM -> in-place restart
ha-failover-drill.sh --mode nodeloss   # cordon + SIGKILL + force-delete -> standby failover
```

The `crash` and `nodeloss` modes SIGKILL the JVM's host PID from the node over SSH, resolve timing from the logs, and — for node loss — cordon the node so the pod genuinely cannot come back, then un-cordon afterwards. The full walkthrough, including the dashboard to watch and the gotchas, is in the playground runbook in the repository.

## What's coming next

> **Update (2026-07-02):** this shipped. The in-place engine restart is no longer exploratory, and the round-trip described below is no longer always a process restart — an active that loses its role can now rebuild its engine inside the live JVM. The same work lifted the two-standby constraint. See [In-place engine restart: the primitive behind multi-standby HA](https://stoatflow.io/blog/in-place-restart-multi-standby). The section is kept as written for the record.

Across all four scenarios, one round-trip recurs: an active that loses its role *exits the process and lets Kubernetes restart it*. A graceful demote self-terminates and comes back as a fresh pod; a crash is a container restart; even an in-place recovery is the orchestrator rebuilding the whole container. ~~The engine teardown and re-initialisation happen, but always wrapped in a process restart.~~

The work in progress — an **in-place engine restart** — removes that wrapper. The idea is to tear down and rebuild the processing engine *in the same process*, without exiting the JVM or recreating the application instance, and resume from the last committed transactional offsets: an in-process crash recovery, exactly-once intact via the same `transactional.id` epoch-bump fence that protects a cross-pod handoff today. It is the shared primitive behind three otherwise-separate needs — the faithful analogue of Kafka Streams' `REPLACE_THREAD` (which StoatFlow currently degrades to a full application shutdown), the hot-standby promotion path, and dynamic lane re-tuning. Build it once, cleanly, and all three get faster.

~~It is exploratory — a draft on the roadmap, pending a feasibility spike and an ownership decision, not a committed release.~~ But it is the natural next step the measurements point at: the promotion path is already fast; the remaining cost on a crash is the process round-trip, and that is the round-trip this would remove.

For the shipped behaviour, the [High availability guide](https://stoatflow.io/docs/operating/high-availability) has the configuration reference and the operator endpoints. The full measurement data — every raw timestamp, the stop-the-world boundaries, and the corrections this testing forced — lives in the repository's failover-timing report. The model holds: one active, a warm passive spare, the fence as the backstop, and now numbers behind the word "seconds".


# How we obfuscate StoatFlow — and what the public-API boundary taught us

> **TL;DR**
>
> - **What we obfuscate:** the engine internals in our core module — not the public API.
> - **How:** a rename-only pass, no shrinking and no optimisation, so the bytecode still behaves identically and stack traces still map back to source.
> - **The hard part:** the boundary. A rename pass is indiscriminate; the public API has to be held still on purpose, and getting that list wrong breaks customers in ways your in-repo tests never see.
> - **The fix that stuck:** stop testing against your own source tree. Test against the obfuscated artefact you actually ship.
> - **On protection:** rename-only obfuscation is a deterrent and a legal signal, not a wall — and in the AI age that distinction matters more, not less.

Obfuscating a library is one line in a build file. Keeping a byte-stable public API in front of the obfuscated code is the part that took ten iterations to get right.

StoatFlow ships a Kafka Streams-compatible DSL — you swap an import and recompile — over an engine that is the actual product. The DSL surface has to stay exactly where customers expect it, method for method, or their code stops compiling. The engine beneath it has to be unreadable in the published jar. Those two goals pull in opposite directions, and every interesting bug we hit lived on the seam between them: public types that vanished from the shipped jar, a constructor the bytecode verifier rejected, a test library that linked the wrong artefact. None of them showed up in our own test suite. All of them showed up the moment someone compiled against what we actually published.

## What we protect, and why

StoatFlow runs a Kafka Streams topology on a single instance, using JDK virtual threads for parallelism and Flink-style commit barriers for exactly-once — the engineering we describe elsewhere as cutting the *distribution tax*. The DSL you write against is deliberately the Kafka Streams DSL; the value we sell is what happens underneath it.

That split maps cleanly onto what gets obfuscated. The public API — the DSL, the configuration types, the processor and state-store interfaces you compile against — must stay legible and stable to the byte. The engine internals must not be self-documenting from the jar. Rename them, and reading the published artefact tells you nothing about how the thing works.

The problem is that a rename pass does not know which side of that line a class sits on. It renames everything by default. So the entire exercise becomes drawing one precise boundary: hold the public surface completely still, and let the tool mangle the rest. Draw it a hair too wide and you leak internals into the jar; a hair too narrow and you rename something a customer was relying on. We drew it wrong, in both directions, more than once.

## How the obfuscation works

We use ProGuard, in **obfuscation-only mode**: renaming on, shrinking off, optimisation off. That combination is deliberate. Shrinking removes code the tool believes is unreachable; optimisation rewrites method bodies, inlines, and restructures. Both are destructive in ways that cost us more than they buy. They complicate stack traces, they can perturb the JIT's view of the code, and — for a Kotlin library — they risk invalidating the metadata the compiler emits for reflection and interop. Name mangling is what actually hides the architecture. The rest is downside without upside, so we leave it off. The published bytecode is byte-for-byte equivalent in behaviour to what we built; only the names change.

What we hold still falls into a few categories, described here at the level that matters:

- **The public API packages** — the DSL, config, processor, exception, watermark, and metrics surfaces a customer names in their code. These stay exactly as written.
- **A cross-module SPI.** StoatFlow is split into several published modules, and some of them — the runtime wrapper, the test driver — reference a small shared set of engine-facing types. Bytecode records those references by name, so that shared set has to survive un-renamed or the modules stop resolving each other at a customer's runtime. We carved those types into their own package precisely so the keep rule for them is one clean glob rather than a scattering of exceptions.
- **Interop invariants.** Kotlin metadata, companion-object members, and the annotations that make static-style access work from Java all have to be preserved, or the jar compiles but reads wrong from the other language.
- **Attributes for stack-trace fidelity.** We keep the attributes that let a stack frame point back at a source position, even though the names in that frame are mangled.

That last point is where the mapping file comes in. Renaming would make production stack traces useless — every frame a soup of single letters — so ProGuard emits a mapping from original names to obfuscated ones. We keep that file private: it is never committed, never shipped inside any jar, never published to the repository. It lives in a private store with indefinite retention, and when a customer sends us an obfuscated trace we run it back through `retrace` against the mapping for that exact release to recover real class names, methods, and line numbers. The jar stays opaque; support stays possible.

One more leak to close: source and documentation. Obfuscating the bytecode is pointless if the sources jar or the generated API docs ship the same class names in clear text. So both are filtered to the public API and the cross-module SPI, and the documentation tool is told to suppress the internal packages outright. What the bytecode pass hides, the docs pass must not hand back.

There is a version constraint worth flagging for anyone attempting this on a modern stack: an obfuscator has to understand both your bytecode version and your Kotlin metadata version, and support lags the language. Recent Java and recent Kotlin needed a recent-enough ProGuard; older releases simply did not recognise the class format and refused to run. Pin the version, and treat bumping it as a change that needs re-verifying, not a routine dependency update.

## The hard part: the public-API boundary

The mechanics above are the easy 20%. Here is the 80% — four ways the boundary bit us, each one invisible in-repo and obvious in the shipped jar.

The picture below is the boundary as it sits in the published jar. The distinction that matters runs through the top band: a region is held still either by a **package glob**, which sweeps every public type in a package including the nested ones, or by a **named list**, which holds exactly what it names and nothing else. Both of the first two failures below landed in list-kept regions, and neither could have landed in a glob-kept one.

![The published core jar in three regions: a public API kept by eight package globs plus two named lists, a cross-module SPI package kept so the sibling modules still resolve, and a renamed engine interior with a few kept islands inside it. Four numbered annotations mark where the boundary failed, and an inset shows why a keep rule naming one class does not reach that class's nested types.](https://stoatflow.io/assets/blog/obfuscation_public-api-boundary.svg)

**Types that vanished from the jar.** Our keep list started as a hand-curated enumeration of public types. It was incomplete. A batch of public state-store types — iterators, the read-only store interfaces, a versioned-record type — were never listed, so the rename pass quietly renamed them. In our own build everything compiled and every test passed, because our tests compiled against the raw, un-obfuscated classes. A customer who swapped their imports and compiled against the *published* jar got "cannot find symbol" for a type that, as far as our CI was concerned, existed and worked. The types were present in the jar — just under a name nobody could write.

**The nested-class trap.** A keep rule that names one class and says "hold it and all its members" does not hold that class's *nested* types. We had a public status enum nested inside a public class, kept — we thought — by the wildcard on its parent. It was renamed anyway, its constants along with it, because nested types are a separate scope that wildcard never reached. Ported code calling the enum by name stopped compiling. The fix is to name each nested type explicitly, with its qualified `$`-name — and, more usefully, to test the public surface by *naming specific constants*, not just checking that the parent class survived. Worth knowing which rules are exposed: a *package* glob does sweep nested types along with their parents, so the packages kept that way were never at risk. The types that broke were the ones held by a hand-maintained list — which is the same root cause as the bug above.

**A constructor the verifier rejected.** One type on the public surface — our Kafka Streams-compatible configuration object — had grown an unusually wide constructor, over a hundred parameters. After the rename pass, ProGuard recomputed the stack-map frames for that method and got them wrong at that width. The result was bytecode the JVM refused to load at all — `VerifyError: Inconsistent stackmap frames`, under the *default* verifier, not only a strict one. It stayed invisible for the same reason as the first bug: our tests ran against the raw classes, which were never rewritten and so never carried the recomputed frames. It compiled, it passed every test, and it would not load. We fixed the immediate problem by grouping the configuration into a set of smaller typed objects so no single constructor was wide enough to trip the recomputation, and the deeper problem by class-loading the shipped type under strict verification in CI. The general lesson: kept does not mean untouched. The pass still rewrites the methods it keeps, and bytecode transformation is fragile at the extremes of what the format allows.

**The wrong artefact linked.** Our test-driver module is compiled against the engine. In-repo, that meant the raw engine. At publish time, the module was linked against the *obfuscated* engine — and it referenced internal types by their original names, which no longer existed. A customer pulling in the test driver got `NoSuchMethodError` and `NoClassDefFoundError` at class-init time. The real fix was structural: we reworked the test driver so it no longer reaches into engine internals at all — it runs on in-memory stores and talks to the engine through a small kept interface — which both removed the fragile references and made the driver simpler. The skew was the tell: any time module B is built against a different form of module A than the one it ships beside, you have a publish-time consistency bug waiting.

## The fix that stuck: test against the artefact you ship

Every bug above has the same shape. It was invisible in the repository because the repository tests the *source*, and it was live in production because customers link the *artefact*. Passing tests locally proved nothing about the thing we shipped.

So the durable fix was not any one keep rule. It was moving the truth-source. We added a set of build gates that operate on the obfuscated jar, not the raw classes:

| Gate                 | What it does                                                                                 | What it catches                                                                       |
| -------------------- | -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
| Contract check       | Introspects the shipped jar and asserts every public type is present under its original name | A missing keep rule that renames a customer-visible type                              |
| Strict-verifier load | Class-loads key public types under strict bytecode verification                              | Stack-map corruption introduced when the pass rewrites a method it kept               |
| Corpus compile       | Compiles a corpus of Kafka Streams-shaped Java against the obfuscated jar                    | Signature and accessor drift a raw-class test would never see                         |
| Reflection smoke     | Exercises a diagnostics path that looks up fields reflectively by name                       | A rename that silently breaks a string-based reference the tool won't rewrite for you |

That last gate is worth dwelling on, because it is a trap the other three don't cover. A rename pass rewrites *references it can see* — a method call, a field access in bytecode. It cannot rewrite a field name you pass as a *string* to a reflective lookup, because to the tool that's just a string. We had exactly this: a diagnostics endpoint that resolved fields by their literal names. The rename pass renamed the fields and left the strings untouched, so the lookup failed on every shipped build. Nothing but loading the obfuscated class and running the path would have found it.

The framing we settled on: the keep boundary is a public API, and it deserves the same treatment as any other API — an explicit contract, tested against the real artefact, on every build.

## Compatibility is a contract at the source and the binary level

Obfuscation was the sharpest version of a lesson that ran through the whole compatibility effort: matching another library's API exactly is deceptively hard, and the compiler passing is not the same as the contract holding. A few of the non-obfuscation ones, because they generalise:

- **Kotlin properties are not Kafka Streams accessors.** A Kotlin `val value` compiles to a Java `getValue()`. Kafka Streams exposes a bare `value()`. On a value type like `ValueAndTimestamp`, that mismatch means ported code calling `.value()` simply doesn't compile. The mapping from Kotlin property to Java getter is one-way and invisible to the consumer; if you're matching another library's naming, you have to add the bare-method aliases explicitly and check them.
- **Variance is part of a functional interface's contract.** Kafka Streams declares its functional interfaces with bounded wildcards — `ValueJoiner<? super V1, ? super V2, ? extends VR>`. Ours were invariant. Inline lambdas compiled fine, because Java infers around it; a *pre-typed* functional object — the kind real ported code reuses across a helper layer — did not. The fix was declaration-site variance on the Kotlin interfaces, which the compiler turns into the matching Java wildcards for free.
- **A test harness has to be faithful to the data path, not just the control path.** Our in-memory test driver originally handed records straight to the topology without serialising them — the serdes you passed in were accepted and then ignored. Around half the driver's tests were passing without ever exercising a serde. The real engine serialises at every source and sink boundary, so we made the driver do the same. It was a large migration, and it turned a pile of false green into real coverage of the exact boundary where serde bugs live.

Two smaller ones in the same vein: a windowed aggregation's `Materialized` should take the plain key type the user actually writes, not the framework's internal windowed representation; and null-key records have to be dropped consistently across *every* aggregation operator, not just the ones you remembered. Each is minor on its own. Together they make the point — API parity is a hundred small contracts, and any one of them left implicit is a place the compatibility quietly frays.

## Why obfuscate at all, in the AI age

Rename-only obfuscation raises the cost of reading a shipped jar. It does not remove the possibility. A decompiler has always turned bytecode back into readable structure; what obfuscation takes away is the *names* — the part that makes the structure quick to understand. Historically that was enough friction to matter, because renaming thousands of symbols back into something meaningful by hand is slow, dull work.

AI tooling has lowered that cost again. The tedious part — inferring what a mangled class probably does and giving it a sensible name — is exactly the kind of task these models are good at. So the honest position is the same one it always should have been: obfuscation is one layer among several, not a wall. It deters casual reading and it demonstrates that we treat the engine as a protected asset — which is a real, and increasingly the more durable, benefit. The protections we actually lean on are commercial and legal: the licence terms, and the fact that reconstructing a snapshot of today's engine is not the same as being able to ship and support it. We keep the bytecode layer because it is cheap and it raises the floor, not because we imagine it is the ceiling.

## What it costs

Obfuscation is not free, and the costs are worth naming plainly.

- **Release complexity.** There is a whole verification stage that only exists because we obfuscate — the gates above, plus the discipline of building them against the artefact rather than the source.
- **Opaque stack traces.** Every production trace that touches an internal frame needs deobfuscating through the private mapping before it means anything. That is an operational tax on every support interaction, and it depends on retaining the mapping for every release forever.
- **A permanent balancing act.** The keep boundary has two failure modes, and they pull against each other:

| If you keep… | You get…                                                  |
| ------------ | --------------------------------------------------------- |
| too much     | internals leak into the published jar in clear text       |
| too little   | a customer-visible type is renamed and their build breaks |

There is no setting that is safe in both directions. The only safety is the contract test on the real artefact.

And the ceiling is real. Rename-only is a deterrent, and heavier obfuscation — string encryption, control-flow flattening — buys time rather than immunity, while fighting the rest of a modern build (ahead-of-time native compilation, in particular, does not take kindly to it). For a library whose value is an architecture rather than a single secret algorithm, rename-only at the right boundary is the proportionate choice. If your value is one hot loop that must never be read, this is not the technique you want.

## Where this leaves us

If you take one thing from this: the moment you transform your artefact after building it — obfuscation, shrinking, shading, any of it — your source tree stops being the truth. Test against the thing you ship. Treat the boundary between what you hide and what you expose as an API in its own right, with its own contract tests, run on the published bytecode. And be clear-eyed about what the hiding buys you — a deterrent and a signal of intent, not a guarantee — so you invest the rest of your protection where it actually holds.

**Read on:**

- [StoatFlow: Kafka Streams compatible engine built to scale up — not out](https://stoatflow.io/blog/introducing-stoatflow)
- [ProGuard manual](https://www.guardsquare.com/manual/home){rel="&#x22;nofollow&#x22;"}
- [ProGuard retrace — deobfuscating stack traces](https://www.guardsquare.com/manual/tools/retrace){rel="&#x22;nofollow&#x22;"}
- [The Java Virtual Machine Specification — stack-map frames and verification](https://docs.oracle.com/javase/specs/jvms/se21/html/jvms-4.html#jvms-4.10.1){rel="&#x22;nofollow&#x22;"}


# In-place engine restart: the primitive behind multi-standby HA

The [failover-testing](https://stoatflow.io/blog/ha-failover-testing) post ended on a piece of work that was still exploratory, and a constraint it told you to live with. The work: tear down and rebuild the processing engine *in the same process*, without exiting the JVM, and resume from the last committed transactional offsets — an in-place restart. The constraint, stated plainly in its tradeoffs: *two is the supported topology; stay at two*. Both have moved. The in-place restart shipped, and a real leader election shipped on top of it — so scaling a StoatFlow HA cluster past two standbys, which used to crash-loop, now elects exactly one successor and holds.

Two changes, and they belong together. This is what they are, why the pair needed them, and what a live three-pod cluster does with them — measured on StoatFlow :stoatflow-version, an exactly-once word-count deployment on Kubernetes.

> **TL;DR**
>
> - **What:** An **in-place engine restart** — rebuild the engine inside the live JVM, no process exit — and a **leader election** built on it that makes one active plus K warm standbys a supported topology.
> - **Why:** Every role change used to wrap engine teardown in a process restart, so a graceful demote bounced the pod and paid a cold store open. And scaling the pair past two crash-looped: competing standbys promoted at once and the exactly-once fence resolved the losers by killing them.
> - **How:** The restart swaps the engine instance in place and resumes from the last committed offsets. Promotion now goes through a token claimed *before* the fence, as the sole authority to promote; a standby that loses the claim stands down as a return value, not a crash.
> - **The guarantee:** Exactly-once is unchanged — the same `transactional.id` epoch fence that protects a cross-pod handoff protects an in-place one, and it stays the backstop behind the election.
> - **The catch:** Still exactly one active. K standbys are redundancy, not scale — and each one is another changelog reader. To go faster, scale up, not out.
> - **Docs:** [High availability](https://stoatflow.io/docs/operating/high-availability).

## In this post

- [The process round-trip we set out to remove](https://stoatflow.io/#the-process-round-trip-we-set-out-to-remove)
- [Restarting in place](https://stoatflow.io/#restarting-in-place)
- [From a pair to a cluster](https://stoatflow.io/#from-a-pair-to-a-cluster)
- [A real election](https://stoatflow.io/#a-real-election)
- [What the cluster does now](https://stoatflow.io/#what-the-cluster-does-now)
- [Tradeoffs and limits](https://stoatflow.io/#tradeoffs-and-limits)
- [Running it](https://stoatflow.io/#running-it)

## The process round-trip we set out to remove

Look at the four ways a StoatFlow active can lose its role — a graceful `/ha/switch`, a SIGTERM from a rolling deploy, a JVM crash, a lost node — and one round-trip recurs underneath all of them: the active *exits the process and lets Kubernetes restart it*. A graceful demote self-terminated and came back as a fresh pod. A crash was a container restart. Even an in-place recovery was the orchestrator rebuilding the whole container. The engine teardown and re-initialisation always happened, but always wrapped in a process restart.

That wrapper is expensive in exactly the case you most want to be cheap. A `/ha/switch` is a planned, graceful handoff — it should cost about one commit. But if the demoted active has to exit and be rescheduled, it pays the full price of a cold start on the way back: a new container, a fresh JVM, and a RocksDB store that opens and replays the changelog before the pod is a warm standby again. On large state that is minutes, for an operation whose useful work took milliseconds. The failover-testing measurements named this as the last remaining cost on the fast paths — the promotion itself was already sub-second; what was left was the process round-trip around it.

## Restarting in place

The in-place engine restart removes the wrapper. Instead of exiting, the instance swaps its engine: it tears down the old `StreamProcessingEngine` and builds a fresh one inside the same live process, then resumes from the last committed transactional offsets. The JVM never exits, the container is never rescheduled, and — the part that matters for recovery time — the state stores are never closed and reopened. The store registry is reused across the swap, so a fresh engine attaches to the already-open, already-warm RocksDB rather than cold-starting it.

Two dispositions cover the reasons an engine gets rebuilt. A **graceful** restart drains the lanes, broadcasts a final commit barrier, and commits the in-flight transaction before tearing down — the new engine resumes from a clean, committed boundary with nothing to reprocess. An **abort** restart throws the in-flight epoch away: it aborts the open transaction, tears down, and the fresh engine reprocesses the uncommitted tail. That is an in-process crash recovery, and it is what a transient processing fault gets — the faithful analogue of Kafka Streams' `REPLACE_THREAD`, which recycles the failed stream thread and keeps the application up. StoatFlow has no per-partition thread to recycle, so until now a fault that Kafka Streams would have shrugged off took the whole instance down. Now it rebuilds the engine and stays running, behind a fault-only budget that escalates to a real shutdown if restarts start looping.

Exactly-once holds across the swap the same way it holds across a pod restart. The fresh engine's transactional producer is fenced by its `transactional.id` epoch — the identical mechanism that fences a crashed pod's producer when a standby takes over — so an aborted epoch's writes can never reach a `read_committed` consumer, whether the engine that wrote them died with the JVM or was torn down inside it. The restart is scheduler-agnostic: the same primitive serves a transient fault, a hot-standby demote, and — later — dynamic lane re-tuning. Build it once, cleanly, and each of those gets faster.

## From a pair to a cluster

Hot standby shipped as a pair: one active, one warm standby. One spare is enough to survive a lost node, but operators wanted more — a second spare so a rolling deploy never drops to zero redundancy, and headroom to lose a node *during* a deploy. The obvious move is to raise the replica count. It didn't work.

Scaling the pair to three replicas crash-looped. Two standbys would decide to promote at the same moment; the exactly-once fence did its job and let only one commit — but it resolved the losers by fencing their producers, which surfaced as `ProducerFencedException`, flipped them to `ERROR`, and let Kubernetes restart them straight back into the same race. The failover-testing post measured the pair and, in its tradeoffs, said exactly this: *stay at two*. On the live cluster the failure was not subtle — three pods logged **52, 53, and 53 restarts in twelve hours** before the fix.

The root cause is worth stating precisely, because it explains the shape of the fix. StoatFlow had a strong *fencing* layer and a weak *election* layer. Fencing — the transactional producer epoch — guarantees at most one instance ever commits, and it is airtight. But it is a safety mechanism, not a selection one: when two standbys promote together, the fence decides the winner by *who called `initTransactions()` last*, which is a function of timing, not of which standby is furthest caught up. And its way of saying no to a loser is to kill it. With one standby there is never a second promoter, so the gap never showed. With two, the election collapsed onto the fence — and the fence elects by accident and resolves by casualty.

## A real election

The fix is to add the election layer the pair never needed: pick the successor deliberately, and *before* anyone touches the fence.

Promotion now goes through a **promotion token** — a single record on the same compacted coordination topic the pair already uses, claimed by a compare-and-set. Winning the claim is the sole authority to promote: a standby reads the current holder, and if its claim goes through it proceeds to fence-and-restore; if it loses, it **stands down** — it becomes a standby again and waits, as an ordinary return value, with no fenced producer, no `ERROR`, no restart. That single reordering — claim the token, *then* fence — is what turns the crash-loop into a clean stand-down. The loser never reaches the fence, so the fence never has to kill it.

Who claims? The token is contended by the election winner, and the winner is chosen the way the [hot-standby post](https://stoatflow.io/blog/hot-standby-high-availability) said it would be — lag-aware, so the freshest standby wins. Candidates are ranked by replication lag; the most caught-up wins, and an exact tie breaks on a stable hash of the pod identity, so the choice always converges on one even with no coordinator to ask. Lag narrows the field to who *should* take over; the hash guarantees the field narrows to *one*; the token compare-and-set serialises the claim, so that even if two pods pick differently under a brief split view, only one wins and the other stands down.

The load-bearing rule is when a claimant may take the token from a holder that already has it. The answer: only if the holder is *unprotected* — either stale, its heartbeats gone quiet past the staleness window so it has likely crashed, or gracefully draining, on its way out and saying so. Against a live, healthy holder, a claimant stands down. That one rule does double duty. It lets a genuinely dead active's token be reclaimed quickly, and it lets a *graceful* demote hand off immediately: the demoting active publishes that it is draining, which marks its token takeable, so the designated successor claims it at once instead of waiting out the staleness clock. This is exactly where the in-place restart plugs in — a graceful `/ha/switch` now demotes the old active *in place*: it drains, drops the token, rebuilds its engine as a standby in the same pod, and rejoins warm. No exit, no reschedule, no cold store open.

The fence does not go away. It is still there, still airtight, still the thing that makes split-brain impossible under exactly-once. The token and the fence divide the work cleanly: the token is an **availability** mechanism — it bounds how often two pods ever contend to promote — and the fence is the **correctness** mechanism — it bounds what happens if they do. Correctness never depends on the token being perfect. If the election ever misfires, the fence still guarantees one committer; the token just means the loser learns it lost by reading a record instead of by being executed.

## What the cluster does now

The proof is the same three-replica deployment, on the same cluster, with only the build changed. On the old image it logged 52, 53, and 53 restarts across the three pods in twelve hours. On the fixed image it logs **zero** — steady state is one `ACTIVE` and two `READY_STANDBY`, a single token holder, and a token epoch that ticks up once per election rather than climbing without bound.

Every way an active can lose its role was run against the three-pod cluster, under load, exactly-once, with the three pods co-located on one worker. In each, the election picked one successor and no pod ever reached `ERROR` or CrashLoopBackOff.

| Scenario                     | Mechanism                         | Outcome                                                        | Pods restarted                                         |
| ---------------------------- | --------------------------------- | -------------------------------------------------------------- | ------------------------------------------------------ |
| **Graceful switch**          | `/ha/switch` → in-place role swap | freshest standby elected; old active demotes in place          | none                                                   |
| **SIGTERM / rolling deploy** | shutdown-hook handoff             | a standby elected on the `DRAINING` signal                     | old active recreated by the StatefulSet; standbys none |
| **JVM crash**                | SIGKILL the active's JVM          | same pod restarts in place and re-promotes; standbys untouched | the crashed pod only                                   |
| **Node loss**                | cordon + SIGKILL + force-delete   | a standby elected via staleness detection                      | crashed pod recreated; standbys none                   |
| **Rolling upgrade**          | `rollout restart`                 | one pod not-ready at a time, active rolled last                | one at a time, redundancy preserved                    |
| **Repeated switch ×3**       | `/ha/switch` in a loop            | one successor each time; never deadlocked in all-standby       | none                                                   |

Two honesty notes on the numbers behind that table. The promotion critical path — from deciding to promote to serving — measured between roughly 0.75 and 1.8 seconds across these runs, but this is a CPU-constrained box: three engines and a load generator sharing one eight-core worker, heavier than the two-worker pair the [earlier measurements](https://stoatflow.io/blog/ha-failover-testing) ran on. Don't read these as faster than those — it is a different, tighter test. And the wall-clock on the ungraceful paths is dominated by things that are not the election: a crashed pod carrying twelve hours of state spends most of its recovery in RocksDB restore, and a lost node spends most of its stop-the-world in the \~7-second staleness detection window. The election itself is the fast part; the numbers around it are state size and detection, as they were for the pair.

The rolling upgrade is where the extra standby earns its place. Readiness is gated not only on being caught up but on redundancy: a caught-up standby reports ready to roll only if rolling it would still leave the active plus at least one other caught-up standby. So a three-pod roll takes one pod down at a time, highest ordinal first, the active last — and never drops below an active and a warm spare while it runs. The pair had no spare to preserve during a roll; a cluster does, and the roll refuses to spend it.

## Tradeoffs and limits

The election and the in-place restart change how a cluster behaves; they do not change what StoatFlow is. The edges are worth stating plainly.

- **It is still exactly one active.** K standbys are redundancy, not throughput — each is a warm spare, none processes source records. This is the single-instance model intact: you don't pay the *distribution tax* for scale-out you don't need, and to go faster you give the active more cores, not more replicas.
- **Each standby is another changelog reader.** A standby stays warm by streaming the changelog the active writes, so K standbys multiply that read fan-out K-fold. `ha.max-standbys` bounds it — pick the redundancy you need, not the maximum you can spell.
- **A busy active on a shared node has less headroom than it looks.** The co-located drill surfaced this: three engines and a load generator on one worker, and a fan-out workload can starve the active of CPU until a commit misses its window and the exactly-once machinery times the transaction out. That now triggers a clean in-place restart rather than a hung pod — but it is still a restart. Give the active room; the in-place restart makes the failure graceful, not free.
- **Node loss on a co-located cluster is a lost *active*, not a lost node.** If every standby shares a worker with the active, losing that worker loses all of them. A standby is insurance against losing a node only if it lives on a different one — spread the replicas across nodes for real node-loss protection.
- **The token is availability; the fence is correctness.** The election reduces how often two pods contend to promote; it does not, and is not relied upon to, guarantee one committer. The transactional fence does that — under a network partition and under a misfiring election alike. Under at-least-once there is no fence, so the token is the whole story; treat it as bounding contention, not eliminating it.
- **A `ProducerFencedException` on the demoting pod during a graceful switch is expected, not a fault.** The successor fences the old active's producer while it drains; the demoted pod moves through restore and back to a warm standby without ever reaching `ERROR`. It is the fence doing its job on a handoff, not a crash.

## Running it

The topology is a configuration change on the deployment you already run. Hot standby is still one knob — `stoatflow.ha.mode: active-standby` — and going past two is the replica count plus a small amount of intent: `ha.desired-standbys` sets the redundancy the readiness gate protects, `ha.max-standbys` caps the changelog fan-out, and `ha.failover-priority` nudges which equally-caught-up standby is preferred. The election, the token, the staleness thresholds, and the in-place demote all have working defaults; a typical deployment sets the mode and the replica count and leaves the rest.

The model is the one the [hot-standby post](https://stoatflow.io/blog/hot-standby-high-availability) described, now without its two footnotes. One active, warm spares, the election picks the freshest, the fence backstops — and a planned role change no longer bounces a pod or waits on a cold store. The two footnotes it shipped with — *promotion will become lag-aware* and *stay at two* — are both closed. The [High availability guide](https://stoatflow.io/docs/operating/high-availability) has the configuration reference and the operator endpoints, and [Exactly-once](https://stoatflow.io/docs/concepts/exactly-once) covers the guarantee the fence enforces. The failover drill in the repository runs every scenario in the table above against a live cluster, if you want the numbers on your own hardware rather than mine.


# Internal consistency on Kafka: emitting a correct answer at every commit

> **TL;DR**
>
> - **The test:** a stream of `$1` transfers between 10 accounts. `credits − debits = balance` per account; `total = Σ balance&#x60;. Money is only moved, so **`total` must be 0 at every consistent cut**.
> - **What everyone else does:** the **Flink Table API** emits 37,978,385 `total` values of which &#x2A;*13,325 are correct — 0.035%**; **ksqlDB** emits 340, of which only the last is right; our own **Kafka Streams** twin sends `total&#x60; to **−1,619 … +1,792** and emits **160 impossible balances** on a dataset where the only valid values are `{−1, 0, +1}`.
> - **What StoatFlow does:** **zero impossible balances in every mode**, and under `emit.mode = on-barrier&#x60; **`total` is exactly 0 at every committed cut** — measured on a 5-node cluster at 2,000 tx/s, verified across churn and unique-key workloads.
> - **Why:** the commit barrier is a genuine Chandy–Lamport cut aligned to Kafka offsets, and the cascade-join withholds it from a downstream operator until *every* contributing chain has pushed its epoch into it. That cross-stream synchronization is exactly what the other engines lack.
> - **The honest part:** getting to *perfect* took one real bug fix (a barrier-cut race across sub-topology boundaries). We found it, root-caused it, fixed it, and re-ran the benchmark to zero.

This post is about a property Jamie Brandon named [**internal consistency**](https://www.scattered-thoughts.net/writing/internal-consistency-in-streaming-systems/){rel="&#x22;nofollow&#x22;"}, the fact that no engine with a Kafka Streams–shaped API has it, and the measured demonstration that StoatFlow does.

## The money test

Take the simplest invariant in accounting: &#x2A;*money is only moved, never created or destroyed.** Model it as a stream.

```kotlin
// transactions: (from, to) — each moves $1
val tx = builder.stream<String, Transfer>("transactions")

val credits = tx.map { _, t -> KeyValue(t.to,   1L) }.groupByKey().reduce { a, b -> a + b }   // received
val debits  = tx.map { _, t -> KeyValue(t.from, 1L) }.groupByKey().reduce { a, b -> a + b }   // sent

val balance = credits.outerJoin(debits) { c, d -> (c ?: 0) - (d ?: 0) }   // per-account net
val total   = balance.groupBy { _, b -> KeyValue("ALL", b) }.reduce({ a, b -> a + b }, { a, b -> a - b })
```

`balance` for any account is always in `{−1, 0, +1}` for a ring of `$1` transfers. And `total`, the sum over all accounts, is **always 0** — every credit has a matching debit. That is the invariant. It is not an approximation that holds "eventually"; it holds at *every* point in time, because it is a conservation law.

So here is the question: when your streaming engine emits a `total` value, &#x2A;*is it 0?**

## What "internal consistency" means

Brandon's definition: a system is **internally consistent** if *every* output it produces is the correct output for **some** subset of the inputs it has seen so far. Not the latest subset, necessarily — but *a* coherent one. Every emitted row is a photograph of a real moment, never a double-exposure that blends two.

This is **strictly stronger than eventual consistency**, and the two don't imply each other. An eventually-consistent system is allowed to emit nonsense in the meantime as long as it converges. An internally consistent system is never allowed to emit nonsense at all — every intermediate output is itself a valid answer.

For the money test, internal consistency means: &#x2A;*every emitted `total` is 0.** If you ever see `total = 3`, the engine has shown you a world in which three dollars materialized from nothing — a subset of the inputs that never existed. That output is not "a little stale". It is *wrong*.

Brandon ran the test across engines, and the counts are the argument.

**Materialize** (and the differential dataflow it is built on) passes: 599 balance updates, and `total` is emitted exactly once, because the correct answer never changes.

**The Flink Table API** emits 18,999,012 balance updates and 37,978,385 `total` values. **13,325 of them are 0** — 0.035%. In one run, 26.3% of the emissions were a single value, 256, which the author could not explain and which differs run to run.

**ksqlDB** emits 340 `total&#x60; values, and in Brandon's words, &#x2A;"none of the outputs are correct until the last."* On his simplified dataset its per-account balances — which can only ever be `−1`, `0` or `+1&#x60; — reach **−135,051 … +2,848**.

**Kafka Streams** he couldn't get to produce output for the query at all ([KAFKA-12594](https://issues.apache.org/jira/browse/KAFKA-12594){rel="&#x22;nofollow&#x22;"}), which is why we built our own twin below.

## Why no Kafka-shaped engine has it

The failure isn't a bug in any of these engines. It's architectural, and it has the same root everywhere.

`balance = credits ⋈ debits` is a join of two aggregations that **fan out from one source**. In Kafka Streams (and anything shaped like it), the re-key sends the two halves of a single transfer down **two separate repartition topics**, consumed independently, each feeding an aggregation with its own state store and its own record cache that **evicts and emits an intermediate** whenever it fills. The join is driven by whichever half arrives — so when a transaction updates both, the join will, sooner or later, read the *new* credit against the *old* debit — one side at a different logical time than the other. The result is a `balance` that never existed, and a `total` that isn't 0.

There is **no cut that spans the topology**. Each operator has its own notion of "now". The record cache consolidates the *volume* of output — Kafka Streams emits far fewer rows than Flink — but each row it does emit is still a blend of two moments. Consolidated garbage is still garbage.

Our own Kafka Streams twin of the benchmark reproduces exactly this: **160 impossible balances** (individual values reaching 180 where only `{−1, 0, +1}` is possible), and `total&#x60; wandering to **−1,619 … +1,792**. The same phenomenon the article finds in ksqlDB, whose balances reach 135,051 on the same impossible quantity — less extreme here only because our churn rate is lower.

![Two arms of one fan-out — credits and debits — reaching a join at different moments. A step chart tracks how many transfers each arm has delivered; wherever the two lines sit apart, the shaded gap is exactly how far the sum of all balances is from zero. At the highlighted moment the join has absorbed five credits against two debits and answers with a balance of +2 for an account that can only ever be −1, 0 or +1.](https://stoatflow.io/assets/blog/internal-consistency_fan-out-blend.svg)

### The load-bearing insight: exactly-once is not internal consistency

It is tempting to think exactly-once processing fixes this. It does not, and the reason is worth internalizing:

**Flink's `EXACTLY_ONCE` sink atomically commits all 19 million garbage rows.** Transactional atomicity of an output *batch* says nothing about whether the *contents* of that batch are a coherent view of anything. Exactly-once guarantees you won't see a row twice or lose one. It says nothing about whether any individual row was ever true. They are orthogonal properties, and every Kafka-shaped engine ships the first without the second.

### The fair caveat: this is a property of the DSL, not of the engine

Brandon's article carries an update worth reading before you conclude Flink can't do this. Vasia Kalavri showed him an internally consistent implementation of the same example in Flink's **DataStream** API, built out of custom operators with `ProcessFunction` — balances only ever `−1`, `0` or `+1`, and `total&#x60; always 0. His own conclusion: &#x2A;"given that the datastream api gives you the tools to build consistent systems, I'm curious as to why the table api doesn't use them."*

So Flink's checkpoint barriers can carry a consistent cut. What doesn't carry it is the **declarative layer** — the Table API, Flink SQL, the Kafka Streams DSL, ksqlDB. Every one of them forwards eagerly, and getting a coherent answer out of them means dropping a level and hand-compiling your topology into operators you write yourself.

That is the honest shape of the claim, and it's the one we're making: not that nobody else can be internally consistent, but that nobody else gives it to you **from the DSL, on Kafka, as a flag you turn on**. Our advantage over Flink here is narrower than our advantage over Kafka Streams, and it is worth being precise about which one it is.

## Why StoatFlow is different

StoatFlow is a single-instance engine: one replica per application, all state global, parallelism from Project Loom virtual threads rather than from partitioned tasks. That architecture turns out to be exactly what internal consistency needs, for four independent reasons:

1. **The commit barrier is a genuine consistent cut.** StoatFlow processes in epochs delimited by a barrier that flows through the whole topology; when it commits, it commits a **per-partition offset prefix** — a Chandy–Lamport cut of the input, snapshotted on a single thread between the last dispatched record and the barrier broadcast. There is one "now" for the entire topology, not one per operator.
2. **The cascade-join synchronizes across streams.** When two chains fan out from a source and merge at a join, StoatFlow &#x2A;*withholds the barrier from the join's sub-topology until *both* contributing chains have pushed their epoch-N output into it.** The join never processes its barrier — never commits — having seen the new credit but not yet the new debit. This cross-stream synchronization is precisely the primitive the article says nobody has. We didn't build it for this; we built it for exactly-once, and it happens to be exactly what internal consistency requires.

![The same fan-out, now with a commit barrier travelling downstream. The barrier is injected while the two arms are still pushing epoch-N credits and debits into the join, and the join's sub-topology is shaded for the whole span in which it is deliberately not given the barrier. Only once the contributing sub-topology has delivered every last half does the gate release it — so the committed cut holds six credits against six debits, and the balances sum to exactly zero.](https://stoatflow.io/assets/blog/internal-consistency_cascade-join.svg)

3. **Watermarks are inserted at the edge.** One global watermark, the minimum across non-idle partitions, broadcast identically to every sub-topology. Brandon's fourth failure mode — per-operator watermarks that drop different data at different operators — cannot occur here by construction.
4. **The cache never evicts.** When StoatFlow's state cache fills, it does **not** flush an eldest entry and emit an intermediate (the Kafka Streams behavior that produces mid-epoch blends). It **cuts the epoch short** — fires an early barrier — and commits a coherent cut. Under memory pressure it commits *sooner*; it never leaks a garbage intermediate. This is what turns barrier-gated emission from best-effort into a *guarantee*.

What was missing was small by comparison: like every Kafka-shaped engine, **every non-monotonic operator forwarded its output eagerly** — one row per input change. So we added the gate: an opt-in `emit.mode = on-barrier` (and a per-operator `Suppressed.untilBarrier()`) that holds a KTable operator's emissions and releases a single consolidated value **per consistent cut**.

## The measured result

We ported Brandon's benchmark verbatim — 10 accounts, `$1` transfers, the `credits`/`debits`/`balance`/`total` topology above — and ran it on a 5-node cluster at 2,000 transactions/second under exactly-once, with a `read_committed` verifier that flags any `balance` outside `[−1, 1]` and any `total ≠ 0`.

**Per-account balances — StoatFlow is consistent even without the valve:**

| Metric                                | StoatFlow EAGER | StoatFlow ON\_BARRIER | **Kafka Streams** |
| ------------------------------------- | --------------- | --------------------- | ----------------- |
| Impossible balances (\|balance\| > 1) | **0**           | **0**                 | **160**           |
| Max \|individual balance\|            | 1 (in range)    | 1                     | **180**           |
| Max \|Σ balance\| excursion           | **5**           | **0**                 | **1,792**         |
| Balance output rate                   | 6,860/s         | 6,876/s               | 45/s              |

Even in **EAGER** mode — one emission per change, the same eager forwarding Kafka Streams uses — StoatFlow emits **zero impossible balances** — none in 1.67 million emissions — and `Σ balance` never leaves `[0, +5]`. That residual +5 isn't an inconsistency: it's the verifier observing an epoch's \~10 per-account updates one at a time. At every *committed* cut, `total` is exactly 0. Kafka Streams, on the identical workload, emits 160 balances that cannot exist.

The two engines from the article aren't columns here, because Brandon doesn't report per-account balance statistics in a form that lines up with ours — and his dataset isn't ours, so putting his numbers in the same row would invite a comparison neither of us measured. For the record: the Flink Table API's balance histogram runs to **33*&#x2A;, and on his simplified dataset ksqlDB's balances reach **−135,051 … +2,848**. Both on a quantity whose only legal values are `−1`, `0` and `+1`.

**The `total` view — the article's headline test.** Now consolidate to one `total` per cut with `emit.mode = on-barrier`:

| Metric (`total` view)        | StoatFlow EAGER | **StoatFlow ON\_BARRIER** | **Kafka Streams**   | Flink Table API (article) | ksqlDB (article) |
| ---------------------------- | --------------- | ------------------------- | ------------------- | ------------------------- | ---------------- |
| Max \|total\|                | +5…8            | **0**                     | **−1,619 … +1,792** | not published             | not published    |
| `total` emissions in the run | 1,674,738       | 210                       | 191                 | 37,978,385                | 340              |
| Of those, exactly 0          | 266,731 (15.9%) | **210 (100%)**            | 175 (91.6%)         | 13,325 (0.035%)           | 1                |

Two of those columns are Brandon's, not ours, and they carry a gap: neither the Flink nor the ksqlDB `total&#x60; has a published minimum and maximum. He reports Flink's qualitatively — the error &#x2A;"seems to be increasing over time"&#x2A;, oscillating &#x2A;"between a number of stable attractors"* — and ksqlDB's not at all. What both do publish is how many values came out and how many of them were right, which is the row that matters most anyway.

Under `on-barrier`, **every emitted `total` is exactly 0** — verified across both a high-churn workload and a unique-key (worst-case) workload, at \~one emission per epoch (a \~680× reduction in output volume versus EAGER, each one a true consistent-cut value). This is the property the article says no Kafka-shaped engine has: `total` never leaves 0, on Kafka.

Kafka Streams, meanwhile, sends the *same* `total` view to −1,619 … +1,792. Its record cache consolidates the stream — few emissions — but the emissions it does make are garbage totals.

Those numbers span four orders of magnitude and end at zero, which is hard to see in a table. On a log axis, where every gridline is a 10× step, the shape of the result is the whole argument:

:consistency-excursion

Read it as an error bar: the invariant says `total = 0`, so a bar's length *is* how wrong the engine got. Two decades separate Kafka Streams from StoatFlow's eager mode — but that is a difference of degree. The step that matters is the last one, from small to none.

Magnitude is only half of the question, though, and it's the half where the engines Brandon tested can't be plotted at all. The other half — how often an engine is right in the first place — is the one axis where every engine in this story has published numbers:

:consistency-correctness

That chart is the reason both halves are here. Kafka Streams is right 91.6% of the time and StoatFlow's eager mode only 15.9% — but the eager mode's errors are never larger than +5, and Kafka Streams' reach ±1,792. Being wrong rarely and being wrong slightly are different virtues, and neither is the one you want. There is exactly one row that needs no such trade-off.

A single number per engine still understates it, because it suggests a system that is steadily a bit wrong. That is not the failure mode. Here is the same measurement over time, one engine per chart — same run, same 210-second window, same invariant. These are the three engines we measured ourselves; the Flink and ksqlDB figures above are Brandon's.

### Kafka Streams: right almost always, and wildly wrong the rest of the time

:consistency-time-series{panel="ks"}

175 of its 191 emissions read exactly 0. The other sixteen reach −1,619 and +1,792. That is the shape of the problem: not a drift you could bound and compensate for, but a correct-looking line that is briefly, unpredictably, wildly wrong.

The record cache is what makes it look so calm. It consolidates each commit interval back to a consistent 0, so the garbage totals surface only on the mid-interval evictions — which is precisely what makes this failure mode hard to catch in production. If you sampled this dashboard once a second you would call it healthy and be wrong about one sample in twelve.

### StoatFlow EAGER: wrong constantly, and never by more than +5

:consistency-time-series{panel="eager"}

Read the axis before the shape: this chart is zoomed 333× against the one above, because at Kafka Streams' amplitude StoatFlow's eager mode is indistinguishable from the zero line. Its `+5` would be a fortieth of a pixel up there.

On that axis, `EAGER` looks worse than Kafka Streams, and on one measure it is: 84.1% of its 1.67 million emissions are non-zero, against Kafka Streams' 8.4%. But the band never leaves `[0, +5]`, and it never goes negative. That residual isn't an inconsistency in the engine — it's the verifier watching an epoch's \~10 per-account updates land one at a time. At every *committed* cut, `total` is already exactly 0. Wrong often and wrong slightly is a different failure from wrong rarely and wrong by 1,792, and neither is the one you want.

### StoatFlow ON\_BARRIER: exactly 0, at every emission

:consistency-time-series{panel="on-barrier"}

Same zoom, same axis, same 210 seconds as the chart above — so the empty space here is precisely the space `EAGER`'s band fills. Hold the two side by side and that gap is the whole feature.

There is no line to trace because there is no variation to show. All 210 emissions are 0, one per commit barrier, \~1.00 s apart, each one a true consistent-cut value. Not a small error. The absence of one.

## The honest part: the bug we had to fix

The first time we ran the `on-barrier` benchmark, it wasn't perfect. `total` was 0 at 406 of 412 emissions on the churn workload — and at 390 of 412 on unique keys. A small residue of tiny (±2…±4) violations survived the valve: &#x2A;*6 on churn, 22 on unique keys.**

That residue was easy to hand-wave away as a startup transient. It wasn't. The full-topic dump showed the violations were scattered through the run, all positive (a credit reflected without its matching debit), and — the tell — the *slower* workload had *more* of them. That ruled out startup and churn and pointed at something structural.

It was: the commit barrier was a consistent cut *within* a sub-topology, but **not across a sub-topology boundary**. The `total` view sits two aggregation levels below the fan-out-merge join, behind a repartition. When the barrier cascaded across that boundary on a detached thread, an upstream lane could resume the instant it reported the barrier and forward a *next*-epoch record that overtook the barrier into the downstream cut — leaking a sliver of the following epoch into the committed transaction. Under crash recovery those leaked records would even double-count.

The fix (we call it receiver-side epoch hold-back) tags every record that crosses a sub-topology boundary with the epoch it belongs to; the receiving side briefly holds back any record that ran ahead of the barrier until the barrier arrives there too. No pauses, no coordinator changes — the cut is realigned across the boundary by construction.

We re-ran the benchmark on the fixed build. &#x2A;*Churn: 6 → 0. Unique keys: 22 → 0.** Every one of the \~412 committed `total` emissions is now exactly 0, on both workloads, at 2,000 tx/s. We're writing this down because "near-perfect" is not the claim — *perfect, and here's the run that proves it* is.

## How to turn it on

Internal consistency is opt-in and per-operator, because eager emission is the right default for latency-sensitive views and matches Kafka Streams exactly. When you want a coherent view, gate it:

```kotlin
// Per operator:
total.suppress(Suppressed.untilBarrier())   // emit one consolidated value per consistent cut

// Or topology-wide, in application.yaml:
stoatflow:
  emit:
    mode: on-barrier
```

Under the hood this buffers a KTable operator's emissions per lane and releases them at the commit barrier — riding the same Kafka transaction as the offsets that produced them, so the consolidated emission is exactly-once *and* internally consistent. The buffer is bounded by the same early-barrier mechanism as everything else: it never grows without cutting the epoch.

## Credits

This work started with Jamie Brandon's [*Internal consistency in streaming systems*](https://www.scattered-thoughts.net/writing/internal-consistency-in-streaming-systems/){rel="&#x22;nofollow&#x22;"} — the benchmark, the definition, and the framing are his, and the article is the clearest statement of the problem we know of. Read it.

The investigation was prompted by [Ralph M. Debusmann](https://www.linkedin.com/in/ralph-m-debusmann-63204675/){rel="&#x22;nofollow&#x22;"}, whose Berlin Buzzwords 2026 talk [*Kafi Streams – Complex Stream Processing Made Simple*](https://2026.berlinbuzzwords.de/session/kafi-streams-complex-stream-processing-made-simple/){rel="&#x22;nofollow&#x22;"} ([recording](https://www.youtube.com/watch?v=_oKQaDyP1CQ){rel="&#x22;nofollow&#x22;"}) and the conversations that followed put the question of whether a Kafka-shaped engine *could* be internally consistent in front of us. The answer, it turns out, is yes — and it was mostly already there.

---

*StoatFlow is a single-replica, Kafka Streams–compatible stream processor built on JDK 25 virtual threads. Internal-consistency mode is available now. [Get in touch](https://stoatflow.io/contact) for early access, or head to [Getting Started](https://stoatflow.io/docs/getting-started).*


# StoatFlow AI Assistant Skills: correct StoatFlow, not hallucinated Kafka Streams

> **TL;DR**
>
> - **What:** a public, versioned pack — six task-cut skills (build, test, port, configure, set up, operate) plus a shared primer and editor rule files, all Apache-2.0.
> - **Install (Claude Code):** `/plugin marketplace add stoatflow/skills`, then `/plugin install stoatflow@stoatflow`.
> - **Install (any agent):** `npx skills add stoatflow/skills`, or copy `AGENTS.md` into your repo.
> - **Pin the version:** the pack ships in lockstep with StoatFlow releases — use the tag matching yours.
> - **The catch:** it documents StoatFlow; you still need customer credentials to resolve the library from `maven.stoatflow.io`.

We've published the **StoatFlow AI Assistant Skills**: a free, open pack of six task-cut skills and editor rule files that make Claude Code, Cursor, Copilot, and other coding assistants write correct StoatFlow instead of hallucinated Kafka Streams.

The pack exists because of a specific problem. StoatFlow is source-compatible with Kafka Streams — swap the import root and your topology compiles — and that same compatibility is why an AI assistant gets it confidently wrong.

## Why your AI assistant writes Kafka Streams, not StoatFlow

StoatFlow ships the Kafka Streams DSL over a different engine. You swap `org.apache.kafka.streams.*` for `io.stoatflow.core.*`, recompile, and existing topology code runs. That drop-in compatibility is the whole point of the DSL — and it's exactly what trips an assistant up.

Every mainstream model trained on years of Kafka Streams code and documentation. Ask one for a StoatFlow topology and it reaches for what it already knows:

- the wrong import root — `org.apache.kafka.streams.*` rather than `io.stoatflow.core.*`;
- the Kafka Streams `at_least_once` default, where StoatFlow defaults to exactly-once;
- `replicas: N` horizontal scaling, where StoatFlow runs one instance per app and scales up, not out;
- Maven Central coordinates, where StoatFlow resolves only from the private `maven.stoatflow.io`;
- Kafka Streams watermark and timestamp semantics, where StoatFlow follows Flink instead.

Every one of these compiles. Every one is wrong in a way you catch only if you already know StoatFlow — a default baked into a config, a scaling model that doesn't exist here, an exactly-once guarantee downgraded to at-least-once. Source-compatibility removes the friction that would otherwise flag the mismatch, so the assistant never gets the signal that it's on unfamiliar ground.

## What's in the pack

The guidance is cut the way you actually work — one skill per task, so the relevant rules load when they matter instead of arriving as one wall of text. Each skill fires on what you're doing:

| Skill                      | Loads when you're…                                                                 |
| -------------------------- | ---------------------------------------------------------------------------------- |
| `stoatflow-build-topology` | writing topology code — DSL, Processor API, serdes, state stores, DLQ              |
| `stoatflow-test`           | writing tests — `TopologyTestDriver`, integration tests                            |
| `stoatflow-port-from-ks`   | porting a Kafka Streams app — code and state                                       |
| `stoatflow-configure`      | configuring an app — `application.yaml`, guarantees, lanes, HA                     |
| `stoatflow-project-setup`  | wiring the build — the private Maven repo, licence, JDK 25, Docker, native image   |
| `stoatflow-operate`        | deploying and running it — single-instance Kubernetes, HA, probes, metrics, tuning |

Under the skills sits a shared primer — the identity and the hard rules every skill inherits — and a set of editor rule files generated from that same source: an `AGENTS.md` for Codex and the AGENTS.md-aware tools, plus Cursor, Copilot, and JetBrains variants. The same facts, whichever assistant reads them.

## How it stays correct

Documentation that has drifted from the code is worse than none: it's wrong with authority, and an assistant repeats it without the hesitation a human reader might feel. The pack is built against that failure mode. On every StoatFlow release we drift-check it against the three sources it can't afford to lag — the porting guide, the compatibility matrix, and the configuration schema — and ship it with a version that *is&#x2A; the StoatFlow version it targets. Each artefact prints a &#x2A;"Targets StoatFlow :stoatflow-version "* line. Pin the tag to your release, and the divergence rules and config reference match the library you're actually running.

## Install it

The pack lives in the public repo [`stoatflow/skills`](https://github.com/stoatflow/skills){rel="&#x22;nofollow&#x22;"}. Install it the way your assistant expects:

| Tool                        | Install                                                                                |
| --------------------------- | -------------------------------------------------------------------------------------- |
| **Claude Code**             | `/plugin marketplace add stoatflow/skills`, then `/plugin install stoatflow@stoatflow` |
| **Any agent (skills CLI)**  | `npx skills add stoatflow/skills`                                                      |
| **Codex / AGENTS.md tools** | copy `AGENTS.md` into your app repo                                                    |
| **Cursor**                  | copy `cursor/rules/stoatflow.mdc` to `.cursor/rules/`                                  |
| **GitHub Copilot**          | copy `copilot/stoatflow.instructions.md` to `.github/instructions/`                    |
| **JetBrains AI / Junie**    | copy `jetbrains/guidelines.md` to `.junie/guidelines.md`                               |

## What it doesn't do

A few boundaries worth stating plainly.

The pack teaches the public API and the concepts, and nothing below them. StoatFlow's engine internals are obfuscated in the shipped jar and off-limits by design; the pack holds the same line, so it won't explain — or invent — how the engine works inside.

A skill shifts what your assistant reaches for first. That's most of the battle, not a guarantee: a determined model can still produce Kafka Streams under a StoatFlow prompt. Read the output, especially around guarantees, scaling, and imports — the three it gets wrong most.

StoatFlow itself is commercial; the pack is free and Apache-2.0. Installing the skills documents the library, it doesn't unlock it — you still need customer credentials to resolve StoatFlow from `maven.stoatflow.io`.

And it's new. It shipped with this release and improves with each one, so pin the matching tag and expect the rules to sharpen release over release.

## Where to go next

- The pack, with every skill and rule file: [`stoatflow/skills`](https://github.com/stoatflow/skills){rel="&#x22;nofollow&#x22;"}.
- The setup guide, including the per-editor install: [AI assistants](https://stoatflow.io/docs/getting-started/ai-assistants).
- Found something the pack gets wrong? [Reach out](https://stoatflow.io/contact) — it's maintained upstream, and a concrete miss is the most useful thing you can send.

For the running commentary on how StoatFlow gets built, [follow along on LinkedIn](https://www.linkedin.com/in/hartmut-co-uk/){rel="&#x22;nofollow&#x22;"}.