# StoatFlow

> JVM stream processing — Kafka Streams DSL + Flink-style commit-barrier reliability on Loom.

## Documentation Sets

- [StoatFlow — full documentation](https://stoatflow.io/llms-full.txt): The complete StoatFlow docs, changelog, and published blog as one markdown bundle for direct LLM ingestion.

## Getting Started

- [Getting started](https://stoatflow.io/raw/docs/getting-started.md): Install StoatFlow and ship your first topology — or migrate an existing Kafka Streams app.
- [Modules overview](https://stoatflow.io/raw/docs/getting-started/modules-overview.md): The StoatFlow module landscape — core, runtime, the two testing modules, and the io.stoatflow Gradle plugin — and how to decide which to depend on.
- [Installation](https://stoatflow.io/raw/docs/getting-started/installation.md): Add the private StoatFlow Maven repository, pull the dependency, and set up your Gradle or Maven build.
- [License configuration](https://stoatflow.io/raw/docs/getting-started/license-configuration.md): Configure your StoatFlow license key for local development and CI/CD — environment variables, system properties, key files, and application.yaml.
- [Your first app](https://stoatflow.io/raw/docs/getting-started/first-app.md): Build and run a complete word-count stream processor on the StoatFlow runtime — in Kotlin or Java.
- [Project setup](https://stoatflow.io/raw/docs/getting-started/project-setup.md): Apply StoatFlow's build conventions with Gradle or Maven — the JDK 25 toolchain, the required JVM flags, a runnable fat JAR, and opt-in reproducible Docker and GraalVM native images, from a few lines of config.
- [AI assistants](https://stoatflow.io/raw/docs/getting-started/ai-assistants.md): Install the StoatFlow skills pack so Claude Code, Cursor, Copilot, and other AI coding assistants write correct StoatFlow code instead of hallucinated Kafka Streams.

## Core Concepts

- [Core concepts](https://stoatflow.io/raw/docs/concepts.md): A map of the StoatFlow concept pages and the order to read them in — from the single-instance architecture through exactly-once, lanes, state, event time, configuration, and error handling.
- [Architecture](https://stoatflow.io/raw/docs/concepts/architecture.md): How StoatFlow runs your Kafka Streams topology as a single replica — the conceptual model, processing lanes, commit barriers, state, and operational surface.
- [Exactly-once semantics](https://stoatflow.io/raw/docs/concepts/exactly-once.md): How StoatFlow's commit barrier delivers end-to-end exactly-once — one Kafka transaction commits state, output, and offsets atomically — plus crash recovery and the at-least-once trade-off.
- [Lanes and parallelism](https://stoatflow.io/raw/docs/concepts/lanes-and-parallelism.md): How key-affinity lanes give StoatFlow per-key ordering and cross-key parallelism — decoupled from partition count, cheap to scale with cores, and friendly to in-line blocking I/O.
- [State and thread-safety](https://stoatflow.io/raw/docs/concepts/state-and-thread-safety.md): How StoatFlow's global state model works — why concurrent access across lanes is safe, when you need the key-lock utility, and which store types are available.
- [Event time and watermarks](https://stoatflow.io/raw/docs/concepts/event-time-and-watermarks.md): How StoatFlow models time — event time vs processing time, record timestamps, watermark strategies, window closing, late-record handling, and event/processing-time timers.
- [The configuration model](https://stoatflow.io/raw/docs/concepts/configuration-model.md): How StoatFlow layers configuration — application.yaml, overlay files, environment variables, and programmatic overrides — the precedence order between them, and why some behaviour is adaptive rather than configured.
- [The error-handling model](https://stoatflow.io/raw/docs/concepts/error-handling-model.md): How StoatFlow classifies failures — deserialization, processing, and production — the skip / fail / dead-letter policies you choose, DLQ semantics, and the commit failure that ends in a restart.
- [How StoatFlow differs from Kafka Streams](https://stoatflow.io/raw/docs/concepts/how-stoatflow-differs-from-ks.md): The conceptual deltas if you already know Kafka Streams — single instance instead of a rebalancing cluster, in-memory re-keying instead of repartition topics, barrier-based exactly-once, global state, and lanes instead of partition-bound tasks. The DSL carries over unchanged.

## Building Topologies

- [Building topologies](https://stoatflow.io/raw/docs/building.md): How to define a StoatFlow topology — StreamsBuilder, the KStream/KTable abstractions, and the fan-out rule that reuses a KStream reference for multiple branches.
- [Error handling and DLQ](https://stoatflow.io/raw/docs/building/error-handling-dlq.md): Configure deserialization, processing, and production exception handlers — log-and-continue, log-and-fail, or dead-letter-queue — in code or YAML, and inspect the DLQ headers StoatFlow writes.
- [State stores](https://stoatflow.io/raw/docs/building/state-stores.md): Use the Stores factory and Materialized to configure persistent and in-memory key-value, window, session, and versioned stores — and read them back via interactive queries.
- [Testing topologies](https://stoatflow.io/raw/docs/building/testing.md): Unit-test a StoatFlow topology in-memory with TopologyTestDriver — pipe records, advance time and watermarks, read state stores, and drive YAML config through StoatFlowTestDriver. No broker required.
- [Sources and sinks](https://stoatflow.io/raw/docs/building/streams-builder.md): Read records into a topology with builder.stream / table / globalTable and Consumed, and write them back out with to() and Produced — serdes, offset reset, watermarks, partitioners, and the fan-out rule.
- [KStream and KTable operations](https://stoatflow.io/raw/docs/building/kstream-ktable.md): Stateless stream transforms (map, filter, flatMap, selectKey, branch, merge) and KTable basics — with the rules for when a key change forces records onto a different lane.
- [Aggregations](https://stoatflow.io/raw/docs/building/aggregations.md): Group records by key and fold them into running totals — count, reduce, aggregate on grouped streams and grouped tables, with the result backed by a state store.
- [Windowing](https://stoatflow.io/raw/docs/building/windowing.md): Windowed aggregations in StoatFlow — tumbling, hopping, sliding, and session windows; emit strategies, suppression, Windowed keys, and WindowedSerdes.
- [Joins](https://stoatflow.io/raw/docs/building/joins.md): Combine two streams or tables on a shared key — stream-stream windowed joins, stream-table enrichment lookups, table-table joins, foreign-key joins, and co-grouping.
- [The Processor API](https://stoatflow.io/raw/docs/building/processor-api.md): Write custom record-by-record logic with Processor and FixedKeyProcessor — forward, access state, schedule punctuators, and register key-based timers.
- [Scheduled sources](https://stoatflow.io/raw/docs/building/scheduled-sources.md): Generate records on an interval or a cron schedule with StreamsBuilder.scheduled() — a topology source that emits without consuming from Kafka.
- [Serdes and Avro](https://stoatflow.io/raw/docs/building/serdes.md): Configure default and per-operator serdes, use the built-in Serdes, write custom serdes, serialize windowed keys, and wire up Avro with Confluent Schema Registry.

## Configuration

- [How configuration works](https://stoatflow.io/raw/docs/configuration.md): How the StoatFlow runtime loads configuration — application.yaml on the classpath, overlay files, environment variables, system properties, and the layered precedence order.
- [Engine configuration (:core)](https://stoatflow.io/raw/docs/configuration/core-config.md): The :core engine knobs every StoatFlow app sets — application id, brokers, default serdes, lanes, commit-barrier cadence, state dir, changelog, processing guarantee, and event-time coordination — with defaults from source.
- [Runtime configuration (:runtime)](https://stoatflow.io/raw/docs/configuration/runtime-config.md): The runtime.* section of application.yaml — HTTP server, metrics, info/config endpoints, health checks — plus logging levels and Kafka client passthrough.
- [Defaults, adaptivity, and presets](https://stoatflow.io/raw/docs/configuration/defaults-and-presets.md): Why a minimal StoatFlow config runs well — the opinionated defaults, the parts that self-tune within bounds you set, and the RocksDB presets and tuning knobs you can change.
- [Kafka client configuration](https://stoatflow.io/raw/docs/configuration/kafka-client-config.md): Pass arbitrary consumer, producer, and restoration-consumer properties through to the Kafka clients, and understand which properties StoatFlow forces for correctness.

## Running in Production

- [The runtime](https://stoatflow.io/raw/docs/runtime.md): The batteries-included :runtime module — StoatFlowRuntime.fromConfig, the lifecycle (start / awaitTermination / stop), and what it adds over the bare :core engine.
- [The REST API](https://stoatflow.io/raw/docs/runtime/rest-api.md): A task-oriented tour of the runtime's built-in HTTP endpoints — metadata, topology, state and progress, health, control, license, metrics, and debug.
- [Health checks](https://stoatflow.io/raw/docs/runtime/health-checks.md): Liveness and readiness probes on the StoatFlow runtime — the built-in indicators, what flips each, and how they behave during state restoration.
- [Metrics](https://stoatflow.io/raw/docs/runtime/metrics.md): Scrape StoatFlow's Prometheus /metrics endpoint — Micrometer integration, the key meters the engine emits, license metrics, and common tags.
- [Pause and resume](https://stoatflow.io/raw/docs/runtime/pause-unpause.md): Pause and resume record processing at runtime with POST /pause and /unpause — for maintenance windows and downstream backpressure, without restarting the instance.
- [Plugins and lifecycle hooks](https://stoatflow.io/raw/docs/runtime/plugins.md): Extend the StoatFlow runtime with custom metrics, HTTP endpoints, and health indicators via the RuntimePlugin interface and lifecycle listeners.
- [Docker images](https://stoatflow.io/raw/docs/runtime/docker.md): Build a container image for your StoatFlow app with the StoatFlow build conventions (Jib) — base image, JVM flags, reproducible builds, container-aware heap sizing, ports and env — on Gradle or Maven.
- [GraalVM native image](https://stoatflow.io/raw/docs/runtime/native-image.md): Compile your StoatFlow app to a GraalVM native binary — a one-line opt-in. The framework ships its own reflection/JNI/FFM metadata; you choose the GC, optionally PGO, and add metadata only for your own serde value classes.

## Deploying & Operating

- [Deploying and operating](https://stoatflow.io/raw/docs/operating.md): The deployment model for operators — one active instance and why two corrupt state, the two HA tiers (fast restart and opt-in hot standby), and a map of the operating pages.
- [Running on Kubernetes](https://stoatflow.io/raw/docs/operating/kubernetes.md): Deploy a StoatFlow application as a single-replica StatefulSet — persistent state volume, termination grace aligned with shutdown, secure license injection, and a representative manifest.
- [High availability](https://stoatflow.io/raw/docs/operating/high-availability.md): Two HA tiers for StoatFlow — fast restart by default, or an opt-in hot-standby cluster (one active plus one or more warm standbys) with near-instant, lag-aware failover — how to choose, configure, deploy, and operate each.
- [Liveness and readiness probes](https://stoatflow.io/raw/docs/operating/probes.md): Wire StoatFlow's /health/live and /health/ready endpoints to Kubernetes probes — what readiness gates, why a frozen commit pipeline fails liveness, and how to tune probe periods.
- [Observability](https://stoatflow.io/raw/docs/operating/observability.md): Operating-time visibility for a single-instance app — scrape /metrics into Prometheus and Grafana, the meters worth alerting on, correlating logs to lanes and barriers via opt-in MDC, and the /debug/threads and /debug/barriers endpoints for diagnosing a frozen or slow pipeline.
- [Tuning under load](https://stoatflow.io/raw/docs/operating/tuning.md): Practical tuning for StoatFlow under load — lane count, commit-cadence bounds, the memory and uncommitted-state knobs, and RocksDB sizing for large state, each tied to the symptom that tells you to reach for it.
- [Production checklist](https://stoatflow.io/raw/docs/operating/production-checklist.md): A go-live checklist for StoatFlow — replica count, license, probes, metrics, error policies, tuning, and thread safety — each item linking to the canonical page that covers it in full.

## Migrating from Kafka Streams

- [Migrating from Kafka Streams](https://stoatflow.io/raw/docs/migration.md): What carries over unchanged when you move a Kafka Streams topology to StoatFlow, what changes operationally, and how to decide between a green-field cutover and a state-carrying migration.
- [Automated port](https://stoatflow.io/raw/docs/migration/automated-port.md): Migrate a Kafka Streams application's imports, entry point and build to StoatFlow with a single OpenRewrite command, across Java and Kotlin, with every judgment call flagged rather than guessed.
- [Migration without carrying state](https://stoatflow.io/raw/docs/migration/without-data-migration.md): Green-field cutover from Kafka Streams — point StoatFlow at the same source topics with a fresh consumer group, let stateful operators rebuild from changelog/source, then switch traffic. The dependency, build, and config swap, with the before/after grounded in the map-filter example.
- [Migration carrying state](https://stoatflow.io/raw/docs/migration/with-data-migration.md): The supported paths for carrying Kafka Streams state into StoatFlow — the reprocess | translate | engage matrix, per-store-type support, the classification table for the migration tool, and the semantics caveats at the cutover boundary.
- [The migration tool](https://stoatflow.io/raw/docs/migration/migration-tool.md): How to run stoatflow-migration-tool — the offline Kafka Streams changelog translator. Config reference, the plan / translate / seed-offsets / verify commands, the cutover runbook, and the rollback line.
- [Reusing your Kafka Streams dashboards](https://stoatflow.io/raw/docs/migration/reusing-kafka-streams-dashboards.md): Turn on Kafka Streams-compatible metrics so your existing Grafana dashboards, Prometheus rules, and alerts light up against a StoatFlow app with at most threshold-level edits.

## Reference

- [Reference](https://stoatflow.io/raw/docs/reference.md): Lookup material for StoatFlow — configuration keys, REST API, the Gradle plugin and Maven build references, the Kafka Streams compatibility matrix, the metrics catalogue, and the glossary.
- [Configuration reference](https://stoatflow.io/raw/docs/reference/configuration-reference.md): Curated reference of the StoatFlow configuration keys — stoatflow.* (engine) and runtime.* (HTTP, metrics, health) — with type, default, and a one-line description per key.
- [REST API reference](https://stoatflow.io/raw/docs/reference/rest-api-reference.md): Every HTTP endpoint the StoatFlow runtime exposes — path, method, purpose, response shape, content negotiation, status codes, and gating.
- [Gradle plugin reference](https://stoatflow.io/raw/docs/reference/gradle-plugin-reference.md): The io.stoatflow convention plugin — the full stoatflow { } extension DSL, the plugins and JVM flags it applies, the tasks it contributes, and the Docker and native-image defaults.
- [Maven reference](https://stoatflow.io/raw/docs/reference/maven-reference.md): StoatFlow's Maven support — the stoatflow-bom, the stoatflow-parent convention POM, and the stoatflow-maven-plugin. The three artifacts, every <properties> key, the four goals, and the pinned plugin versions.
- [Kafka Streams compatibility matrix](https://stoatflow.io/raw/docs/reference/ks-compatibility-matrix.md): Operator-by-operator parity between the StoatFlow DSL and the Kafka Streams DSL — implemented, partial, not yet implemented, deprecated, incompatible, and StoatFlow extensions.
- [Metrics reference](https://stoatflow.io/raw/docs/reference/metrics-reference.md): The full catalogue of Prometheus meters StoatFlow exposes on /metrics — name, type, tags, and what each one measures, grouped by area.
- [Glossary](https://stoatflow.io/raw/docs/reference/glossary.md): Concise definitions of the terms used across the StoatFlow docs — lanes, commit barriers, epochs, watermarks, event time, changelog, state stores, serdes, repartitioning, and exactly-once.

## Blog

- [StoatFlow: Kafka Streams compatible engine built to scale up — not out](https://stoatflow.io/raw/blog/introducing-stoatflow.md): The first alpha is here. Kafka Streams DSL on a single-replica runtime with virtual-thread parallelism — measurably less CPU, memory, and latency on the same hardware.
- [llms.txt for StoatFlow: docs your AI agent can fetch](https://stoatflow.io/raw/blog/llms-txt-machine-readable-docs.md): The StoatFlow documentation is now published as llms.txt, llms-full.txt, and raw markdown — the retrieval-side complement to the AI Assistant Skills pack. Nothing to install.
- [Three gates, one transaction: error handling in StoatFlow](https://stoatflow.io/raw/blog/error-handling-three-gates-one-transaction.md): A record can fail on the way in, in the middle, or on the way out. StoatFlow gives every gate the same verdicts — continue, fail, or dead-letter — and settles them all inside the exactly-once transaction: DLQ records commit on the same barrier as your output, and what cannot be handled kills the epoch, never the guarantee. The full model, from one bad record to a Kubernetes restart.
- [From first alpha to release candidate: the road to StoatFlow 1.0.0](https://stoatflow.io/raw/blog/road-to-1-0-0.md): StoatFlow 1.0.0-rc.1 is cut and the feature set is frozen — what separates the candidate from GA is proof, not features. The twelve weeks from the first alpha: 26 releases, 1,086 commits, a compatibility matrix grown from 417 to 619 tracked entries, three full-codebase review rounds — and the framework integration that kept us honest.
- [KIP-1035: Why Kafka Streams 4.3 lets state stores own their offsets](https://stoatflow.io/raw/blog/kip-1035-state-store-managed-offsets.md): Kafka Streams kept each task's changelog offsets in a separate checkpoint file that could drift from the state it described. KIP-1035, in Kafka 4.3, moves them inside the state store — atomic with the data, and the keystone for transactional state stores.
- [Compiling StoatFlow to a GraalVM native image: G1, PGO, and why JNI beats FFM under AOT](https://stoatflow.io/raw/blog/native-image-g1-pgo-jni-vs-ffm.md): Native image is viable for a throughput-first stream processor — but only with Oracle G1 + PGO. The wrong turn (we blamed FFM; it was the GC) and the reversal (FFM is faster on the JVM, slower under AOT) are the most interesting parts.
- [KIP-1271: record headers in state stores, and the cost of a value format](https://stoatflow.io/raw/blog/kip-1271-record-headers-in-state-stores.md): KIP-1271/1285 let a state store keep a record's headers next to its value — shipped in Apache Kafka 4.3.0. What the KIP is, where Kafka Streams has got to (and what is still in flight), how StoatFlow 1.0.0 compares — and why a one-varint format change becomes an engine-wide one.
- [Hot standby for StoatFlow: failover in seconds, not a cold restore](https://stoatflow.io/raw/blog/hot-standby-high-availability.md): StoatFlow recovery has always been fast restart — but restart time scales with state. Hot standby is an opt-in active/passive pair that fails over in seconds regardless of state size, with exactly-once preserved across the handoff. Redundancy, not scale-out.
- [Measuring StoatFlow failover: four scenarios, from the logs](https://stoatflow.io/raw/blog/ha-failover-testing.md): The obvious way to test HA failover — kubectl delete pod — is a graceful SIGTERM handoff, not a crash. Here are the four real failover scenarios, the millisecond timing measured from pod logs (because a 10-second Prometheus scrape cannot resolve a one-second event), and what the numbers revealed: a single JVM crash recovers in place without failing over, and stop-the-world is not the latency blip.
- [How we obfuscate StoatFlow — and what the public-API boundary taught us](https://stoatflow.io/raw/blog/obfuscation-and-the-public-api-boundary.md): A drop-in Kafka Streams DSL has to stay byte-stable while the engine beneath it stays hidden. Those two goals pull apart — and every interesting bug we hit while obfuscating StoatFlow lived on the seam between them.
- [In-place engine restart: the primitive behind multi-standby HA](https://stoatflow.io/raw/blog/in-place-restart-multi-standby.md): The failover-testing post ended on an exploratory idea — rebuild the processing engine in the same process, no JVM exit — and a constraint: stay at two standbys. Both have moved. The in-place restart shipped, and a lag-aware leader election shipped on top, so a StoatFlow HA cluster now runs one active and any number of warm standbys, scaling past two elects exactly one successor instead of crash-looping, and a graceful role swap no longer bounces the pod.
- [Internal consistency on Kafka: emitting a correct answer at every commit](https://stoatflow.io/raw/blog/internal-consistency.md): Money can only be moved, never created — so a stream that tracks balances should read total = 0 at every consistent cut. The Flink Table API gets it right 0.035% of the time; our Kafka Streams twin sends total to −1,619 … +1,792. StoatFlow holds it at exactly 0, at every one of its committed cuts. Here is why, and the measured proof.
- [StoatFlow AI Assistant Skills: correct StoatFlow, not hallucinated Kafka Streams](https://stoatflow.io/raw/blog/ai-assistant-skills.md): A free, open skills pack that makes AI coding assistants write correct StoatFlow instead of hallucinated Kafka Streams.

## Changelog

Release notes, one page per StoatFlow version.



## AI assistants

- [AI assistants guide](https://stoatflow.io/docs/getting-started/ai-assistants): Install the StoatFlow skills pack so AI coding assistants write correct StoatFlow code.
- [Skills pack (instruction-side)](https://github.com/stoatflow/skills): The public skills/rules bundle — the instruction-side complement to this index.