Engine configuration (:core)

These are the :core engine settings — the stream-processing knobs that exist whether you embed stoatflow-core directly or run on the batteries-included stoatflow-runtime. On the runtime they live under the stoatflow: block of application.yaml; in plain :core code you set the same values on StreamsConfig (or its Builder). This page groups the ones you'll actually reach for, with their defaults.

Identity and connection

Only application-id is required — it has no default, so the engine won't start without it. bootstrap-servers defaults to localhost:9092, which is fine for local development but you'll set it explicitly for any real broker.

stoatflow:
  application-id: order-processor
  bootstrap-servers: kafka-1:9092,kafka-2:9092,kafka-3:9092

Default serdes

When an operator doesn't specify a serde via Consumed, Produced, Materialized, etc., these defaults apply. Both default to ByteArray — set them to the type your topology mostly uses.

In YAML the values are fully qualified serde class names; in code you pass Serde instances:

val config = StreamsConfig.builder("order-processor", "localhost:9092")
    .defaultKeySerde(Serdes.String())
    .defaultValueSerde(Serdes.String())
    .build()

var config = StreamsConfig.builder("order-processor", "localhost:9092")
    .defaultKeySerde(Serdes.String())
    .defaultValueSerde(Serdes.String())
    .build();

stoatflow:
  default-key-serde: org.apache.kafka.common.serialization.Serdes$StringSerde
  default-value-serde: org.apache.kafka.common.serialization.Serdes$StringSerde

Topology

How the compiler shapes and checks your topology. All of these are StoatFlow-only — Kafka Streams has no equivalent.

sub-topology-split selects where sub-topology boundaries are opened. lazy (the default) opens one only where a downstream operator genuinely needs the re-keyed record's lane affinity — a grouped aggregation, any join, toTable() — plus every explicit repartition(). That is the Kafka Streams repartitionRequired model, so describe() lines up with a KS original. eager restores the pre-1.0.0 boundary at every key-changing node.

Changing it renumbers sub-topology ids, which show up in thread names, the sub_topology metric tag and the topology endpoints. See Lanes and parallelism — in particular the skew note, if you relied on a re-key to spread work across lanes.

processor-api-key-affinity selects whether a Processor API node's key affinity is presumed from the adapter's shape. off (the default) matches Kafka Streams, which never repartitions before process() / processValues() even with connected stores. presumed restores the pre-1.0.0 boundary. It is inert under sub-topology-split: eager, which splits at every key-changing node regardless.

off extends the lazy split's skew trade to Processor API nodes, and after a many-to-one re-key it gives up key affinity there — a real trade-off, spelled out in Key affinity after a re-key. StoatFlow refuses to compile the provable case (papi-key-affinity-presumed, below). It also puts the node in its upstream sub-topology, so it inherits that sub-topology's lane count rather than starting its own — re-check any per-sub-topology numberOfLanes override you set deliberately.

validation.<rule-id> turns individual topology checks off, or promotes them to build failures:

stoatflow:
  topology:
    sub-topology-split: lazy
    processor-api-key-affinity: off
    validation:
      unresolved-boundary-key-serde: error   # off | warn | error
      inherited-boundary-key-serde: off
      papi-key-affinity-presumed: error      # the default for this one

The two boundary-serde rules and the chain rule default to warn, so upgrading changes nothing about what you are told — only where. papi-key-affinity-presumed is the one that defaults to error, because the shape it names is silent data loss at runtime rather than a diagnostic. All severities are decided at topology compilation rather than at engine start, so TopologyTestDriver reaches them with no broker: at error the driver refuses to build, naming every offending site in one message rather than failing on the first; at warn the same line lands in your own test output. off silences the condition outright. An unknown rule id is rejected at startup rather than silently ignored.

Lanes (parallelism)

Lanes are the engine's virtual partitions — records with the same key always route to the same lane, preserving per-key order while running many keys in parallel. See Lanes and parallelism for the why.

stoatflow:
  lanes:
    count: 32
    queue-capacity: 500

Commit-barrier cadence

The commit barrier is what makes StoatFlow exactly-once: when a barrier flows through the topology, state flushes and the Kafka transaction commits as one atomic unit. The cadence self-tunes at runtime within the bounds you set — interval-ms is the seed, and the engine adapts between min-interval-ms and max-interval-ms. You set the envelope, not the algorithm; how the engine schedules within it is an implementation concern (see Architecture).

stoatflow:
  commit-barrier:
    interval-ms: 500
    min-interval-ms: 150
    max-interval-ms: 5000
    timeout-ms: 60000

There are several finer self-tuning knobs under commit-barrier.* (e.g. commit-duration-ema-alpha, initial-max-epoch-records). They have sensible defaults and rarely need changing — the full list is in the reference.

Processing guarantee

Controls the commit protocol. Both modes use barriers for state-flush and offset-commit synchronization; the difference is the transaction boundary.

stoatflow:
  processing-guarantee: EXACTLY_ONCE

See Exactly-once for the full semantics and the at-least-once recovery contract.

State and changelog

State stores are RocksDB-backed by default and durably backed by Kafka changelog topics, so state survives restarts. See State stores and State and thread safety.

State directory and memory

stoatflow:
  state:
    dir: /var/lib/stoatflow/state
    uncommitted-max-bytes: 268435456
  rocks-db:
    preset: DEFAULT

Changelog topics

stoatflow:
  changelog:
    enabled: true
    replication-factor: 3

Event-time and watermarks

Watermarks track event-time progress for windowing, late-record handling, and event-time timers. The engine auto-detects whether your topology needs event-time tracking and bypasses it on the hot path when it doesn't. See Event time and watermarks.

stoatflow:
  watermark:
    max-out-of-orderness-ms: 10000
    idleness-timeout-ms: 300000
  processing:
    timestamp-coordination-mode: AUTO

Putting it together

A representative production stoatflow: block:

stoatflow:
  application-id: order-processor
  bootstrap-servers: kafka-1:9092,kafka-2:9092,kafka-3:9092

  default-key-serde: org.apache.kafka.common.serialization.Serdes$StringSerde
  default-value-serde: org.apache.kafka.common.serialization.Serdes$StringSerde

  processing-guarantee: EXACTLY_ONCE

  lanes:
    count: 32
    queue-capacity: 500

  commit-barrier:
    interval-ms: 500
    min-interval-ms: 150
    max-interval-ms: 5000
    timeout-ms: 60000

  state:
    dir: /var/lib/stoatflow/state
    uncommitted-max-bytes: 268435456

  changelog:
    enabled: true
    replication-factor: 3

  watermark:
    max-out-of-orderness-ms: 10000
    idleness-timeout-ms: 300000

Next steps

• Configuration reference — every :core key with type, default, and bounds.
• Runtime config () — the runtime.* block (HTTP, metrics, endpoints, health).
• Kafka client config — passthrough consumer/producer properties and the layered merge order.
• Defaults and presets — the opinionated defaults and how to override them.
• Tuning — picking lane count, barrier cadence, and memory caps from observed behaviour.