Tuning under load

StoatFlow ships with defaults that work for most topologies, and the parts that need to adapt to your workload — commit cadence and epoch size — self-tune within the bounds you set. This page covers the knobs you do set: lane count, the commit-cadence bounds, the memory and uncommitted-state limits, and RocksDB sizing. Each section starts from a symptom so you know when to reach for it.

For the why behind these mechanics, see Lanes and parallelism, Exactly-once, and State and thread safety. For where each value comes from and how it's merged, see the Configuration model and Defaults and presets.

Before you tune

Tune one thing at a time, and tune against a representative load, not a synthetic burst. The defaults are deliberately conservative; most workloads need at most one or two of the knobs below. Always confirm the symptom first — reach for /metrics and the /debug/barriers endpoint (see REST API) rather than guessing.

Lane count vs cores

Knob: stoatflow.lanes.count — default max(2, CPU cores).

Lanes are the unit of per-key parallelism: every record is routed to a lane by its key, and each lane is a virtual thread that processes its keys in order. More lanes means more keys processed concurrently, at the cost of more queues and a little more dispatch overhead.

stoatflow:
  lanes:
    count: 16

How to size it:

• CPU-bound processing (pure transforms, local aggregation): start at the number of CPU cores. Going much higher just adds scheduling overhead without more throughput, because the work is already saturating the cores.
• I/O-bound processing (enrichment calls, blocking lookups, external APIs): go well above core count — virtual threads park cheaply while blocked, so 4×–8× cores keeps cores busy while many lanes wait on I/O. This is the case where raising the lane count pays off most.
• Few distinct keys: lanes only help if keys spread across them. If your traffic is dominated by a handful of hot keys, extra lanes sit idle — the hot keys still serialize onto their assigned lanes. Fix the key distribution first.

Lane queue capacity

Knob: stoatflow.lanes.queue-capacity — default 300.

Each lane has a bounded queue; when a lane fills, the dispatcher applies backpressure. Raise it if you see bursty input stalling dispatch and you have memory headroom; the default is sufficient for steady-state load.

stoatflow:
  lanes:
    queue-capacity: 500

Commit cadence

StoatFlow commits in epochs — the work between two commit barriers becomes one Kafka transaction plus one state-store flush (see Exactly-once). The cadence — how often a barrier fires — is self-tuning: the engine adapts it within the bounds you configure, so you set the envelope, not the exact interval. The algorithm itself stays in the source; what you control is the floor and ceiling.

The bounds you set:

Defaults and StreamsConfig field names for every commit-barrier.* knob are in Engine configuration → Commit-barrier cadence.

stoatflow:
  commit-barrier:
    min-interval-ms: 150
    max-interval-ms: 5000
    interval-factor: 1.5

The trade-off is the classic latency-vs-throughput one:

• Tighter, more frequent commits (lower max-interval-ms, lower interval-factor) reduce end-to-end latency and bound how much work a crash replays — at the cost of more transaction and flush overhead per record.
• Looser, less frequent commits (higher bounds) amortize commit overhead over more records, raising throughput — at the cost of higher latency and a larger replay window after a restart.

Epoch size

You normally don't touch epoch size — it self-tunes within the bounds above. Two knobs exist for the edges:

stoatflow:
  commit-barrier:
    initial-max-epoch-records: 1024   # conservative startup for heavy-state topologies
    # max-epoch-records: 25000        # optional deterministic ceiling

Memory and uncommitted state

Between commits, a stateful topology accumulates uncommitted state in memory — the writes buffered since the last barrier, held until the transaction commits (KIP-892 transactional state stores). On topologies where one input record fans out into many state writes (multi-way joins, foreign-key joins, high-cardinality aggregations), this is the value most likely to threaten container memory.

Knob: stoatflow.state.uncommitted-max-bytes — default 268435456 (256 MiB).

When buffered uncommitted bytes approach this limit, the engine triggers an early commit barrier to flush state and bound peak memory. This is the backstop that keeps a high-fan-out epoch from running the container out of memory. With the default persistent timer backend, timer registrations and deletions count toward it as well, so a topology that registers many timers per record can trigger these barriers even when it has no other state store.

stoatflow:
  state:
    uncommitted-max-bytes: 268435456   # 256 MiB (default)

The global ceiling has two per-store companions under stoatflow.caching: max-estimated-bytes (default 256 MiB per caching store) and max-entries (default unbounded). Exceeding a per-store cap fires the same early commit barrier — tagged CACHE_PRESSURE on the barrier-trigger metric — so one hot store can't monopolise the buffer. In practice uncommitted-max-bytes measures a superset and fires first; leave max-entries unbounded unless a specific store must bound its per-epoch key cardinality.

Sizing it against the container:

Changing a store's retention

Symptom: you want to keep windowed or session state for longer (or shorter) than you originally configured, on a store that already has data in it.

Change retention and restart. On the segmented backend — the default for every windowed aggregation and every session store — StoatFlow re-reads the existing state as-is: each segment records its own time range on disk, so a changed retention (which also changes the segment interval) remaps rather than being reinterpreted. Nothing is restored, nothing is re-read from the changelog, and no data becomes invisible. Kafka Streams requires wiping local state for the same change.

That covers the stream-stream join buffers too: they are segmented window stores that retain duplicates, so a changed grace remaps the same way. Changing before or after is different: the buffer's window size is max(before, after), so that is a window-size change and takes the full restore below. Only hand-built TransactionalRocksDBWindowStore and TransactionalRocksDBSessionStore instances — the non-segmented backend, which no Stores factory and no DSL operator builds any more — keep the interval as an 8-byte prefix on every stored key, where a retention change that moves it cannot remap and is handled as a restore (below) instead.

Three consequences worth knowing before you do it:

• Retention is a minimum, not an exact bound. Whole segments are dropped, so the segment holding watermark − retention is kept in full. Actual retention lands anywhere in [retention, retention + segmentInterval), and exactly 1× on an in-memory store. Size disk against the upper bound.
• Segments are sized by one rule, not configurable:

  Backend	Rule	Worst-case retained
  Segmented window stores (every persistent RocksDB DSL or Stores-built window store, the default join buffers included)	max(retention / 3, 60s, windowSize)	up to 2× when windowSize governs; otherwise up to retention + 60s while the 60s floor governs, or 4/3× once retention / 3 does
  Segmented session stores (every persistent RocksDB DSL or Stores-built session store — no window-size term)	max(retention / 3, 60s)	up to retention + 60s while the floor governs, else up to 4/3×
  StoreType.IN_MEMORY window and session stores	not segmented	exactly retention
  Non-segmented (hand-built TransactionalRocksDBWindowStore / TransactionalRocksDBSessionStore only)	max(retention / 3, 60s)	up to retention + 60s on the floor, else up to 4/3×

  
  The windowSize term binds whenever windowSize > retention / 3 — including the common default, since TimeWindows.ofSizeWithNoGrace(d) gives retention == windowSize. There the interval equals the retention and the store holds two live segments, so plan disk for twice the window's data. Kafka Streams has no such term (its rule is max(retention / 2, 60s)), so StoatFlow retains less than KS on session stores (equal while both sit on the 60s floor, less once KS's /2 term exceeds it), the same on a symmetric no-grace join buffer (window size max(before, after) against a retention of before + after + grace lands exactly on KS's retention / 2), and more on a default windowed aggregation.
• Watch the segment count, not the disk. stoatflow.store.segments{store_name} is the live number of RocksDB instances a segmented store holds — about seven file descriptors each, with no upper bound. It plateaus at retention / segmentInterval + 1 while event time advances and climbs for as long as the watermark is pinned by a lagging-but-live partition or, with watermark.idleness-timeout-ms: 0, an idle one. Alert on stoatflow_watermark_lag_ms (sampled every second, so it climbs during the stall it reports) and read the segment gauge to see which stores are paying for it.
  Below about three minutes of retention the 60-second floor governs provided windowSize ≤ 60s — a window larger than the floor still wins on a segmented store. It is the same floor Kafka Streams applies (though KS reaches its own floor below 120 seconds, since its rule is retention / 2), and it is deliberately not clamped down to the retention, so a 10-second store still gets a 60-second interval.
  Being on the floor says nothing about how many segments are live. The number of live segments is set by where the watermark sits: retention / segmentInterval whole intervals plus the one holding the boundary when it lands on a segment edge, and ceil(retention / segmentInterval) + 1 in the worst phase — the same number when the retention divides evenly, one higher when it does not, since segments are fixed-width and epoch-aligned. Size against the maximum. A single segment is never guaranteed — 30 seconds of retention on a 60-second interval still spans two whenever the watermark is off a boundary. At retention = 2min, windowSize = 90s the interval is 90 seconds, the store holds two segments, and effective retention is 3min 30s. And because the floor is ours as much as KS's, short retentions are a large multiple rather than a small one — a 30-second persistent session store keeps up to 30s + 60s, i.e. 3×, not 4/3×; the in-memory one still keeps exactly 30 seconds.
• EmitStrategy.onWindowClose() adds a third term. Under the default onWindowUpdate(), expiry runs on the committed watermark. Under onWindowClose() it is gated on the durable emit frontier instead, so a window is never expired before its final result has been emitted:

  Store family	Emit strategy	Expiry driven by	Effective retention
  Time / sliding	onWindowUpdate() (default)	committed watermark	≤ retention + segmentInterval
  Session	onWindowUpdate() (default)	observed stream time	≤ retention + segmentInterval
  Either	onWindowClose()	durable emit frontier	≤ retention + segmentInterval + (watermark − emit frontier)

  
  The extra term is normally small and is zero-slack by design — the gate exists so expiry can never overtake an un-emitted close — but it is not bounded by the retention contract. Size for it if you run onWindowClose() over a large keyspace. The gate is in the processor rather than the store, so it delays in-memory expiry too: the exactly-1× property above is an onWindowUpdate() statement.
• Narrowing retention frees disk gradually. A segment keeps the span it was written with, so after cutting retention the old, wider segments expire only once their whole span has cleared the new boundary. Until then effective retention is bounded by retention + the widest existing span, not by retention + the new interval. Cutting 30 days to 1 hour therefore keeps answering reads from ~10-day segments for up to 10 days. The store logs one INFO line at startup whenever that is the case. If retention is a deletion deadline for you, plan for the transition rather than the steady state.

Changes that do need a restore

These changes alter a layout that cannot be remapped. StoatFlow detects each at startup, logs a WARN naming the field and both values, and rebuilds the store from its changelog:

The restore is the ordinary changelog replay, so it costs restore time proportional to the store's changelog, and the application starts once it completes. Plan these alongside a deploy that can absorb it, not as a hot config edit.

When there is no changelog to restore from

A store built with withLoggingDisabled(), or any store under changelog.enabled = false, has no recovery source: its local state is the only copy. StoatFlow will not delete it for you. Startup fails with a message naming every affected store, the field that moved and both values, and you choose:

• put the previous setting back and restart — instant, no data movement;
• enable the store's changelog so it has something to rebuild from;
• accept the loss with stoatflow.state.layout-mismatch: wipe-and-restore, which starts those stores empty. Remove the setting afterwards: it does not expire, and left in place it silently converts a later accidental change into a wipe.

Segmented-store checkpoint cadence

Time-windowed and session stores are segmented, and their RocksDB segments run with the write-ahead log off — so they only become durable on a clean close(). A periodic checkpoint (flush dirty segments, then the offsets) makes them durable between commits and caps how much changelog a pod replays after an ungraceful restart, and how much a hot-standby standby drains on promotion — to at most one checkpoint interval.

Knobs: stoatflow.state.segment-checkpoint-records (default 100000) and stoatflow.state.segment-checkpoint-ms (default 30000), whichever fires first.

The defaults are deliberately coarse: a checkpoint flushes SST files, so a tight cadence trades a smaller recovery window against more SST/compaction churn in steady state. Lower them only if your recovery-time objective needs a shorter ungraceful-restart catch-up than ~30 s / 100k records of windowed writes; raise them if you see checkpoint-driven write amplification and can absorb a longer catch-up.

RocksDB for large state

Persistent state lives in RocksDB. Unlike Kafka Streams' unbounded default, StoatFlow caps RocksDB memory by default to keep the process within its container. You pick a preset; the framework wires the block cache and memtable budget for you.

Knob: stoatflow.rocks-db.preset — default DEFAULT.

stoatflow:
  rocks-db:
    preset: HIGH_PERFORMANCE

When state grows large (millions of keys, hot read paths), DEFAULT's 256 MiB block cache becomes a bottleneck: working-set reads start missing the cache and hit disk. HIGH_PERFORMANCE widens both the block cache and the memtable budget. Budget for it — it is real resident memory on top of the JVM heap and uncommitted-max-bytes, so size the container accordingly.

Confirm the diagnosis before you resize. Enable the RocksDB metrics (rocks-db.metrics.enabled + statistics-enabled) and watch stoatflow_rocksdb_block_cache_hit_ratio under load: a ratio falling toward the floor while stoatflow_rocksdb_shared_block_cache_usage_bytes sits at capacity is the signal that the cache is too small — the metric that turns the symptom below from a guess into a measurement.

Backend selection

Knob: stoatflow.rocks-db.backend — default AUTO.

AUTO picks each runtime's faster path: the FFM (Foreign Function & Memory API) backend on the JVM — lower per-call overhead than JNI and no virtual-thread pinning during native RocksDB calls — and JNI under a native image, where FFM's critical-downcall advantage doesn't apply. Leave it on AUTO unless you have a specific reason to pin a backend; if you force JNI on the JVM, also set stoatflow.lanes.thread-type: PLATFORM to avoid pinning carrier threads.

Custom RocksDB budget (programmatic)

The YAML preset is the supported path for most deployments. If you need a memory budget between the presets, set RocksDBConfig directly when building the engine config — RocksDBConfig.withTotalMemory(...) splits the budget evenly between block cache and memtables:

import io.stoatflow.core.config.StreamsConfig
import io.stoatflow.core.state.rocksdb.RocksDBConfig

val config = StreamsConfig.builder("my-app", "localhost:9092")
    .rocksDbConfig(RocksDBConfig.withTotalMemory(512L * 1024 * 1024)) // 512 MiB total
    .build()

import io.stoatflow.core.config.StreamsConfig;
import io.stoatflow.core.state.rocksdb.RocksDBConfig;

var config = StreamsConfig.builder("my-app", "localhost:9092")
    .rocksDbConfig(RocksDBConfig.withTotalMemory(512L * 1024 * 1024L)) // 512 MiB total
    .build();

Watching the effect

Tune against signals, not hunches. After each change, watch:

• /metrics — throughput, consumer lag, commit timing, and JVM/container memory, scraped by Prometheus. See Metrics and Observability.
• /debug/barriers — the live commit-pipeline state, which tells you whether barriers are firing on schedule and what's triggering them. See REST API.
• /health/ready — readiness flips DOWN if the engine can't keep up or a commit stalls. See Probes.

A clean tuning loop: change one knob → run representative load → read the metric that maps to the symptom → keep or revert. Resist changing several knobs at once; you won't know which one moved the needle.

Next steps

• Defaults and presets — where every default comes from and how layers merge.
• Configuration reference — every key, type, range, default, and environment override.
• Production checklist — the pre-flight list before you take a topology to production.
• Observability — wiring metrics and dashboards so the symptoms above show up before users notice.