Metrics reference

The full catalogue of Prometheus meters StoatFlow exposes on /metrics — name, type, tags, and what each one measures, grouped by area.

The :runtime module exports metrics on GET /metrics in Prometheus text format, backed by a Micrometer PrometheusMeterRegistry. This page is the meter catalogue: every StoatFlow meter the current release registers, grouped by area, with its type, tags, and a one-line description. For how to enable, scrape, and configure metrics — recording levels, naming rewrites, PromQL examples — see Metrics. For what to alert on, see Observability.

Contract. This catalogue lists the stable meter set. The live /metrics scrape is authoritative for the exact tag sets and any point-release additions — when in doubt, trust the scrape.

How to read this page

Meter — the canonical dotted metric ID. Prometheus rewrites . to _ and appends the type suffix (_total for counters; _seconds_count / _seconds_sum / _seconds_max for timers), so stoatflow.barrier.commit.latency is scraped as stoatflow_barrier_commit_latency_seconds_*.
Tags — per-meter tags beyond the common ones. Every StoatFlow meter carries application_id; the registry-level application tag plus anything under runtime.metrics.common-tags is applied to every meter on the endpoint, including JVM and Kafka-client meters.
Recording level — meters marked (debug) in the description only appear with runtime.metrics.recording-level: debug; everything else is exported at the default info level.
The commit-path latency timers publish client-side P50/P95/P99 quantiles as {quantile="…"} series on the base _seconds name — there are no _bucket histograms.

Application and engine

Meter	Type	Tags	What it measures
`stoatflow.client.state`	Gauge	—	Application state ordinal: `CREATED=0`, `STARTING=1`, `VALIDATING_STATE=2`, `RESTORING=3`, `RUNNING=4`, `DRAINING=5`, `PAUSED=6`, `STOPPING=7`, `STOPPED=8`, `ERROR=9`.
`stoatflow.client.lane.alive`	Gauge	—	Currently alive/healthy lanes.
`stoatflow.client.uptime.seconds`	Gauge	—	Seconds since the application started.
`stoatflow.client.start.time.epoch.seconds`	Gauge	—	Application start time (epoch seconds).
`stoatflow.lanes`	Gauge	`sub_topology`, `chain`	Configured key-affinity lanes per sub-topology; `sum(stoatflow_lanes)` gives the cross-topology total.
`stoatflow.engine.restart.total`	Counter	`disposition`, `target`, `trigger`	Successful in-place engine restarts (fault recovery, programmatic, HA demote, poison replay, source-topic discovery under HA, and `trigger=source_topic_deleted` when a pattern-matched source topic is deleted without HA). Topic-set triggers do not consume the restart budget.
`stoatflow.topology.stateless`	Gauge	—	1 if the topology has no state stores, 0 otherwise.
`stoatflow.topology.event.time.tracking`	Gauge	—	1 if watermark tracking is active, 0 if bypassed.
`stoatflow.topology.dispatch.mode`	Gauge	—	The dispatch mode resolved at start-up: 0 for `LANE`, 1 for `DIRECT` (`AUTO` resolves to one of the two).

Lanes and throughput

Meter	Type	Tags	What it measures
`stoatflow.lane.queue.size`	Gauge	`lane_id`, `chain`	Current lane queue depth (backpressure signal).
`stoatflow.lane.queue.capacity`	Gauge	`sub_topology`, `chain`	Configured lane queue capacity per sub-topology.
`stoatflow.lane.queue.full.total`	Counter	`lane_id`	Times a lane queue was full (backpressure events).
`stoatflow.lane.records.processed.total`	Counter	`lane_id`	Records processed by a lane.
`stoatflow.lane.bytes.processed.total`	Counter	`lane_id`	Bytes processed by a lane.
`stoatflow.lane.process.latency`	Timer	`lane_id`	Per-record processing latency.
`stoatflow.e2e.latency`	Timer	—	Per-record end-to-end latency, record timestamp to processing completion (KS `record-e2e-latency` analogue).

Lane IDs follow the {subtopology}_{lane} convention — e.g. 0_0, 1_7. Aggregate with a regex matcher: sum(stoatflow_lane_queue_size{lane_id=~"1_.*"}).

Commit barriers and transactions

Meter	Type	Tags	What it measures
`stoatflow.barrier.initiated.total`	Counter	—	Barriers initiated.
`stoatflow.barrier.completed.total`	Counter	—	Barriers successfully completed.
`stoatflow.barrier.failed.total`	Counter	`error_type`	Barrier failures, by error type: `timeout`, `bounded_await_timeout`, `commit_stall`, `commit_failed`, `producer_fenced`, `tx_error`, `state_flush_failed` (a previous epoch's state flush failed), `poison_epoch`, `poison_halt`, `abandoned_epoch_refused` (refused after an earlier epoch on the engine was aborted), `abandonment_invariant`, `dispatch_failed` (the dispatcher failed between registering a barrier and broadcasting it), `fatal_error` (an `Error` such as `TODO()` or `AssertionError` thrown inside the commit — typically user code: a serde, a custom store, a consumer interceptor; a JVM-fatal error such as `OutOfMemoryError` is not counted — it ends the engine), `shutdown` (queued barriers drained by a failure shutdown), `unknown`.
`stoatflow.barrier.abandoned.total`	Counter	`reason`	Barriers deliberately not committed, by reason: `pre_assignment` (shutdown before the first partition assignment), `poison_abort` (the epoch was aborted around a poison record), `abort_teardown` (an ABORT engine teardown — a fault-triggered in-place restart — was under way, or a graceful shutdown ran out of time for a barrier whose output it could no longer send) or `not_broadcast` (the engine stopped before the barrier left the dispatcher). Not a failure — the epoch replays on restart.
`stoatflow.epoch.invariant.violations.total`	Counter	`kind`	Epoch-isolation tripwires that fired, by kind: `rotation_over_pending` (a state store started a new epoch while the previous one was still uncommitted), `skipped_commit_with_pending` (a commit was skipped while state stores still held its epoch's writes), `lane_epoch_lag` (a lane reported a barrier after a newer one was already dispatched), `holdback_epoch_gap` (a lane held records from two different epochs ahead of it at once), `offsets_send_refused` (a second barrier's offsets were about to share an open transaction with another barrier's — the send is refused, or the commit fails and replays its epoch), `partition_offset_regression` (a partition dispatched a record at a lower offset than one it had already dispatched — both processing guarantees. A broker-side seek-back — log truncation after an unclean leader election, an offset reset — fires it too, once per seek-back, and is not a StoatFlow bug: offset tracking follows the partition back, so the re-fetched records are processed again rather than skipped by the next commit. Logged at WARN, at most once a minute per partition), `partition_buffer_over_bound` (a poll delivered records to a partition whose input buffer was already full — the pause that bounds the buffer did not take effect. The records are not buffered: they are fetched again once the buffer has room, so memory stays bounded at `buffered-records-per-partition` − 1 + `max.poll.records`. Logged at WARN once until the buffer drains; an earlier WARN about a failed consumer pause usually names the cause), and five timer kinds — `timer_claim_without_slot` (a due timer reached its lane's `onTimer` without having been handed out by the timer store; it fires anyway), `timer_claim_while_firing` (a timer's fire arrived while the same timer was already firing; the second is skipped), `timer_registry_not_empty_at_reload` (a standby promoted to active found timers in flight, which a standby never has), `timer_event_without_handler` (a timer's fire reached a lane with no timer callback; with a persistent timer backend it fires again after a restart, a `Heap` timer is lost), `timer_claim_behind_queue_epoch` (a timer's fire was about to run in an older commit than writes made before the timer became due; it is skipped and the timer fires again once its lane has caught up). Every other kind is unreachable by design, so any non-zero value there is a bug — each is also logged at ERROR; please report it. Recorded at every recording level.
`stoatflow.barrier.in.flight`	Gauge	—	Currently pending barriers.
`stoatflow.barrier.latency`	Timer	—	Barrier initiation to completion.
`stoatflow.barrier.alignment.latency`	Timer	—	Time for all lanes to receive the barrier.
`stoatflow.lane.holdback.records`	DistributionSummary	`sub_topology`	Records and timer fires held back and replayed per barrier at a sub-topology boundary — receiver-side epoch hold-back that keeps the commit a consistent cut across the cascade (ADR-135; timer fires since ADR-147's 2026-10-13 amendment).
`stoatflow.lane.holdback.drains.total`	Counter	`sub_topology`	Non-empty hold-back drains — a receiving lane replayed stashed ahead-of-epoch records or timer fires after crossing its barrier (ADR-135).
`stoatflow.barrier.commit.latency`	Timer	—	Commit latency: producer drain + Kafka TX + state flush (KS `commit-latency` analogue).
`stoatflow.barrier.records.committed.total`	Counter	—	Records committed across barriers.
`stoatflow.barrier.bytes.committed.total`	Counter	—	Bytes committed across barriers.
`stoatflow.barrier.subtopology.received.total`	Counter	`subtopology_id`	Barriers received per sub-topology. (debug)
`stoatflow.barrier.subtopology.pending.lanes`	Gauge	`subtopology_id`	Pending lane completions per sub-topology. (debug)
`stoatflow.barrier.interval.target.ms`	Gauge	—	Current target barrier-to-barrier interval (ms).
`stoatflow.barrier.commit.duration.estimate.ms`	Gauge	—	Smoothed estimate of commit duration (ms).
`stoatflow.barrier.epoch.max.records`	Gauge	—	Current dynamic epoch record limit.
`stoatflow.barrier.epoch.actual.records`	Gauge	—	Records dispatched in the last epoch.
`stoatflow.barrier.epoch.memory.cap`	Gauge	—	Memory-derived epoch record cap; `-1` = uncapped.
`stoatflow.barrier.epoch.bytes.per.record`	Gauge	—	Smoothed uncommitted state bytes per record (feeds the memory cap).
`stoatflow.s1.epoch.records.smoothed`	Gauge	—	Smoothed epoch-records average used by the epoch-size controller.
`stoatflow.s1.cap.grow.events.total`	Counter	`reason`	Epoch-cap grow events by binding signal (`record_count`, `memory_pressure`, `stall`, `soft_binding`, `time_grow`).
`stoatflow.s1.cap.decay.events.total`	Counter	—	Epoch-cap decay events.
`stoatflow.barrier.trigger.total`	Counter	`trigger`	Barrier creation count by trigger: `TIME`, `RECORD_COUNT`, `MEMORY_PRESSURE`, `CACHE_PRESSURE`.
`stoatflow.dispatcher.stall.latency`	Timer	—	Dispatcher stall time waiting for an active commit after hitting the epoch record limit.
`stoatflow.barrier.commit.changelog.latency`	Timer	—	Aggregate changelog serialization+send latency per commit epoch.
`stoatflow.barrier.commit.state.flush.latency`	Timer	—	Aggregate state-store flush latency per commit epoch.
`stoatflow.barrier.commit.producer.flush.latency`	Timer	—	Pre-commit producer flush latency.
`stoatflow.barrier.commit.send.offsets.latency`	Timer	—	`sendOffsetsToTransaction` RPC latency.
`stoatflow.barrier.commit.queue.wait.latency`	Timer	—	Barrier wait in the commit queue between alignment and commit-thread pickup.
`stoatflow.barrier.commit.post.tx.latency`	Timer	—	Post-transaction phase latency (guard wait + epoch prepare + send-buffer drain).
`stoatflow.barrier.commit.window.duration`	Timer	—	Duration of the commit-in-progress blocking window. (debug)
`stoatflow.commit.state.flush.guard.wait`	Timer	—	Time the next transaction waited on the previous epoch's async state flush.
`stoatflow.commit.state.flush.async.latency`	Timer	—	Async (off-commit-path) state flush latency.
`stoatflow.commit.state.flush.overlap`	Counter	—	Commits that overlapped a still-running previous flush.
`stoatflow.commit.stall.detected.total`	Counter	—	Commit-pipeline stalls detected by the in-process watchdog.
`stoatflow.commit.stall.duration.ms`	Gauge	—	Age of the most recently observed commit stall (ms).
`stoatflow.transaction.begin.total`	Counter	—	Kafka transactions begun.
`stoatflow.transaction.commit.total`	Counter	—	Kafka transactions committed.
`stoatflow.transaction.abort.total`	Counter	—	Kafka transactions aborted.
`stoatflow.transaction.commit.latency`	Timer	—	Kafka transaction commit latency.
`stoatflow.alo.commit.delegation.wait`	Timer	—	Commit delegation wait (at-least-once mode only).
`stoatflow.alo.commit.sync.latency`	Timer	—	`commitSync` latency (at-least-once mode only).
`stoatflow.commit.barrier.idle.throttled`	Gauge	—	`1` while the barrier scheduler is parked on the stretched warm-idle cadence, else `0` (ADR-134). Expected on an idle exactly-once application, and — since reclamation became idle-safe — on an idle at-least-once one that still has a versioned store left to sweep.

State stores and caching

Most store meters are per-store (store_name) and require recording-level: debug; the changelog counters, store.num.keys and store.segments are exported at info.

Meter	Type	Tags	What it measures
`stoatflow.store.get.total`	Counter	`store_name`	Get operations. (debug)
`stoatflow.store.put.total`	Counter	`store_name`	Put operations. (debug)
`stoatflow.store.delete.total`	Counter	`store_name`	`delete()` calls from user code only — StoatFlow's DSL tombstones through `put(key, null)`, counted under `put`. (debug)
`stoatflow.store.get.latency`	Timer	`store_name`	Point-read latency. (debug)
`stoatflow.store.put.latency`	Timer	`store_name`	Write latency. (debug)
`stoatflow.store.delete.latency`	Timer	`store_name`	Delete latency, user-code `delete()` only. A `delete()` is a get plus a tombstone put, so it also lands in the get and put timers. (debug)
`stoatflow.store.range.latency`	Timer	`store_name`	Range query latency. (debug)
`stoatflow.store.all.latency`	Timer	`store_name`	Full-iteration latency. (debug)
`stoatflow.store.flush.latency`	Timer	`store_name`	Store flush latency. (debug)
`stoatflow.store.cache.hit.total`	Counter	`store_name`	Store cache hits. (debug)
`stoatflow.store.cache.miss.total`	Counter	`store_name`	Store cache misses. (debug)
`stoatflow.store.changelog.records.written.total`	Counter	`store_name`	Changelog records written.
`stoatflow.store.changelog.bytes.written.total`	Counter	`store_name`	Changelog bytes written.
`stoatflow.store.commit.changelog.latency`	Timer	`store_name`	Per-store changelog serialization+send latency during the commit epoch. (debug)
`stoatflow.store.commit.flush.latency`	Timer	`store_name`	Per-store flush latency during the commit epoch. (debug)
`stoatflow.store.flush.entries`	DistributionSummary	`store_name`	Entry count per epoch flush. (debug)
`stoatflow.store.flush.sub.batches`	DistributionSummary	`store_name`	Sub-batch count per flush (1 = sequential, N = parallel). (debug)
`stoatflow.store.num.keys`	Gauge	`store_name`, `store_type`	Committed key count of an in-memory store (KIP-1250 parity); silent for RocksDB-backed stores.
`stoatflow.store.expired.records`	Counter	`store_name`, `store_type`	Records event-time-expired (pruned past retention) from an in-memory window or session store; `0` for plain key-value stores (ADR-133).
`stoatflow.store.segments`	Gauge	`store_name`, `store_type`	Live segment count of a segmented RocksDB window or session store — one RocksDB instance per segment, ~7 file descriptors and 10 files each, no upper bound. Writes open segments and watermark-driven expiry drops them, so the count is bounded only while the watermark advances: in steady state it is phase-dependent and bounded by `ceil(retention / segmentInterval) + 1` (wider pre-existing segments can raise it through a retention change); a stalled or lagging-but-live partition pins the watermark, nothing expires, and every write into a new time range adds a segment — the count climbs past that bound for as long as the lag lasts (`RS-03`). The value is the data segments the store holds (the offsets holder is one more RocksDB instance per store; a segment whose deletion waits behind an open iterator has already left the count). Sampled ~1 Hz from an O(1) counter. Pair it with `stoatflow.watermark.lag.ms`, which names the cause.
`stoatflow.store.versioned.sweep.lag.ms`	Gauge	`store_name`	Milliseconds since the versioned store's last completed history-reclamation pass. Versioned-store retention is a minimum bound — an expired version stops being readable immediately, but its bytes are reclaimed by a rolling budgeted sweep whose target is one `stoatflow.state.versioned.sweep-period-ms`. This is that lag, made observable: alert when it exceeds the configured period for a sustained interval.
`stoatflow.store.versioned.sweep.budget.exhausted.total`	Counter	`store_name`	Retention-sweep ticks that hit a per-barrier byte, entry or deletion cap before finishing their slice. A steady rate means the configured sweep period is not achievable at this store's size — raise `stoatflow.state.versioned.max-scan-bytes-per-barrier`, or accept a longer period.
`stoatflow.store.versioned.sweep.position`	Gauge	`store_name`	Entries the retention sweep has visited since the current pass began. Progress detail; the cursor is a byte array, so there is no cheap ordinal to report instead. (debug)
`stoatflow.store.versioned.sweep.failures.total`	Counter	—	Retention-sweep ticks that failed. The tick catches every error by design, so that a reclamation problem can never disturb the commit protocol — which also means a persistently-broken sweep is invisible outside the log while the app looks healthy. Untagged (a failure can happen before the store is even resolved; the accompanying warning names the sub-topology). Any sustained increase means reclamation is not running.
`stoatflow.store.versioned.sweep.reclaimed.total`	Counter	`store_name`	Versions the retention sweep removed from the local store. Pairs with `sweep.tombstones.total` below: in a healthy steady state the two track each other, and the local one running ahead means the store is shedding while its changelog is not.
`stoatflow.store.versioned.sweep.tombstones.total`	Counter	`store_name`	Reclamation tombstones the sweep handed to the changelog — one Kafka null per reclaimed version, which is what lets log compaction bound a topic whose composite key it can never collapse on its own. Zero for a store with logging disabled (nothing to write to); otherwise a sustained gap below `sweep.reclaimed.total` means reclamation is not reaching the changelog and the topic will grow. Counts records handed to the changelog writer: under exactly-once the send rides the epoch transaction, so a broker-side failure surfaces as a failed commit rather than as a gap here.
`stoatflow.store.format.downgrade.total`	Counter	`outcome`	State stores whose on-disk format was newer than the topology asked for — record headers (KIP-1271) turned off over a store that carries them. Reported once per affected store at startup, before it opens. `outcome` is `dropped-empty` (the headers keyspace was empty and was dropped in place — free, no restore), `wiped` (acknowledged via `stoatflow.state.format-downgrade: wipe-and-restore` in YAML, or the `state.format.downgrade` Properties key; local state deleted and rebuilt from the changelog) or `refused` (startup failed). Do not alert on `refused`: the process exits moments later, so a scrape almost never lands in the window — treat a refusal as a log/crash-loop signal, and use this counter for the `dropped-empty` and `wiped` outcomes, which run on a process that goes on to serve. See state stores.
`stoatflow.fk.enumeration.behind.scans`	Counter	`store_name`	Foreign-key join evaluations that enumerated the subscription store as-of an earlier committed epoch because the reader was behind (ADR-136).
`stoatflow.suppress.buffer.count`	Gauge	`buffer_name`	Records currently held in a suppression buffer.
`stoatflow.suppress.buffer.bytes`	Gauge	`buffer_name`	Bytes currently held in a suppression buffer.
`stoatflow.suppress.emit.total`	Counter	`buffer_name`	Records released from a suppression buffer.

Suppression buffers never drop records, so there is no drop counter here: a late record is dropped upstream by the windowed-aggregation guards and counted as stoatflow.dropped.records.total with reason=late_window or late_session. On a full buffer, EAGER early-emits the oldest entry (a forward, not a drop) while STRICT and maxBytes throw.

RocksDB

Opt-in, and off by default — set stoatflow.rocks-db.metrics.enabled: true for the property and cache gauges (info, cheap), and additionally statistics-enabled: true for the ticker, ratio and histogram meters (debug; enabling RocksDB statistics costs roughly 5–10% on the write path, and takes effect at store open, so it needs a restart). A recorder thread samples every interval-ms (default 60 s).

This section elides related meters into one row — compaction.time.{avg,min,max}.ms is three meters, not one. That is a deliberate exception to the rest of this page, which spells every name out: the family is ~43 series per store, and elision keeps it readable. All meters carry store_name except the three global shared-cache gauges.

Two caveats worth knowing before you build a dashboard on these. open.files is not published — it is derived from a RocksDB ticker (NO_FILE_CLOSES) that no longer exists in the bundled RocksJava. And *.min.ms is JNI-only: the default FFM backend reads statistics from a text dump that carries no MIN token, so those gauges stay silent unless you set stoatflow.rocks-db.backend: JNI. Averages, maxima and totals are unaffected.

Meter	Type	Tags	What it measures
`stoatflow.rocksdb.{bytes.written,bytes.read}.total`	Counter	`store_name`	Bytes written / read by RocksDB. (debug)
`stoatflow.rocksdb.memtable.bytes.flushed.total`	Counter	`store_name`	Bytes written to disk by memtable flushes. (debug)
`stoatflow.rocksdb.compaction.bytes.{read,written}.total`	Counter	`store_name`	Bytes read / written by compaction — the write-amplification signal. (debug)
`stoatflow.rocksdb.file.errors.total`	Counter	`store_name`	RocksDB file errors. (debug)
`stoatflow.rocksdb.memtable.hit.ratio`	Gauge	`store_name`	Memtable hit ratio over the last sample interval. (debug)
`stoatflow.rocksdb.block.cache.{,data.,index.,filter.}hit.ratio`	Gauge	`store_name`	Block-cache hit ratio over the last sample interval — overall, then per block role. A falling data-block ratio with a healthy index ratio usually means the working set outgrew the cache. (debug)
`stoatflow.rocksdb.compaction.time.{avg,min,max}.ms`	Gauge	`store_name`	Compaction time. (debug)
`stoatflow.rocksdb.compaction.time.total.ms`	Counter	`store_name`	Cumulative compaction time. (debug)
`stoatflow.rocksdb.flush.time.{avg,min,max}.ms`	Gauge	`store_name`	Memtable-flush time. (debug)
`stoatflow.rocksdb.flush.time.total.ms`	Counter	`store_name`	Cumulative memtable-flush time. (debug)
`stoatflow.rocksdb.write.stall.duration.avg.ms`	Gauge	`store_name`	Write-stall duration. Any sustained value is RocksDB backpressuring your commit path. (debug)
`stoatflow.rocksdb.write.stall.duration.total.ms`	Counter	`store_name`	Cumulative time writes were stalled. (debug)
`stoatflow.rocksdb.num.immutable.mem.table`	Gauge	`store_name`	Immutable memtables not yet flushed.
`stoatflow.rocksdb.{cur.size.active.mem.table,cur.size.all.mem.tables,size.all.mem.tables}.bytes`	Gauge	`store_name`	Memtable sizes: active, active plus unflushed, and including pinned.
`stoatflow.rocksdb.num.entries.{active.mem.table,imm.mem.tables}`	Gauge	`store_name`	Entries in the active / unflushed immutable memtables.
`stoatflow.rocksdb.num.deletes.{active.mem.table,imm.mem.tables}`	Gauge	`store_name`	Delete entries in the active / unflushed immutable memtables.
`stoatflow.rocksdb.{mem.table.flush.pending,compaction.pending}`	Gauge	`store_name`	1 while a flush / compaction is pending, else 0.
`stoatflow.rocksdb.num.running.{flushes,compactions}`	Gauge	`store_name`	Currently running flushes / compactions.
`stoatflow.rocksdb.estimate.pending.compaction.bytes`	Gauge	`store_name`	Bytes compaction must rewrite to bring the LSM tree back to target size. A climbing value means compaction is losing ground.
`stoatflow.rocksdb.{total.sst.files.size,live.sst.files.size}.bytes`	Gauge	`store_name`	SST bytes on disk: all files, and those in the latest version.
`stoatflow.rocksdb.num.live.versions`	Gauge	`store_name`	Live versions — more than one means an open iterator is pinning old files.
`stoatflow.rocksdb.estimate.num.keys`	Gauge	`store_name`	Estimated live keys. This is the RocksDB counterpart of `store.num.keys`, which covers in-memory stores only.
`stoatflow.rocksdb.estimate.table.readers.mem.bytes`	Gauge	`store_name`	Memory held by table readers (indexes and filters), outside the block cache.
`stoatflow.rocksdb.background.errors`	Gauge	`store_name`	Accumulated background errors. Alert on any non-zero value.
`stoatflow.rocksdb.block.cache.{capacity,usage,pinned.usage}.bytes`	Gauge	`store_name`	The shared block cache, repeated per store so Kafka Streams dashboards port unchanged.
`stoatflow.rocksdb.shared.block.cache.{capacity,usage,pinned.usage}.bytes`	Gauge	—	The shared block cache, reported once. StoatFlow runs one LRU cache across all stores, so these are the real numbers — prefer them over the per-store duplicates above.

State restoration

Meter	Type	Tags	What it measures
`stoatflow.restoration.in.progress`	Gauge	—	1 while restoration is running, 0 otherwise.
`stoatflow.restoration.stores.total`	Gauge	—	Total stores to restore.
`stoatflow.restoration.stores.completed`	Gauge	—	Stores that have finished restoring.
`stoatflow.restoration.records.restored.total`	Counter	`store_name`	Records restored, per store.
`stoatflow.restoration.bytes.restored.total`	Counter	`store_name`	Changelog bytes restored, per store — serialized key plus value as read off the changelog, excluding record and batch overhead. Compare against changelog volume to size a cold start.
`stoatflow.restoration.remaining.records`	Gauge	`store_name`	Changelog records the store still has to restore, across all its changelog partitions (queued ones included); counts down during restoration and reads 0 once the store is restored. An estimate — end offset minus start offset minus records restored, so a transactional changelog's commit markers keep it slightly above the true count until the store completes. The Kafka Streams–compatible metrics mode sums it into `restore-remaining-records-total`.
`stoatflow.restoration.duration.seconds`	Timer	`store_name`	Per-store restoration duration, recorded once when the store finishes. A store's changelog partitions restore in parallel, so this is the slowest partition, not their sum.

Consumer, producer, and buffers

Meter	Type	Tags	What it measures
`stoatflow.consumer.poll.latency`	Timer	—	Consumer poll latency.
`stoatflow.consumer.poll.total`	Counter	—	Poll invocations.
`stoatflow.consumer.poll.records`	DistributionSummary	—	Records per poll.
`stoatflow.consumer.records.consumed.total`	Counter	`topic`, `partition`	Records consumed.
`stoatflow.consumer.bytes.consumed.total`	Counter	`topic`, `partition`	Bytes consumed.
`stoatflow.consumer.lag.records`	Gauge	`topic`, `partition`	Consumer lag, in records.
`stoatflow.consumer.assigned.partitions`	Gauge	—	Number of assigned partitions.
`stoatflow.source.pattern.matched.topics`	Gauge	`source`	Topics currently matched by a pattern-subscribed source node; updated on every (re)assignment. `0` means the pattern matches nothing yet.
`stoatflow.consumer.partition.forgotten.total`	Counter	`reason`	Source partitions dropped from offset tracking because their topic no longer exists. `reason=topic_deleted` is a pattern-matched topic deleted on the default `subscribe()` path, answered by an in-place engine restart; `reason=ha_source_shrink` is the HA `assign()`-mode active's periodic detector confirming a shrink and forgetting the offsets in place. Counts partitions actually dropped, not drops requested — the HA detector re-confirms the same shrink every two ticks, and a partition that was assigned but never produced a record has nothing to drop — so a `rate()` over this meter reflects real shrink activity.
`stoatflow.consumer.partitions.paused`	Gauge	—	Currently paused partitions.
`stoatflow.consumer.partitions.delay.pending`	Gauge	—	Partitions with a pending resume delay.
`stoatflow.consumer.partition.paused.total`	Counter	`topic`, `partition`, `reason`	A partition paused because its input buffer filled (`reason=capacity`). Counted even when the partition is already paused for another reason, so `paused − resumed` is the number of partitions currently paused for capacity. The application pause (`POST /pause`) and state restoration at start-up pause partitions too but are not counted here — the application state reports them. Watermark alignment never pauses a partition (it holds its records at dispatch instead — see the `watermark.alignment` meters).
`stoatflow.consumer.partition.resumed.total`	Counter	`topic`, `partition`, `reason`	A partition released from its capacity pause, including when it is revoked — the counterpart of `paused.total`.
`stoatflow.producer.records.produced.total`	Counter	`topic`	Records produced.
`stoatflow.producer.bytes.produced.total`	Counter	`topic`	Bytes produced.
`stoatflow.producer.send.latency`	Timer	`topic`	Producer send latency. (debug)
`stoatflow.buffer.partition.size`	Gauge	`topic`, `partition`	Current partition buffer depth. (debug)
`stoatflow.buffer.partition.capacity`	Gauge	`topic`, `partition`	Partition buffer capacity. (debug)
`stoatflow.buffer.offer.total`	Gauge	`topic`, `partition`	Cumulative records offered to the partition buffer. (debug)
`stoatflow.buffer.poll.total`	Gauge	`topic`, `partition`	Cumulative records polled from the partition buffer. (debug)
`stoatflow.buffer.oldest.age.ms`	Gauge	`topic`, `partition`	Age (by its timestamp) of the next record to leave the partition buffer — the lowest buffered offset; 0 when the buffer is empty or that record is stamped in the future. Sampled about twice a second; if sampling stops while records wait (a stalled dispatcher), it reports at least the time since the last sample rather than 0. (debug)
`stoatflow.buffer.scheduled.source.size`	Gauge	—	Scheduled-source dispatch buffer depth.
`stoatflow.buffer.timer.event.size`	Gauge	—	Event-time timer buffer depth.
`stoatflow.buffer.timer.processing.size`	Gauge	—	Processing-time timer buffer depth.

Dispatcher internals

Meter	Type	Tags	What it measures
`stoatflow.dispatcher.batch.size`	DistributionSummary	—	Records per dispatch batch. (debug)
`stoatflow.dispatcher.batch.held.total`	Counter	—	Batches held because a target lane lacked capacity. (debug)
`stoatflow.dispatcher.dispatch.latency`	Timer	—	Per-batch dispatch latency.
`stoatflow.parallel.deser.activated.total`	Counter	—	Times parallel deserialization was activated.
`stoatflow.queue.offer.failure.total`	Counter	`queue_type`	Internal queue offer rejections (`lane`, `partition_buffer`, `commit`).
`stoatflow.coordinator.select.latency`	Timer	—	Time spent selecting the next timestamp-ordered batch. (debug)
`stoatflow.coordinator.partition.switches`	DistributionSummary	—	Partition switches per selected batch. (debug)

Watermarks and event time

Meter	Type	Tags	What it measures
`stoatflow.watermark.current`	Gauge	—	Current global watermark (epoch ms).
`stoatflow.watermark.lag.ms`	Gauge	—	Wall-clock time minus the global watermark (event-time lag). Sampled ~1 Hz while event-time tracking is on — independently of whether the watermark advanced, so during a stall it climbs rather than freezing at its last healthy value (`RS-23`). `-1` until the first watermark has been emitted.
`stoatflow.watermark.advance.total`	Counter	—	Watermark advancement events.
`stoatflow.watermark.preview.partitions`	Gauge	—	Partitions whose next record waits behind older records of other partitions — a record stamped in the future, or a partition far ahead during a skewed catch-up or replay — and whose watermark meanwhile counts as if that record had been processed, so event time follows the other partitions instead of standing still. Sampled ~1 Hz while event-time tracking is on; idle partitions are not counted, nor one whose own watermark is already as high. See its `previewWatermarkMs` / `headHeldSinceMs` on `/watermarks`. Can also read non-zero while the application is paused or fully backpressured.
`stoatflow.watermark.partition.current`	Gauge	`topic`, `partition`	Per-partition watermark. (debug)
`stoatflow.watermark.late.records.total`	Counter	`topic`, `partition`	Records arriving after the watermark. (debug)
`stoatflow.watermark.out.of.orderness.max`	Gauge	—	Maximum observed out-of-orderness (ms). (debug)
`stoatflow.watermark.alignment.drift.ms`	Gauge	`topic`, `partition`, `group`	Per-partition drift above the alignment group's minimum watermark. (debug)
`stoatflow.watermark.alignment.group.min`	Gauge	`group`	Minimum watermark across the alignment group (epoch ms). (debug)
`stoatflow.watermark.alignment.partitions.held`	Gauge	`group`	Partitions currently held for watermark alignment: their watermark is ahead of a member of their group by more than the drift, so their records are dispatched only in timestamp order. A held partition keeps being fetched.
`stoatflow.watermark.alignment.held.time.total`	Counter	`group`	Seconds during which a hold in the group stopped all dispatch: the held partition's next record was the oldest buffered one, and a lagging member of its group had nothing buffered. Only one hold binds at a time, so `rate()` is the fraction of time the application waited on the group. Accrues while the hold lasts, not only when it ends. Counts no time spent waiting for any other reason (a readiness wait, backpressure, a pause).
`stoatflow.watermark.alignment.yields.total`	Counter	`group`	Times a held partition's records were dispatched in order although its group's laggards had not caught up — every laggard had data, so waiting could not release the hold. At most once per partition per alignment `updateInterval`. A flat `held.time.total` with a rising `yields.total` usually means the group's sources have different out-of-orderness bounds.

Timers, punctuators, and scheduled sources

Meter	Type	Tags	What it measures
`stoatflow.timer.registered.total`	Counter	`type`	Keyed timers registered (`event_time`, `processing_time`).
`stoatflow.timer.fired.total`	Counter	`type`	Timers fired.
`stoatflow.timer.deleted.total`	Counter	`type`	Timers explicitly deleted.
`stoatflow.timer.lane.mismatch.total`	Counter	`processor`	Record keys whose re-serialization with the processor's timer key serde picks another lane than the record's — the key's timers would run on another lane. Pre-registered at 0 for every timer-declaring Processor API processor. A split for a timer on the record's own key stops the app with `TimerLaneAffinityException` by default (`timer.lane-affinity.guard`); a sampled split is WARN-only by default (`timer.lane-affinity.sampled-check`). Recorded at every level. Alert on any increase and on application state — a fatal detection can stop the app before the next scrape.
`stoatflow.timer.queue.size`	Gauge	`type`	Timer dispatch-buffer depth, by timer type.
`stoatflow.timer.fire.latency`	Timer	`type`	Delay between scheduled and actual fire time. (debug)
`stoatflow.timer.fire.late.total`	Counter	`type`	Timers that fired late. (debug)
`stoatflow.punctuator.invocation.total`	Counter	`punctuator_id`, `type`, `mode`	Punctuator invocations.
`stoatflow.punctuator.skip.total`	Counter	`punctuator_id`	Punctuator invocations skipped (previous run still active).
`stoatflow.punctuator.active`	Gauge	—	Currently registered punctuators.
`stoatflow.punctuator.execution.latency`	Timer	`punctuator_id`, `type`	Punctuator execution time. (debug)
`stoatflow.scheduled.source.trigger.total`	Counter	`source_name`, `type`	Scheduled source triggers.
`stoatflow.scheduled.source.records.emitted.total`	Counter	`source_name`	Records emitted by scheduled sources.

Error handling

Meter	Type	Tags	What it measures
`stoatflow.error.total`	Counter	`error_type`, `topic`	Errors by type (`topic` omitted when unknown).
`stoatflow.error.typed.total`	Counter	`category`, `error_type`	Categorised errors (e.g. production failures), by category and type.
`stoatflow.dropped.records.total`	Counter	`reason`	Records dropped before processing (KS `dropped-records` parity). The `reason` tag is the semantics — seven values: `null_key` (a record whose key is null, or serializes to null, reaching an aggregation), `late_window` / `late_session` (a record arriving after its window or session already closed), `join_null_key` / `join_null_value` (a join input the join cannot match on), `table_source_null_key` (a null-keyed record on a `KTable` source topic), `null_window_value` (a time or sliding window whose stored value your value serde reads back as `null`, so its final `OnWindowClose` result is not emitted — counted once per window, at close; also logged at WARN once per processor, and again after a restart; a closed session in that state is forwarded as `null`, as in Kafka Streams).
`stoatflow.sink.records.dropped.total`	Counter	`sink_name`, `topic`	Records dropped because a custom partitioner returned no partition (KIP-837).
`stoatflow.sink.records.multicast.total`	Counter	`sink_name`, `topic`	Output records produced via multicast — a custom partitioner returned multiple partitions (KIP-837).
`stoatflow.dlq.poison.quarantined.total`	Counter	—	Source records quarantined as poison after an asynchronous broker-side production failure returned `continue`. The epoch's source offsets are held and the epoch is replayed with exactly these records skipped, so the epoch's other records survive — but a quarantined record loses all of its outputs, across every sink it fans out to. See Error-handling model.
`stoatflow.dlq.poison.replays.total`	Counter	—	Poison-epoch replays: in-place engine restarts that reprocess an aborted epoch with the quarantined records skipped. Bounded by `stoatflow.commit-barrier.max-poison-replays` within `poison-replay-window-ms`. That budget is in-memory — exhausting it ends the process and the pod restarts with a clean budget — so `continue` is bounded per process, never end to end. Alert on this meter.
`stoatflow.dlq.poison.replay.budget.exhausted.total`	Counter	—	The replay budget ran out: `max-poison-replays` replays were already consumed within `poison-replay-window-ms`, so the instance shut down with the source offsets held rather than replaying again. Terminal — any non-zero value is an incident. Alert on it alongside `…replays.total`, which counts replays and therefore stops incrementing at the moment a non-converging poison starts crash-looping the pod.
`stoatflow.dlq.poison.quarantine.size`	Gauge	—	Source offsets currently quarantined and skipped by a replay. `0` in steady state; entries retire as soon as a commit moves permanently past them.

High availability

Registered only when hot standby is enabled (ha.mode != off).

Meter	Type	Tags	What it measures
`stoatflow.ha.role`	Gauge	—	`1` when this pod is the active, `0` when standby.
`stoatflow.ha.standby.replication.lag.records`	Gauge	—	Total committed-changelog lag in records across all changelog partitions; `0` when active or caught up.
`stoatflow.ha.standby.replication.applied.records`	Counter	—	Cumulative changelog records the standby applier has applied; `rate(...)` = apply throughput.
`stoatflow.ha.standby.replication.lag.ms`	Gauge	—	Standby replication lag in milliseconds.
`stoatflow.ha.peer.freshness.ms`	Gauge	—	Time since the freshest observed peer's last heartbeat.
`stoatflow.ha.peers`	Gauge	—	Number of observed HA peers.
`stoatflow.ha.restart.required`	Gauge	—	`1` when this pod needs a restart (e.g. after a source-topic partition-count change).
`stoatflow.ha.restarting`	Gauge	—	`1` while this pod is restarting its engine in place to absorb a processing fault. The pod stays `ACTIVE` throughout, so `stoatflow.ha.role` does not move — this is what distinguishes "absorbing a fault" from "healthy".
`stoatflow.ha.redundancy.ready.standbys`	Gauge	—	Caught-up (`READY_STANDBY`) fresh standby count, including self.
`stoatflow.ha.redundancy.below.desired`	Gauge	—	`1` when the active has fewer ready standby spares than `ha.desired-standbys`.
`stoatflow.ha.token.epoch`	Gauge	—	Current promotion-token epoch on the metadata topic.
`stoatflow.ha.election.outcome`	Counter	`outcome`	Cumulative election outcomes: `won`, `reclaimed`, `stood_down_fresh`, `stood_down_lost`.
`stoatflow.ha.election.latency.ms`	Gauge	—	Last claim-to-ACTIVE promotion latency (ms); `-1` until the first promotion.

License

Registered once at startup; exported at every recording level. See License configuration.

Meter	Type	Tags	What it measures
`stoatflow.license.valid`	Gauge	`tier`	`1` under a VALID or GRACE_PERIOD license, else `0`.
`stoatflow.license.expiry_timestamp_seconds`	Gauge	—	Unix epoch seconds of the license entitlement deadline.
`stoatflow.license.days_remaining`	Gauge	—	Whole days until expiry; negative within the post-expiry grace window.
`stoatflow.license.grace_remaining_seconds`	Gauge	—	Seconds left in the offline-grace budget; floored at 0.
`stoatflow.license.heartbeat_consecutive_failures`	Gauge	—	Heartbeat ticks failed since the last success.
`stoatflow.license.heartbeat_last_success_timestamp_seconds`	Gauge	—	Unix epoch seconds of the last successful heartbeat.
`stoatflow.license.validations_total`	Counter	`result`	Cold-start validation outcomes, by result.
`stoatflow.license.heartbeats_total`	Counter	`result`	Heartbeat tick outcomes, by result.

JVM, system, and Kafka-client meters

Beyond the stoatflow.* meters above, the same endpoint carries:

JVM and system meters (jvm_*, system_*, process_*) — bound by default (runtime.metrics.bind-jvm-metrics: true): heap/non-heap memory, GC pauses, thread counts, classloading, CPU usage.
Native Kafka client meters (kafka_consumer_*, kafka_producer_*) — the Kafka clients' own JMX metrics surfaced through Micrometer.

These carry the registry common tags (application plus your common-tags) but not application_id. See Metrics for details.

Next steps

Metrics — enabling, scraping, recording levels, naming and tag conventions, PromQL examples.
Observability — wiring /metrics into a monitoring stack and what to alert on.
Configuration reference — the runtime.metrics.* keys.

Kafka Streams compatibility matrix

Operator-by-operator parity between the StoatFlow DSL and the Kafka Streams DSL — implemented, partial, not yet implemented, deprecated, incompatible, and StoatFlow extensions.

Glossary

Concise definitions of the terms used across the StoatFlow docs — lanes, commit barriers, epochs, watermarks, event time, changelog, state stores, serdes, repartitioning, and exactly-once.