Metrics

The :runtime module exports metrics in Prometheus text format on GET /metrics. Internally it's a Micrometer PrometheusMeterRegistry — the engine reports events to a Micrometer instrumentation callback, the runtime binds JVM and Kafka-client metrics to the same registry, and the handler serves the registry's scrape output. Point your existing Prometheus stack at the endpoint and the StoatFlow meters appear alongside everything else.

Enable and scrape

Metrics are enabled by default and live under runtime.metrics:

runtime:
  http:
    enabled: true
    port: 8080            # default
  metrics:
    enabled: true         # default
    prefix: stoatflow     # default — prefix on all metric names
    recording-level: info # info | debug | trace (default: info)
    bind-jvm-metrics: true # default — JVM memory/GC/thread binders
    common-tags:          # extra tags applied to every meter
      environment: production
      service: word-count

Scrape it from the shell:

curl -s localhost:8080/metrics | head -40

A minimal Prometheus scrape config:

scrape_configs:
  - job_name: stoatflow
    metrics_path: /metrics
    static_configs:
      - targets: ["my-app:8080"]

The endpoint returns the standard exposition format, so any Prometheus-compatible collector (Prometheus, Grafana Agent, OpenTelemetry Collector with the Prometheus receiver, the VictoriaMetrics agent) scrapes it without extra translation. On Kubernetes, add prometheus.io/scrape / prometheus.io/path pod annotations or a ServiceMonitor pointing at the HTTP port — see Observability.

Recording levels

recording-level controls how much detail the engine reports, mirroring Kafka Streams' levels. A higher level is a superset of the ones below it.

Tags like lane_id, store_name, topic, and partition multiply series count. On large topologies, keep production at info and switch to debug only while investigating.

Naming and tags

All StoatFlow meters use a dotted name with the configured prefix (stoatflow by default). The Prometheus registry rewrites dots to underscores and appends the conventional suffix (_total for counters; _seconds_count/_seconds_sum/_seconds_max for timers). So stoatflow.barrier.completed.total is scraped as stoatflow_barrier_completed_total, and the timer stoatflow.barrier.commit.latency produces stoatflow_barrier_commit_latency_seconds_* series. The latency timers publish client-side percentiles as {quantile="0.5|0.95|0.99"} labels on the base _seconds name rather than _bucket histograms, so use rate(_sum)/rate(_count) (or the {quantile=...} series) — histogram_quantile() over _bucket won't match.

Two application tags are present on StoatFlow meters:

Anything you add under common-tags (for example environment, region, service) is also applied to every meter — handy for selecting one deployment in a shared Prometheus. Per-meter tags vary by group: lane_id, store_name, topic, partition, error_type, trigger, type, result, and others are listed with each meter below.

Key meters

The dotted names below are the canonical metric IDs the engine emits. Remember the Prometheus rewrite (. → _, plus the type suffix) when writing queries. This is a working set, not the exhaustive list — the full catalogue is on the metrics reference, and your live scrape stays authoritative for exact tag sets.

Two families are deliberately left to the reference rather than repeated here: stoatflow.rocksdb.* (~43 series per store, opt-in via stoatflow.rocks-db.metrics.enabled — reach for it when tuning RocksDB, not on a general dashboard) and stoatflow.suppress.* (a suppression-tuning signal).

Throughput and end-to-end latency

# Total processing throughput (records/sec)
sum(rate(stoatflow_lane_records_processed_total[1m]))

# Average per-record processing latency (this timer emits _sum/_count/_max, not _bucket)
sum(rate(stoatflow_lane_process_latency_seconds_sum[5m])) / sum(rate(stoatflow_lane_process_latency_seconds_count[5m]))

Barrier commits

Commit barriers are StoatFlow's exactly-once mechanism (see Exactly-once). These meters track barrier lifecycle and commit-phase latency. The latency timers publish client-side P50/P95/P99 percentiles.

# Commit rate vs. failure rate
rate(stoatflow_barrier_completed_total[1m])
rate(stoatflow_barrier_failed_total[5m])

Watermarks

Event-time progress for windowing and late-event detection (see Event time and watermarks).

# Alert when event time falls more than 5 minutes behind wall clock while the app is running (RUNNING == 4).
# The lag meter climbs during a stall — a lagging-but-live partition pins the global watermark and every
# window/session store stops expiring, so segments accumulate (see stoatflow_store_segments below).
stoatflow_watermark_lag_ms > 300000 and stoatflow_client_state == 4

# Fraction of time watermark alignment kept the application waiting, per group (0..1).
rate(stoatflow_watermark_alignment_held_time_total[5m])

Changelog writes and state restoration

Changelog write counters track state durability traffic; restoration meters track recovery progress on startup (see State stores and State and thread safety).

# Restoration progress as a fraction of stores completed
stoatflow_restoration_stores_completed / stoatflow_restoration_stores_total

Consumer, producer, and dispatch

# Worst-case consumer lag across partitions
max(stoatflow_consumer_lag_records)

Errors

Error classification for alerting (see Error handling).

Client state

High availability

Bound only when hot standby is enabled (ha.mode != off).

License metrics

When a license is configured, the runtime binds license meters to the same registry (see License configuration). These use the <prefix>.license prefix (default stoatflow.license, tracking runtime.metrics.prefix) and carry the registry common tags (application, plus your common-tags). They're exported even when recording-level is info.

# Alert when the license is not valid, or when expiry is within 14 days.
# Use a `for:` clause of at least 10m on the valid==0 rule: every deploy
# reads 0 for the ~5-minute deferred-validation PENDING window.
stoatflow_license_valid == 0
stoatflow_license_days_remaining < 14

JVM and Kafka-client metrics

With bind-jvm-metrics: true (the default), the runtime binds Micrometer's standard JVM binders to the registry, so the usual jvm_* and system_* series appear on the same endpoint:

• jvm_memory_used_bytes / jvm_memory_max_bytes (heap and non-heap)
• jvm_gc_pause_seconds and related GC metrics
• jvm_threads_live_threads, jvm_threads_states_threads
• jvm_classes_loaded_classes
• system_cpu_usage, process_cpu_usage, system_load_average_1m

These carry the registry common tags (application + your common-tags) but not application_id. Native Kafka client metrics are also surfaced through Micrometer, so producer/consumer client internals appear alongside the StoatFlow meters.

Dashboards and alerts

A starting overview combines throughput, latency, backpressure, and commit health:

# Throughput
sum(rate(stoatflow_lane_records_processed_total[1m]))

# Backpressure — total queued vs. capacity
sum(stoatflow_lane_queue_size) / sum(stoatflow_lane_queue_capacity)

# Commit health
rate(stoatflow_barrier_completed_total[1m])
rate(stoatflow_barrier_failed_total[5m])

# Event-time lag
stoatflow_watermark_lag_ms

Useful alert conditions:

Next steps

• Observability — wiring /metrics, logs, and probes into a monitoring stack.
• Health checks — the readiness and liveness signals that complement metrics.
• Tuning — which bound knobs the self-tuning cadence respects.
• The REST API — the other operational endpoints on the same HTTP server.