Observability

A StoatFlow application is one process, so observability is simpler than a clustered stream processor — there is one set of metrics, one log stream, and one HTTP surface to point your monitoring stack at. This page wires those three together: scraping /metrics into Prometheus and Grafana, the meters worth alerting on, reading and correlating the logs, and using the /debug/* endpoints when a pipeline goes quiet or slow.

The three signals

The runtime exposes everything an operator needs over the same HTTP server (default port 8080):

Metrics and logs are continuous and cheap; the /debug/* snapshots are pull-on-demand and read state the engine already keeps, so there is no steady-state cost to having them enabled.

Scrape metrics into Prometheus

Metrics are enabled by default under runtime.metrics. The endpoint serves the standard Prometheus exposition format, so any Prometheus-compatible collector scrapes it without translation.

runtime:
  http:
    enabled: true
    port: 8080            # default
  metrics:
    enabled: true         # default
    recording-level: info # info | debug | trace (default: info)
    common-tags:          # applied to every meter — handy in a shared Prometheus
      environment: production
      service: word-count

A minimal static scrape config:

scrape_configs:
  - job_name: stoatflow
    metrics_path: /metrics
    static_configs:
      - targets: ["my-app:8080"]

On Kubernetes, prefer service discovery over static targets. With a Prometheus Operator, point a ServiceMonitor at the HTTP port; without it, the pod annotations are enough for the default Kubernetes SD relabeling:

# Pod template metadata — picked up by Prometheus pod service-discovery
metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/path: "/metrics"
    prometheus.io/port: "8080"

# Or, with the Prometheus Operator: a ServiceMonitor selecting the app Service
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: word-count
spec:
  selector:
    matchLabels:
      app: word-count
  endpoints:
    - port: http        # the named container/Service port that maps to 8080
      path: /metrics
      interval: 15s

Build the Grafana view

Grafana reads the same Prometheus series. A single-instance app means no per-replica aggregation to reason about — one target, one set of series, filtered by application_id (and your common-tags) when several apps share a Prometheus.

The names below are the canonical dotted metric IDs; Prometheus rewrites . to _ and appends the type suffix (_total for counters, _seconds_count / _seconds_sum / _seconds_max for timers). The complete catalogue and every tag are on the metrics reference; recording levels and scrape setup are on Metrics — this is the operator's working subset.

A starter overview row, top to bottom:

# Throughput — records processed per second across all lanes
sum(rate(stoatflow_lane_records_processed_total[1m]))

# End-to-end latency — mean over the scrape window, source timestamp to processing completion.
# e2e.latency is a plain timer: it exports _seconds_count / _seconds_sum / _seconds_max, no
# _bucket series — so use the sum/count ratio for the average and the _max gauge for the peak.
rate(stoatflow_e2e_latency_seconds_sum[5m]) / rate(stoatflow_e2e_latency_seconds_count[5m])
stoatflow_e2e_latency_seconds_max

# Commit health — barriers completing vs. failing
rate(stoatflow_barrier_completed_total[1m])
rate(stoatflow_barrier_failed_total[5m])

# Backpressure — total queued depth as a fraction of capacity
sum(stoatflow_lane_queue_size) / sum(stoatflow_lane_queue_capacity)

# Event-time progress — how far behind wall-clock the watermark is
stoatflow_watermark_lag_ms

# Consumer lag — worst partition
max(stoatflow_consumer_lag_records)

With bind-jvm-metrics: true (default) the standard jvm_* and system_* series land on the same endpoint, so a second row covers the runtime itself:

# Heap used vs. max
jvm_memory_used_bytes{area="heap"} / jvm_memory_max_bytes{area="heap"}

# GC pause rate
rate(jvm_gc_pause_seconds_sum[5m])

# CPU
process_cpu_usage

Alert on the right meters

For a single-instance app, the alerts that matter are the ones that tell you the one process is stuck, falling behind, or about to be evicted — and the license, because an expired license takes the readiness probe down. Wire these into Alertmanager (or your equivalent); the full rationale for each meter is on Metrics.

The standby alerts apply only when hot standby is enabled. The stoatflow.client.state gauge reports the ordinal of the application lifecycle state. The states are numbered in the order the app moves through them on startup — CREATED=0, STARTING=1, VALIDATING_STATE=2, RESTORING=3, RUNNING=4 — so == 4 is the "healthy and processing" predicate the alerts above gate on, and any value below 4 means the app has not yet reached (or has left) RUNNING. The complete state-to-ordinal mapping and the readiness contract are on Health checks and Probes.

Logs and correlation

The runtime logs to stdout. Out of the box that is plain text — Logback's pattern encoder, one line per event — which is fine for kubectl logs and local development. If you want structured ingestion into Loki, Elasticsearch, CloudWatch, or any pipeline that parses JSON, add a JSON encoder to your application's Logback configuration; StoatFlow does not impose one.

Set per-package log levels through config (applied to Logback at startup) without rebuilding or editing logback.xml:

logging:
  level:
    root: INFO
    io.stoatflow: INFO
    # Quieten the Kafka clients in steady state:
    org.apache.kafka: WARN

Correlating a log line to a lane and barrier

Lane and barrier identifiers are not in the log output by default. The engine can publish them via SLF4J's MDC (Mapped Diagnostic Context), but MDC is off unless you opt in — set stoatflow.processing.mdc-enabled: true in your config:

stoatflow:
  processing:
    mdc-enabled: true   # default: false

With it enabled, the engine populates these MDC keys while processing:

• laneId — the lane handling the record or barrier, in the {subtopology}_{lane} form (0_0, 1_7). This is the same value the metrics carry as the lane_id tag, so you can pivot from a per-lane Grafana panel to that lane's log lines — note the key name differs by case: MDC uses laneId, the metric tag is lane_id.
• barrierId — the monotonically increasing commit-barrier identifier. The same id appears in the /debug/barriers snapshot, so a slow commit can be followed across logs and the debug endpoint.
• sourceTopic, sourcePartition, sourceOffset — the input record's coordinates, for tracing a specific record through the topology.

These keys only reach your log lines if your Logback pattern references them — for example %X{laneId} and %X{barrierId} in a pattern encoder, or the equivalent fields in a JSON encoder. Operator names you set explicitly (Named.as(...), Consumed.as(...), …) appear in the topology-related log messages themselves, so a log entry can still be tied back to a specific node in the DAG without MDC.

Diagnose a frozen or slow pipeline

Metrics tell you that throughput dropped to zero or a commit stopped advancing. The two /debug/* endpoints tell you why, by snapshotting state the engine already holds. Reach for them when a dashboard shows a stall — stoatflow_barrier_completed_total flat, stoatflow_consumer_lag_records climbing, throughput at zero — and you need to see what the process is actually doing right now.

Both are gated by runtime.http.debug.enabled (default true); when disabled they return 404. They expose internal engine state, so keep them on the in-cluster network and off public ingress.

/debug/threads — what every lane is doing

A JSON snapshot of engine-owned and platform threads with their current state and stack traces. Use it to answer "is a lane blocked, and on what?" — a lane parked in a user Processor waiting on a slow REST call looks different from one waiting on a commit, and the stack tells you which.

# Full snapshot
curl -s localhost:8080/debug/threads

# Only non-runnable threads (the usual starting point for a freeze)
curl -s 'localhost:8080/debug/threads?filter=stuck'

# Engine-named threads and lanes only, deeper stacks
curl -s 'localhost:8080/debug/threads?filter=known&depth=80'

Query params (both optional):

filter=stuck is the fast triage view: if throughput is zero, the threads that aren't RUNNABLE are where the work has piled up. The response also surfaces any threads the JVM has detected as deadlocked.

/debug/barriers — where a commit is stuck

A JSON snapshot of the commit pipeline, built to answer "which barrier is stuck, at what phase, and what's holding it up?" from a single call. When stoatflow_barrier_completed_total has gone flat, this is the endpoint that tells you whether the commit is waiting on lanes, queued, mid-transaction, or simply idle.

curl -s localhost:8080/debug/barriers

The snapshot includes the last committed barrier and its age, any in-flight commit progress, and a derived phase field — a single-string summary of where the pipeline is:

A healthy app cycles through these quickly; a stuck app sits in one. If phase is AWAITING_LANE_ACKS and not moving, cross-reference /debug/threads?filter=stuck to find which lane is blocked and on what. If a commit is genuinely frozen, the runtime detects the stall, aborts the in-flight transaction, and exits so the orchestrator restarts the process — you'll see this as a spike in stoatflow.barrier.failed.total and the restarting instance entering its restoration phase.

A diagnosis workflow

When the dashboards say something is wrong, the order that gets to a cause fastest:

1. Check stoatflow.client.state — is the app RUNNING (4)? If it's validating, restoring, or stopping (any value below 4), the answer is lifecycle, not a freeze. Confirm against /health/ready.
2. Check throughput and lag — rate(stoatflow_lane_records_processed_total[1m]) at zero with rising stoatflow_consumer_lag_records means the process stopped making progress.
3. Check commit progress — flat stoatflow_barrier_completed_total points at the commit pipeline. Pull /debug/barriers and read the phase.
4. Find the blocked lane — if phase is waiting on lanes, curl '…/debug/threads?filter=stuck' and read the stacks for the lane that isn't making progress.
5. Correlate with logs — with MDC enabled, filter the logs to the offending laneId and barrierId to see the records and errors around the stall.

RocksDB metrics

For persistent (RocksDB-backed) stores, StoatFlow can expose the same signals Kafka Streams does — cache hit ratios, compaction and flush activity, write stalls, memtable and SST sizes — under stoatflow.rocksdb.*, tagged store_name. They are off by default; two flags gate them:

stoatflow:
  rocks-db:
    metrics:
      enabled: true             # property + block-cache gauges (recording-level info) — negligible cost
      statistics-enabled: true  # + ticker/histogram/ratio family (recording-level debug) — RocksDB write-path
                                # cost, commonly cited ~5–10% on write-heavy loads; wired at store open (restart to toggle)

The property/cache half (enabled) is cheap — a handful of native reads per store per minute — and is what confirms a sizing diagnosis: stoatflow_rocksdb_shared_block_cache_usage_bytes vs ..._capacity_bytes tells you whether the block cache is full, and stoatflow_rocksdb_estimate_num_keys / ..._total_sst_files_size_bytes track state growth. The statistics half (statistics-enabled) adds the health signals that need RocksDB's internal counters:

• stoatflow_rocksdb_block_cache_hit_ratio (+ data/index/filter variants) — per-interval; a falling ratio under load is the classic "cache too small" symptom (see Tuning → RocksDB).
• stoatflow_rocksdb_write_stall_duration_avg_ms and ..._total_ms — writes stalling on compaction back-pressure.
• rate(stoatflow_rocksdb_compaction_bytes_written_total[5m]) and the compaction/flush time histograms — compaction load, useful when segment-checkpoint churn is suspected.

Ratios and histogram avg/min/max are per-interval gauges; tickers are .total counters — use PromQL rate(). Two caveats: the counters reset on an in-place engine restart (a normal Prometheus counter reset, absorbed by rate()), and on the default (FFM) backend the histogram *.min.ms series is unavailable — use avg/max. The block cache is shared across all stores, so stoatflow_rocksdb_shared_block_cache_* (no store_name) is the global truth; the per-store stoatflow_rocksdb_block_cache_* gauges repeat it for Kafka Streams dashboard portability.

Next steps

• Metrics — scraping, recording levels, tags, and license metrics; the complete meter catalogue is on the metrics reference.
• Probes — liveness and readiness, and how readiness holds traffic off during restoration.
• The REST API — every operational endpoint, with the in-cluster vs. public exposure split.
• Tuning — the bound knobs behind the self-tuning commit cadence, and the latency/throughput/restart trade-offs.
• High availability — the hot-standby meters and what to alert on for the cluster.
• Production checklist — the pre-flight list before going live.