Liveness and readiness probes

StoatFlow exposes two HTTP health endpoints designed to drive Kubernetes liveness and readiness probes: /health/live decides whether the pod should be kept alive, /health/ready decides whether it should receive traffic and count toward a rollout. This page shows how to wire them, what each one gates on, and how to set the probe periods.

For the response shape and the full component breakdown, see Health checks. This page is about wiring and tuning the probes.

The two endpoints

Both endpoints aggregate every registered health indicator. The overall status is UP (HTTP 200) only when every component is UP; if any component is DOWN, the endpoint returns DOWN (HTTP 503). The JSON body always lists the per-component status so you can see which one tripped.

The built-in indicators are stoatflow (engine state), license, kafka-broker, and — when a Schema Registry URL is configured — schema-registry. Each contributes to both probes, but several of them deliberately report different liveness vs. readiness status (table below).

What readiness gates

Readiness answers "should this pod receive traffic and count as available?" StoatFlow returns /health/ready = DOWN (503) until the application is genuinely able to process records:

• State restoration. On cold start or restart the engine rebuilds state stores from their changelog topics before processing begins. Readiness stays DOWN for the whole STARTING / VALIDATING_STATE / RESTORING window. During a rolling update this is what keeps Kubernetes from cutting traffic over to a pod that hasn't caught up yet. See Architecture — lifecycle for the cold-start sequence.
• Broker availability. The kafka-broker indicator checks broker connectivity. If the broker is unreachable, readiness goes DOWN — the pod can't make progress, so it shouldn't be considered ready.
• License. The license indicator returns readiness DOWN when the runtime license is EXPIRED, REVOKED, INVALID, or absent — the orchestrator stops routing to a pod whose license has been pulled. (VALID and GRACE_PERIOD are treated as operational.) See License configuration.
• Schema Registry (only when configured). If a Schema Registry URL is set and the registry is unreachable, readiness goes DOWN.
• Pause and shutdown. While paused (DRAINING / PAUSED) or shutting down (STOPPING), readiness is DOWN so load balancers drain the pod cleanly. See Pause / unpause.

Readiness returns UP only when the engine is in the RUNNING state and every other component is healthy.

What liveness gates

Liveness answers a narrower question: "is this process broken in a way that only a restart can fix?" It deliberately stays UP through the normal transient states — STARTING, VALIDATING_STATE, RESTORING, DRAINING, PAUSED, STOPPING — so the orchestrator doesn't kill a pod that's simply busy restoring state or draining for a clean shutdown.

Liveness returns DOWN in two situations:

1. Hard dependency failure — the broker (or a configured Schema Registry) is unreachable. These can't be made healthy by anything the pod is doing, so a restart is the right response.
2. A frozen commit pipeline (see below).

Note one deliberate asymmetry: the license indicator never fails liveness. An expired or revoked license fails readiness (traffic stops) but keeps liveness UP — restarting the pod won't fix a license problem, and a restart loop would only hide it.

Stall-aware liveness

The commit barrier is the heart of exactly-once processing: a transaction periodically commits state, output, and offsets together (see Exactly-once semantics). If that pipeline were ever to freeze — a commit that starts but never finishes — the pod would sit there alive but making no progress, the worst kind of failure for an orchestrator to miss.

StoatFlow guards against this. The runtime tracks how long it has been since the last successful commit while commit work is pending. If that age exceeds a configured threshold, /health/live flips to DOWN — Kubernetes restarts the pod, and the restarted process resumes from the last successful barrier with no duplicates downstream. The threshold is stoatflow.commit-stall.threshold-ms (default 45000 ms; set to 0 to disable the check, which re-introduces the silent-freeze failure mode and is discouraged).

Liveness vs. readiness, per component

Kubernetes probe configuration

Point both probes at the HTTP server's port (default 8080). A typical container spec:

# Deployment / Pod container spec
ports:
  - name: http
    containerPort: 8080

readinessProbe:
  httpGet:
    path: /health/ready
    port: http
  periodSeconds: 5
  timeoutSeconds: 3
  failureThreshold: 3        # 3 × 5s = ~15s of consecutive failures before "not ready"

livenessProbe:
  httpGet:
    path: /health/live
    port: http
  periodSeconds: 10
  timeoutSeconds: 3
  failureThreshold: 3        # ~30s of consecutive failures before restart

startupProbe:
  httpGet:
    path: /health/ready
    port: http
  periodSeconds: 10
  failureThreshold: 60       # allow up to ~10 minutes for state restoration

Tuning the periods

• Readiness period controls how quickly traffic is cut from a pod that goes unhealthy and restored when it recovers. A short periodSeconds (e.g. 5s) reacts fast at the cost of slightly more probe load; the probe is cheap (an in-process status aggregation), so erring short is fine.
• Liveness period and failureThreshold govern how long a wedged pod survives before restart: roughly periodSeconds × failureThreshold. Keep this comfortably longer than the stall threshold (stoatflow.commit-stall.threshold-ms, default 45s) so a genuine stall is what trips the restart, not a single slow probe — for example a 10s period × 3 failures (~30s) layered on top of the stall threshold means a real freeze is reported by the endpoint well before the probe budget would have expired on its own.
• timeoutSeconds should leave headroom over the broker health-check timeout, since the readiness aggregation includes the kafka-broker connectivity check. The default broker check timeout is short, but set timeoutSeconds to at least 2-3s.

Verifying locally

With the app running, the endpoints answer over HTTP. A ready pod returns 200:

# Returns 200 + {"status":"UP",...} when RUNNING; 503 during restoration/pause/shutdown
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/health/ready

# Returns 200 unless the process is wedged (commit stall) or a hard dependency is down
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/health/live

# Inspect the component breakdown
curl -s localhost:8080/health/ready | jq

During state restoration /health/ready returns 503 with a body identifying the in-progress restorations; once the engine reaches RUNNING it flips to 200.

See also

• Health checks — response format, all indicators, registering custom ones
• High availability — how readiness gates the hot-standby rolling update
• Kubernetes — deploying StoatFlow on Kubernetes
• Observability — metrics and the /debug/* endpoints
• Pause / unpause — how pause affects readiness
• Core configuration — commit-stall and commit-barrier settings