Health checks

The :runtime module exposes two HTTP health endpoints — /health/live and /health/ready — backed by a registry of health indicators. Each indicator answers two independent questions: is the process alive? (liveness) and is the process ready to serve traffic? (readiness). This page covers the built-in indicators, exactly what flips each one, and how the probes behave while state is restoring.

Liveness vs readiness

The two probes serve different orchestrator decisions, so they have different failure semantics:

The split matters because a restart does not fix every problem. A pod that is still starting up, restoring its state stores, paused, or whose license has expired should not be restarted — restarting either wastes the work in progress or loops forever without resolving the issue. Those conditions flip readiness DOWN (stop sending it traffic) while keeping liveness UP (leave it running).

Both endpoints return the same JSON shape — an overall status plus one entry per indicator:

{
  "status": "UP",
  "components": [
    { "name": "stoatflow", "status": "UP", "details": { "state": "RUNNING" } },
    { "name": "license", "status": "UP", "details": { "status": "VALID", "tier": "production", "daysRemaining": 281 } },
    { "name": "kafka-broker", "status": "UP", "details": { "broker": "reachable" } }
  ]
}

HTTP status follows the overall status: 200 when UP, 503 when DOWN. The format is compatible with Spring Boot Actuator, so existing dashboards and probe tooling that parse Actuator health work unchanged.

The built-in indicators

StoatFlowRuntime registers four indicators automatically at startup (a fifth, ha, when hot standby is enabled). Each is registered for both liveness and readiness, but most return different verdicts for the two checks — that is the whole point of the split.

App state (stoatflow)

The canonical indicator — it reads the engine's lifecycle state and maps it to liveness and readiness:

Readiness is UP in exactly one state — RUNNING. Every transient lifecycle state (starting, restoring, draining, pausing, stopping) is readiness DOWN but liveness UP, so the orchestrator routes traffic away without killing a process that is doing legitimate work.

*The asterisk on RUNNING liveness: if the engine's commit pipeline stalls past a configured threshold, liveness flips DOWN so the orchestrator can restart a wedged process. This is an independent safety net layered on top of the in-process supervision the engine already runs — the probe exists so Kubernetes has its own recovery path even if internal recovery is itself stuck. The threshold is the engine's commitStallThresholdMs field (default 45 000 ms; set 0 to disable the check), set in YAML via stoatflow.commit-stall.threshold-ms. When it fires, the liveness response carries stall_age_ms, stall_threshold_ms, and related detail fields so the cause is visible in the probe payload.

Kafka broker (kafka-broker)

Verifies broker connectivity by listing topics through a dedicated AdminClient. A StoatFlow app cannot make progress without its broker, so this indicator returns the same verdict for both probes: unreachable broker → DOWN for liveness and readiness. Loss of broker connectivity is treated as fatal — a restart is a reasonable response.

The check is bounded by a timeout (runtime.health.kafka-broker.timeout-ms, default 2 000 ms). On success the detail is { "broker": "reachable" }; on failure it is { "broker": "unreachable", "error": "…" }.

Schema Registry (schema-registry)

Self-enabling — this indicator is registered only when a Schema Registry URL is configured (stoatflow.schema-registry-url). If you don't use Schema Registry, it never appears in the components array. When present, it does an HTTP GET /subjects against the registry; like the broker check it returns the same verdict for liveness and readiness (unreachable → DOWN for both). Its timeout is runtime.health.schema-registry.timeout-ms (default 5 000 ms).

License (license)

The license indicator is the clearest example of the liveness/readiness split:

Readiness goes DOWN the moment the license is no longer operational — EXPIRED, REVOKED, INVALID, or NO_KEY — so orchestrators stop routing traffic to a pod whose license has been pulled. VALID and GRACE_PERIOD are both treated as operational (the grace period exists precisely so a transient validation outage doesn't take you down).

Liveness stays UP regardless of license status. Restarting a pod will not fix an expired or revoked license — it would just produce a restart loop that hides the real problem. The readiness payload carries the status and tier; the operational (UP) readiness payload additionally carries daysRemaining.

Within the renewal-warning window, the liveness payload adds a licenseRenewalWarning detail (e.g. "expires in 9 days") so the nudge is visible on the liveness response too — even though liveness itself stays UP.

Hot standby (ha)

Registered only when hot standby is enabled (ha.mode != off). It encodes the readiness contract that makes the rolling update order-independent:

• Readiness is UP when the pod is the active and processing, or a standby and caught up. A still-catching-up standby is not ready, so Kubernetes won't advance the roll until the restarted pod is a caught-up standby. With more than one standby it also enforces the redundancy floor — a caught-up standby is ready to roll only if rolling it leaves the active plus at least one other caught-up standby (ha.desired-standbys).
• Liveness stays UP while a standby is catching up (so it isn't killed mid-catch-up); it goes DOWN only when the pod is genuinely down or a restart is required (for example, after a source topic's partition count changes).

The payload carries role and ha_state details; readiness also reports replication_lag_records. See High availability for the full operational picture.

How the probes behave during restoration

State restoration is the case the readiness/liveness split is designed for. On startup — and again after a crash recovery — StoatFlow rebuilds its state stores from their changelog topics before it begins processing. While that runs, the engine is in RESTORING (preceded by STARTING / VALIDATING_STATE):

• Readiness is DOWN for the entire restoration. The app is alive but not yet processing, so the orchestrator must not count it as in service or route admin traffic to it as "ready".
• Liveness is UP throughout. Restoration is expected, legitimate work — restarting mid-restore would throw away progress and start over, potentially never converging for a large state store.

This is why you map the readiness probe to load-balancer membership and the liveness probe to restart decisions. A correctly-wired deployment lets a restoring pod take all the time it needs (within the readiness probe's failureThreshold × periodSeconds budget) without being killed. Size the readiness probe's startup tolerance to your largest expected restore — see Probes for the startupProbe / initialDelaySeconds guidance.

# During restoration: ready is 503, live is 200.
curl -s -o /dev/null -w "ready=%{http_code}\n" localhost:8080/health/ready   # ready=503
curl -s -o /dev/null -w "live=%{http_code}\n"  localhost:8080/health/live    # live=200

# The readiness body shows why:
curl -s localhost:8080/health/ready | jq '.components[] | select(.name=="stoatflow")'
# { "name": "stoatflow", "status": "DOWN", "details": { "state": "RESTORING" } }

The same pattern applies to pause/unpause: a paused engine (PAUSED / DRAINING) is readiness DOWN, liveness UP — traffic is steered away but the process is left running so you can /unpause it. See Pause and unpause.

The indicator registry

Indicators are held in a registry that keeps separate collections for liveness and readiness. Registering an indicator with register(name, indicator) adds it to both; the registry also supports liveness-only and readiness-only registration. /health/live evaluates every liveness indicator's livenessHealth(); /health/ready evaluates every readiness indicator's health(). The overall status is UP only when all indicators in that collection report UP.

Adding your own indicator

A health indicator implements one interface — health() for readiness and an optional livenessHealth() (defaults to the same verdict as health()):

import io.stoatflow.runtime.health.Health
import io.stoatflow.runtime.health.HealthIndicator

class DownstreamApiHealthIndicator(
    private val client: DownstreamClient,
) : HealthIndicator {
    // Readiness: stop routing traffic if the dependency is down.
    override fun health(): Health =
        if (client.ping()) {
            Health.up().withDetail("downstream", "reachable").build()
        } else {
            Health.down().withDetail("downstream", "unreachable").build()
        }

    // Liveness: a downstream outage should NOT restart us — leave the default,
    // or override to return UP explicitly.
    override fun livenessHealth(): Health =
        Health.up().withDetail("downstream", "checked-on-readiness-only").build()
}

import io.stoatflow.runtime.health.Health;
import io.stoatflow.runtime.health.HealthIndicator;

public class DownstreamApiHealthIndicator implements HealthIndicator {

    private final DownstreamClient client;

    public DownstreamApiHealthIndicator(DownstreamClient client) {
        this.client = client;
    }

    // Readiness: stop routing traffic if the dependency is down.
    @Override
    public Health health() {
        return client.ping()
            ? Health.up().withDetail("downstream", "reachable").build()
            : Health.down().withDetail("downstream", "unreachable").build();
    }

    // Liveness: a downstream outage should NOT restart us.
    @Override
    public Health livenessHealth() {
        return Health.up().withDetail("downstream", "checked-on-readiness-only").build();
    }
}

Register it on the runtime builder with addHealthIndicator(name, indicator), which puts it in both the liveness and readiness collections alongside the built-ins:

val runtime = StoatFlowRuntime.fromConfig(
    topologyBuilder = { buildTopology(it) },
    configure = {
        addHealthIndicator("downstream-api", DownstreamApiHealthIndicator(client))
    },
)

var runtime = StoatFlowRuntime.fromConfig(
    Main::buildTopology,
    builder -> builder.addHealthIndicator("downstream-api", new DownstreamApiHealthIndicator(client))
);

Configuration

The health checks themselves need no configuration to work — the four built-in indicators are wired automatically. The only tunables are the connectivity-check timeouts and the broader HTTP-server enablement:

stoatflow:
  # Schema Registry health auto-enables when this is set:
  schema-registry-url: ${SCHEMA_REGISTRY_URL:-}

runtime:
  http:
    enabled: true              # /health/live and /health/ready live on the runtime HTTP server
    port: ${HTTP_PORT:-8080}
  health:
    kafka-broker:
      timeout-ms: 2000         # broker connectivity check timeout (default 2000)
    schema-registry:
      timeout-ms: 5000         # schema-registry check timeout (default 5000)

Quick reference

# Liveness — should the orchestrator restart this pod?
curl -s localhost:8080/health/live  | jq .

# Readiness — should the orchestrator route traffic to this pod?
curl -s localhost:8080/health/ready | jq .

# Just the HTTP status (what a probe actually checks):
curl -s -o /dev/null -w "%{http_code}\n" localhost:8080/health/ready

Next steps

• Probes — the Kubernetes livenessProbe / readinessProbe / startupProbe wiring, with budgets sized for restoration.
• REST API — the full endpoint catalogue, including /state, /license, and /pause.
• Metrics — Prometheus gauges and counters, including engine state and license metrics.
• Pause and unpause — how pausing interacts with readiness.