Production checklist

Before you put a StoatFlow application in front of production traffic, walk this list. Each item is a decision or a wiring step you can verify, and each links to the page that covers it in full — this page is the index, not the explanation.

Deployment and the HA tier

The one rule an operator must not break: run exactly one active instance. Two active instances pointed at the same source topics corrupt state. Then choose your HA tier — fast restart (default) or the opt-in hot-standby cluster — deliberately.

• HA tier chosen deliberately. Fast restart (default, replicas: 1) for most workloads, or the opt-in hot-standby cluster (replicas: 2 or more, ha.mode: active-standby) for near-instant failover on large state / tight RTO. → High availability
• replicas: 1 on the workload (default mode), with a rollout strategy that tears the old pod down before the new one starts processing — never two active concurrently. Hot standby is the exception: a readiness-gated cluster of replicas: 2 or more. → Kubernetes, High availability
• Failure mode of two active instances understood, and why the standby is passive. → Architecture, Deploying and operating
• Stable application-id. It is the application's identity — the consumer group and changelog topics derive from it. Two active processes sharing it collide. → Core configuration
• Graceful shutdown on SIGTERM. The runtime drains in-flight records and commits a final barrier before exiting; give the orchestrator's termination grace period enough headroom (and, under hot standby, room for the handoff). → Kubernetes

State persistence and restart

Fast restart — the default HA tier — depends on whether the new pod starts with a warm local store or rebuilds state from the changelog cold. (Hot standby sidesteps the restore entirely by keeping one or more warm standbys ready — see High availability.)

• Persistent volume for warm restart. A PersistentVolumeClaim retained across restarts means restoration only reads the changelog gap since the last commit, not the whole topic. → Kubernetes
• Restart window understood. Cold-start time scales with state size; the levers are state size, disk persistence, and broker read throughput. → Tuning, Benchmarks
• Recovery semantics confirmed for your chosen processing guarantee — the last successful commit barrier is the recovery point. → Exactly-once

Processing guarantee

Pick exactly-once or at-least-once deliberately — it is the central reliability decision and it shapes your latency floor.

• Exactly-once vs at-least-once chosen on purpose, not by default. Exactly-once commits state, output, and offsets atomically on a barrier; at-least-once trades possible duplicates for a lower commit-cadence floor on latency. → Exactly-once
• Downstream consumers match the mode. Under exactly-once, downstream reads with read_committed isolation see no duplicates; under at-least-once, downstream must be idempotent or duplicate-tolerant. → Exactly-once

License

A production instance needs a valid runtime license, injected as a secret — never baked into an image or committed.

• License key injected as an environment variable / secret, not hardcoded. → License configuration
• License health is wired into readiness. An expired, revoked, or missing license flips /health/ready to not-ready; the renewal window surfaces a warning before it bites. → License configuration, Health checks

Probes

Readiness is the lever that keeps traffic off a process that hasn't caught up. Wire both probes.

• /health/ready wired as the readiness probe. It returns 503 throughout state restoration, so nothing routes to a process that isn't caught up. → Probes, Health checks
• /health/live wired as the liveness probe. Lets the orchestrator restart a genuinely stuck process. → Probes
• Probe timing tuned so a long but legitimate restoration isn't killed by an aggressive liveness deadline. → Probes

Observability

You operate one process, so its metrics and introspection endpoints are your whole picture.

• /metrics scraped into your monitoring stack — JVM, Kafka-client, and StoatFlow internal counters in Prometheus format. → Metrics, Observability
• Alerts set for the signals that matter on a single-instance app — consumer lag, commit stalls, error rates, restoration progress. → Observability
• Logs collected — plain text by default; add a JSON encoder to your Logback config for structured ingestion, and enable MDC to correlate by lane and barrier. → Observability
• Introspection endpoints reachable for diagnosis — /topology, /state, /watermarks, and the /debug/* views. → REST API, Observability

Error handling

Decide what happens to a bad record before one arrives — silent skipping should be a choice, not a default.

• Processing-exception handler chosen — log-and-continue, log-and-fail, or route to a dead-letter queue. → Error handling (DLQ), Error-handling model
• Deserialization-error handler chosen for malformed source records, with its own policy. → Error handling (DLQ)
• DLQ topic provisioned if you route to one, and monitored — a filling DLQ is a signal. → Error handling (DLQ), Observability

Configuration and tuning

Start from the presets, then tune against your workload rather than guessing.

• Defaults and presets reviewed — you know which preset you're running and why. → Defaults and presets
• Resources sized — CPU and memory for the single process, lane count for your core count. → Tuning
• Commit-cadence bounds set appropriately for your latency-vs-throughput target. The cadence self-tunes within the configured bounds; you set the bounds. → Tuning, Configuration model
• Kafka client config reviewed — bootstrap servers, security, and any client overrides. → Kafka client configuration
• -XX:+UseG1GC set so GC pauses stay short and predictable (the Gradle plugin applies it to the Docker image; set it explicitly otherwise). → Installation, Docker

Custom processor correctness

If you wrote custom Processor code, make sure it holds under concurrent lane execution.

• Thread safety understood — key affinity makes per-key updates serial, but custom processors that read-modify-write across multiple keys need the runtime's key-lock utility. → State and thread safety
• No cross-lane state assumptions in custom code. → State and thread safety, Processor API
• Topology tested with the in-memory test driver before deploying. → Testing

Where to go next

• Kubernetes — the manifest that satisfies the deployment and persistence rows above.
• Observability — what to scrape and what to alert on once you're live.
• Tuning — turn the sizing and cadence rows into concrete numbers for your workload.
• High availability — the opt-in hot-standby cluster, if fast restart isn't fast enough.
• Deploying and operating — the operating section overview, including both HA tiers in full.