Posts tagged “Deep Dive”

Three gates, one transaction: error handling in StoatFlow

A record can fail on the way in, in the middle, or on the way out. StoatFlow gives every gate the same verdicts — continue, fail, or dead-letter — and settles them all inside the exactly-once transaction: DLQ records commit on the same barrier as your output, and what cannot be handled kills the epoch, never the guarantee. The full model, from one bad record to a Kubernetes restart.

Internal consistency on Kafka: emitting a correct answer at every commit

Money can only be moved, never created — so a stream that tracks balances should read total = 0 at every consistent cut. The Flink Table API gets it right 0.035% of the time; our Kafka Streams twin sends total to −1,619 … +1,792. StoatFlow holds it at exactly 0, at every one of its committed cuts. Here is why, and the measured proof.

KIP-1271: record headers in state stores, and the cost of a value format

KIP-1271/1285 let a state store keep a record's headers next to its value — shipped in Apache Kafka 4.3.0. What the KIP is, where Kafka Streams has got to (and what is still in flight), how StoatFlow 1.0.0 compares — and why a one-varint format change becomes an engine-wide one.

KIP-1035: Why Kafka Streams 4.3 lets state stores own their offsets

Kafka Streams kept each task's changelog offsets in a separate checkpoint file that could drift from the state it described. KIP-1035, in Kafka 4.3, moves them inside the state store — atomic with the data, and the keystone for transactional state stores.