A use-after-free in a RocksDB iterator, and the test that passed on Linux

A StoatFlow RocksDB store does not write each update straight into RocksDB. Underneath its caching layer it keeps epoch machinery of its own: it collects an epoch's writes in a native WriteBatchWithIndex, and only the commit at the end of the epoch writes the batch — data and changelog offsets together, in one atomic write. An iterator therefore reads a stack: RocksDB itself, then the batch being committed, then the batch waiting to commit, then the batch being written. Each layer is a native iterator over native memory, and the batches are freed when their epoch is done.

Four defects let a batch be freed or rewritten while an iterator was still reading it. The symptoms were the ones native code gives you: segmentation faults in RocksDB's iterator, keys read twice, a value read back as ??. Up front, because it matters most: no store StoatFlow builds could reach the reproductions that crashed or misread data — the key-value, window and session stores sit behind a caching layer that never writes these batches, so they stay empty, and the two stores that do write their own, versioned and timer, never hold an iterator across a write. What production could reach was an iterator holding an empty batch that might be freed under it: undefined behaviour, never observed as a crash. The one user path to the last defect is now refused at startup. This post is about how we proved them anyway, and the test that would not fail on Linux.

TL;DR

• What: an iterator over a StoatFlow RocksDB store chains native write batches, one per epoch layer. Ref-counting gaps and concurrent appends let a batch be freed or reallocated under an open iterator.
• Symptoms: SIGSEGV in RocksDB's write-batch iterator, keys read twice, a value read back as ??. Reproduced on raw stores only.
• The harness: each scenario runs in its own child JVM with the allocator scribbling freed memory; the test asserts the child exits cleanly.
• The fix: references travel with the batch, one shared chaining helper, and the commit writes a copy whenever an iterator holds the batch.
• What you might notice: a custom StateStoreBuilder whose build() returns a persistent supplier's get() is refused at startup.
• Docs: State stores · Exactly-once.

Four ways to lose a batch under an iterator

RocksDB's write-batch iterator holds the entry it is positioned on as a slice into the batch's buffer. Free the batch, or append to it and let the buffer reallocate, and that slice points at memory that no longer holds the entry.

(a) Two iterators and one counter. Each epoch's batches were reference-counted, so a batch whose commit finished while an iterator still read it was deferred until the last reference went. Closing one iterator and opening another on the same epoch could interleave: the release decremented to zero, the new iterator incremented the same counter, then the release removed the counter, and the batch was freed with the new iterator still on it. The new iterator's next step crashed in WBWIIteratorImpl::AdvanceKey.

(b) The newest layer had no reference at all. The batch being written was chained on the assumption that it lives as long as the store. It does not: at a commit barrier it rotates into "pending", and that epoch's commit frees it — under any iterator opened before the rotation. The crashes landed in WBWIIteratorImpl::Entry(), BaseDeltaIterator::Valid and the batch's entry comparator; the non-segmented window and session stores returned wrong results instead — in one run a window scan read 2 of its 1,024 entries.

(c) A writer and a reader on one batch. Writes into the current batch took the store's write lock; iterator steps read the same batch with no lock. WriteBatchWithIndex does not promise to tolerate that. Ten runs of ten read some keys twice — between 1,030 and 1,333 entries over 1,024 distinct keys, values intact — and one earlier run crashed.

(d) An append under a positioned iterator. Found while fixing the others: any append to a batch an iterator is positioned on can reallocate the buffer and corrupt the one entry the iterator is on, including an append between two steps that a per-step lock does not cover. Measured on both of StoatFlow's RocksDB bindings, JNI and the Foreign Function & Memory API, single-threaded and deterministic: the first value read back as ??.

Why none of it reached production

Every RocksDB key-value, window and session store StoatFlow builds sits behind a caching layer, and the caching layer commits through its own path: it never writes the raw batches. In production the batch being written is always empty. The one shape that remained reachable was an iterator chaining an empty batch that could be freed under it — undefined behaviour, never observed as a crash. The versioned and timer stores have no caching layer and do write their batches, but they build and consume their iterators inside the store's read lock, and every write, rotation and commit takes the write lock, so no batch changes or goes away under an open iterator.

We fixed them anyway. Every store StoatFlow builds is built on these raw stores, a future store might write the batches, and the engine's own commit path did append to a batch an iterator could hold — the changelog offsets go into the same batch as the data.

A test that has to crash

A use-after-free is undefined behaviour: it may crash, return the old bytes, return garbage, or work. A test that runs it in the JVM under test can take the test runner down with it. So each scenario runs in its own child JVM, with native access enabled and an error-file path set, and the test asserts what a stale read must break: the child read every entry correctly and exited 0. That only holds if freed memory is actually unusable, which is what the allocator settings below are for. A crash, a wrong read or a hang is red; the controls, without the triggering interleaving, are green.

The tests force their interleavings. For (a), the test swaps the per-epoch counter map for one whose remove parks until a new iterator has opened — a deterministic reproduction of a race that would otherwise need luck.

The first version of the (d) probe passed on Linux. It failed on macOS, where the allocator reuses freed memory at once, but glibc left the old bytes in place, so the dangling read returned the right value. The CI job that checks every registered red probe flagged it the only way it could: every non-control test passed, so the probe no longer proved anything. The fix was to make freed memory unusable on both platforms:

There is no jemalloc in the picture: the RocksDB JNI library does not link one, so the system allocator is what the scribbling has to reach. With it, the probe went red on both platforms, and the native crash frames from the child's error file are attached to the test report.

The fix

References travel with the batch. Every batch gets an id when it is created and keeps it from current, to pending, to committing. A rotation carries its references over by construction, and every reference change and the free are a single atomic update per batch — the counter can no longer be removed between a decrement and an increment.

One chaining helper instead of three. The key-value store and the non-segmented window and session stores each had their own copy of the layer chaining, and the copies had drifted: window and session also chained the committing batch without a reference. One shared helper now opens, references and releases the layers for all three. It chains only non-empty layers, so behind the caching layer a production scan builds one bare RocksDB iterator — no layers, no reference updates, no lock.

The commit writes a copy. The commit must stay one atomic write of data and offsets. When an open iterator holds the batch being committed, the commit copies the batch, appends the offsets to the copy and writes the copy — the batch under the iterator is never appended to. The FFM binding copies native to native; the JNI binding copies through a Java byte array.

A per-step lock only where it is needed. An iterator takes the store's read lock per step only if it chained a non-empty current batch, decided once when it opens. No store StoatFlow builds chains one, so production iterators take no lock.

Tradeoffs and limits

One residual stays open, by decision. A raw put or delete into the current batch between two steps of an open iterator still corrupts the positioned entry. Fixing it would mean building an indexed copy of the batch whenever a write lands under an open iterator, a cost paid for a path no supported topology takes. No supported path reaches it, and a still-red test pins it, so CI notices if anything ever does.

The one user path is now refused. A custom StateStoreBuilder whose build() returns a persistent supplier's get() — for example Stores.persistentKeyValueStore("counts").get() — is refused at startup with a TopologyValidationException. Such a store never took part in the epoch lifecycle: it never committed or wrote its changelog, so its state was lost on restart anyway. Register the supplier through Stores.keyValueStoreBuilder(supplier, keySerde, valueSerde), the window or session equivalent, or Materialized.as(supplier). In-memory and LRU suppliers are not affected.

A green probe proves the scenarios, not the absence of others. The child-JVM harness makes these four interleavings reproducible. It does not search for new ones.

Read on:

• State stores — suppliers, builders and caching.
• Exactly-once — the commit barrier and the epoch a batch belongs to.
• Compiling StoatFlow to a GraalVM native image: G1, PGO, and why JNI beats FFM under AOT — the two RocksDB bindings.
• KIP-1035: Why Kafka Streams 4.3 lets state stores own their offsets — why data and offsets go into one atomic write.