Compiling StoatFlow to a GraalVM native image: G1, PGO, and why JNI beats FFM under AOT

TL;DR

• Native image is viable today — but only with Oracle GraalVM + G1 + PGO. GraalVM Community Edition ships Serial GC only, and Serial GC can't carry an allocation-heavy stateful workload.
• We blamed FFM; it was the GC. A stateful app that "couldn't keep up" under native looked like a RocksDB FFM problem. Switching Serial → G1 fixed it outright, to JVM parity. The bottleneck was the garbage collector, not the foreign-function path.
• JNI beats FFM under AOT — the reverse of the JVM. On HotSpot the JIT intrinsifies FFM's critical(false) downcalls; SubstrateVM has no JIT, so a plain JNI call ends up cheaper. This drove a new AUTO backend default: FFM on the JVM, JNI under native image.
• The cost, across all four apps: CPU within −4% to +10% of the JVM (the Avro stateful app is cheaper), latency equal-or-better everywhere, after PGO. Down from +42% / +75% on the Serial floor.
• Memory is tunable, not free. Serial native is tiny (78 MB RSS for a stateless app vs 496 MB on the JVM). G1 sizes its heap from -Xmx and sprawls unless you pin it; the stateful app still lands 23% below the JVM.
• The serde / security surface is still being widened — Avro + Schema Registry, SASL, OAuth, mTLS, and non-zstd codecs all need native metadata we haven't fully mapped yet.

We've been building and running StoatFlow applications as GraalVM native images. For a throughput-first stream processor the interesting question isn't the one native image is usually sold on — cold-start and footprint — it's whether steady-state CPU and tail latency hold up under sustained load. They do: with Oracle GraalVM, G1, and Profile-Guided Optimisation, the four apps we measured run within −4% to +10% CPU of the JVM, latency equal-or-better everywhere — after one wrong turn worth the telling.

Where we started

A StoatFlow app doesn't begin from a blank GraalVM configuration. The framework already bundles native-image support, discovered automatically the moment an app depends on it:

• :core ships the metadata for everything the engine touches — the Kafka serdes, assignors and partitioners, and the RocksDB API behind both backends: the JNI bindings and the FFM foreign downcalls to RocksDB's C API, plus JCTools and zstd. GraalVM finds it by classpath (META-INF/native-image/io.stoatflow/core/), so any dependent app inherits it.
• :runtime adds the configuration layer — the Hoplite YAML binding for the runtime config classes, Logback, Micrometer/HdrHistogram, and the rest of the runtime stack.
• The io.stoatflow Gradle plugin turns it on in one line: nativeImage { enabled = true } applies GraalVM's build plugin, the --initialize-at-run-time flags that RocksDB and the Kafka SASL client need, and a nativeDockerBuild task that drives a multi-stage Dockerfile.native.

So compiling a StoatFlow app to a native image is a one-line opt-in: the framework covers its own reflective, JNI and FFM surface, and the only per-app work is registering the metadata for your own serde value classes. That gets you a binary that runs. This post is about the next question — does it run well? The honest answer runs through a wrong turn, a reversal, and a memory result you have to tune for.

We benchmarked four apps; the deep dive that follows uses two representatives:

• stateless-simple — a stateless 1 KB transform at 20K msg/s. No state stores.
• word-count — stateful, RocksDB-backed, ~300K records/s processed (7.5K msg/s in, fanned out by aggregation).

Both ran under exactly-once semantics, four configurations each, 10-minute warmup + 10-minute measurement on a Hetzner ccx33 worker (8 dedicated vCPU); the full four-app matrix closes the post. What follows is the honest version, wrong turns included — because the wrong turns are where the interesting engineering lives.

Act 1 — The floor: GraalVM CE and Serial GC

GraalVM Community Edition only ships Serial GC for native image. G1 and the parallel collector are Oracle-GraalVM-only. So the first build we measured had no choice in collector — Serial it was.

The stateless app worked, but at a price: +42% CPU vs the JVM (1.7 cores vs 1.2). Tolerable, if not free.

The stateful app didn't work at all. It couldn't keep up:

That's a 112× regression in end-to-end latency and a 175× regression in lag. The app was falling behind the input and never catching up.

The obvious suspect was the RocksDB FFM (Foreign Function & Memory API) backend. StoatFlow talks to RocksDB's C API through FFM downcalls, and native image has to do special gymnastics to make those work (more on that below). It's the most exotic part of the native build, and when the workload that touches state explodes, you blame the exotic thing.

We were wrong.

Act 2 — The reversal: Oracle GraalVM and G1

Oracle GraalVM is now free for production use under the GFTC licence, and it unlocks two things CE doesn't: the G1 garbage collector and PGO (Profile-Guided Optimisation).

We rebuilt the stateful app with nothing changed but --gc=G1:

G1 fixed it outright. 128 ms P50, 91 lag — that's JVM parity. The 112× latency blowup was gone with a one-flag change to the garbage collector.

So the "FFM is the bottleneck" hypothesis was simply false. The stateful failure was GC-bound, not FFM-bound. Serial GC, single-threaded and stop-the-world, was choking on the allocation churn of the stateful path (RocksDB read/write buffers, aggregation state, serialisation). G1's concurrent and parallel collection kept up where Serial stalled. The FFM path had been carrying the heat for the GC the whole time.

This is the beat worth slowing down on: the exotic component is rarely the bottleneck. The boring component — the collector that runs on every allocation — was. We reached for the interesting explanation and missed the mundane one. The benchmark matrix is what caught it, not intuition.

One caveat we name in the report: this comparison mixes two variables. Serial was GraalVM CE; G1 was Oracle GraalVM, which also applies optimisations CE lacks. So "CE-Serial → Oracle-G1" is edition and collector. The actionable comparison is JVM vs the best shippable native build — and that's what every headline number here uses.

Act 3 — The CPU lever: PGO

G1 brought latency and throughput to parity, but CPU was still high: +33% (stateless) and +45% (stateful) vs the JVM. The JVM's advantage here is the JIT — it profiles the running code and recompiles hot paths with real branch and type information. A native image is compiled ahead of time with static heuristics; it never sees the workload.

PGO closes that gap. It's a two-pass build:

1. Compile an instrumented binary (--pgo-instrument).
2. Run it under representative load to collect a .iprof profile.
3. Rebuild with the profile (--pgo=<app>.iprof) so the AOT compiler gets the hot-path inlining and code layout the static heuristics miss.

PGO is purely a CPU lever — it doesn't touch latency or throughput, which were already at parity. It cut CPU:

That's −13% and −17% on top of G1. The residual gap to the JVM lands at +17% (stateless) and +20% (stateful) — down from the Serial floor of +42% / +75%. That residual is the irreducible AOT-vs-JIT peak-code difference, plus the FFM downcall path under SubstrateVM, which is the next act.

Act 4 — The counterintuitive finding: JNI beats FFM under AOT

Here's the part that inverts conventional wisdom.

On the HotSpot JVM, StoatFlow's FFM RocksDB backend is faster than the JNI one. The reason is Linker.Option.critical(false) — it marks a downcall as not needing a full safepoint transition, and the JIT intrinsifies it, compiling the foreign call down to nearly a direct C call. FFM on a warm JVM is genuinely excellent.

Under SubstrateVM (native image), there is no JIT. That intrinsification never happens — there's no profiling compiler to perform it. So the critical(false) optimisation that makes FFM win on the JVM simply doesn't fire. And once you strip it away, an FFM downcall ends up slower than a plain JNI call. (Both pin the virtual thread for the duration of the native call anyway, so there's no Loom advantage either way.)

We measured it on word-count, G1+PGO, swapping only the RocksDB backend:

JNI uses ~8% less CPU (2.2 vs 2.4 cores) with a better P99 (211.8 vs 226.9 ms, −6.7%) and marginally higher throughput. The optimisation that's correct on one runtime is wrong on the other.

So the fix is to pick per-runtime. We added a new RocksDbBackendType.AUTO default that resolves to:

• FFM on the JVM — where the JIT intrinsifies it and it wins.
• JNI under a native image — where there's no JIT and JNI wins.

Each runtime gets its faster path automatically. This also has a pleasant side effect on the native build's fragility, which the snares section gets into: the FFM-under-native machinery is genuinely brittle, and AUTO takes it off the default path for native images entirely.

Memory: real, but GC-dependent

Native image's headline memory win is real — and more nuanced than the marketing suggests.

Under Serial GC, native is tiny. The stateless app's RSS was 78 MB vs the JVM's 496 MB — an 84% reduction. Serial keeps a compact heap and reclaims aggressively. That is a genuine, large win.

Under G1, it depends on the workload's live footprint, because G1 sizes its heap reservation from -Xmx — and the container entrypoint hands -Xmx almost the whole container (≈15.6 GB). So G1 reserves generously:

• Stateless app (live heap ~40 MB): G1's reservation dominates RSS, which jumps to 587 MB — above the JVM's 496 MB, and far above Serial's 78 MB. The Serial memory win is gone.
• Stateful app (larger live footprint): G1's reservation is comparable-to-lower than the JVM's heap + metaspace + JIT code cache, so RSS lands at 606 MB — 23% below the JVM's 791 MB.

The takeaway: G1 native wants a tight -Xmx to keep the memory win. The stateless app needs ~40 MB of heap; giving G1 15 GB to reserve from is the whole problem. That's a tuning follow-up — an env-var override, no rebuild — and we haven't done it yet. So the honest statement is: native memory is a win you tune for under G1, not one you get for free.

The snares

These are the discoveries — the things that cost an afternoon each and that you only hit because native image moves failures from compile time to first-record time. Closed-world compilation means anything reflective, JNI, or FFM must be declared up front; miss one and it surfaces as a runtime error the first time that code path executes, never at build.

glibc version pinning

Oracle's default native-image:25 builder image is built on Oracle Linux 10 (glibc 2.39). The binary it produces then requires GLIBC_2.38 at runtime. But the distroless cc-debian12 runtime image ships glibc 2.36 — so the binary crashloops on startup with version GLIBC_2.38 not found.

Fix: pin the builder to the :25-ol9 tag (Oracle Linux 9, glibc 2.34). 2.34 ≤ 2.36, so the binary runs. The rule: a native binary requires the build host's glibc version or newer at runtime — build on the oldest glibc you can, run anywhere newer.

reachability-metadata.json silently shadows the legacy reflection-config.json

GraalVM has two metadata formats: the older reflection-config.json and the newer reachability-metadata.json. When both are present, the newer file wins — and silently drops the old file's registrations, including the allDeclaredConstructors: true on our runtime config data classes. The symptom: a runtime MissingReflectionRegistrationError on a config class that was registered, in the file that used to win.

Fix: consolidate the registrations into the reachability-metadata file. Don't run the two formats side by side and expect them to merge — they don't.

Everything reflective must be declared, and you find out at first contact

Each of these surfaced only when its code path first executed, never at build time:

• JCTools lane-queue field-holder classes (accessed via Unsafe) → NoSuchFieldException
• HdrHistogram ConcurrentHistogram → NoSuchFieldException
• zstd-jni native field handles → NoSuchFieldError
• Kafka serialisers instantiated by class name → NoSuchMethodException

There's no shortcut here other than discipline: a closed world means the failure budget moves entirely from compile time to first-record time. Drive every code path before you trust the binary.

FFM under native image needs hand-rolled symbol resolution

This is the brittle one. SymbolLookup.loaderLookup() — the normal way to find a native function — can't see the symbols of a JNI library loaded at runtime under SubstrateVM. librocksdbjni is loaded by RocksDB's JNI bootstrap at runtime; the FFM backend then can't resolve rocksdb_* symbols against it.

The workaround:

1. Detect native image via the org.graalvm.nativeimage.imagecode system property.
2. Read the library's mapped path out of /proc/self/maps, then libraryLookup it explicitly.
3. --initialize-at-run-time for the FFM holder class.
4. Register every foreign downcall descriptor — and the leaf type includes the implicit function-pointer first argument, which is trivially easy to miss on one of forty descriptors.

It works, but it's fragile and Linux-only (the /proc/self/maps trick has no portable equivalent). The good news: the AUTO backend change (JNI under native) means this whole apparatus is off the default path now. It's there if you force FFM on native, but you have to ask for it.

Avro + Schema Registry is the worst offender

Confluent's Avro stack is reflection-heavy in three compounding ways, and native image trips on all of them:

• The serde config instantiates classes by name from its own defaults, even properties you never set — e.g. NullContextNameStrategy. Symptom: ConfigException: … could not be found at startup, for a class you've never heard of.
• The schema-id serialisers and the registry REST entities (Jackson-deserialised from the registry's HTTP responses) all need registering.
• Native image disables URL protocols by default, so the HTTP schema fetch fails until you add --enable-url-protocols=http,https.

Out of this fell a clean ownership pattern: keep serde reachability metadata in the app module, not in the engine. The app is the closed-world boundary. StoatFlow ships metadata for everything it touches — the engine, Kafka client, RocksDB (both backends), JCTools, Micrometer/HdrHistogram, the config layer. Customers add only their own value classes. The tracing agent catches the config-default classes in one pass and is the surer route for anything Avro. (Full recipe in the custom-serde native-image guide.)

Extracting a PGO profile from a server that never exits

PGO's instrumented binary only writes its .iprof profile on clean shutdown. There's no on-demand dump API. A streaming app runs forever and never exits on its own, so getting the profile out takes some plumbing.

The trick that's race-free:

• An entrypoint that runs the binary as a child process (not exec), and forwards SIGTERM to it — so kubectl exec -- kill -TERM 1 drains and dumps the profile.
• After the child exits, the entrypoint runs sleep infinity so the pod stays alive (and the liveness probe's eventual restart doesn't race the extraction).
• Dump to an emptyDir so the file survives that restart, then kubectl cp it out.

Drive ~4 minutes of representative load, kill -TERM 1, copy the .iprof, rebuild with it. Done.

Results

The full four-configuration matrix, both apps, JVM as the baseline. Every number traces to the benchmark report; deltas are vs the JVM.

stateless-simple (20K msg/s, stateless)

Throughput and latency parity (P99 actually beats the JVM by 9%) at +17% CPU. The memory catch is real: G1's heap reservation pushes RSS 18% above the JVM. Serial native's 78 MB is the memory-optimal build here; G1+PGO is the CPU/latency-optimal one.

word-count (7.5K msg/s in, ~300K/s processed, stateful — RocksDB)

Serial couldn't keep up (112× E2E latency). G1 fixed it to parity; PGO trimmed CPU from +75% to +20%. And because this app's live footprint is larger, G1's reservation lands below the JVM's total — so the stateful native image keeps a 23% memory advantage and keeps up.

FFM vs JNI backend, under native image (word-count, G1+PGO)

JNI is the better native backend on every axis that moves — which is exactly the inversion of the JVM, and exactly why AUTO resolves to JNI under native image.

All four apps under G1+PGO (with the AUTO backend)

We then re-benchmarked all four apps on the best config — G1 + PGO + AUTO (so the two stateful apps run on JNI). The two serde-heavy apps ran as native images for the first time; Protobuf and Avro + Schema Registry both processed cleanly under load. Numbers are native vs the JVM baseline:

That is the bottom line of this whole exercise. CPU lands between −4% and +10% of the JVM; latency is equal-or-better on every app (P99 −8% to −16% on three, parity on the fourth); and memory beats the JVM on both stateful apps (−20%, −24%), where the RocksDB working set is large enough that G1's reservation comes in under the JVM's heap + metaspace + code cache. The Protobuf app is at outright parity on CPU. Only the two tiny-heap stateless apps give back a little memory — the same G1 -Xmx sprawl, tunable. Throughput and lag are at parity throughout.

The standout: the Avro stateful-joins app — the most complex of the four, with windowed joins over five Avro streams — runs leaner than the JVM on CPU, memory, and tail latency at once. A long way from "native can't do stateful."

Smaller on disk — and what the build costs

There's a footprint win too, the one native image is actually known for. word-count's native image is 147 MB on disk against 222 MB for the JVM build — a 34% reduction. The other three apps land within a few MB of that same split (the native binaries are near-identical in size, ~147–155 MB; the JVM images ~222–233 MB), so one number stands for all four. Most of the native image is the GraalVM binary plus the distroless base; the JVM image carries a JRE and the shadow JAR.

What you pay for it is build time. A native PGO build is two passes — an instrumented compile, a few minutes of profiling load, then the optimised compile — each running minutes on a dedicated amd64 worker, where the JVM image is a shadow JAR plus a Jib layer in seconds. Call it a 10–50× build-time multiplier, on Oracle GraalVM with a buildx-on-Kubernetes builder to keep alive. It's a release-time cost rather than a per-change one — but it's real, and it's the honest other half of the footprint win.

Can FFM be fixed on native?

The honest answer, after digging into it: not from our code, not today. The slowness is GraalVM's, and Oracle says so — their FFM tracking issue (oracle/graal #8113) lists "improve downcall performance (currently always unoptimized)" as an open goal, undelivered in GraalVM 25, with no committed date, and they explicitly intend to keep JNI and the @CFunction C-interface as the performance-grade native call paths. SubstrateVM emits a generic downcall stub and has no JIT to intrinsify the critical() hint the way HotSpot's C2 does.

We tried the one lever that lives in application code: critical(true) (allow-heap-access), which lets the hot path pass the on-heap key directly instead of copying it off-heap first. It doesn't recover the intrinsification — it only saves the copy — and a RocksDB key is a handful of bytes, so we didn't expect much. It turned out to be worse than "not much": under native image critical(true) changes the downcall's leaf types, so it needs a whole new set of foreign-metadata registrations and crashes at startup without them. It's not a flag, it's a rebuild-per-missing-entry slog — and it leaves the actual cost, the unoptimised stub, untouched. The other option, a @CFunction no-transition binding, would match HotSpot's fast path, but it's native-image-only: a third call path, not "FFM everywhere."

So the foreign-function path doesn't have a cheap fix on native, and — this is the part that matters — it doesn't need one to ship today. JNI already gets the stateful apps to JVM-parity-or-better. AUTO (JNI on native, FFM on the JVM) is the pragmatic default that routes each runtime to its faster path without anyone having to know any of this.

But AUTO is a stopgap, not the destination. The goal is the single most efficient, most resource-frugal path on each runtime — and on the JVM that's FFM, so what we actually want is FFM everywhere, not a runtime-dependent split. The GraalVM team has named the route: the slow stub is what bound downcall handles (the jextract pattern we use today) compile to; an unbound handle — the function pointer passed as a call argument, the holder class initialised at build time — is reported to reach near-@CFunction speed under native image. There's no flag for it yet, and the registerForDirectDowncall API that would formalise it is proposed but uncommitted (oracle/graal #12219 / GR-75754). So what's next is concrete and not ours to ship: rewrite the RocksDB FFM backend to unbound handles, validate it on GraalVM 25.1+, and the day native FFM matches JNI, AUTO collapses to FFM everywhere — one path, the most efficient one. Until that API lands, AUTO routes each runtime to its current best — eyes open that "current best" isn't yet "best possible."

What's next

Native image is viable today for the surface we've covered: G1 + PGO + the AUTO backend get you JVM-parity-or-better latency and throughput, CPU within −4% to +10%, and memory that beats the JVM on the stateful apps. The work from here is widening that covered surface, and being honest that it isn't complete:

• Kafka client security isn't covered yet. SASL mechanisms (SCRAM, GSSAPI/Kerberos), OAuth bearer (OAUTHBEARER), and mTLS each pull in JAAS, login-module, and JCA reflection that will need native metadata. None of it is mapped yet.
• Compression codecs beyond zstd. snappy, lz4, and gzip each have their own JNI/native or reflective surface to register. Only zstd is covered today.
• More as real workloads exercise more of the Kafka client and serde surface. Closed-world compilation guarantees that every code path a customer hits for the first time is a potential MissingReflectionRegistrationError until we (or they) declare it.
• The G1 -Xmx tuning follow-up. A tight -Xmx should recover the stateless memory win that G1's default container-fill reservation currently gives away — cheap to do, not yet done.

The shape of the thing is clear: native image earns its keep for the covered workloads, the wrong turns taught us where the real costs live (the collector, not the foreign-function path), and the AUTO backend means you don't have to know any of this to get the faster path on each runtime. The surface we widen next is set by the workloads that hit it first — which serde and security paths a real deployment exercises is exactly what tells us what to map.