How we obfuscate StoatFlow — and what the public-API boundary taught us

TL;DR

• What we obfuscate: the engine internals in our core module — not the public API.
• How: a rename-only pass, no shrinking and no optimisation, so the bytecode still behaves identically and stack traces still map back to source.
• The hard part: the boundary. A rename pass is indiscriminate; the public API has to be held still on purpose, and getting that list wrong breaks customers in ways your in-repo tests never see.
• The fix that stuck: stop testing against your own source tree. Test against the obfuscated artefact you actually ship.
• On protection: rename-only obfuscation is a deterrent and a legal signal, not a wall — and in the AI age that distinction matters more, not less.

Obfuscating a library is one line in a build file. Keeping a byte-stable public API in front of the obfuscated code is the part that took ten iterations to get right.

StoatFlow ships a Kafka Streams-compatible DSL — you swap an import and recompile — over an engine that is the actual product. The DSL surface has to stay exactly where customers expect it, method for method, or their code stops compiling. The engine beneath it has to be unreadable in the published jar. Those two goals pull in opposite directions, and every interesting bug we hit lived on the seam between them: public types that vanished from the shipped jar, a constructor the bytecode verifier rejected, a test library that linked the wrong artefact. None of them showed up in our own test suite. All of them showed up the moment someone compiled against what we actually published.

What we protect, and why

StoatFlow runs a Kafka Streams topology on a single instance, using JDK virtual threads for parallelism and Flink-style commit barriers for exactly-once — the engineering we describe elsewhere as cutting the distribution tax. The DSL you write against is deliberately the Kafka Streams DSL; the value we sell is what happens underneath it.

That split maps cleanly onto what gets obfuscated. The public API — the DSL, the configuration types, the processor and state-store interfaces you compile against — must stay legible and stable to the byte. The engine internals must not be self-documenting from the jar. Rename them, and reading the published artefact tells you nothing about how the thing works.

The problem is that a rename pass does not know which side of that line a class sits on. It renames everything by default. So the entire exercise becomes drawing one precise boundary: hold the public surface completely still, and let the tool mangle the rest. Draw it a hair too wide and you leak internals into the jar; a hair too narrow and you rename something a customer was relying on. We drew it wrong, in both directions, more than once.

How the obfuscation works

We use ProGuard, in obfuscation-only mode: renaming on, shrinking off, optimisation off. That combination is deliberate. Shrinking removes code the tool believes is unreachable; optimisation rewrites method bodies, inlines, and restructures. Both are destructive in ways that cost us more than they buy. They complicate stack traces, they can perturb the JIT's view of the code, and — for a Kotlin library — they risk invalidating the metadata the compiler emits for reflection and interop. Name mangling is what actually hides the architecture. The rest is downside without upside, so we leave it off. The published bytecode is byte-for-byte equivalent in behaviour to what we built; only the names change.

What we hold still falls into a few categories, described here at the level that matters:

• The public API packages — the DSL, config, processor, exception, watermark, and metrics surfaces a customer names in their code. These stay exactly as written.
• A cross-module SPI. StoatFlow is split into several published modules, and some of them — the runtime wrapper, the test driver — reference a small shared set of engine-facing types. Bytecode records those references by name, so that shared set has to survive un-renamed or the modules stop resolving each other at a customer's runtime. We carved those types into their own package precisely so the keep rule for them is one clean glob rather than a scattering of exceptions.
• Interop invariants. Kotlin metadata, companion-object members, and the annotations that make static-style access work from Java all have to be preserved, or the jar compiles but reads wrong from the other language.
• Attributes for stack-trace fidelity. We keep the attributes that let a stack frame point back at a source position, even though the names in that frame are mangled.

That last point is where the mapping file comes in. Renaming would make production stack traces useless — every frame a soup of single letters — so ProGuard emits a mapping from original names to obfuscated ones. We keep that file private: it is never committed, never shipped inside any jar, never published to the repository. It lives in a private store with indefinite retention, and when a customer sends us an obfuscated trace we run it back through retrace against the mapping for that exact release to recover real class names, methods, and line numbers. The jar stays opaque; support stays possible.

One more leak to close: source and documentation. Obfuscating the bytecode is pointless if the sources jar or the generated API docs ship the same class names in clear text. So both are filtered to the public API and the cross-module SPI, and the documentation tool is told to suppress the internal packages outright. What the bytecode pass hides, the docs pass must not hand back.

There is a version constraint worth flagging for anyone attempting this on a modern stack: an obfuscator has to understand both your bytecode version and your Kotlin metadata version, and support lags the language. Recent Java and recent Kotlin needed a recent-enough ProGuard; older releases simply did not recognise the class format and refused to run. Pin the version, and treat bumping it as a change that needs re-verifying, not a routine dependency update.

The hard part: the public-API boundary

The mechanics above are the easy 20%. Here is the 80% — four ways the boundary bit us, each one invisible in-repo and obvious in the shipped jar.

The picture below is the boundary as it sits in the published jar. The distinction that matters runs through the top band: a region is held still either by a package glob, which sweeps every public type in a package including the nested ones, or by a named list, which holds exactly what it names and nothing else. Both of the first two failures below landed in list-kept regions, and neither could have landed in a glob-kept one.

Types that vanished from the jar. Our keep list started as a hand-curated enumeration of public types. It was incomplete. A batch of public state-store types — iterators, the read-only store interfaces, a versioned-record type — were never listed, so the rename pass quietly renamed them. In our own build everything compiled and every test passed, because our tests compiled against the raw, un-obfuscated classes. A customer who swapped their imports and compiled against the published jar got "cannot find symbol" for a type that, as far as our CI was concerned, existed and worked. The types were present in the jar — just under a name nobody could write.

The nested-class trap. A keep rule that names one class and says "hold it and all its members" does not hold that class's nested types. We had a public status enum nested inside a public class, kept — we thought — by the wildcard on its parent. It was renamed anyway, its constants along with it, because nested types are a separate scope that wildcard never reached. Ported code calling the enum by name stopped compiling. The fix is to name each nested type explicitly, with its qualified $-name — and, more usefully, to test the public surface by naming specific constants, not just checking that the parent class survived. Worth knowing which rules are exposed: a package glob does sweep nested types along with their parents, so the packages kept that way were never at risk. The types that broke were the ones held by a hand-maintained list — which is the same root cause as the bug above.

A constructor the verifier rejected. One type on the public surface — our Kafka Streams-compatible configuration object — had grown an unusually wide constructor, over a hundred parameters. After the rename pass, ProGuard recomputed the stack-map frames for that method and got them wrong at that width. The result was bytecode the JVM refused to load at all — VerifyError: Inconsistent stackmap frames, under the default verifier, not only a strict one. It stayed invisible for the same reason as the first bug: our tests ran against the raw classes, which were never rewritten and so never carried the recomputed frames. It compiled, it passed every test, and it would not load. We fixed the immediate problem by grouping the configuration into a set of smaller typed objects so no single constructor was wide enough to trip the recomputation, and the deeper problem by class-loading the shipped type under strict verification in CI. The general lesson: kept does not mean untouched. The pass still rewrites the methods it keeps, and bytecode transformation is fragile at the extremes of what the format allows.

The wrong artefact linked. Our test-driver module is compiled against the engine. In-repo, that meant the raw engine. At publish time, the module was linked against the obfuscated engine — and it referenced internal types by their original names, which no longer existed. A customer pulling in the test driver got NoSuchMethodError and NoClassDefFoundError at class-init time. The real fix was structural: we reworked the test driver so it no longer reaches into engine internals at all — it runs on in-memory stores and talks to the engine through a small kept interface — which both removed the fragile references and made the driver simpler. The skew was the tell: any time module B is built against a different form of module A than the one it ships beside, you have a publish-time consistency bug waiting.

The fix that stuck: test against the artefact you ship

Every bug above has the same shape. It was invisible in the repository because the repository tests the source, and it was live in production because customers link the artefact. Passing tests locally proved nothing about the thing we shipped.

So the durable fix was not any one keep rule. It was moving the truth-source. We added a set of build gates that operate on the obfuscated jar, not the raw classes:

That last gate is worth dwelling on, because it is a trap the other three don't cover. A rename pass rewrites references it can see — a method call, a field access in bytecode. It cannot rewrite a field name you pass as a string to a reflective lookup, because to the tool that's just a string. We had exactly this: a diagnostics endpoint that resolved fields by their literal names. The rename pass renamed the fields and left the strings untouched, so the lookup failed on every shipped build. Nothing but loading the obfuscated class and running the path would have found it.

The framing we settled on: the keep boundary is a public API, and it deserves the same treatment as any other API — an explicit contract, tested against the real artefact, on every build.

Compatibility is a contract at the source and the binary level

Obfuscation was the sharpest version of a lesson that ran through the whole compatibility effort: matching another library's API exactly is deceptively hard, and the compiler passing is not the same as the contract holding. A few of the non-obfuscation ones, because they generalise:

• Kotlin properties are not Kafka Streams accessors. A Kotlin val value compiles to a Java getValue(). Kafka Streams exposes a bare value(). On a value type like ValueAndTimestamp, that mismatch means ported code calling .value() simply doesn't compile. The mapping from Kotlin property to Java getter is one-way and invisible to the consumer; if you're matching another library's naming, you have to add the bare-method aliases explicitly and check them.
• Variance is part of a functional interface's contract. Kafka Streams declares its functional interfaces with bounded wildcards — ValueJoiner<? super V1, ? super V2, ? extends VR>. Ours were invariant. Inline lambdas compiled fine, because Java infers around it; a pre-typed functional object — the kind real ported code reuses across a helper layer — did not. The fix was declaration-site variance on the Kotlin interfaces, which the compiler turns into the matching Java wildcards for free.
• A test harness has to be faithful to the data path, not just the control path. Our in-memory test driver originally handed records straight to the topology without serialising them — the serdes you passed in were accepted and then ignored. Around half the driver's tests were passing without ever exercising a serde. The real engine serialises at every source and sink boundary, so we made the driver do the same. It was a large migration, and it turned a pile of false green into real coverage of the exact boundary where serde bugs live.

Two smaller ones in the same vein: a windowed aggregation's Materialized should take the plain key type the user actually writes, not the framework's internal windowed representation; and null-key records have to be dropped consistently across every aggregation operator, not just the ones you remembered. Each is minor on its own. Together they make the point — API parity is a hundred small contracts, and any one of them left implicit is a place the compatibility quietly frays.

Why obfuscate at all, in the AI age

Rename-only obfuscation raises the cost of reading a shipped jar. It does not remove the possibility. A decompiler has always turned bytecode back into readable structure; what obfuscation takes away is the names — the part that makes the structure quick to understand. Historically that was enough friction to matter, because renaming thousands of symbols back into something meaningful by hand is slow, dull work.

AI tooling has lowered that cost again. The tedious part — inferring what a mangled class probably does and giving it a sensible name — is exactly the kind of task these models are good at. So the honest position is the same one it always should have been: obfuscation is one layer among several, not a wall. It deters casual reading and it demonstrates that we treat the engine as a protected asset — which is a real, and increasingly the more durable, benefit. The protections we actually lean on are commercial and legal: the licence terms, and the fact that reconstructing a snapshot of today's engine is not the same as being able to ship and support it. We keep the bytecode layer because it is cheap and it raises the floor, not because we imagine it is the ceiling.

What it costs

Obfuscation is not free, and the costs are worth naming plainly.

• Release complexity. There is a whole verification stage that only exists because we obfuscate — the gates above, plus the discipline of building them against the artefact rather than the source.
• Opaque stack traces. Every production trace that touches an internal frame needs deobfuscating through the private mapping before it means anything. That is an operational tax on every support interaction, and it depends on retaining the mapping for every release forever.
• A permanent balancing act. The keep boundary has two failure modes, and they pull against each other:

There is no setting that is safe in both directions. The only safety is the contract test on the real artefact.

And the ceiling is real. Rename-only is a deterrent, and heavier obfuscation — string encryption, control-flow flattening — buys time rather than immunity, while fighting the rest of a modern build (ahead-of-time native compilation, in particular, does not take kindly to it). For a library whose value is an architecture rather than a single secret algorithm, rename-only at the right boundary is the proportionate choice. If your value is one hot loop that must never be read, this is not the technique you want.

Where this leaves us

If you take one thing from this: the moment you transform your artefact after building it — obfuscation, shrinking, shading, any of it — your source tree stops being the truth. Test against the thing you ship. Treat the boundary between what you hide and what you expose as an API in its own right, with its own contract tests, run on the published bytecode. And be clear-eyed about what the hiding buys you — a deterrent and a signal of intent, not a guarantee — so you invest the rest of your protection where it actually holds.

Read on:

• StoatFlow: Kafka Streams compatible engine built to scale up — not out
• ProGuard manual
• ProGuard retrace — deobfuscating stack traces
• The Java Virtual Machine Specification — stack-map frames and verification