Scheduled sources

Generate records on an interval or a cron schedule with StreamsBuilder.scheduled() — a topology source that emits without consuming from Kafka.

StreamsBuilder.scheduled() is a StoatFlow extension — it has no Kafka Streams equivalent. It creates a topology source that periodically generates records and feeds them downstream, without consuming from any Kafka topic. Use it for heartbeats, periodic state-store scans, external polling, or any time-driven event you want to flow through the same DAG as your Kafka-sourced records.

Interval-based sources

The interval overload fires every Duration according to a TimeNotion:

  • WALL_CLOCK_TIME — fires on the system clock: at start-up, then every interval, regardless of data flow. Use for heartbeats, metrics, or polling that must happen on real-world time.
  • STREAM_TIME — fires on the event-time watermark: at the first watermark, then at every multiple of the interval in event time. Data-driven: with no records flowing, the watermark doesn't move and the source doesn't fire.

The interval must be positive (scheduled() throws IllegalArgumentException otherwise).

Fire times

  • A WALL_CLOCK_TIME source fires at start, then every interval after that.
  • A STREAM_TIME source fires at the first watermark, then each time the watermark passes a multiple of the interval — counted from event time 0, so an hourly source fires at each event-time hour, and the boundaries do not move when the application restarts. Nothing fires before the first watermark.
  • When time jumps past several intervals at once — a backlog replayed, a gap in the data, a record with a far-future timestamp — the source fires once, with the current time, and the next fire is the next point on its grid. A jump of a week fires it once, not once per interval.
  • A cron source fires at its occurrences, and also once per jump: if wall-clock time passes several occurrences at once (a long pause), it fires once and then at the next occurrence.
  • Start is when the application begins processing — start-up, or with high availability the promotion of a standby. A standby fires nothing, and its sources start when it is promoted.

Every start fires an interval source again at once — at start-up, or at that run's first watermark. That includes the restarts StoatFlow makes in place, not only a redeploy: after a fault handled with REPLACE_THREAD, for a poison-record replay, after a pattern-matched source topic is deleted, and at an HA promotion. Under exactly-once, a restart that aborted the open epoch has dropped that epoch's output, which the new fire replaces; otherwise it is one fire more.

A STREAM_TIME source fires at the first watermark whatever its sign. With small event times — smaller than the source's out-of-orderness bound, as in many tests — the first watermark is negative, and a record the source forwards carries that time. Kafka cannot store a negative timestamp, so a sink sends such a record without one and the broker stamps the wall-clock time, as for any negative event time.

To wait for the first point of the grid instead — one interval after start, or the first multiple of the interval at or after the first watermark (at once when the first watermark is one) — start the source at its next interval: per source with ScheduledSourceOptions, or for every source that does not set it with stoatflow.dsl.scheduled-source-start: next-interval (default immediately).

import io.stoatflow.core.processor.ScheduledStart
import io.stoatflow.core.topology.ScheduledSourceOptions

builder.scheduled<String, String>(
    interval = Duration.ofHours(1),
    type = PunctuationType.STREAM_TIME,
    options = ScheduledSourceOptions.start(ScheduledStart.NEXT_INTERVAL),
    emitter = { context -> context.forward("hourly", "report") },
)

Scheduled sources and punctuators share the once-per-jump rule and, for STREAM_TIME, the grid aligned to 0. A non-anchored STREAM_TIME punctuator also fires at the first watermark, whatever its sign; a WALL_CLOCK_TIME punctuator waits one interval, like a scheduled source started at its next interval.

import io.stoatflow.core.processor.PunctuationType
import io.stoatflow.core.topology.Produced
import io.stoatflow.core.topology.StreamsBuilder
import org.apache.kafka.common.serialization.Serdes
import java.time.Duration

val builder = StreamsBuilder()

// Emit a heartbeat every 30 seconds on wall-clock time
builder
    .scheduled<String, String>(
        interval = Duration.ofSeconds(30),
        type = PunctuationType.WALL_CLOCK_TIME,
        emitter = { context ->
            context.forward("heartbeat", "alive-${context.currentWallClockTime()}")
        },
    )
    .to("heartbeats", Produced.with(Serdes.String(), Serdes.String()))
For Java, the emitter is delivered as a java.util.function.Consumer<ScheduledEmitterContext<K, V>>, so a plain lambda works. The Kotlin lambda overload is hidden from Java (@JvmSynthetic) to avoid ambiguity. The named and keySerde parameters have Kotlin defaults; from Java, pass null to skip them.

Naming the source

Pass a Named to give the source a stable name in the topology graph and metrics. Without it, sources are auto-named scheduled-source-0, scheduled-source-1, … in declaration order — which is the name you'll reference when testing.

import io.stoatflow.core.processor.PunctuationType
import io.stoatflow.core.topology.Named
import java.time.Duration

builder.scheduled<String, String>(
    named = Named.`as`("heartbeat-source"),
    interval = Duration.ofSeconds(30),
    type = PunctuationType.WALL_CLOCK_TIME,
    emitter = { context -> context.forward("heartbeat", "alive") },
)

Cron-based sources

The cron overload fires according to a CronExpression. Cron scheduling always uses wall-clock time — there is no TimeNotion parameter, and cron sources do not fire on watermark advance.

import io.stoatflow.core.processor.CronExpression
import io.stoatflow.core.topology.Named
import io.stoatflow.core.topology.Produced
import org.apache.kafka.common.serialization.Serdes

// Fire every day at midnight (Unix 5-field cron)
builder
    .scheduled<String, String>(
        named = Named.`as`("daily-report-source"),
        cron = CronExpression.unix("0 0 * * *"),
        emitter = { context ->
            context.forward("report", "daily-${context.currentWallClockTime()}")
        },
    )
    .to("daily-reports", Produced.with(Serdes.String(), Serdes.String()))

Cron expressions

CronExpression wraps three cron dialects. Pick the factory that matches the dialect of your expression — the field counts differ, so an expression valid in one dialect is usually invalid in another. An unparseable expression throws IllegalArgumentException.

FactoryFieldsFormatExampleMeaning
CronExpression.unix(...)5min hour day-of-month month day-of-week"0 * * * *"top of every hour
CronExpression.quartz(...)6–7sec min hour day-of-month month day-of-week [year]"0 0 0 * * ?"every day at midnight
CronExpression.spring(...)6sec min hour day-of-month month day-of-week"0 0 0 * * *"every day at midnight

All three factories are @JvmStatic, so they're called identically from Kotlin and Java. Day-of-week numbering differs between dialects (Unix 0–6 with 0 = Sunday; Quartz 1–7 with 1 = Sunday), so consult the dialect when porting expressions.

CronExpression resolves the next fire time in the system default time zone by default. The next-execution helpers (nextExecutionTime, timeUntilNextExecution) accept an explicit ZoneId if you need a fixed zone.

The emitter and its context

The emitter is a ScheduledEmitter<K, V> — a single-method functional interface. Each invocation receives a ScheduledEmitterContext<K, V> for emitting records and reading state.

Forwarding records

The context offers three forward overloads:

MethodTimestampHeaders
forward(key, value)the fire time — the watermark that fired a STREAM_TIME source, the wall-clock time otherwisenone
forward(key, value, timestamp)explicitnone
forward(record: Record<K, V>)from the Recordfrom the Record

A single emitter invocation may forward zero, one, or many records. The key drives downstream routing, exactly as for Kafka-sourced records, so per-key ordering downstream behaves the same way — see Lanes & parallelism.

To pick a record's lane, StoatFlow serializes its key with the source's key serde — the keySerde you pass, else default.key.serde — so that serde must be able to serialize the keys you forward. A String key with neither set fails the fire (Scheduled source '…' could not serialize the key it emitted), and the test driver fails the same way. Such a failure, like an emitter that throws, goes to the processing exception handler with the synthetic __scheduled_source context: the default handler stops the application; a handler that returns CONTINUE writes its dead letters and the fire ends there, keeping what it forwarded before. A null key is not serialized. A Schema Registry key serde registers the key under the subject <application.id>-<source>-scheduled-key (Subjects StoatFlow registers).

An explicit timestamp can be any Long, negative included, except Long.MIN_VALUE, which marks a missing event time: forward(key, value, Long.MIN_VALUE), or a Record built with it, throws IllegalArgumentException (see Which event times are valid).

import io.stoatflow.core.processor.Record
import org.apache.kafka.common.header.internals.RecordHeaders

builder.scheduled<String, String>(
    interval = Duration.ofSeconds(10),
    type = PunctuationType.WALL_CLOCK_TIME,
    emitter = { context ->
        // simplest: key + value, stamped with the fire time (here the wall-clock time)
        context.forward("k1", "v1")

        // explicit event-time timestamp
        context.forward("k2", "v2", context.currentWallClockTime())

        // full control via Record (key, value, timestamp, headers)
        val headers = RecordHeaders().apply { add("source", "scheduled".toByteArray()) }
        context.forward(Record("k3", "v3", context.currentWallClockTime(), headers))
    },
)

Reading state stores

The context exposes getStateStore(name) for read-only access. Write operations (put, delete) on a store obtained this way throw UnsupportedOperationException. This makes the scheduled source a natural fit for periodically scanning state and emitting derived events — for example, scanning pending items and emitting a "publish" event once their scheduled time has passed.

The write refusal covers every store family. Plain key-value, window and session stores, and equally the timestamped, versioned and headers-aware ones — including the timestamped store a DSL Materialized aggregation or table produces, which is the common case for exactly the "scan state, emit derived events" pattern above. Whatever store type you ask for, you get an object of that type back, and its writes throw.

The store you get back keeps its declared type, so getStateStore<TimestampedKeyValueStore<String, Long>>("counts") returns something that really is a TimestampedKeyValueStore — reads work as usual, and only the write methods throw. This mirrors Kafka Streams, whose global-store decorators work the same way.

import io.stoatflow.core.state.ReadOnlyKeyValueStore

builder.scheduled<String, String>(
    named = Named.`as`("publish-scanner"),
    interval = Duration.ofMinutes(1),
    type = PunctuationType.WALL_CLOCK_TIME,
    emitter = { context ->
        val now = context.currentWallClockTime()
        val pending = context.getStateStore<ReadOnlyKeyValueStore<String, Long>>("pending-publishes")
        pending.all().use { it ->
            it.forEach { entry ->
                if (entry.value <= now) {
                    context.forward(entry.key, "published")
                }
            }
        }
    },
)

all(), range(), prefixScan(), and the reverse variants all return a KeyValueIterator<K, V>, which is Closeable — close it (Kotlin use {}, Java try-with-resources) to release resources. See State stores for the full read-only query surface.

For the time helpers: currentWallClockTime() and currentSystemTimeMs() return system time (under the TopologyTestDriver, both return the driver's wall clock); currentWatermarkMs() (also currentStreamTimeMs()) returns the global event-time watermark. For a STREAM_TIME source, currentWatermarkMs() is at least the watermark value that triggered the firing (it may have moved on while the emitter runs); for a WALL_CLOCK_TIME source it's whatever the global watermark currently is (Long.MIN_VALUE if no events have been processed yet).

Processing the result

scheduled() returns a KStream<K, V> — there's nothing special about it downstream. Map, filter, group, aggregate, join, or merge it with a Kafka-sourced stream, all with the operators from KStream & KTable.

val ticks = builder.scheduled<String, Int>(
    interval = Duration.ofSeconds(5),
    type = PunctuationType.WALL_CLOCK_TIME,
    emitter = { context ->
        context.forward("a", 1)
        context.forward("b", 2)
    },
)

ticks
    .filter { _, value -> value > 1 }
    .mapValues { value -> value * 10 }
    .to("output", Produced.with(Serdes.String(), Serdes.Integer()))

To combine a scheduled source with a Kafka topic, keep a reference to each KStream and merge them:

import io.stoatflow.core.topology.Consumed

val scheduled = builder.scheduled<String, String>(
    interval = Duration.ofSeconds(5),
    type = PunctuationType.WALL_CLOCK_TIME,
    emitter = { context -> context.forward("scheduled", "tick") },
)
val kafka = builder.stream<String, String>("input", Consumed.with(Serdes.String(), Serdes.String()))

scheduled
    .merge(kafka)
    .to("output", Produced.with(Serdes.String(), Serdes.String()))

Testing scheduled sources

The TopologyTestDriver drives scheduled sources deterministically — no clock waiting. Three controls:

  • triggerScheduledSource(name) — fires a source immediately, once, by name, without moving its schedule. With no Named, the auto name is scheduled-source-0, -1, … in declaration order. A record forwarded without a timestamp carries the current watermark for a STREAM_TIME source (the driver's wall clock before the first watermark) and the driver's wall clock otherwise.
  • advanceWallClockTime(duration) — advances the test clock; a WALL_CLOCK_TIME interval source fires on the first advance (it is due at start) and then once per advance that passes its next interval; a cron source once per advance that passes an occurrence.
  • advanceWatermark(timestamp) — advances the event-time watermark; the first advance of stream time fires a STREAM_TIME interval source, and later ones fire it once each time they pass a multiple of its interval. Under the default test.driver.commit-per-pipe, every pipeInput advances stream time to its record's timestamp too, so the first piped record can be the one that fires it — once for a jump over many intervals, advanceWatermark(Long.MAX_VALUE) included. Cron sources do not fire on watermark advance.

A source started at its next interval (ScheduledSourceOptions or dsl.scheduled-source-start in the driver's config) does not fire on the first advance — unless it is a STREAM_TIME source whose first watermark is a multiple of its interval. While the driver is paused (pause()), both advances move their clock and fire nothing; unpause() then fires each clock once at its current time, as the engine does when it resumes.

Pass initialWallClockTime to fromBuilder to pin the starting instant for cron and wall-clock assertions. In Kotlin it's a named, defaulted parameter. From Java, fromBuilder is @JvmOverloads over (builder, config, initialWallClockTime) — to set the instant you must also pass the StreamsConfig (there is no two-argument (builder, Instant) overload), so construct one explicitly.

import io.stoatflow.testutils.TopologyTestDriver
import java.time.Instant

val builder = StreamsBuilder()
builder
    .scheduled<String, String>(
        interval = Duration.ofSeconds(10),
        type = PunctuationType.WALL_CLOCK_TIME,
        emitter = { context -> context.forward("key1", "tick") },
    )
    .to("output", Produced.with(Serdes.String(), Serdes.String()))

val driver = TopologyTestDriver.fromBuilder(builder, initialWallClockTime = Instant.ofEpochMilli(0L))
try {
    val output = driver.createOutputTopic("output", Serdes.String(), Serdes.String())

    // Option A: fire once, immediately, by name
    driver.triggerScheduledSource("scheduled-source-0")
    output.readRecord()?.key shouldBe "key1"

    // Option B: advance the clock and let the schedule fire
    driver.advanceWallClockTime(Duration.ofSeconds(10))
    output.readRecord()?.value shouldBe "tick"
} finally {
    driver.close()
}

Execution semantics

A few behaviours worth knowing when you write an emitter:

  • One run at a time. Each scheduled source uses locked-run semantics: if the previous invocation is still running when the next fire is due, the fire waits for it, then fires once — at the time of the first clock tick or watermark advance after the run ends, however many points of the grid passed meanwhile. Keep emitters reasonably quick, or expect late, coalesced fires under load.
  • No Kafka source is required. A topology fed only by scheduled sources runs on its own, under both processing guarantees: it subscribes to no topic, and its output commits at the normal barrier cadence. A STREAM_TIME source is the exception — see below.
  • Records flow through the engine like any source. Emitted records participate in the same exactly-once commit boundaries as Kafka-sourced records — see Exactly-once semantics and the architecture overview.
  • State access is read-only. Use a scheduled source to observe state and emit derived events; mutate state from the processors that handle those events, not from the emitter. Enforced for every store family — plain key-value, window and session as well as timestamped, versioned and headers-aware (see Reading state stores).
  • STREAM_TIME is data-driven. A STREAM_TIME source only fires as the watermark advances, so it won't fire at all when no records are flowing. Choose WALL_CLOCK_TIME or a cron schedule when you need firing independent of throughput.
  • STREAM_TIME needs a watermark. An application with a STREAM_TIME source refuses to start under timestampCoordinationMode = DISABLED (event-time tracking off), when a source is read with WatermarkStrategy.noWatermarks() (which holds the watermark at its minimum), or when it reads no Kafka topic (the watermark comes from records) — the source would never fire. The refusal is a TopologyValidationException, and the TopologyTestDriver refuses the same topologies when it is built.

See also

Deep dives

  • Scheduled sources: a topology source that isn't a topicEngineering — Every source in a Kafka Streams topology is a topic. The calendar and your own state — a snapshot across keys, a sweep over what you hold — needed a job outside the topology or a punctuator bent into a source. StoatFlow's scheduled sources run your code on a schedule and turn what it forwards into an ordinary KStream: keyed, processed in parallel on the lanes, committed with the stream. What that opens up, how it works, two verified examples, and the catch — the emitter's view is read-only, a run does not yield to live traffic on its own, and a read from outside the engine is not in the transaction.