Flink Dynamic Kafka

Flink discovers PicoMQ streams by prefix, consumes them over the Kafka listener, and writes one file bucket per topic. Streams added or removed while the job runs are picked up on the next poll.

Source: examples/connectors/flink-dynamic-kafka.

Run

cd examples/connectors/flink-dynamic-kafka
docker compose up -d --build
./seed.sh
./verify.sh a b

Service	Host
PicoMQ	http://localhost:9090
Kafka listener	localhost:9092
Files	./out/fleets.<id>/

--build compiles the Flink job image.

Job

flink-job/src/main/java/picomq/example/IngestionJob.java:

Piece	Setting
Discovery	HttpKafkaMetadataService polls GET /?prefix=/fleets. on the Pico HTTP listener
Source	DynamicKafkaSource, bootstrap pico:9092, allow.auto.create.topics=false
Sink	FileSink under ./out, bucket by topic name
Checkpoint	2s, so files appear without a shutdown

Add a stream

./add.sh          # pico append /fleets.c
./verify.sh a b c

Topic fleets.c appears after the next poll.

Remove a stream

./remove.sh       # pico delete /fleets.a, append to /fleets.b
./verify.sh b c

Topic fleets.a is dropped after the next poll. fleets.b keeps receiving.