A mini kafka flink demo stack with avro as the serialization format based on https://github.com/garystafford/flink-kafka-demo/
- Basic development env setup (Ubuntu guest os, docker engine, git, maven/gradle, ide, java, python3)
- Bring up the mini dev stack with docker-compose (the stack should work both in x86-64 and ARM64(apple silicon chip))
docker-compose -f docker/flink-kafka-stack.yml up -d
- Run the python data generator script to simulate data ingestion
cd streaming-data-generator/sales_generator python3 ./producer.py
- Build the Kafka-Flink-demo job with cradle to generate fat/uber jar to be deployed to the Flink cluster
cd flink-kafka-demo ./gradlew clean shadowJar
- Deploy Flink jobs via shell scripts
./deploy.sh
- Submit flink jobs
./submit-join-streams-job.sh ./submit-running-totals-job.sh
- Go to http://localhost:9080 to check the running status of the single-node kafka cluster
- Go to http://localhost:9081 to check the running status of the flink cluster
The stack consists of a single-node Kafka cluster with KRaft(without ZooKeeper), a schema registry service instance, a Flink Job Manager instance, a Flink Task Manager instance, and a Kafka UI service
The serialization format used is βAVROβ
The use case is to about joining two data streams by productId that come out of Kafka and sending the enriched messages to another Kafka topic