This is an example project that demos that it is possible to integrate Kafka Streams, Confluent Schema Registry, and Avro4s.
We have a simple Kafka Streams application, for which serialization to Kafka will be done by the KafkaAvroDeserializer
and KafkaAvroSerializer
, which leverage the schema validation in Confluent Schema Registry.
We use Avro4s to easily add Avro support to our Scala case classes.
During this demo, we assume that an installation of Confluent Platform is installed in $CONFLUENT_HOME
- Start up Zookeeper
$CONFLUENT_HOME/bin/zookeeper-server-start $CONFLUENT_HOME/etc/kafka/zookeeper-properties
- Start up Kafka
$CONFLUENT_HOME/bin/kafka-server-start $CONFLUENT_HOME/etc/kafka/server.properties
- Start up Schema Registry
$CONFLUENT_HOME/bin/schema-registry-start $CONFLUENT_HOME/etc/schema-registry/schema-registry.properties
- Boot Writer application
- Start up ordinary console-producer and produce on input topic, and publish some messages in the
key;value
format
$CONFLUENT_HOME/bin/kafka-console-producer --topic values --broker-list localhost:9092 --property "parse.key=true" --property "key.separator=;"
key;value
- Use avro-console-consumer to consume from output topic
$CONFLUENT_HOME/bin/kafka-avro-console-consumer --topic avro-values --from-beginning --zookeeper localhost:2181
- Note that the schemas have been registered at Schema Registry
curl localhost:8081/subjects/
- Boot Reader application
- Start up ordinary console-consumer to consume from second output topic
$CONFLUENT_HOME/bin/kafka-console-consumer --topic upper-case-values