batuhankucukali/spring-boot-cdc-example-with-debezium

Spring Boot Change Data Capture Example with Debezium (PostgreSQL + Debezium + Kafka)

Spring Boot Change Data Capture Example with Debezium (PostgreSQL + Debezium + Kafka)

Debezium

Debezium is an open source distributed platform for change data capture. Start it up, point it at your databases, and your apps can start responding to all of the inserts, updates, and deletes that other apps commit to your databases. Debezium is durable and fast, so your apps can respond quickly and never miss an event, even when things go wrong.

Technologies

• Spring Boot

Prerequisites

• Git
• JDK
• Maven
• Docker

Run this project

Note

Modify the KAFKA_ADVERTISED_HOST_NAME in docker-compose.yml to match your docker host IP (Note: Do not use localhost or 127.0.0.1 as the host ip if you want to run multiple brokers.)

Set your local ip to env variable.

export KAFKA_ADVERTISED_HOST_NAME=$(ifconfig | grep "inet " | grep -Fv 127.0.0.1 | awk '{print $2}')

Check the env variable

echo $KAFKA_ADVERTISED_HOST_NAME

Steps

1 . Clone project on your machine

git clone [email protected]:BatuhanKucukali/spring-boot-cdc-example-with-debezium.git

2 . Change directory

cd spring-boot-cdc-example-with-debezium

3 . Run Dependencies

docker-compose up

4 . Run Projects

cd customer-api
./mvnw spring-boot:run

cd customer-projection-consumer
./mvnw spring-boot:run

5. Create Connector

curl --location --request POST 'http://localhost:8083/connectors' \
--header 'Content-Type: application/json' \
--data-raw '{
    "name": "customer-connector",
    "config": {
        "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
        "tasks.max": "1",
        "database.hostname": "postgres",
        "database.port": "5432",
        "database.user": "postgres",
        "database.password": "postgres",
        "database.dbname": "user",
        "database.server.name": "master-db"
    }
}'

You can also check api-doc

6. Create User with API

curl --location --request POST 'http://localhost:7010' \
--header 'Content-Type: application/json' \
--data-raw '{
    "firstName": "John",
    "lastName": "Doe",
    "email": "[email protected]"
}'

You can also check api-doc

7. Check Database

customer-api database (PostgreSQL)

id	first_name	last_name	email
1	John	Doe	[email protected]

customer-projection-consumer database (MariaDB)

id	first_name	last_name	email
1	John	Doe	[email protected]

Sources

• https://debezium.io
• https://github.com/debezium/debezium-examples/tree/master/tutorial
• https://github.com/codefreshdemo/example-springboot-kafka
• https://reflectoring.io/spring-boot-kafka/
• https://github.com/codefreshdemo/example-springboot-kafka

Getting help

If you're having trouble getting this project running, feel free to open an issue or email me!