bellmit/vaccine-order-mgr-pg

Order management with postgresql db and debezium outbox

Vaccine Order Manager event-driven microservice

This service is responsible to manage the Vaccine Order entity for the Vaccine Delivery Solution. It is using the transactional outbox pattern to generate order events to Kafka while writing to its persistence layer, which is Postgresql here. It is implemented with Smallrye microprofile, Quarkus, reactive messaging using Kafka, Debezium outbox extension and Debezium change data capture deployed within Kafka Connector, hibernate ORM with panache for Posgresql database, and git action pipeline.

Visit detail implementation approach, design and different deployment model, read explanations of this service in the main solution documentation.

The goal of this project is to present:

• a Quarkus app with the Debezium outbox extension to create order events in the same transaction as writing the business entity within Postgresql
• how to use the Reactive RESTs API with Mutiny and JAXRS
• the JPA with Hibernate and Panache implementation for Postgresql database
• Debezium Postgres Change Data Capture Kafka connector to publish OrderEvents to Kafka topic
• Consume ShipmentPlan from Kafka using reactive messaging, schema registry

This service integrates with Kafka and may be completed with the order optimization service to demonstrate the vaccine order fulfillment use case. The component view may look like the figure below:

Pre-requisites

Be sure to have a Kafka cluster deployed with the order and shipment plan topics defined. This can be done using the vaccine-gitops repository where dependent components such as Kafka (Strimzi) and Postgresql are defined as code, and the order management microservice deployment config yaml files are also defined in this gitops project.

Build

Build and package locally

We have a script to compile the user interface, package the Quarkus app, and build a docker image.

./buildAll.sh

Source to image

The application uses Quarkus OpenShift extension to create yaml files for OpenShift and deploy the application using source to image capability of OpenShift. The following command will build the Vuejs UI, the quarkus app and then perform a source to image so the service will be deployed as pod inside OpenShift.

# You need to be connected to the OpenShift Cluster.
./mvnw clean package -Dui.deps -Dui.dev -Dquarkus.kubernetes.deploy=true -DskipTests

The -Dui.deps -Dui.dev arguments are used to prepare and build the Vue.js app from the ui folder. The packaging builds a runner jar and pushes it to the private image registry in OpenShift.

Be sure to get the Order Microservice URL to access the user interface, using oc get routes on the project.

Run

Local execution

If you want to develop with existing code, running: mvn quarkus:dev will start the app but you need to get access to a Postgresql database and Kafka. For that we have integrated different docker compose files depending of the target environment.

• Staging profile is with a remote Kafka and Schema Registry but local services and Postgresql.
• Dev profile is for get all running locally

Staging mode

• You need to specify the URL and credential to connect to Kafka bootstrap server, user and Schema registry with an .env file:

export KAFKA_USER=scram-user
export KAFKA_PASSWORD=
export KAFKA_BOOTSTRAP_SERVERS=......appdomain.cloud:443
export KAFKA_SSL_TRUSTSTORE_LOCATION=${PWD}/certs/truststore.p12
export KAFKA_SSL_TRUSTSTORE_PASSWORD=
export KAFKA_SSL_TRUSTSTORE_TYPE=PKCS12
export SCHEMA_REGISTRY_URL=
export KAFKA_SASL_MECHANISM=SCRAM-SHA-512
export KAFKA_SECURITY_PROTOCOL=SASL_SSL
export QUARKUS_PROFILE=staging

• Use the docker-compose.yaml docker compose to start Postgresql, Postgresql admin console, and the transportation service.

 # Start local environment 
 docker-compose up -d 

• Load the env variable with: source .env
• Start the app using ./mvnw quarkus:dev
• Upload schema definitions to the Schema registry (the name of the ArtifactId is the name of the topic + "-value")

Example of commands:

curl -X POST -H "Content-type: application/json; artifactType=AVRO" \
   -H "X-Registry-ArtifactId: vaccine.reefers-value" \
   --data @${scriptDir}/../data/avro/schemas/reefer.avsc http://${SCHEMA_REGISTRY_URL}/api/artifacts

Dev mode

• Start docker compose

docker-compose -f dev-docker-compose.yaml up -d

• Create needed topics: ./scripts/createTopics.sh
• Verify created topics: ./scripts/listTopics.sh
• Upload schema definitions to the Schema registry
• Create some shipment plan:
• Start the app using ./mvnw quarkus:dev

UI development

For UI development start the components as explained in the previous section and then under the ui folder, do the following:

yarn install
yarn serve

Use the web browser and developer console to the address http://localhost:4545. The Vue app is configured to proxy to localhost:8080.

Git Action

This repository includes a Github workflow to build the app and push a new docker image to the public registry. To do that we need to define four secrets in the github repository:

• DOCKER_IMAGE_NAME the image name to build. Here it is vaccineorderms
• DOCKER_USERNAME: user to access docker hub
• DOCKER_PASSWORD: and his/her password.
• DOCKER_REPOSITORY for example the organization we use is ibmcase
• DOCKER_REGISTRY which point to quay.io

OpenShift Deployment

Using the gitops repository, we can have an OpenShift project created with Postgresql and Kafka deployed.

Demonstration

See the script in this section.

Debezium CDC connector

The Debezium Postgres connector is a Kafka Connector. So to deploy to Event Streams or Strimzi Connector we will use source to image approach.

• From the Event Streams UI > Toolbox > Kafka Connect menu, download the source to image yaml file.

• Modify the file to get connection to the Kafka brokers

• Add declaration to use Postgres connector

• Deploy with oc apply -f environment/cdc/kafka-connect-s2i.yaml

• Start a consumer on the CDC topic for the order events

docker-compose exec kafka /opt/kafka/bin/kafka-console-consumer.sh     --bootstrap-server kafka:9092     --from-beginning     --property print.key=true     --topic ...

• Add new order from the user interface: http://localhost:8080/#/Orders, or...

• Post an order using the API: http://localhost:8080/swagger-ui/#/default/post_orders. Use the following JSON

{
   "deliveryDate": "2021-07-25",
   "deliveryLocation": "Milano",
   "askingOrganization": "Italy gov",
   "priority": 1,
   "quantity": 100,
   "type": "COVID-19"
}

• The expected result should have the following records in the Kafka topic:

{"ID":"lvz4gYs/Q+aSqKmWjVGMXg=="}	
{"before":null,"after":{"ID":"lvz4gYs/Q+aSqKmWjVGMXg==","AGGREGATETYPE":"VaccineOrderEntity","AGGREGATEID":"21","TYPE":"OrderCreated","TIMESTAMP":1605304440331350,"PAYLOAD":"{\"orderID\":21,\"deliveryLocation\":\"London\",\"quantity\":150,\"priority\":2,\"deliveryDate\":\"2020-12-25\",\"askingOrganization\":\"UK Governement\",\"vaccineType\":\"COVID-19\",\"status\":\"OPEN\",\"creationDate\":\"13-Nov-2020 21:54:00\"}"},"source":{"version":"1.3.0.Final","connector":"db2","name":"vaccine_lot_db","ts_ms":1605304806596,"snapshot":"last","db":"TESTDB","schema":"DB2INST1","table":"ORDEREVENTS","change_lsn":null,"commit_lsn":"00000000:0000150f:0000000000048fca"},"op":"r","ts_ms":1605304806600,"transaction":null}