hvalfangst/kotlin-spring-batch-etl

HTTP-triggered Spring Batch tasklet job which performs ETL

HTTP triggered Spring Batch ETL Job using Tasklet

Overview

The main purpose of this application is to perform data migration and aggregation from a source database to a target database using Spring Batch. It follows the ETL pattern, where data is extracted from the source database, transformed, and loaded into the target database. The application provides an easy-to-use RESTful API to trigger the ETL job.

Requirements

• x86-64
• JDK 17
• Docker

Startup

The script "up" starts the application by executing the following:

1. docker-compose -f db/source/docker-compose.yml up -d
2. docker-compose -f db/target/docker-compose.yml up -d
3. mvn clean install
4. mvn spring-boot:run

Usage

The application exposes an HTTP endpoint for triggering the ETL job. You can use any REST client, like Postman, to make a POST request to the /batch/start endpoint with the target date as a parameter.

Endpoint details

Endpoint: POST /batch/start

Parameters:

targetDate: The date for which the ETL job should be executed. It should be in the format 'yyyy-MM-dd'.

Response: The endpoint will respond with a 200 status code if the job succeeded and a 400 status code if the job failed.

Shutdown

The script "down" wipes the source and target databases executing the following:

1. docker-compose -f db/source/docker-compose.yml down
2. docker-compose -f db/target/docker-compose.yml down

Postman Collection

The repository includes a Postman collection in the 'postman' directory.