https://github.com/hvalfangst/spring-batch
HTTP-triggered Spring Batch tasklet job which performs ETL
https://github.com/hvalfangst/spring-batch
etl exposed flyway kotlin spring-6 spring-batch spring-boot spring-boot-3
Last synced: 7 months ago
JSON representation
HTTP-triggered Spring Batch tasklet job which performs ETL
- Host: GitHub
- URL: https://github.com/hvalfangst/spring-batch
- Owner: hvalfangst
- Created: 2023-07-30T08:17:30.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2023-08-15T12:52:48.000Z (about 2 years ago)
- Last Synced: 2025-01-25T06:25:10.525Z (9 months ago)
- Topics: etl, exposed, flyway, kotlin, spring-6, spring-batch, spring-boot, spring-boot-3
- Language: Kotlin
- Homepage:
- Size: 27.3 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# HTTP triggered Spring Batch ETL Job using Tasklet
## Overview
The main purpose of this application is to perform data migration and aggregation from a source database to a target database using Spring Batch.
It follows the ETL pattern, where data is extracted from the source database, transformed, and loaded into the target database.
The application provides an easy-to-use RESTful API to trigger the ETL job.
## Requirements
* x86-64
* JDK 17
* Docker
## Startup
The script "up" starts the application by executing the following:
```
1. docker-compose -f db/source/docker-compose.yml up -d
2. docker-compose -f db/target/docker-compose.yml up -d
3. mvn clean install
4. mvn spring-boot:run
```
## Usage
The application exposes an HTTP endpoint for triggering the ETL job.
You can use any REST client, like Postman, to make a POST request to the /batch/start endpoint with the target date as a parameter.
## Endpoint details
Endpoint: POST /batch/start
Parameters:
targetDate: The date for which the ETL job should be executed. It should be in the format 'yyyy-MM-dd'.
Response: The endpoint will respond with a 200 status code if the job succeeded and a 400 status code if the job failed.
## Shutdown
The script "down" wipes the source and target databases executing the following:
```
1. docker-compose -f db/source/docker-compose.yml down
2. docker-compose -f db/target/docker-compose.yml down
```
## Postman Collection
The repository includes a Postman collection in the 'postman' directory.