Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/swapnilvnandane/file-reader-spring-batch
This project is a Spring Boot application that processes files using Spring Batch. It includes validation, database persistence, and file archiving/error handling. Along with file system, it also supports reading files from AWS S3 bucket with the help of AWS SQS queue service.
https://github.com/swapnilvnandane/file-reader-spring-batch
aws aws-s3 aws-sqs jasypt java liquibase lombok maven mysql spring-batch spring-boot spring-data-jpa sql
Last synced: 21 days ago
JSON representation
This project is a Spring Boot application that processes files using Spring Batch. It includes validation, database persistence, and file archiving/error handling. Along with file system, it also supports reading files from AWS S3 bucket with the help of AWS SQS queue service.
- Host: GitHub
- URL: https://github.com/swapnilvnandane/file-reader-spring-batch
- Owner: swapnilvnandane
- License: mit
- Created: 2024-08-22T13:33:17.000Z (4 months ago)
- Default Branch: main
- Last Pushed: 2024-08-24T11:04:16.000Z (4 months ago)
- Last Synced: 2024-10-15T20:41:11.868Z (2 months ago)
- Topics: aws, aws-s3, aws-sqs, jasypt, java, liquibase, lombok, maven, mysql, spring-batch, spring-boot, spring-data-jpa, sql
- Language: Java
- Homepage:
- Size: 27.3 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# File Reader Application Using Spring Batch Processing
## Overview
This project is a Spring Boot application that processes files using Spring Batch. It includes validation, database persistence, and file archiving/error handling.
Along with file system, it also supports reading files from AWS S3 bucket with the help of AWS SQS queue service.
## Technologies Used
- Java
- Spring Boot
- Spring Batch
- Spring Data JPA
- Liquibase
- Maven
- Lombok
- Jasypt
- SQL
- AWS S3 and SQS
## Setup and Installation
1. **Clone the repository:**
```sh
git clone https://github.com/swapnilvnandane/file-reader-spring-batch.git
cd file-reader-spring-batch
```
2. **Build the project:**
```sh
mvn clean install
```
3. **Generate SALT for Jasypt**
- This SALT key required to encrypt and decrypt the username and password using Jasypt. This key also use by Spring boot to decode the encrypted properties.
- Pass this SALT to application while runtime.
```sh
-Djasypt.encryptor.password=
```
4. **Run the application:**
```sh
mvn spring-boot:run -Dspring.profiles.active= -Djasypt.encryptor.password=
```
## Configuration
- **Spring profiles:**
- `aws` - To read / watch files on AWS S3 bucket.
- `filesystem` - To read / watch files from file system.
- **Application YML properties:** Configure the application in `src/main/resources/application.yml`.
```properties
com:
file:
location:
separator: "," # file content separator
archive: ${com.file.location}archive/
error: ${com.file.location}error/
```
- **Database properties:** Put either same way mentioned above in `application.yml` or in application runtime environment (Environment Varable).
```properties
com.filereader.mysql.driverClassName=com.mysql.cj.jdbc.Driver
com.filereader.mysql.username=
com.filereader.mysql.password=
com.filereader.mysql.connectionUrl=jdbc:mysql://localhost:3306/file-reader?useSSL=false&autoReconnect=true
com.filereader.mysql.maxWait=10000
com.filereader.mysql.maxActive=30
com.filereader.mysql.validationQuery=SELECT 1
com.filereader.mysql.minIdle=0
com.aws.accessKey.id=
com.aws.accessKey.secret=
com.aws.s3.bucket-name=
com.aws.region=
com.aws.sqs.queuename=
```
## [Liquibase](https://www.liquibase.com/)
- Database schema changes are managed using Liquibase.
- Configuration is defined in `src/main/resources/liquibase/liquibase-config.xml`.
- ChangeSets are defined in `src/main/resources/liquibase/changelog.xml`.
- SQL files for ChangeSets are located in `src/main/resources/sql/`.
- Run the following command to know status of the database sql scripts:
```sh
mvn liquibase:status -f liquibase-config.xml
```
- Run the following command to update the database with the sql scripts:
```sh
mvn liquibase:update -f liquibase-config.xml
```
## [Jasypt Encryption and Decryption](http://www.jasypt.org/)
To encrypt and decrypt the SQL username and password using Jasypt, use the following commands:
1. **Encrypt:**
- This will generate the encrypted text like `ENC()`. Use this encrypted text in the application properties.
```sh
mvn jasypt:encrypt-value -Djasypt.encryptor.password= -Djasypt.plugin.value=
```
2. **Decrypt:**
```sh
jasypt:decrypt-value -Djasypt.encryptor.password= -Djasypt.plugin.value=
```
## License
This project is licensed under the MIT License.