https://github.com/thusithz/spring-boot-data-processing-batch
This Spring Batch application processes server metrics data from Datadog logs and generates performance summaries. It includes scheduled jobs for processing server metrics and generating analytical reports.
https://github.com/thusithz/spring-boot-data-processing-batch
spring-backend spring-batch spring-batch-application spring-batch-example spring-batch-jobs spring-boot
Last synced: 7 months ago
JSON representation
This Spring Batch application processes server metrics data from Datadog logs and generates performance summaries. It includes scheduled jobs for processing server metrics and generating analytical reports.
- Host: GitHub
- URL: https://github.com/thusithz/spring-boot-data-processing-batch
- Owner: thusithz
- Created: 2025-01-16T01:25:52.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2025-01-16T02:42:25.000Z (about 1 year ago)
- Last Synced: 2025-03-13T15:43:57.403Z (11 months ago)
- Topics: spring-backend, spring-batch, spring-batch-application, spring-batch-example, spring-batch-jobs, spring-boot
- Language: Java
- Homepage:
- Size: 35.2 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Data Processing Batch Application
This Spring Batch application processes server metrics data from Datadog logs and generates performance summaries. It includes scheduled jobs for processing server metrics and generating analytical reports.
## Features
1. **Server Metrics Processing**
- Processes Datadog log files for server performance metrics
- Monitors CPU and memory usage with configurable thresholds
- Tracks request counts and response times
- Generates performance summaries in CSV format
2. **Alert Monitoring**
- Automatic alerts for high resource usage
- CPU Usage threshold > 90%
- Memory Usage threshold > 85%
3. **Batch Processing**
- Chunk-based processing with configurable batch sizes
- Efficient file reading with FlatFileItemReader
- JSON parsing for Datadog log format
- CSV output generation
## Project Structure
```
src/main/java/com/tk/batch/data_processing_batch/
├── config/
│ ├── BatchConfig.java # Batch job configurations
│ ├── MetricProcessor.java # Server metrics processor
│ └── DatadogLogReader.java # Datadog log reader
├── controller/
│ └── BatchController.java # REST endpoints
├── model/
│ ├── ServerMetric.java # Server metric model
│ └── MetricSummary.java # Processed summary model
└── DataProcessingBatchApplication.java
```
## Configuration
### Application Properties
```properties
spring.batch.jdbc.initialize-schema=always
spring.batch.job.enabled=false
spring.task.scheduling.pool.size=2
```
### Required Files
1. `datadog.log` - Place in `src/main/resources/`
2. `datadog_metrics.csv` - Place in `src/main/resources/`
## REST Endpoints
1. **Start Batch Processing**
```
POST /start-batch
```
## Dependencies
- Spring Boot 3.1.0
- Spring Batch
- Spring Web
- H2 Database
- Lombok
- Jackson Databind
## Output
The application generates a `metric_summary.csv` file containing:
- Server ID
- Average CPU Usage
- Average Memory Usage
- Total Request Count
- Average Response Time
## Error Handling
- Failed jobs are automatically retried
- Job execution history maintained in H2 database
- Detailed logs generated for debugging
- Exception handling for JSON parsing errors
## Support
For issues and support:
1. Check application logs
2. Review job execution history
3. Verify input file formats (Datadog log format)
4. Monitor batch processing status