{"id":31047759,"url":"https://github.com/amitnema/spark-etl-framework","last_synced_at":"2026-05-18T06:09:06.342Z","repository":{"id":312915741,"uuid":"1049278897","full_name":"amitnema/spark-etl-framework","owner":"amitnema","description":"A generic, multimodule framework for building scalable ETL (Extract, Transform, Load) processes using Apache Spark and Java. This framework is designed to be cloud-agnostic, maintainable, testable, and near-production-ready.","archived":false,"fork":false,"pushed_at":"2025-09-11T05:31:05.000Z","size":213,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-11T08:53:05.874Z","etag":null,"topics":["etl","etl-framework","etl-pipeline","general-purpose","generalization","java-11","maven","spark-3x","spark-submit"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/amitnema.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-02T18:36:59.000Z","updated_at":"2025-09-04T19:16:34.000Z","dependencies_parsed_at":"2025-09-02T20:41:12.296Z","dependency_job_id":"fc2e4f23-7e8a-4e76-ac4a-8138951cdaea","html_url":"https://github.com/amitnema/spark-etl-framework","commit_stats":null,"previous_names":["amitnema/spark-etl-framework"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/amitnema/spark-etl-framework","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amitnema%2Fspark-etl-framework","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amitnema%2Fspark-etl-framework/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amitnema%2Fspark-etl-framework/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amitnema%2Fspark-etl-framework/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/amitnema","download_url":"https://codeload.github.com/amitnema/spark-etl-framework/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amitnema%2Fspark-etl-framework/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":275160087,"owners_count":25415764,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-14T02:00:10.474Z","response_time":75,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["etl","etl-framework","etl-pipeline","general-purpose","generalization","java-11","maven","spark-3x","spark-submit"],"created_at":"2025-09-14T19:55:34.206Z","updated_at":"2025-10-06T00:24:18.198Z","avatar_url":"https://github.com/amitnema.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Java CI with Maven](https://github.com/amitnema/spark-etl-framework/actions/workflows/maven.yml/badge.svg)](https://github.com/amitnema/spark-etl-framework/actions/workflows/maven.yml)\n[![codecov](https://codecov.io/gh/amitnema/spark-etl-framework/branch/main/graph/badge.svg?token=YOUR_CODECOV_TOKEN)](https://codecov.io/gh/amitnema/spark-etl-framework)\n[![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n\n# Spark ETL Framework\n\n**Author:** Amit Prakash Nema\n\nA generic, multimodule framework for building scalable ETL (Extract, Transform, Load) processes using Apache Spark and\nJava. This framework is designed to be cloud-agnostic, maintainable, testable, and production-ready.\n\n## Architecture Overview\n\nThe framework follows enterprise design patterns and consists of the following modules:\n\n- **etl-core**: Core framework components including configuration, validation, transformation interfaces, and I/O\n  operations\n- **etl-jobs**: Concrete job implementations and custom transformers\n\n## Architecture Diagram\n\n```mermaid\nflowchart TD\n    A[Job Config] --\u003e B[ETLEngine]\n    B --\u003e C[Reader]\n    B --\u003e D[Transformer]\n    B --\u003e E[Writer]\n    B --\u003e F[Validation]\n    B --\u003e G[Monitoring]\n    B --\u003e H[Cloud Integration]\n    C --\u003e|Reads Data| D\n    D --\u003e|Transforms Data| E\n    E --\u003e|Writes Data| F\n    F --\u003e|Validates| G\n    G --\u003e|Monitors| H\n```\n\n## Key Features\n\n- ✅ **Cloud Agnostic**: Supports AWS S3, Google Cloud Storage, Azure Blob Storage\n- ✅ **Modular Design**: Separation of concerns with pluggable components\n- ✅ **Configuration-Driven**: YAML-based job configuration with parameter substitution\n- ✅ **Data Validation**: Built-in validation rules (NOT_NULL, UNIQUE, RANGE, REGEX)\n- ✅ **Multiple Data Sources**: File systems, databases (JDBC), streaming sources\n- ✅ **Format Support**: Parquet, JSON, CSV, Avro, ORC\n- ✅ **Transformation Framework**: Abstract transformer pattern with custom implementations\n- ✅ **Error Handling**: Comprehensive exception handling and logging\n- ✅ **Production Ready**: Docker containerization, monitoring, and deployment scripts\n- ✅ **Testing Support**: Unit and integration testing capabilities\n\n## Project Structure\n\n```\nspark-etl-framework/\n├── etl-core/                    # Core framework module\n│   └── src/main/java/com/etl/core/\n│       ├── config/              # Configuration management\n│       ├── model/               # Data models and POJOs\n│       ├── io/                  # Data readers and writers\n│       ├── validation/          # Data validation components\n│       ├── transformation/      # Transformation interfaces\n│       ├── factory/             # Factory pattern implementations\n│       ├── exception/           # Custom exceptions\n│       └── utils/               # Utility classes\n├── etl-jobs/                    # Job implementations\n│   └── src/main/java/com/etl/jobs/\n│       └── sample/              # Sample job implementation\n├── dist/                        # Distribution packages\n│   └── sample-job/              # Ready-to-deploy job package\n├── docker/                      # Docker configuration\n├── scripts/                     # Build and deployment scripts\n└── README.md                    # This file\n```\n\n## Design Patterns Used\n\n1. **Factory Pattern**: For creating readers, writers, and transformers\n2. **Abstract Template Pattern**: For transformation pipeline\n3. **Strategy Pattern**: For different I/O implementations\n4. **Builder Pattern**: For configuration objects\n5. **Singleton Pattern**: For configuration management\n6. **Command Pattern**: For job execution\n\n## Getting Started\n\n### Prerequisites\n\n- Java 11 (or higher)\n- Maven 3.6+\n- Docker (for containerized deployment)\n\n### Build \u0026 Test\n\n```sh\nmvn clean verify\n```\n\nThis runs all unit/integration tests, code quality checks, coverage, and security scans.\n\n### Configuration\n\nAll configuration values can be overridden via environment variables. Example:\n\n```sh\nexport DATABASE_URL=\"jdbc:postgresql://prod-db:5432/etl_db\"\nexport AWS_ACCESS_KEY=\"your-access-key\"\nexport AWS_SECRET_KEY=\"your-secret-key\"\n```\n\nSee `src/main/resources/application.properties` and `etl-jobs/src/main/resources/jobs/sample-job/job-config.yaml` for\nall available variables.\n\n### Running Locally\n\n```sh\njava -jar etl-jobs/target/etl-jobs-*.jar\n```\n\n### Running with Docker\n\nBuild and run the container:\n\n```sh\ndocker build -t etl-framework:latest -f docker/Dockerfile .\ndocker run --rm -e DATABASE_URL=\"jdbc:postgresql://prod-db:5432/etl_db\" etl-framework:latest\n```\n\n### CI/CD Workflows\n\n- **Build, Test, Quality, Security**: See `.github/workflows/maven.yml`\n- **Docker Build \u0026 Publish**: See `.github/workflows/docker.yml`\n\n### Cloud Agnostic Deployment\n\n- Supports AWS, GCP, Azure, and local deployments via environment variables.\n- Containerized for Kubernetes, Docker Compose, or serverless platforms.\n\n## Development Guide\n\n### Creating Custom Transformers\n\nExtend the `AbstractDataTransformer` class:\n\n```java\npublic class MyCustomTransformer extends AbstractDataTransformer {\n    @Override\n    protected Dataset\u003cRow\u003e doTransform(Dataset\u003cRow\u003e input, Map\u003cString, Object\u003e parameters) {\n        // Your transformation logic here\n        return input.filter(functions.col(\"status\").equalTo(\"active\"));\n    }\n}\n```\n\n### Adding New Data Sources\n\nImplement the `DataReader` interface:\n\n```java\npublic class MyCustomReader implements DataReader {\n    @Override\n    public Dataset\u003cRow\u003e read(InputConfig config) {\n        // Your data reading logic here\n        return dataset;\n    }\n}\n```\n\n### Configuration Management\n\nThe framework uses a hierarchical configuration approach:\n\n1. `application.properties` - Application-wide settings\n2. `job-config.yaml` - Job-specific configuration\n3. Command-line parameters - Runtime overrides\n\n## Cloud Deployment\n\n### AWS Configuration\n\n```properties\ncloud.provider=aws\naws.access.key=YOUR_ACCESS_KEY\naws.secret.key=YOUR_SECRET_KEY\naws.region=us-east-1\n```\n\n### GCP Configuration\n\n```properties\ncloud.provider=gcp\ngcp.project.id=your-project-id\ngcp.service.account.key=/path/to/service-account.json\n```\n\n### Azure Configuration\n\n```properties\ncloud.provider=azure\nazure.storage.account=your-storage-account\nazure.storage.access.key=your-access-key\n```\n\n## Docker Deployment\n\n```bash\n# Build Docker image\ncd docker/\ndocker-compose build\n\n# Run with Docker Compose\ndocker-compose up -d\n\n# Execute ETL job in container\ndocker-compose exec etl-framework ./sample-job/spark-submit.sh\n```\n\n## Monitoring and Logging\n\nThe framework includes comprehensive logging using Logback:\n\n- **Console Output**: Real-time job progress\n- **File Logging**: Detailed logs with rotation\n- **Structured Logging**: JSON format for log aggregation\n- **Metrics**: Job execution metrics and validation results\n\n## Testing\n\n```bash\n# Run unit tests\nmvn test\n\n# Run integration tests\nmvn verify -P integration-tests\n```\n\n## Performance Tuning\n\n### Spark Configuration\n\nKey configuration parameters in `application.properties`:\n\n```properties\nspark.sql.adaptive.enabled=true\nspark.sql.adaptive.coalescePartitions.enabled=true\nspark.sql.adaptive.skewJoin.enabled=true\nspark.serializer=org.apache.spark.serializer.KryoSerializer\n```\n\n### Memory Settings\n\nAdjust memory settings in spark-submit script:\n\n```bash\n--driver-memory 4g\n--executor-memory 8g\n--executor-cores 4\n```\n\n## Contributing\n\n- Fork the repo and create a feature branch.\n- Run `mvn clean verify` before submitting a PR.\n- Ensure new code is covered by tests and passes code quality checks.\n\n## License\n\nApache 2.0\n\n## Contact\n\nFor questions or support, open an issue or contact the maintainer\n\n## Roadmap\n\n- [ ] Kafka streaming support\n- [ ] Delta Lake integration\n- [ ] Advanced data profiling\n- [ ] ML pipeline integration\n- [ ] REST API for job management\n- [ ] Web UI for monitoring\n\n---\n\nBuilt with ❤️ using Apache Spark and Java\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famitnema%2Fspark-etl-framework","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Famitnema%2Fspark-etl-framework","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famitnema%2Fspark-etl-framework/lists"}