https://github.com/dadananjesha/azuredataengine
AzureDataEngine is a robust, scalable batch processing data architecture built on the Azure platform. It efficiently extracts, transforms, and loads massive datasets for machine learning applications, leveraging Azure Blob Storage, PostgreSQL, Databricks, and Key Vault to ensure reliability and maintainability.
https://github.com/dadananjesha/azuredataengine
azure batch-processing blob-storage databricks etl etl-framework key-vault postgresql-database spark vnet
Last synced: 8 months ago
JSON representation
AzureDataEngine is a robust, scalable batch processing data architecture built on the Azure platform. It efficiently extracts, transforms, and loads massive datasets for machine learning applications, leveraging Azure Blob Storage, PostgreSQL, Databricks, and Key Vault to ensure reliability and maintainability.
- Host: GitHub
- URL: https://github.com/dadananjesha/azuredataengine
- Owner: DadaNanjesha
- License: mit
- Created: 2024-04-11T09:18:23.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-03-08T21:59:25.000Z (9 months ago)
- Last Synced: 2025-03-08T22:25:50.578Z (9 months ago)
- Topics: azure, batch-processing, blob-storage, databricks, etl, etl-framework, key-vault, postgresql-database, spark, vnet
- Language: Python
- Homepage:
- Size: 1.04 MB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Batch Processing Data Architecture 🚀📊
---
## 📖 Introduction
**Batch Processing Data Architecture** is a robust project that builds a scalable, dependable, and maintainable data processing backend on the Azure platform. Designed as the backbone for a machine learning application, it efficiently processes enormous amounts of data, performs necessary preprocessing, and aggregates it for downstream ML tasks.
The system leverages canonical software components and data engineering best practices to integrate multiple Azure services for a comprehensive solution.
---
## ✨ Key Features
- **Scalable Batch Processing:** Efficiently processes massive datasets in scheduled batches.
- **ETL Workflows:** Custom Python scripts for data extraction, transformation, and loading.
- **Azure Integration:** Leverages Blob Storage, PostgreSQL, Databricks, Key Vault, and more.
- **Modular Design:** Easy-to-maintain code structure with dedicated ETL and loading scripts.
---
## 🛠️ Technologies Used
- **Azure DevOps**
- **Azure Repos**
- **Azure Pipelines**
- **Azure Portal**
- **Azure Database for PostgreSQL**
- **Azure Databricks**
- **Azure Blob Storage**
- **Azure Key Vault**
- **Network Watcher & Network Security**
- **Resource Group**
- **Python**
---
## 🔄 Flow Diagram
```mermaid
flowchart TD
A[📄 CSV Data Source] --> B[🔄 ETL_batchdata.py]
B --> C[🧹 Data Transformation & Aggregation]
C --> D[📤 loadtoblobtable.py]
D --> E[💾 Storage :Azure Blob/PostgreSQL]
E --> F[📈 Machine Learning Application]
```
---
## 🗂️ Project Structure
```plaintext
batch-processing/
├── .gitignore # Git ignore file
├── ETL_batchdata.py # Main ETL script for batch data processing
├── loadtoblobtable.py # Script to load processed data into storage
├── GoudaShanbog_DadaNanjesha_10220129_Data Engineering_Phase1.pdf # Phase 1 design document
├── GoudaShanbog_DadaNanjesha_10220129_Data Engineering_Phase2.pdf # Phase 2 design document
├── GoudaShanbog_DadaNanjesha_10220129_Data Engineering_Phase3.pdf # Phase 3 design document
├── Project structure.png # Visual diagram of project architecture
└── output file.pdf # Sample output report from data aggregation
```
---
## 💻 Setup Steps
Before getting started, ensure you have an active [Azure subscription](https://azure.microsoft.com/).
1. **Create Your Azure Environment:**
- Set up your Azure subscription and create a Resource Group.
- Provision necessary services such as Azure Blob Storage, PostgreSQL, Databricks, Key Vault, etc.
2. **Prepare Your Data:**
- Deploy your CSV data into the PostgreSQL database or Blob Storage as needed.
3. **Run the ETL Process:**
- Execute the `ETL_batchdata.py` script to extract, transform, and prepare your data.
- Run `loadtoblobtable.py` to load the processed data into your target storage.
4. **Integrate with ML Application:**
- Ensure your machine learning application can access the processed data from the designated storage.
---
## ⭐️ Support & Call-to-Action
If you find this project useful, please consider:
- **Starring** the repository ⭐️
- **Forking** the project to contribute enhancements
- **Following** for updates on future improvements
Your engagement helps increase visibility and encourages further collaboration!
---
## 📜 License
This project is licensed under the [MIT License](LICENSE).
---
## 🙏 Acknowledgements
- **Azure Services:** For providing a robust, scalable infrastructure.
- **Data Engineering Principles:** Guiding our modular and reliable architecture.
- **Contributors:** Thank you to everyone who supported and contributed to this project.
---
*Happy Data Processing! 🚀📊*