https://github.com/emmanuelkdev/data-engineering-etl-weather-project
https://github.com/emmanuelkdev/data-engineering-etl-weather-project
Last synced: about 1 year ago
JSON representation
- Host: GitHub
- URL: https://github.com/emmanuelkdev/data-engineering-etl-weather-project
- Owner: EmmanuelKdev
- Created: 2025-03-20T10:49:21.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-03-20T10:52:56.000Z (over 1 year ago)
- Last Synced: 2025-03-20T11:36:57.106Z (over 1 year ago)
- Language: Python
- Size: 0 Bytes
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Data-Engineering-ETL-Weather-Project 🌦️🚀
## Project Highlights ✨
This project is a simple ETL pipeline built using **Apache Airflow** on **Astronomer** to extract, transform, and load weather data into a **PostgreSQL** database. Here's what makes it awesome:
- **Technologies Used**:
- 🐍 **Python**: For scripting the ETL tasks.
- 🐳 **Docker**: To containerize the entire setup for easy deployment.
- 🌬️ **Apache Airflow**: To orchestrate and automate the ETL workflow.
- 🗄️ **PostgreSQL**: As the database to store the transformed weather data.
- 🌌 **Astronomer**: As the platform to run and manage Airflow seamlessly.
- **ETL Workflow**:
1. **Extract**: Fetch daily weather data from the Open-Meteo API.
2. **Transform**: Process and structure the data for storage.
3. **Load**: Insert the transformed data into a PostgreSQL database.
- **Key Features**:
- Fully automated daily weather data ingestion. ⏰
- Scalable and containerized using Docker. 🐳
- Easy-to-use and deploy with Astronomer. 🌌
---
### How to Run the Project 🛠️
1. Clone the repository.
2. Start the services using Docker Compose.
3. Access the Airflow UI to monitor and trigger the ETL pipeline.
Enjoy building data pipelines with Airflow and Astronomer! 🚀