{"id":27172937,"url":"https://github.com/waridrox/weather-etl-airflow","last_synced_at":"2025-04-09T10:41:13.365Z","repository":{"id":285834574,"uuid":"331592604","full_name":"waridrox/weather-etl-airflow","owner":"waridrox","description":null,"archived":false,"fork":false,"pushed_at":"2025-04-02T22:44:37.000Z","size":232,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-02T23:28:05.557Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/waridrox.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-01-21T10:28:29.000Z","updated_at":"2025-04-02T22:44:41.000Z","dependencies_parsed_at":"2025-04-02T23:38:09.338Z","dependency_job_id":null,"html_url":"https://github.com/waridrox/weather-etl-airflow","commit_stats":null,"previous_names":["waridrox/weather-etl-airflow"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/waridrox%2Fweather-etl-airflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/waridrox%2Fweather-etl-airflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/waridrox%2Fweather-etl-airflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/waridrox%2Fweather-etl-airflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/waridrox","download_url":"https://codeload.github.com/waridrox/weather-etl-airflow/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248023266,"owners_count":21035034,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-09T10:41:12.653Z","updated_at":"2025-04-09T10:41:13.351Z","avatar_url":"https://github.com/waridrox.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Data Engineering Project: Automated ETL Pipeline with Apache Airflow\n\u003cimg width=\"952\" alt=\"data flow dia\" src=\"https://github.com/user-attachments/assets/f822c04e-c941-4a92-95d7-57b051a21091\" /\u003e\n\nIn this project, I built and automate an ETL process that extracts current weather data from the Open Weather Map API, transforms it, and loads the csv results obtained into an AWS S3 bucket. The entire project runs on the AWS cloud platform and leverages Apache Airflow to orchestrate and schedule the workflows.\n\n\u003cimg width=\"1362\" alt=\"airflow functions\" src=\"https://github.com/user-attachments/assets/71bd32cd-31cc-45be-a4ff-5067b92695f4\" /\u003e\n\n## Key Concepts\n\n- **ETL Process**: \n  - **Extract**: Pull weather data from the Open Weather Map API.\n  - **Transform**: Clean and prepare the data.\n  - **Load**: Upload the transformed data into an AWS S3 bucket.\n  \n- **Apache Airflow**:\n  - **DAG (Directed Acyclic Graph)**: Represents the workflow structure.\n  - **Operators**: Tasks that define each step of the pipeline.\n  - **Sensors**: Special operators that wait for a particular condition or event before proceeding.\n\n- **AWS Cloud Platform**: \n  - Utilized AWS S3 for data storage.\n\n## Project Workflow\n\n1. **Environment Setup**  \n   - Install Apache Airflow.\n   - Configure AWS credentials and Openweathermap API key.\n\n2. **Extract Data**  \n   - Connect to the Open Weather Map API to fetch current weather data.\n\n3. **Transform Data**  \n   - Process and clean the data to match requirements.\n\n4. **Load Data**  \n   - Upload the transformed data into an S3 bucket on AWS.\n\n5. **Schedule and Monitor the Workflow**  \n   - Create an Airflow DAG to schedule the ETL process.\n   - Use Airflow operators and sensors to automate and monitor each step.\n\n## Installation and Setup\n\n```bash\n# Install Apache Airflow (using pip)\npip install apache-airflow\n\n# Initialize the Airflow database\nairflow db init\n\n# Run the dag file\npython weather_dag.py\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwaridrox%2Fweather-etl-airflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwaridrox%2Fweather-etl-airflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwaridrox%2Fweather-etl-airflow/lists"}