https://github.com/vaxdata22/city-weather-and-s3file-rds-s3-bigquery-by-airflow-on-ec2
This is my third industry-level ETL project. This data pipeline orchestration uses Apache Airflow on AWS EC2. It demonstrates how to build an ETL data pipeline that would perform data extraction to a database in parallel to a loading process into the same database, join the tables, copy joined data to S3 and finally copy the S3 file to BigQuery DW.
https://github.com/vaxdata22/city-weather-and-s3file-rds-s3-bigquery-by-airflow-on-ec2
apache-airflow aws-ec2 aws-rds-postgres aws-s3 bigquery business-intelligence dags data-warehousing etl-pipeline openweathermap-api orchestration python3 sql
Last synced: 7 months ago
JSON representation
This is my third industry-level ETL project. This data pipeline orchestration uses Apache Airflow on AWS EC2. It demonstrates how to build an ETL data pipeline that would perform data extraction to a database in parallel to a loading process into the same database, join the tables, copy joined data to S3 and finally copy the S3 file to BigQuery DW.
- Host: GitHub
- URL: https://github.com/vaxdata22/city-weather-and-s3file-rds-s3-bigquery-by-airflow-on-ec2
- Owner: vaxdata22
- Created: 2025-03-17T19:33:12.000Z (7 months ago)
- Default Branch: main
- Last Pushed: 2025-03-17T19:51:04.000Z (7 months ago)
- Last Synced: 2025-03-17T20:38:07.775Z (7 months ago)
- Topics: apache-airflow, aws-ec2, aws-rds-postgres, aws-s3, bigquery, business-intelligence, dags, data-warehousing, etl-pipeline, openweathermap-api, orchestration, python3, sql
- Language: Python
- Homepage:
- Size: 2.4 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0