https://github.com/undisputed-jay/airflow-etl-pipeline-with-pyspark-and-google-cloud-dataproc

This project automates daily vehicle data processing on Google Cloud using Apache Airflow. It uploads scripts to Google Cloud Storage, runs specific PySpark jobs on Dataproc based on the day, and shuts down resources when done for efficiency.

automated-etl-airflow-dataproc cost-effective-data-processing daily-data-analysis-airflow-pyspark

Last synced: 20 Jul 2025