{"id":15060602,"url":"https://github.com/zeinhasan/etl-using-airflow","last_synced_at":"2026-01-02T18:03:55.766Z","repository":{"id":254527913,"uuid":"846610630","full_name":"zeinhasan/ETL-Using-AirFlow","owner":"zeinhasan","description":"Extract Transform Load Using Airflow ","archived":false,"fork":false,"pushed_at":"2024-08-25T02:53:47.000Z","size":715,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-22T13:52:25.104Z","etag":null,"topics":["airflow","bigquery","etl"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zeinhasan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-23T15:14:27.000Z","updated_at":"2024-08-25T02:53:50.000Z","dependencies_parsed_at":"2024-09-24T23:01:13.659Z","dependency_job_id":"80c1b5f1-3499-4f2a-ae54-75f6946c539e","html_url":"https://github.com/zeinhasan/ETL-Using-AirFlow","commit_stats":null,"previous_names":["zeinhasan/etl-using-airflow"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zeinhasan%2FETL-Using-AirFlow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zeinhasan%2FETL-Using-AirFlow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zeinhasan%2FETL-Using-AirFlow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zeinhasan%2FETL-Using-AirFlow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zeinhasan","download_url":"https://codeload.github.com/zeinhasan/ETL-Using-AirFlow/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243809888,"owners_count":20351407,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","bigquery","etl"],"created_at":"2024-09-24T23:01:04.683Z","updated_at":"2026-01-02T18:03:55.689Z","avatar_url":"https://github.com/zeinhasan.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ETL-Using-Airflow\nThis is a simple ETL (Extract, Transform, \u0026 Load) using Airflow. Step :\n1. Extract dataset from the API \n2. Store the dataset to Google Cloud Storage\n3. Retreive the dataset from Google Cloud Storage and clean it\n4. Store cleaned dataset to stagging (Google Cloud Storage)\n5. Store the final dataset in Google BigQuery\n\n# Prerequisite\n\n## Docker Desktop\nInstall docker deskto from official docker website\n```bash\nhttps://www.docker.com/products/docker-desktop/\n```\n\n## Service Account Credentials\nGenerate key of Service Account on IAM \u0026 Admin Google Cloud Console and store it as Connection in Airflow Admin\n```Json\n{\n  \"type\": \"service_account\",\n  \"project_id\": \"Your Project ID\",\n  \"private_key_id\": \"Your Private ID\",\n  \"private_key\": \"Your Private Key\",\n  \"client_email\": \"Your Client Key\",\n  \"client_id\": \"Your Client ID\",\n  \"auth_uri\": \"https://accounts.google.com/o/oauth2/auth\",\n  \"token_uri\": \"https://oauth2.googleapis.com/token\",\n  \"auth_provider_x509_cert_url\": \"https://www.googleapis.com/oauth2/v1/certs\",\n  \"client_x509_cert_url\": \"https://www.googleapis.com/robot/v1/metadata/xxx.iam.gserviceaccount.com\",\n  \"universe_domain\": \"googleapis.com\"\n}\n\n```\n# Step \nStep by step to use Airflow\n1. Clone this repository\n2. Install  \u0026 Open Docker\n3. Install \u0026 Open Visual Studio Code\n4. Open Docker \u0026 Visual Studio Code\n5. Create a new file .env and add the following lines\n```console\nAIRFLOW_IMAGE_NAME=apache/airflow:2.4.2\nAIRFLOW_UID=50000\n```\n6. Build container using this command on command:\n```console\ndocker-compose up -d\n```\n7. Create Admin user using bellow command:\n```console\ndocker-compose run airflow-worker airflow users create --role Admin --username admin --email admin --firstname admin --lastname admin --password admin\n```\n8. Login to Airflow on and insert username and password:\n```\nhttp://localhost:8080/\n```\n9. Setting Up Connection, You can create a connection to Google Cloud from Airflow webserver admin menu. In this menu you can pass the Service Account key file path:\n![Connection](Connection0.png) In this picture, the keyfile Path is '/usr/local/airflow/dags/gcp.json.' Beforehand you need to mount your key file as a volume in your Docker container with the previous path. You can also directly copy the key json content in the Airflow connection, in the keyfile Json field :\n![Connection](Connection.png)\n\n10. Run the DAG by trigger the button like below picture\n ![Connection](DAG.png)\n\n\n# Deploying on Google Cloud Composer\nGoogle Cloud Composer is a managed workflow orchestration service built on Apache Airflow, an open-source platform designed for orchestrating complex workflows.\nAt its core, Composer simplifies the process of scheduling, managing, and monitoring workflows and data pipelines. Here are step to deploy Airflow DAGs on Cloud Composer:\n1. Activate the Cloud Composer API on Google Cloud Platform\n2. Go to Cloud Composer Services and create new environtment\n ![Composser](CloudComposer1.png)\n3. Choose the environments version, for this tutorial use the Composer 3\n![Composser](CloudComposer2.png) \n4. Define environment name\n![Composser](CloudComposer3.png)\n5. Choose environment resources\n![Composser](CloudComposer4.png)\n6. Setup network \u0026 advance configuration if necessary and create environment\n![Composser](CloudComposer5.png)\n7. Wait until environtment created\n8. Open the created environment and go to DAGs folder\n![Composser](CloudComposer6.png)\n9. Upload the python file for DAGS\n![Composser](CloudComposer7.png)\n10. Wait until the new DAG appear\n![Composser](CloudComposer8.png)\n11. Setting Up Connection, You can create a connection to Google Cloud from Airflow webserver admin menu. In this menu you can pass the Service Account key file path:\n![Connection](Connection0.png) In this picture, the keyfile Path is '/usr/local/airflow/dags/gcp.json.' Beforehand you need to mount your key file as a volume in your Docker container with the previous path. You can also directly copy the key json content in the Airflow connection, in the keyfile Json field :\n![Connection](Connection.png)\n\n11. Open the DAG, trigger it, and wait until finish \n![Composser](CloudComposer9.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzeinhasan%2Fetl-using-airflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzeinhasan%2Fetl-using-airflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzeinhasan%2Fetl-using-airflow/lists"}