{"id":15208896,"url":"https://github.com/akarce/udacity-data-pipeline-with-airflow","last_synced_at":"2025-09-19T06:56:02.633Z","repository":{"id":252957358,"uuid":"828654569","full_name":"akarce/Udacity-Data-Pipeline-with-Airflow","owner":"akarce","description":"Udacity Data Engineering Nanodegree Program, Data Pipeline with Airflow project using MinIO and Postgresql.","archived":false,"fork":false,"pushed_at":"2024-08-13T14:20:38.000Z","size":88688,"stargazers_count":5,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-09T07:42:14.552Z","etag":null,"topics":["airflow","minio","postgresql","pyspark","spark"],"latest_commit_sha":null,"homepage":"https://medium.com/@akarce/udacity-data-pipeline-with-airflow-60f04f99efed","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/akarce.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-14T19:46:50.000Z","updated_at":"2025-02-09T21:07:07.000Z","dependencies_parsed_at":"2024-08-13T17:23:13.560Z","dependency_job_id":"574a838c-3e6f-4c8b-9626-bb0f41314d00","html_url":"https://github.com/akarce/Udacity-Data-Pipeline-with-Airflow","commit_stats":null,"previous_names":["akarce/udacity-data-pipeline-with-airflow"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/akarce/Udacity-Data-Pipeline-with-Airflow","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akarce%2FUdacity-Data-Pipeline-with-Airflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akarce%2FUdacity-Data-Pipeline-with-Airflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akarce%2FUdacity-Data-Pipeline-with-Airflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akarce%2FUdacity-Data-Pipeline-with-Airflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/akarce","download_url":"https://codeload.github.com/akarce/Udacity-Data-Pipeline-with-Airflow/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akarce%2FUdacity-Data-Pipeline-with-Airflow/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":275894590,"owners_count":25547859,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-19T02:00:09.700Z","response_time":108,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","minio","postgresql","pyspark","spark"],"created_at":"2024-09-28T07:03:27.441Z","updated_at":"2025-09-19T06:55:57.621Z","avatar_url":"https://github.com/akarce.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Udacity-Data-Pipeline-with-Airflow\nUdacity Data Engineering Nanodegree Program, Data Pipeline with Airflow project using MinIO and Postgresql.\n\n\n## Project Structure:\n![alt text](udacity_overview.jpg)\n\n\n+ dags: Directory containing Airflow DAG scripts.\n    + create_tables_and_buckets.py: DAG for creating tables on PostgreSQL and buckets on MinIO.\n    + create_tables.sql: SQL script containing CREATE TABLE queries.\n    + pipeline_dag.py: Main DAG script for the ETL data pipeline.\n\n+ data: Directory for storing project source data.\n    + log_data: Subdirectory for log data.\n    + song_data: Subdirectory for song data.\n\n+ plugins: Directory for custom Airflow plugins.\n    + operators: Subdirectory for operator scripts.\n        + init.py: Initialization script for operators.\n        + stage_postgresql_operator.py: Operator script for copying data from MinIO to PostgreSQL.\n        + load_fact_operator.py: Operator script for executing INSERT queries into the fact table.\n        + load_dimension_operator.py: Operator script for executing INSERT queries into dimension tables.\n        + data_quality_operator.py: Operator script for performing data quality checks.\n\n    + helpers: Subdirectory for helper scripts.\n        + __init__.py: Initialization script for helpers.\n        + sql_queries.py: Script containing SQL queries for building dimensional tables.\n\n\n\n### Airflow Dag Overview\n\n![alt text](img/dag_overview.png)\n\n### Staging \u0026 Fact Table Schema\n\n![alt text](img/erd1.png)\n\n### Dimension Tables Schema\n\n![alt text](img/erd2.png)\n\n# How to Run\n\n## Clone the project repository\n\n`$ git clone https://github.com/akarce/Udacity-Data-Pipeline-with-Airflow`\n\n### Change directory\n\n`$ cd Udacity-Data-Pipeline-with-Airflow`\n\n## Build the docker images and start airflow:\n\n`$ docker-compose up airflow-init `\n`$ docker compose up -d`\n\n### Create database called sparkifydb:\n\n`$ docker exec -it postgresdev psql -U postgres_user -d postgres_db`\n\n`$ CREATE DATABASE sparkifydb; `\n\n![alt text](img/create_database_sparkifydb.png)\n\n### WebUI links: \\\n`Airflow webserver` http://localhost:8080/\n\n`MinIO` http://localhost:9001/\n\n### Sign in to Airflow webserver:\n\n`Username:` airflow \n`Password:` airflow\n\n![alt text](img/airflow_sign_in.png)\n\n\n### Create Postgres connection:\n\n#### Go to admin -\u003e Connections -\u003e Add a new record\n\nConnection Id: postgres_conn \\\nConnection Type: Postgres \\\nHost: 172.18.0.1 \\\nDatabase: sparkifydb \\\nLogin: postgres_user \\\nPassword: postgres_password \\\nPort: 5432\n\n![alt text](img/postgres_conn.png)\n\n\n### Obtain Access Key from MinIO:\n\nGo to MinIO WebUI using http://localhost:9001/ \\\nFrom Access Keys section -\u003e create access key -\u003e Create \\\nStore you Access Key and Secret Key using Download for Import\n\n![alt text](img/create_access_key.png)\n\n\n### Create MinIo connection:\n\n#### Go to admin -\u003e Connections -\u003e Add a new record\n\nConnection Id: minio_conn \\\nConnection Type: Amazon Web Services \\\nAWS Access Key ID: \u003cyour_key_id\u003e \\\nAWS Secret Access Key: \u003cyour_secret_key\u003e \\\n\n![alt text](img/minio_conn.png)\n\n### Trigger the create_tables_and_buckets_dag using Airflow WebUI\n\n![alt text](img/create_table_and_buckets.png)\n\nThis will create two buckets named \\\n    udacity-dend, processed \\\nand 7 postgres tables named \\\n    artists, songplays, songs, staging_events, staging_songs, times, users\n\n![alt text](img/created_buckets.png)\n\n![alt text](img/created_tables.png)\n\n\n### To see the schemas of tables use \\\n`$ \\d \u003ctable_name\u003e`\n\n\n![schemas](img/schemas.png)\n\n### Upload the log_data and song_data folders from data folder into udacity-dend bucket from MinIO WebUI\n\n`Username:` minioadmin \\\n`Password:` minioadmin\n\n![Uploaded log_data and song_data from UI](img/uploaded_data.png)\n\n### Finally unpause the dag named pipeline:\n\n![alt text](img/unpause.png)\n\n#### After finished successfully you can check the logs for data quality messages, and the all objects will be moved to the processed bucket.\n\n![alt text](img/processed_bucket.png)\n\n\n#### You can see the Airflow logs for data quality results:\n\n![alt text](img/data_quality_result.png)\n\n=======\n\u003e\u003e\u003e\u003e\u003e\u003e\u003e 1b712ebd7a3d8fd64ccb00116f6c053a5f10cde7\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fakarce%2Fudacity-data-pipeline-with-airflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fakarce%2Fudacity-data-pipeline-with-airflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fakarce%2Fudacity-data-pipeline-with-airflow/lists"}