{"id":26977326,"url":"https://github.com/dain55788/elt-data-pipeline","last_synced_at":"2026-01-17T07:27:40.854Z","repository":{"id":285896259,"uuid":"959647957","full_name":"dain55788/ELT-Data-Pipeline","owner":"dain55788","description":"ELT Data Pipeline implementation in Data Warehousing environment","archived":false,"fork":false,"pushed_at":"2025-04-19T02:53:25.000Z","size":1701,"stargazers_count":20,"open_issues_count":0,"forks_count":8,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-19T11:14:00.754Z","etag":null,"topics":["apache-airflow","data-engineering","dbt","great-expectations","postgresql","powerbi"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dain55788.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-03T06:04:06.000Z","updated_at":"2025-04-19T02:53:28.000Z","dependencies_parsed_at":null,"dependency_job_id":"196e967c-9b9d-437e-b50b-d53ac3e5cf58","html_url":"https://github.com/dain55788/ELT-Data-Pipeline","commit_stats":null,"previous_names":["dain55788/elt-data-pipeline"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dain55788/ELT-Data-Pipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dain55788%2FELT-Data-Pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dain55788%2FELT-Data-Pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dain55788%2FELT-Data-Pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dain55788%2FELT-Data-Pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dain55788","download_url":"https://codeload.github.com/dain55788/ELT-Data-Pipeline/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dain55788%2FELT-Data-Pipeline/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28504181,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-17T06:57:29.758Z","status":"ssl_error","status_checked_at":"2026-01-17T06:56:03.931Z","response_time":85,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-airflow","data-engineering","dbt","great-expectations","postgresql","powerbi"],"created_at":"2025-04-03T12:19:13.605Z","updated_at":"2026-01-17T07:27:40.836Z","avatar_url":"https://github.com/dain55788.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ELT-Data-Pipeline 🚀\n## ELT Data Pipeline Project implementation in Data Warehousing environment using DBT, Airflow, PostgreSQL and more!! ⚡\n---\n## 📕 What is covered in this project?\n+ \u003cb\u003eData Orchestration\u003c/b\u003e: Apache Airflow\n+ \u003cb\u003eData Processing\u003c/b\u003e: Apache Spark\n+ \u003cb\u003eOn-Premise Data Lake\u003c/b\u003e: MinIO (Object Storage), Hive Metastore (Metadata Layer)\n+ \u003cb\u003eQuery Engine\u003c/b\u003e: Trino, DBeaver\n+ \u003cb\u003eData Warehousing\u003c/b\u003e: PostgreSQL\n+ \u003cb\u003eData Governance and Data Quality (Staging Area)\u003c/b\u003e: Great Expectations\n+ \u003cb\u003eData Transformaton and Data Modeling (Star Schema)\u003c/b\u003e: DBT (Data Build Tool) 🌟\n+ \u003cb\u003eData Visualization\u003c/b\u003e: PowerBI 📊\n+ \u003cb\u003eELT Data Processing Terminology Implemention\u003c/b\u003e\n\n## ⛅ API Data Source\nIn this project, we will take a look at OpenAQ API, read the document here: https://openaq.org/.\nSpecifically, we will explore the location measurement of air quality of different locations/ countries in the world.\nTake a look on how to create your API Key: https://docs.openaq.org/using-the-api/quick-start \n\n## 🛠️ System Architecture\n![SystemArchitecture](https://github.com/user-attachments/assets/eb814a85-7c58-4a65-b4ef-e54224ad5dc9)\n\n## 📁 Repository Structure\n```shell\nELT-Data-Pipeline/\n├── .env                      /* Environment variables */\n├── .git/\n├── .gitignore\n├── .idea/\n├── Dockerfile\n├── LICENSE\n├── Note.txt\n├── README.md\n├── CHANGELOG.md\n├── airflow/                  /* Airflow configuration and DAGs */\n│   ├── README.md\n│   ├── dags/\n│   │   ├── __pycache__/\n│   │   ├── .gitkeep\n│   │   ├── airquality_pipeline_dag.py\n│   │   ├── data_migration_dag.py\n│   │   └── utils/\n│   │       ├── __pycache__/\n│   │       ├── airquality_collector.py\n│   │       └── migrate_data.py\n│   ├── logs/\n│   └── plugins/\n├── airflow-docker-compose.yaml\n├── data/                     /* Sample or test data */\n│   ├── location_air_quality.json\n│   └── sensor_air_quality.json\n├── data_validation/          /* Data quality check with Great Expectations */\n│   ├── .gitkeep\n│   ├── README.md\n│   ├── gx/\n│   │   ├── .gitignore\n│   │   ├── checkpoints/\n│   │   ├── expectations/\n│   │   ├── great_expectations.yml\n│   │   ├── plugins/\n│   │   ├── uncommitted/\n│   │   └── validation_definitions/\n│   └── openaq_data_quality.ipynb\n├── dbt_openaq/               /* dbt project for transformations and modeling */\n│   ├── .gitignore\n│   ├── .user.yml\n│   ├── README.md\n│   ├── analyses/\n│   ├── dbt_packages/\n│   ├── dbt_project.yml\n│   ├── logs/\n│   ├── macros/\n│   ├── models/\n│   │   └── production/\n│   │       ├── dim_date.sql\n│   │       ├── dim_instrument.sql\n│   │       ├── dim_location.sql\n│   │       ├── dim_owner.sql\n│   │       ├── dim_parameter.sql\n│   │       ├── dim_provider.sql\n│   │       ├── dim_sensor.sql\n│   │       ├── dim_time.sql\n│   │       ├── fact_air_quality_measurement.sql\n│   │       └── schema.yml\n│   ├── package-lock.yml\n│   ├── packages.yml\n│   ├── profiles.yml\n│   ├── seeds/\n│   ├── snapshots/\n│   ├── target/\n│   └── tests/\n├── docker-compose.yaml\n├── images/\n├── requirements.txt          /* Python dependencies */\n├── scripts/                  /* Utility scripts */\n│   ├── __init__.py\n│   ├── __pycache__/\n│   ├── create_schema.py\n│   ├── create_table.py\n│   └── postgresql_client.py\n├── storage-docker-compose.yaml  /* Data lake components (Trino, MinIO, Hive) */\n├── trino-minio/              /* Trino and MinIO configuration */\n│   ├── conf/\n│   │   └── metastore-site.xml\n│   └── etc/\n│       ├── catalog/\n│       │   └── minio.properties\n│       ├── config.properties\n│       ├── jvm.config\n│       ├── log.properties\n│       └── node.properties\n└── venv/\n\n```\n\n## ⚙ Workflow Diagram\n![WorkflowDiagram](https://github.com/user-attachments/assets/b3682736-6543-496f-a7c2-7216af31fc0f)\n\n## 🔥 NOTE\n#### The Project is still in progress, so stay tuned!! 🙌🏻\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdain55788%2Felt-data-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdain55788%2Felt-data-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdain55788%2Felt-data-pipeline/lists"}