{"id":25196159,"url":"https://github.com/dzrekenathan/dbt-data-pipeline","last_synced_at":"2026-04-28T20:02:11.988Z","repository":{"id":242526045,"uuid":"809798767","full_name":"dzrekenathan/dbt-data-pipeline","owner":"dzrekenathan","description":"A dbt data pipeline capstone project.","archived":false,"fork":false,"pushed_at":"2024-06-10T20:38:54.000Z","size":119,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-10T01:40:05.680Z","etag":null,"topics":["airflow","airflow-dags","dataengineering","dbt","docker","python","snowflake","sql"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dzrekenathan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-03T13:22:39.000Z","updated_at":"2024-06-10T20:38:58.000Z","dependencies_parsed_at":"2024-06-10T22:14:24.537Z","dependency_job_id":null,"html_url":"https://github.com/dzrekenathan/dbt-data-pipeline","commit_stats":null,"previous_names":["techwithnate/dbt-data-pipeline","dzrekenathan/dbt-data-pipeline"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dzrekenathan%2Fdbt-data-pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dzrekenathan%2Fdbt-data-pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dzrekenathan%2Fdbt-data-pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dzrekenathan%2Fdbt-data-pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dzrekenathan","download_url":"https://codeload.github.com/dzrekenathan/dbt-data-pipeline/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247208063,"owners_count":20901568,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","airflow-dags","dataengineering","dbt","docker","python","snowflake","sql"],"created_at":"2025-02-10T01:40:15.266Z","updated_at":"2026-04-28T20:02:11.894Z","avatar_url":"https://github.com/dzrekenathan.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# An ELT data pipeline Using DBT, Snowflake and Airflow\nA dbt data pipeline capstone project.\n\n\u003ch1 align=\"center\"\u003eETL Data pipeline Project\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"#technologies\"\u003eTechnologies\u003c/a\u003e •\n  \u003ca href=\"#about-the-project\"\u003eAbout the project\u003c/a\u003e •\n  \u003ca href=\"#conceptual-architecture\"\u003eConceptual architecture\u003c/a\u003e •\n  \u003ca href=\"#conceptual-report-on-the-technologies-used\"\u003eConceptual Report on the Technologies used\u003c/a\u003e •\n  \u003ca href=\"#project-setup\"\u003eProject Set up\u003c/a\u003e\n\u003c/p\u003e\n\n---\n\n## Technologies\n ![Apache Airflow](https://img.shields.io/badge/Apache%20Airflow-017CEE?style=for-the-badge\u0026logo=Apache%20Airflow\u0026logoColor=white)\n ![Docker](https://img.shields.io/badge/docker-%230db7ed.svg?style=for-the-badge\u0026logo=docker\u0026logoColor=white)\n ![MySQL](https://img.shields.io/badge/mysql-4479A1.svg?style=for-the-badge\u0026logo=mysql\u0026logoColor=white)\n ![Python](https://img.shields.io/badge/python-3670A0?style=for-the-badge\u0026logo=python\u0026logoColor=ffdd54)\n ![Snowflake](https://a11ybadges.com/badge?logo=snowflake)\n ![dbt](https://a11ybadges.com/badge?logo=dbt)\n\n ---\n\n## About the project\n\nA data engineering training project to build an end-to-end data pipline for a real-time processing of data. The project implements an ELT batch data processing. \nData is fetched daily using airflow into into a snowflake data warehouse for further analysis.\n\nIn addition, process data can then be used for visual analytics.\n\n---\n\n## Conceptual architecture\n\n```bash\n           +-------------------------+\n           |   External Data Source  |\n           +-------------------------+\n                      |\n                      V\n           +-------------------------+\n           |       Snowflake         |\n           |      Data Warehouse     |\n           +-------------------------+\n                      |\n                      V\n+------------------------------------------------------+\n|                  ELT Pipeline (Airflow)              |\n|                                                      |\n|               +--------------+     +------------+    |\n|               |    Extract   | --\u003e |    Load    |    |\n|               +--------------+     +------------+    |\n|                        |                     |        |\n|                        V                     V        |\n|               +--------------+     +------------+    |\n|               |   Transform  |     | dbt Models |    |\n|               +--------------+     +------------+    |\n+------------------------------------------------------+\n```\n  \n---\n\n## Conceptual Report on the Technologies used\n### dbt (Data Build Tool)\nDefinition: dbt (data build tool) is an open-source command-line tool that enables data analysts and engineers to transform data in their warehouses more effectively. It allows users to write modular, SQL-based transformation logic in the form of \"models\" and run these transformations against their data warehouse.\n\n| Pros | Cons |\n| --- | --- |\n| **SQL-based:** Allows analysts to write transformations using familiar SQL syntax. | **Customization:** Increased customization for incremental models to support larger data sets |\n| **Modular:** Encourages the creation of modular, reusable transformation logic. | **Learning Curve:** Requires knowledge of SQL and familiarity with dbt concepts. |\n| **Testing:** Provides built-in testing capabilities to ensure data quality and correctness. | **Dependency Management:** Managing dependencies between models can become complex in larger projects. |\n\n---\n### Snowflake\nSnowflake is a cloud-based data warehousing platform that provides a fully managed service for storing, analyzing, and processing large volumes of data. It offers features such as elastic scaling, separation of storage and compute, and support for structured and semi-structured data.\n\n| Pros | Cons |\n| --- | --- |\n| Handles infrastructure management, optimization, and tuning, allowing users to focus on data analysis. | Pricing can be higher compared to self-managed solutions, especially for large workloads. |\n| Supports JSON, Avro, Parquet, and other semi-structured data formats. | Managing data pipelines and access controls can be complex in larger deployments. |\n|  Allows users to scale compute independently of storage, providing cost efficiency. | Being a cloud-specific service, migrating away from Snowflake can be challenging. | \n\n### Airflow (Apache Airflow)\nApache Airflow is an open-source workflow orchestration tool used for scheduling, monitoring, and managing complex data pipelines. It allows users to define workflows as directed acyclic graphs (DAGs) and execute tasks in a distributed and fault-tolerant manner.\n| Pros | Cons |\n| --- | --- |\n| Enables the creation and management of complex data pipelines as DAGs. | Setting up and configuring Airflow can be complex, especially for beginners. |\n| Provides built-in monitoring and alerting capabilities to track pipeline progress and detect failures. | Requires resources for running the Airflow scheduler, web server, and worker nodes. |\n| Supports dynamic DAG generation based on runtime parameters and conditions. | Users need to learn Airflow concepts such as DAGs, operators, sensors, and hooks. |\n\n---\n## Airflow Graphs\n![image](https://github.com/TechWithNate/dbt-data-pipeline/assets/81887567/7872ef38-352c-42d0-b414-dd38069b1b51)\n\n![image](https://github.com/TechWithNate/dbt-data-pipeline/assets/81887567/e4de9a90-30e2-4758-a728-b08b50eb2e72)\n\n## Project Setup\n### Step 1: Create a snowflake account and setup snowflake environment\n```bash\n\n-- create accounts\nuse role accountadmin;\n\ncreate warehouse dbt_wh with warehouse_size='x-small';\ncreate database if not exists dbt_db;\ncreate role if not exists dbt_role;\n\nshow grants on warehouse dbt_wh;\n\ngrant role dbt_role to user jayzern;\ngrant usage on warehouse dbt_wh to role dbt_role;\ngrant all on database dbt_db to role dbt_role;\n\nuse role dbt_role;\n\ncreate schema if not exists dbt_db.dbt_schema;\n\n-- clean up\nuse role accountadmin;\n\ndrop warehouse if exists dbt_wh;\ndrop database if exists dbt_db;\ndrop role if exists dbt_role;\n\n```\n### Step 2. Clone the repository\n```bash\ngit clone https://github.com/TechWithNate/dbt-data-pipeline\n```\n### Step 3. Install the requirements \nOpen your terminal and run the command\n```bash\npip install -r requirements.txt\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdzrekenathan%2Fdbt-data-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdzrekenathan%2Fdbt-data-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdzrekenathan%2Fdbt-data-pipeline/lists"}