{"id":26038348,"url":"https://github.com/ndomah/elt-pipeline","last_synced_at":"2025-07-24T02:32:27.843Z","repository":{"id":277386923,"uuid":"932273888","full_name":"ndomah/ELT-Pipeline","owner":"ndomah","description":"Simple ELT pipeline using dbt, Snowflake, and Apache Airflow.","archived":false,"fork":false,"pushed_at":"2025-02-14T17:02:10.000Z","size":250,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-07T09:42:00.495Z","etag":null,"topics":["airflow","dbt","elt","elt-pipeline","python","snowflake","sql"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ndomah.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-13T16:45:34.000Z","updated_at":"2025-02-14T17:04:43.000Z","dependencies_parsed_at":"2025-02-13T17:47:05.276Z","dependency_job_id":"775c1e63-11fa-4cad-ab01-2a2a72d82bc1","html_url":"https://github.com/ndomah/ELT-Pipeline","commit_stats":null,"previous_names":["ndomah/elt-pipeline"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ndomah/ELT-Pipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndomah%2FELT-Pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndomah%2FELT-Pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndomah%2FELT-Pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndomah%2FELT-Pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ndomah","download_url":"https://codeload.github.com/ndomah/ELT-Pipeline/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndomah%2FELT-Pipeline/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266785479,"owners_count":23983824,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-24T02:00:09.469Z","response_time":99,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","dbt","elt","elt-pipeline","python","snowflake","sql"],"created_at":"2025-03-07T09:41:34.229Z","updated_at":"2025-07-24T02:32:27.582Z","avatar_url":"https://github.com/ndomah.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ELT Pipeline using dbt, Snowflake, and Airflow\n\nThis project demonstrates how to build a simple ELT pipeline using [dbt (Data Build Tool)](https://www.getdbt.com/) to transform data in [Snowflake](https://www.snowflake.com/en/), with orchestration managed by [Apache Airflow](https://airflow.apache.org/). This setup showcases a modern data engineering workflow, essential for handling large-scale data transformations efficiently.\n\n\n## 1. Setting up Snowflake\nWe will use Snowflake as our data warehouse, leveraging the sample dataset TPCH_SF1.\n\n### Configuring Access Control\nSnowflake’s RBAC (Role-based Access Control) is utilized to manage permissions. A dedicated `dbt_role` is created and assigned to the user.\n\n```sql\nUSE ROLE accountadmin;\nCREATE ROLE IF NOT EXISTS dbt_role;\nGRANT ROLE dbt_role TO USER \u003cSnowflake username\u003e;\n```\n\n### Creating the Warehouse and Database\nWe create the required warehouse and database for our dbt transformations.\n\n```sql\nCREATE WAREHOUSE dbt_wh WITH WAREHOUSE_SIZE='X-SMALL';\nCREATE DATABASE IF NOT EXISTS dbt_db;\nGRANT USAGE ON WAREHOUSE dbt_wh TO ROLE dbt_role;\nGRANT ALL ON DATABASE dbt_db TO ROLE dbt_role;\nUSE ROLE dbt_role;\nCREATE SCHEMA IF NOT EXISTS dbt_db.dbt_schema;\n```\n\n\n\n## 2. Setting up dbt Project\nTo integrate dbt with Apache Airflow, we use [astronomer-cosmos](https://github.com/astronomer/astronomer-cosmos). The setup involves initializing an Astro project and configuring dbt.\n\n### Initializing the Astro Project\n```bash\ncurl -sSL install.astronomer.io | sudo bash -s  # Install Astro CLI\nmkdir elt_project \u0026\u0026 cd elt_project\nastro dev init\n```\n\nThis generates a directory structure including DAGs, plugins, and dependencies.\n\n### Creating the dbt Project\n```bash\npython -m venv dbt-env  # Create virtual environment\nsource dbt-env/bin/activate  # Activate environment\npip install dbt-core dbt-snowflake  # Install dbt\nmkdir dags/dbt \u0026\u0026 cd dags/dbt\ndbt init  # Initialize dbt project\n```\n\nDuring initialization, provide Snowflake credentials and database details.\n\n\n## 3. Building the Data Model\nWe process the `orders` and `lineitem` tables from TPCH_SF1.\n\n![Data Model](https://github.com/ndomah/ELT-Pipeline/blob/main/img/data-model.png)\n\n### Configuring dbt Models\nWe define sources and transformations in dbt, specifying materialization strategies:\n\n```yaml\nmodels:\n  data_pipeline:\n    staging:\n      +materialized: view\n      snowflake_warehouse: dbt_wh\n    marts:\n      +materialized: table\n      snowflake_warehouse: dbt_wh\n```\n\nThe [`models/staging/stg_tpch_orders.sql`](https://github.com/ndomah/ELT-Pipeline/blob/main/dags/dbt/data_pipeline/models/staging/stg_tpch_orders.sql) transformation extracts required fields from `orders`:\n\n```sql\nSELECT\n    o_orderkey AS order_key,\n    o_custkey AS customer_key,\n    o_orderstatus AS status_code,\n    o_totalprice AS total_price,\n    o_orderdate AS order_date\nFROM {{ source('tpch', 'orders') }}\n```\n\nSimilarly, [`models/staging/stg_tpch_line_item.sql`](https://github.com/ndomah/ELT-Pipeline/blob/main/dags/dbt/data_pipeline/models/staging/stg_tpch_line_item.sql) processes `lineitem` and creates a surrogate key:\n\n```sql\nSELECT\n    {{ dbt_utils.generate_surrogate_key(['l_orderkey', 'l_linenumber']) }} AS order_item_key,\n    l_orderkey AS order_key,\n    l_partkey AS part_key,\n    l_linenumber AS line_number,\n    l_quantity AS quantity,\n    l_extendedprice AS extended_price,\n    l_discount AS discount_percentage,\n    l_tax AS tax_rate\nFROM {{ source('tpch', 'lineitem') }}\n```\n\nThe fact table [`fct_orders.sql`](https://github.com/ndomah/ELT-Pipeline/blob/main/dags/dbt/data_pipeline/models/marts/fct_orders.sql) integrates orders and order summaries:\n\n```sql\nSELECT\n    orders.*,\n    order_item_summary.gross_item_sales_amount,\n    order_item_summary.item_discount_amount\nFROM {{ ref('stg_tpch_orders') }} AS orders\nJOIN {{ ref('dim_order_items_summary') }} AS order_item_summary\n    ON orders.order_key = order_item_summary.order_key\nORDER BY order_date\n```\n\nTests ensure data integrity, checking constraints such as unique `order_key` values.\n\n```yaml\nmodels:\n  - name: fct_orders\n    columns:\n      - name: order_key\n        tests:\n          - unique\n          - not_null\n          - relationships:\n              to: ref('stg_tpch_orders')\n              field: order_key\n```\n\nRunning dbt:\n```bash\ndbt test  # Run tests\ndbt run  # Execute transformations\ndbt build  # Execute and test\n```\n\n\n## 4. Orchestrating with Airflow\nWe configure Airflow to run dbt models using the [astronomer-cosmos](https://github.com/astronomer/astronomer-cosmos) package.\n\n### Configuring Airflow\nModify [`Dockerfile`](https://github.com/ndomah/ELT-Pipeline/blob/main/Dockerfile) to install dbt inside a virtual environment:\n\n```dockerfile\nFROM quay.io/astronomer/astro-runtime:11.4.0\nRUN python -m venv dbt_venv \u0026\u0026 source dbt_venv/bin/activate \u0026\u0026 \\\n    pip install --no-cache-dir dbt-snowflake \u0026\u0026 deactivate\n```\n\nAdd necessary dependencies in [`requirements.txt`](https://github.com/ndomah/ELT-Pipeline/blob/main/requirements.txt):\n\n```\nastronomer-cosmos\napache-airflow-providers-snowflake\n```\n\n### Creating the DAG\nDefine [`dags/dbt/dbt_dag.py`](https://github.com/ndomah/ELT-Pipeline/blob/main/dags/dbt/dbt_dag.py) to run dbt models:\n\n```python\nimport os\nfrom datetime import datetime\nfrom cosmos import DbtDag, ProjectConfig, ProfileConfig, ExecutionConfig, RenderConfig\nfrom cosmos.profiles import SnowflakeUserPasswordProfileMapping\nfrom cosmos.constants import TestBehavior\n\nprofile_config = ProfileConfig(\n    profile_name='default',\n    target_name='dev',\n    profile_mapping=SnowflakeUserPasswordProfileMapping(\n        conn_id='snowflake_conn',\n        profile_args={'database': 'dbt_db', 'schema': 'dbt_schema'}\n    )\n)\n\ndbt_snowflake_dag = DbtDag(\n    project_config=ProjectConfig('/usr/local/airflow/dags/dbt/data_pipeline'),\n    operator_args={'install_deps': True},\n    profile_config=profile_config,\n    execution_config=ExecutionConfig(dbt_executable_path=f\"{os.environ['AIRFLOW_HOME']}/dbt_venv/bin/dbt\"),\n    render_config=RenderConfig(test_behavior=TestBehavior.AFTER_ALL),\n    schedule_interval='@daily',\n    start_date=datetime(2024, 6, 1),\n    catchup=False,\n    dag_id='dbt_dag'\n)\n```\n\n### Running Airflow\nStart Airflow locally:\n\n```bash\nastro dev start\n```\n\nOnce running, access the Airflow UI at [http://localhost:8080](http://localhost:8080).\n\n### DAG Execution\n![DAG Execution](https://github.com/ndomah/ELT-Pipeline/blob/main/img/dbt_dag_success.png)\n\nAfter setting up the Snowflake connection in Airflow, trigger the DAG to orchestrate the ELT pipeline successfully.\n\n## Conclusion\nThis project showcases a modern ELT pipeline with dbt, Snowflake, and Airflow, demonstrating efficient data transformation and orchestration workflows—ideal for building scalable and maintainable data pipelines.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fndomah%2Felt-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fndomah%2Felt-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fndomah%2Felt-pipeline/lists"}