{"id":26820510,"url":"https://github.com/deepakramani/etl_with_dbt_dwh","last_synced_at":"2025-03-30T06:33:10.018Z","repository":{"id":283467732,"uuid":"951828869","full_name":"deepakramani/etl_with_dbt_dwh","owner":"deepakramani","description":"A small scale ETL project that ingests data into the data warehouse using dbt(data build tool) to facilitate transformation and analytics.","archived":false,"fork":false,"pushed_at":"2025-03-27T20:30:37.000Z","size":882,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-27T21:33:32.410Z","etag":null,"topics":["analytics-engineering","data-engineering","dbt","dimensional-modeling","etl","jinja","sql"],"latest_commit_sha":null,"homepage":"","language":"Makefile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deepakramani.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-20T10:02:13.000Z","updated_at":"2025-03-26T11:25:52.000Z","dependencies_parsed_at":"2025-03-21T14:03:10.509Z","dependency_job_id":null,"html_url":"https://github.com/deepakramani/etl_with_dbt_dwh","commit_stats":null,"previous_names":["deepakramani/etl_with_dbt_dwh"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepakramani%2Fetl_with_dbt_dwh","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepakramani%2Fetl_with_dbt_dwh/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepakramani%2Fetl_with_dbt_dwh/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepakramani%2Fetl_with_dbt_dwh/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deepakramani","download_url":"https://codeload.github.com/deepakramani/etl_with_dbt_dwh/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246285632,"owners_count":20752951,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics-engineering","data-engineering","dbt","dimensional-modeling","etl","jinja","sql"],"created_at":"2025-03-30T06:33:09.579Z","updated_at":"2025-03-30T06:33:10.006Z","avatar_url":"https://github.com/deepakramani.png","language":"Makefile","funding_links":[],"categories":[],"sub_categories":[],"readme":"A small-scale ETL project to demonstrate the capabilities of dbt in adding flexibility to data transformations. This is a portfolio project that takes a small dataset and follows some of the industry standards of developing an ETL pipeline and maintaining it. Feel free to suggest improvements or point out mistake from me to learn. \n\n# Instructions to run the ETL pipeline and bring data to data warehouse\n\nThis project is tested with AWS/GCP linux machine running Ubuntu 20.04 and Mac M1. Since the focus is on building an ETL pipeline, the user is advised patience if the setup has bugs or problems.\n\n## Install Docker and Conda\n\n```{.bash filename=\"Clone, install docker and conda\"}\ncd ~\nsudo apt update \u0026\u0026 sudo apt install git make -y\ngit clone https://github.com/deepakramani/etl_with_dbt_dwh.git\ncd etl_with_dbt_dwh\nmake install_conda\nmake install_docker\nsource ~/.bashrc\n```\n\nLogout and log in back to the instance. To test docker if it is working, run\n```\ndocker run --rm hello-world # should return \"Hello from Docker!\" without errors\n```\n**Set environment variables:**\n\n```{.bash filename=\"export env variables\"}\nexport POSTGRES_USER=postgres                                            \nexport POSTGRES_PASSWORD=postgres\nexport POSTGRES_HOST=localhost\nexport POSTGRES_DB=sql_dwh_db\nexport DBT_PROFILES_DIR=$(pwd)\n```\n\nNow we're ready start our project\n\n```\ncd ~/etl_with_dbt_dwh \nmake up # runs the postgres docker container, creates bronze layer tables and loads data from csv files into it.\nconda create -n mydbt python=3.9 dbt-core dbt-postgres -y # install conda env with dbt-core and dbt-postgres packages.\nconda activate mydbt\n\n```\n\nBefore we start `dbt`, make sure the `dbt_project.yml` is no errors, `macros` directory has the three generic tests, `tests` directory has one custom test file, in DB raw data tables are present inside `bronze` schema. \n\n```{.bash filename=\"run dbt commands\"}\ncd ~/etl_with_dbt_dwh  \nmake dbt_setup # cleans, debugs dbt project for errors and installs dependencies\nmake run_silver # runs only silver layer model\nmake run_gold # runs only gold layer model\nmake test_silver # data quality tests for silver layer tables\nmake test_gold # data quality tests for gold layer tables\n```\n\n# Explanation\nKeen observers following or read my other project using SQL, will realise the ingestion into postgres DB is done independent of the data transformation. Indeed, it is so. dbt allows `dbt seed` capability to seed `raw_data` into DB. However, I find this method slower than `bulk insert` mode with `COPY` command in postgres. \n\nNow that the `raw_data` is in the DB, the process is little simpler with just two layers -- staging and serving layer. Here we use `silver` for staging and `gold` for serving.\n\n**NOTE:** Since the `gold` layer is materialised as `view`, having issues with dbt execution order -- dbt executes the dependent `fact_sales` view before `dim_customers` and `dim_produts` view are fully materialised. Hence the split `dbt run` command. \n\nDBT also offers flexibility interms of writing tests -- generic or specific test cases. DBT provide a host of built-in functions that enable faster testing capability than long,cumbersome, repetitive SQL script. \n\n## dbt transformations\n\n\n## dbt data quality checks\n\nWherever possible dbt's built-in generic tests are used. \n\nCommon checks used are - \n- unique, not_null, expression_check, date_range_check, accepted_values_check, whitespaces_check.\n\nWhen built-in tests are available, custom test macro is implemented inside `macros` directory. This way allows testing to be flexible, extensible and non-repetitive.\nThese checks are written inside `schema.yml` allowing schema standardisation and possible evolution. \n\nSingular/specific test come under `tests` directory. We place a test that checks the gold layer view `fact_sales` which depends on `dim_customers` and `dim_products` forming a star schema inside the data warehouse. \n\n\n\n\nTry running the following commands:\n- dbt run\n- dbt test\n\n\n### Resources:\n- Learn more about dbt [in the docs](https://docs.getdbt.com/docs/introduction)\n- Check out [Discourse](https://discourse.getdbt.com/) for commonly asked questions and answers\n- Join the [chat](https://community.getdbt.com/) on Slack for live discussions and support\n- Find [dbt events](https://events.getdbt.com) near you\n- Check out [the blog](https://blog.getdbt.com/) for the latest news on dbt's development and best practices\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeepakramani%2Fetl_with_dbt_dwh","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeepakramani%2Fetl_with_dbt_dwh","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeepakramani%2Fetl_with_dbt_dwh/lists"}