{"id":22003529,"url":"https://github.com/riju18/dashing-datawarehouse-dbt","last_synced_at":"2025-05-05T16:46:00.910Z","repository":{"id":151804064,"uuid":"590325478","full_name":"riju18/Dashing-Datawarehouse-dbt","owner":"riju18","description":"Design the Data Warehouse with minimalistic code.","archived":false,"fork":false,"pushed_at":"2024-10-31T16:48:56.000Z","size":744,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-30T22:32:13.411Z","etag":null,"topics":["cte","data-warehouse","dbt","jinja","postgresql","python","sql"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/riju18.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-01-18T06:30:03.000Z","updated_at":"2024-10-31T16:49:00.000Z","dependencies_parsed_at":null,"dependency_job_id":"bf18c6fe-2af3-45aa-b8d5-93376a5a3d46","html_url":"https://github.com/riju18/Dashing-Datawarehouse-dbt","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/riju18%2FDashing-Datawarehouse-dbt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/riju18%2FDashing-Datawarehouse-dbt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/riju18%2FDashing-Datawarehouse-dbt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/riju18%2FDashing-Datawarehouse-dbt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/riju18","download_url":"https://codeload.github.com/riju18/Dashing-Datawarehouse-dbt/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252535707,"owners_count":21763996,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cte","data-warehouse","dbt","jinja","postgresql","python","sql"],"created_at":"2024-11-30T00:09:57.049Z","updated_at":"2025-05-05T16:46:00.903Z","avatar_url":"https://github.com/riju18.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Welcome to Dashing DataWarehouse with dbt project!\n\n# index\n- [Data Architecture](#data_architecture)\n- [DAG Sample](#dag_sample)\n- [Directory Structure](#dir-structure)\n- [Prerequisites](#prerequisites)\n- [How to Run This Project](#how_to_run_this_project)\n- [Lessons learned](#lessons_learned)\n\n# data_architecture\n```mermaid\nflowchart LR\n\npublic[Prod DB] --\u003e dbt\n\ndbt --clean, raw data--\u003e dbt_src\ndbt --non redundant--\u003e dbt_dim\ndbt --analytical--\u003e dbt_fact\ndbt --scd--\u003e snapshots\n```\n\n# dag_sample\n```mermaid\nflowchart LR\nfilm_category --prod to dwh--\u003e src_film_category --dimension--\u003e dim_film\ncategory --prod to dwh--\u003e src_category --dimension--\u003e dim_film\nfilm --prod to dwh--\u003e src_film --dimension--\u003e dim_film\nlanguage --prod to dwh--\u003e src_language --dimension--\u003e dim_film\n\ndim_film --data tests--\u003e movie_release_year_not_less_than_2006\n```\n\n# dir-structure\n\n```\nproject\n│   README.md\n│   requirements.txt  \n|   .gitignore  \n|   packages.yml\n|   dbt_project.yml\n│\n└───miscellaneous\n│   │   dvdrental.tar     : example DB for postgres\n│   \n└───models\n│   │   schema.yml       : for generic test \u0026 documentation\n|   └───source.yml       : Data sources\n│   └───src              : all core sql file\n│   └───dim              : data generated from src\n│   └───fact             : data generated from src\n│\n└───seeds\n│   │   *.csv     : static file want to export in DWH\n│   \n└───snapshots\n│   │   *.sql     : history/log of tables\n│   \n└───tests\n│   │   *.sql     : custom singular test file \n│\n└───macros\n│   │   test_*.sql     : generic(parameterized) test file \n```\n\n# prerequisites\n- Python\n- SQL\n- PostgreSQL\n- dbt\n- Linux/mac\n\n# how_to_run_this_project\n+ create python venv\n    ```sh\n    python3 -m venv venv_name\n    ```\n+ activate: \n    ```sh \n    source venv_name/bin/activate \n    ```\n+ install dependency\n    ```sh\n    pip install -r requirements.txt\n    ```\n\n+ DB setup(`postgres` )\n    + **DBName**: dvdrental\n    + restore the DB: ```miscellaneous --\u003e dvdrental.tar``` in ```public``` schema\n\n+ clone the repo\n+ clone and rename the `profiles_sample.yml` to `profiles.yml`\n+ Update the credentials in `profiles.yml`\n+ check config: ```dbt debug --target dev```\n+ install dbt packages: ```dbt deps --target dev```\n+ test project: ```dbt test --target dev```\n+ run project: ```dbt run --target dev```\n+ snapshot/log generate: ```dbt snapshot --target dev```\n+ Documentation of project:\n    ```sh\n    dbt docs generate --target dev\n    ```\n    ```sh\n    dbt docs serve --port 8002 --target dev\n    ```\n\n# lessons_learned\n- materialization\n    - table\n    - view\n    - incremental\n    - ephemeral (`not implemented`)\n- macros\n- test\n    - generic\n    - singular\n- snapshot\n    - SCD2\n- dbt docs\n- configurations\n    - `dbt_project.yml`\n    - `source.yml`\n    - `schema.yml`","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Friju18%2Fdashing-datawarehouse-dbt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Friju18%2Fdashing-datawarehouse-dbt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Friju18%2Fdashing-datawarehouse-dbt/lists"}