{"id":19307386,"url":"https://github.com/artefactory/dbt_intro_da","last_synced_at":"2025-04-22T12:36:34.265Z","repository":{"id":203183636,"uuid":"650977194","full_name":"artefactory/dbt_intro_da","owner":"artefactory","description":null,"archived":false,"fork":false,"pushed_at":"2024-03-29T13:41:14.000Z","size":7798,"stargazers_count":1,"open_issues_count":0,"forks_count":6,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-02T00:01:39.032Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/artefactory.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-08T08:13:35.000Z","updated_at":"2023-11-01T12:30:29.000Z","dependencies_parsed_at":"2024-04-02T01:00:33.201Z","dependency_job_id":null,"html_url":"https://github.com/artefactory/dbt_intro_da","commit_stats":null,"previous_names":["artefactory/dbt_intro_da"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/artefactory%2Fdbt_intro_da","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/artefactory%2Fdbt_intro_da/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/artefactory%2Fdbt_intro_da/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/artefactory%2Fdbt_intro_da/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/artefactory","download_url":"https://codeload.github.com/artefactory/dbt_intro_da/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250242077,"owners_count":21398061,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T00:10:45.825Z","updated_at":"2025-04-22T12:36:32.369Z","avatar_url":"https://github.com/artefactory.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"## SETUP:\n\n\n### Requirements\n\n- [Download and install Python 3.11.x](https://www.python.org/downloads/release/python-3116/)\n- [Download and install Git](https://git-scm.com/downloads)\n- [Download and install VScode](https://code.visualstudio.com/download)\n- For windows: [Download and install windows terminal](https://learn.microsoft.com/en-us/windows/terminal/install)\n\n### Get the code and install python environment\n\n- Open a terminal\n- Clone the repository: `git clone https://github.com/artefactory/dbt_intro_da.git`\n- Change directory in the repository folder: `cd dbt_intro_da`\n- If you have conda installed you will need to deactivate it: `conda deactivate`\n- Setup a python environment: \n    - MacOs: `python3 -m venv venv`\n    - Windows: `python -m venv venv`\n- Activate python environment:\n    - MacOs: `source venv/bin/activate`\n    - Windows:\n      - `Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope CurrentUser`\n      - `. .\\venv\\Scripts\\activate`\n- Install python packages `pip install -r requirements.txt`\n\n\u003cdetails\u003e\n  \u003csummary\u003eAdditionnal setup to work with bigquery (optionnal)\u003c/summary\u003e\n\n### Additionnal setup to work with bigquery (optionnal)\n\nIn order to work with bigquery you need to change the target in the profiles to the bigquery target.\nYou also need to update the dataset used in the profile by changing `name` to your name.\n\n```\ndbt_intro_da:\n  target: dev_bigquery\n  outputs:\n    duckdb:\n      path: dbt.duckdb\n      type: duckdb\n      threads: 4\n    dev_bigquery:\n      type: bigquery\n      method: oauth\n      project: formation-sql-316408\n      dataset: dbt_intro_da_name\n      location: EU\n      threads: 4\n\n```\n\n- [Download and install Gcloud](https://cloud.google.com/sdk/docs/install)\n- Connect to gcloud:\n```\ngcloud auth application-default login \\\n--scopes=https://www.googleapis.com/auth/bigquery,\\\nhttps://www.googleapis.com/auth/drive.readonly,\\\nhttps://www.googleapis.com/auth/iam.test\n\n```\n\n\u003c/details\u003e\n\n\n### DBT Setup and test\n\n- `dbt debug` should now tell you everything is OK\n- `dbt deps` to install dbt packages used in this project\n- `dbt run`\n\nThe first `dbt run` should create a file named `dbt.duckdb`.\nThis file will contains all the tables views we are creating.\n\n\u003cdetails\u003e\n  \u003csummary\u003eVScode extension (optionnal but nice to have)\u003c/summary\u003e\n\n### VScode extension (optionnal but nice to have):\n\nTo help us navigate the dbt project we are using the extension dbt power user:\n- Install the extension: https://marketplace.visualstudio.com/items?itemName=innoverio.vscode-dbt-power-user\n\nThere is one setting that should be added\n- Go in settings\n    - ![](./assets/vs_code_extensions/dbt_power_user/01_open_settings.png)\n- Search for `files:assoc` and add `*.sql` in the field *Item* and `jinja-sql` in the *Value*\n    - ![](./assets/vs_code_extensions/dbt_power_user/02_add_in_settings.png)\n- Clicks on *run dbt SQL*\n    - ![](./assets/vs_code_extensions/dbt_power_user/03_query_model.png)\n- View lineage\n    - ![](./assets/vs_code_extensions/dbt_power_user/04_view_lineage.png)\n\n\u003c/details\u003e\n\n### dbt show\n\nTo look explore what is inside the database we can use the `dbt show` command: https://docs.getdbt.com/reference/commands/show\n\nFor example the command:\n```\ndbt show -s local_sales\n```\nshould output something like:\n```\n13:58:16  Running with dbt=1.6.6\n13:58:17  Registered adapter: duckdb=1.6.1\n13:58:17  Found 1 model, 4 sources, 0 exposures, 0 metrics, 351 macros, 0 groups, 0 semantic models\n13:58:17  \n13:58:17  Concurrency: 4 threads (target='dev_duckdb')\n13:58:17  \n13:58:17  Previewing node 'local_sales':\n| product_category     | pdt_SUB_CATEGORY  | barcode_ean13     | site_key          |       DATE |    CA |\n| -------------------- | ----------------- | ----------------- | ----------------- | ---------- | ----- |\n| APERITIFS            | SANS ALCOOL       | f_f_0327607249893 | f_f_8429768983288 | 2023-03-08 | 37.57 |\n| VQPRD ROUGES/HORS... | VINS DU SUD OUEST | f_f_4371089866174 | f_f_8429768983288 | 2023-10-15 | 11.79 |\n| ALCOOLS              | PUNCHS            | f_f_2239755478262 | f_f_8429768983288 | 2023-08-13 | 31.95 |\n| ALCOOLS              | GIN-VODKA-TEQUILA | f_f_6023176352753 | f_f_8429768983288 | 2023-06-20 | 22.36 |\n| VQPRD ROUGES/HORS... | VINS DU SUD OUEST | f_f_2550597061567 | f_f_8429768983288 | 2023-05-22 | 34.23 |\n```\nWhich are a couple of rows from the table defined by the dbt model `local_sales`\n\nWe can also run arbitrary queries:\n```\ndbt show --inline \"select distinct(product_category) from {{ ref('local_sales') }}\"\n```\nshould output something like:\n```\n13:58:20  Running with dbt=1.6.6\n13:58:20  Registered adapter: duckdb=1.6.1\n13:58:20  Found 1 model, 4 sources, 0 exposures, 0 metrics, 351 macros, 0 groups, 0 semantic models\n13:58:20  \n13:58:20  Concurrency: 4 threads (target='dev_duckdb')\n13:58:20  \n13:58:20  Previewing inline node:\n| product_category     |\n| -------------------- |\n| ALCOOLS              |\n| APERITIFS            |\n| VQPRD BORDEAUX RO... |\n| EAUX AROMATISEES     |\n| VINS DE TABLES/PAYS  |\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fartefactory%2Fdbt_intro_da","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fartefactory%2Fdbt_intro_da","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fartefactory%2Fdbt_intro_da/lists"}