{"id":26011449,"url":"https://github.com/gbourniq/dbt-fundamentals","last_synced_at":"2025-06-22T12:08:51.268Z","repository":{"id":199596950,"uuid":"703281632","full_name":"gbourniq/dbt-fundamentals","owner":"gbourniq","description":"Repository to support learning from the dbt courses","archived":false,"fork":false,"pushed_at":"2023-10-16T22:03:28.000Z","size":1693,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-05T23:19:44.855Z","etag":null,"topics":["data-engineering","data-pipeline","dbt","github-actions","poetry-python","snowflake","sql","sqlfluff"],"latest_commit_sha":null,"homepage":"","language":"Makefile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gbourniq.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-10-11T00:31:25.000Z","updated_at":"2024-10-16T19:06:06.000Z","dependencies_parsed_at":"2023-10-14T23:30:12.250Z","dependency_job_id":null,"html_url":"https://github.com/gbourniq/dbt-fundamentals","commit_stats":null,"previous_names":["gbourniq/dbt-demo","gbourniq/dbt-fundamentals"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/gbourniq/dbt-fundamentals","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gbourniq%2Fdbt-fundamentals","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gbourniq%2Fdbt-fundamentals/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gbourniq%2Fdbt-fundamentals/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gbourniq%2Fdbt-fundamentals/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gbourniq","download_url":"https://codeload.github.com/gbourniq/dbt-fundamentals/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gbourniq%2Fdbt-fundamentals/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261289417,"owners_count":23136070,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-engineering","data-pipeline","dbt","github-actions","poetry-python","snowflake","sql","sqlfluff"],"created_at":"2025-03-05T23:19:47.479Z","updated_at":"2025-06-22T12:08:46.250Z","avatar_url":"https://github.com/gbourniq.png","language":"Makefile","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Repository for the dbt courses\n\n![Data Lineage](.github/assets/lineage_graph.png)\n\n## dbt documentation\n\n\u003chttps://courses.getdbt.com/courses\u003e\n\u003chttps://docs.getdbt.com/guides/best-practices\u003e\n\u003chttps://docs.getdbt.com/docs/introduction\u003e\n\u003chttps://docs.getdbt.com/reference/references-overview\u003e\n\n## Snowflake instance\n\n\u003chttps://qr92018.eu-west-2.aws.snowflakecomputing.com/\u003e\n\n# Setup\n\nInstall dbt in a virtual environment\n\n```bash\npip install --no-dependencies -r requirements.txt\n```\n\nCreate a `~/.dbt/profiles.yml`\n\n```\npersonal_profile:\n  target: dev\n  outputs:\n    dev:\n      type: snowflake\n      account: ****\n      user: ****\n      password: ****\n      role: ****\n      database: ****\n      warehouse: ****\n      schema: ****\n      threads: 8\n      client_session_keep_alive: False\n      query_tag: ****\n      connect_retries: 0\n      connect_timeout: 10\n      retry_on_database_errors: False\n      retry_all: False\n      reuse_connections: False\n```\n\nCheck connection\n\n```bash\ndbt deps\ndbt debug\n```\n\nCreate database, schemas and tables\n\n```bash\ndbt run-operation create_db_objects\ndbt run-operation create_sample_src_tables\n```\n\nBuild and test dbt models.\n\n```bash\ndbt build\n```\n\n## Notes on materializations in dbt\n\nMaterialization defines how dbt builds the models (select statements).\n\n* Tables: Copy of the data from upstream into a NEW table.\n  * Higher build time and more storage usage, but shorter query time\n  * If new record added in upstream stable, model/table is rebuilt from scratch.\n* Views: No copy, data remains in the upstream table, and the query is run every time we do a select on the view.\n  * Quicker build time and less storage usage, but longer query time\n  * If new record added in upstream stable, it will be part of the view/query, but slower.\n* Ephemeral: Brings CTE into downstream models (another select statement), but no persistent storage, do not exist in the DB.\n  * Can be used if too many CTEs in a model, and it makes sense to break it down, or that the CTE can be reused\n  in other downstream models, but we don't want to store the data.\n  Help reduces the amount of tables in the WH, but harder to debug, cannot be queried directly.\n* Incremental: Only brings in new records in upstream tables, do not rebuild the table from scratch\n* Snapshot: Look at changed records, if anything changes, bring the updated records as a NEW row - no loss of information\n  * Used for type 2 slowly changing dimension tables: \u003chttps://docs.getdbt.com/docs/build/snapshots\u003e\n  * Preserve the history of changing fields (e.g. dobjects, sec master)\n  * Snapshots: \u003chttps://courses.getdbt.com/courses/take/advanced-materializations/lessons/30195287-implementing-snapshots\u003e\n  * Usually applied on source tables\n\nGenerally, start with a view, when it takes too long to query, \"upgrade\" the model to a table.\nWhen the table takes too long to build, consider upgrading to an incremental table.\nUpgrading to an incremental table is not straightforward, follow the [dbt documentation](https://courses.getdbt.com/courses/take/advanced-materializations/lessons/30195285-incremental-models)\n\n## Node selection syntax\n\n\u003chttps://docs.getdbt.com/reference/node-selection/syntax\u003e\n\n## Tests\n\n* Native tests and packages (dbt_expectations) in yml\n* Singular tests: sql files in the tests/ folder, with reference to models.\n  Tests business logic and shows up in the lineage graph.\n* Generic tests, macro-like, parameterized sql, which are reusable across models.\n  Usually start with a singular test (pure sql) then \"promote\" to a generic tests for reusability.\n* Override native tests by creating a generic tests with the same name, e.g 'not_null', for customization\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgbourniq%2Fdbt-fundamentals","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgbourniq%2Fdbt-fundamentals","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgbourniq%2Fdbt-fundamentals/lists"}