https://github.com/dbt-labs/snowflake-resource-monitoring

https://github.com/dbt-labs/snowflake-resource-monitoring

Last synced: about 1 month ago
JSON representation

• Host: GitHub
• URL: https://github.com/dbt-labs/snowflake-resource-monitoring
• Owner: dbt-labs
• Created: 2022-09-07T15:59:35.000Z (over 2 years ago)
• Default Branch: main
• Last Pushed: 2024-09-02T14:40:25.000Z (5 months ago)
• Last Synced: 2024-10-30T05:42:54.078Z (4 months ago)
• Size: 16.6 KB
• Stars: 7
• Watchers: 1
• Forks: 2
• Open Issues: 1
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

• awesome-dbt - snowflake-resource-monitoring - Yet another package to monitor Snowflake usage. (Packages)

README

        
Welcom to the Snowflake Resource Monitoring Project!

NOTE: This is Snowflake-only at present (with a thought on how to extend to other warehouses :) ) 

### Background

Monitoring data warehouse utilization levels becomes a key consideration as analytics organizations mature and grow. In dbt, the [snowflake spend](https://hub.getdbt.com/gitlabhq/snowflake_spend/latest/) package has gained some traction as an approach for doing this. Snowflake spend is great for 

analyzing workloads according to database, schema, user and warehouse. However, to really identify potential bottlenecks and areas for improvement, it's useful to go deeper by understanding the trends in model run times, configuration, user behavior in development vs. production environments, etc.

### Setup

Add the macro code from `config/set_query_tag.sql` to dbt Snowflake projects you want to monitor. Doing this will override the default set_query_tag() macro that ships with dbt. This version of the package uses some dbt Cloud specific environment variables:

- DBT_CLOUD_JOB_ID

- DBT_CLOUD_RUN_ID

- DBT_CLOUD_RUN_REASON

### Potential Future Directions: 

- Make this approach generic across Data Warehouses by logging the models and test runs in an `on-run-end` hook + macro.

- Build additional reporting and pre-canned alerts

- Build time series anomaly detection on usage data with dbt + Python models 

- Semantic layer integration with partners for pre-canned monitoring dashboards 

- Create similar visuals in Administrative reporting within dbt Cloud?