Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/elementary-data/dbt-data-reliability
dbt package that is part of Elementary, the dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.
https://github.com/elementary-data/dbt-data-reliability
analytics analytics-engineering data data-lineage data-observability data-pipeline-monitoring data-pipelines data-reliability dbt dbt-artifacts dbt-packages dbt-tests
Last synced: 3 months ago
JSON representation
dbt package that is part of Elementary, the dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.
- Host: GitHub
- URL: https://github.com/elementary-data/dbt-data-reliability
- Owner: elementary-data
- License: apache-2.0
- Created: 2022-01-09T13:15:45.000Z (over 2 years ago)
- Default Branch: master
- Last Pushed: 2024-03-17T07:46:55.000Z (3 months ago)
- Last Synced: 2024-03-17T08:51:44.762Z (3 months ago)
- Topics: analytics, analytics-engineering, data, data-lineage, data-observability, data-pipeline-monitoring, data-pipelines, data-reliability, dbt, dbt-artifacts, dbt-packages, dbt-tests
- Language: Python
- Homepage: https://www.elementary-data.com/
- Size: 7.36 MB
- Stars: 327
- Watchers: 5
- Forks: 75
- Open Issues: 23
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Lists
- awesome-stars - elementary-data/dbt-data-reliability - dbt package that is part of Elementary, the dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service wit (Python)
README
![]()
dbt native data observability for analytics & data engineers
Monitor your data quality, operation and performance directly from your dbt project.## Quick start
1. Add to your `packages.yml`:
```yml packages.yml
packages:
- package: elementary-data/elementary
version: 0.13.2
## Docs: https://docs.elementary-data.com
```2. Run `dbt deps`
3. Add to your `dbt_project.yml`:
```yml
models:
## elementary models will be created in the schema '_elementary'
## for details, see docs: https://docs.elementary-data.com/
elementary:
+schema: "elementary"
```4. Run `dbt run --select elementary`
Check out the [full documentation](https://docs.elementary-data.com/) for generating the UI, alerts and adding anomaly detection tests.
## Run Results and dbt artifacts
The package automatically uploads the dbt artifacts and run results to tables in your data warehouse:
Run results tables:
- dbt_run_results
- model_run_results
- snapshot_run_results
- dbt_invocations
- elementary_test_results (all dbt test results)Metadata tables:
- dbt_models
- dbt_tests
- dbt_sources
- dbt_exposures
- dbt_metrics
- dbt_snapshotsHere you can find [additional details about the tables](https://docs.elementary-data.com/guides/modules-overview/dbt-package).
## Data anomalies detection as dbt tests
Elementary dbt tests collect metrics and metadata over time, such as freshness, volume, schema changes, distribution, cardinality, etc.
Executed as any other dbt tests, the Elementary tests alert on anomalies and outliers.**Elementary tests are configured and executed like native tests in your project!**
Example of Elementary test config in `properties.yml`:
```yml
models:
- name: your_model_name
config:
elementary:
timestamp_column: updated_at
tests:
- elementary.table_anomalies
- elementary.all_columns_anomalies
```## Data observability report
## Slack alerts
## How it works?
Elementary dbt package creates tables of metadata and test results in your data warehouse, as part of your dbt runs. The [CLI tool](https://github.com/elementary-data/elementary) reads the data from these tables, and is used to generate the UI and alerts.
## Data warehouse support
- [x] **Snowflake** ![](https://raw.githubusercontent.com/elementary-data/elementary/master/static/snowflake-16.png)
- [x] **BigQuery** ![](https://raw.githubusercontent.com/elementary-data/elementary/master/static/bigquery-16.svg)
- [x] **Redshift** ![](https://raw.githubusercontent.com/elementary-data/elementary/master/static/redshift-16.png)
- [x] **Databricks SQL** ![](https://raw.githubusercontent.com/elementary-data/elementary/master/static/databricks-16.png)
- [x] **Postgres** ![](https://raw.githubusercontent.com/elementary-data/elementary/master/static/postgres-16.png)## Community & Support
- [Slack](https://join.slack.com/t/elementary-community/shared_invite/zt-uehfrq2f-zXeVTtXrjYRbdE_V6xq4Rg) (Talk to us, support, etc.)
- [GitHub issues](https://github.com/elementary-data/elementary/issues) (Bug reports, feature requests)## Contributions
Thank you :orange_heart: Whether it’s a bug fix, new feature, or additional documentation - we greatly appreciate contributions!
Check out the [contributions guide](https://docs.elementary-data.com/general/contributions) and [open issues](https://github.com/elementary-data/elementary/issues) in the main repo.