Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/autotraderuk/oliver-twist

dbt DAG Auditor
https://github.com/autotraderuk/oliver-twist

dag dag-auditing dbt

Last synced: 2 days ago
JSON representation

dbt DAG Auditor

Host: GitHub
URL: https://github.com/autotraderuk/oliver-twist
Owner: autotraderuk
License: apache-2.0
Created: 2020-12-15T12:08:35.000Z (almost 4 years ago)
Default Branch: main
Last Pushed: 2021-11-02T14:30:08.000Z (about 3 years ago)
Last Synced: 2024-09-24T09:16:54.166Z (about 1 month ago)
Topics: dag, dag-auditing, dbt
Language: Python
Homepage: https://olivertwi.st
Size: 1.86 MB
Stars: 41
Watchers: 6
Forks: 4
Open Issues: 16
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE

Awesome Lists containing this project

awesome-dbt - oliver-twist - DAG auditing tool that audits the DBT DAG and generates a summary report. (Utilities)

README

![Alt text](https://github.com/autotraderuk/oliver-twist/blob/main/docs/images/oliver_twist_logo.png)
# oliver-twist

DAG Auditor

[![Build status badge](https://github.com/autotraderuk/oliver-twist/workflows/CI/badge.svg)](https://github.com/autotraderuk/oliver-twist/actions?query=workflow%3ACI)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![PyPI](https://img.shields.io/pypi/v/olivertwist)](https://pypi.org/project/olivertwist/)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/olivertwist)](https://pypi.org/project/olivertwist/)
[![GitHub - License](https://img.shields.io/github/license/autotraderuk/oliver-twist)](https://github.com/autotraderuk/oliver-twist/blob/main/LICENSE)

oliver-twist is a dag auditing tool that audits the [DBT](https://www.getdbt.com/) DAG and generates a summary report. There is a [docs site][1], including descriptions of all [currently implemented rules][2].

![please sir, can I automate my DAG auditing](https://github.com/autotraderuk/oliver-twist/blob/main/docs/images/oliver_dag_meme.jpg)

# Getting Started

To get started, install the package

```shell
$ pip install olivertwist
```

and then run it by passing it your dbt manifest JSON

```shell
olivertwist check manifest.json
```

This will report any failures to the console, and also in HTML format in a directory called `target`. You can optionally auto-open the report in a browser with:

```shell
olivertwist check manifest.json --browser
```

You can also tell Oliver to load and run your own custom rules using the `--add-rules-from` option. See [documentation for full details](https://olivertwi.st/custom_rules/).

Full options are available with:

```shell
olivertwist check --help
```

## Configuration

[All rules][2] are enabled by default. To change this you need a configuration file called `olivertwist.yml` in the same directory you are running `olivertwist`. An example configuration is shown below:

```yaml
version: '1.0'
universal:
- id: no-rejoin-models
enabled: false
- id: no-disabled-models
enabled: true
```

There is a command to help you generate the config automatically:

```shell
olivertwist config
```

This will show all the available rules and allow you to toggle the ones that you want to enforce.

## Local Development

### Clone this repo and install the project:

`poetry install`

### Install pre-commit hooks for linting

This is optional, but highly recommended to avoid annoying linting failure in CI.

`poetry run pre-commit install`

To run the pre-commit hooks locally:

`poetry run pre-commit run`

### To get the latest versions of the dependencies and to update the poetry.lock file run:

`poetry update`

### To run oliver-twist and generate the summary report run:

`poetry run olivertwist example_manifest.json`

### Working with diagrams

To update and regenerate the images that illustrate rule failures in the documentation follow the next steps:
- update the diagrams using the [mermaid syntax](https://mermaid-js.github.io/mermaid/#/)
- install [yarn](https://classic.yarnpkg.com/en/docs/install/)
- `cd docs/diagrams`
- `./generate.sh`
- inspect the generated images in `./docs/diagrams/output/`
- if you're happy with the results, run `./copy.sh` so that they are copied over to `./docs/images`
- you can now reference those images. e.g. in `.docs/rules.md`

### Creating a distribution

```poetry build --format wheel```

[1]: http://olivertwi.st/
[2]: http://olivertwi.st/rules/