Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

https://github.com/autotraderuk/oliver-twist

dbt DAG Auditor
https://github.com/autotraderuk/oliver-twist

dag dag-auditing dbt

Last synced: about 1 month ago
JSON representation

dbt DAG Auditor

Lists

README 

        



![Alt text](https://github.com/autotraderuk/oliver-twist/blob/main/docs/images/oliver_twist_logo.png)

# oliver-twist



DAG Auditor



[![Build status badge](https://github.com/autotraderuk/oliver-twist/workflows/CI/badge.svg)](https://github.com/autotraderuk/oliver-twist/actions?query=workflow%3ACI)

[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)

[![PyPI](https://img.shields.io/pypi/v/olivertwist)](https://pypi.org/project/olivertwist/)

[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/olivertwist)](https://pypi.org/project/olivertwist/)

[![GitHub - License](https://img.shields.io/github/license/autotraderuk/oliver-twist)](https://github.com/autotraderuk/oliver-twist/blob/main/LICENSE)



oliver-twist is a dag auditing tool that audits the [DBT](https://www.getdbt.com/) DAG and generates a summary report. There is a [docs site][1], including descriptions of all [currently implemented rules][2].



![please sir, can I automate my DAG auditing](https://github.com/autotraderuk/oliver-twist/blob/main/docs/images/oliver_dag_meme.jpg)



# Getting Started



To get started, install the package



```shell

$ pip install olivertwist

```



and then run it by passing it your dbt manifest JSON



```shell

olivertwist check manifest.json

```



This will report any failures to the console, and also in HTML format in a directory called `target`. You can optionally auto-open the report in a browser with:



```shell

olivertwist check manifest.json --browser

```



You can also tell Oliver to load and run your own custom rules using the `--add-rules-from` option. See [documentation for full details](https://olivertwi.st/custom_rules/).



Full options are available with:



```shell

olivertwist check --help

```



## Configuration



[All rules][2] are enabled by default. To change this you need a configuration file called `olivertwist.yml` in the same directory you are running `olivertwist`. An example configuration is shown below:



```yaml

version: '1.0'

universal:

  - id: no-rejoin-models

    enabled: false

  - id: no-disabled-models

    enabled: true

```



There is a command to help you generate the config automatically:



```shell

olivertwist config

```



This will show all the available rules and allow you to toggle the ones that you want to enforce.



## Local Development



### Clone this repo and install the project:



`poetry install`



### Install pre-commit hooks for linting



This is optional, but highly recommended to avoid annoying linting failure in CI.



`poetry run pre-commit install`



To run the pre-commit hooks locally:



`poetry run pre-commit run`



### To get the latest versions of the dependencies and to update the poetry.lock file run:



`poetry update`



### To run oliver-twist and generate the summary report run:



`poetry run olivertwist example_manifest.json`



### Working with diagrams

 

To update and regenerate the images that illustrate rule failures in the documentation follow the next steps:

- update the diagrams using the [mermaid syntax](https://mermaid-js.github.io/mermaid/#/)

- install [yarn](https://classic.yarnpkg.com/en/docs/install/)

- `cd docs/diagrams`

- `./generate.sh`

- inspect the generated images in `./docs/diagrams/output/`

- if you're happy with the results, run `./copy.sh` so that they are copied over to `./docs/images`

- you can now reference those images. e.g. in `.docs/rules.md`



### Creating a distribution



```poetry build --format wheel```



[1]: http://olivertwi.st/

[2]: http://olivertwi.st/rules/