Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/Dashlane/dbt-invoke
A CLI for creating, updating, and deleting dbt property files
https://github.com/Dashlane/dbt-invoke
cli dbt python
Last synced: 3 months ago
JSON representation
A CLI for creating, updating, and deleting dbt property files
- Host: GitHub
- URL: https://github.com/Dashlane/dbt-invoke
- Owner: Dashlane
- License: apache-2.0
- Created: 2021-02-26T13:50:52.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2024-04-16T19:39:27.000Z (7 months ago)
- Last Synced: 2024-07-13T22:03:11.086Z (4 months ago)
- Topics: cli, dbt, python
- Language: Python
- Homepage:
- Size: 63.5 KB
- Stars: 66
- Watchers: 10
- Forks: 7
- Open Issues: 10
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-dbt - dbt-invoke - CLI for creating, updating, and deleting dbt property files. (Packages)
README
# dbt-invoke
dbt-invoke is a CLI (built with [Invoke](http://www.pyinvoke.org/)) for
creating, updating, and deleting
[dbt](https://docs.getdbt.com/docs/introduction)
[property files](https://docs.getdbt.com/reference/declaring-properties).
- Supported dbt resource types:
- models
- seeds
- snapshots
- analyses
- Under the hood, this tool works by combining the power of the
[dbt ls](https://docs.getdbt.com/reference/commands/list) and
[dbt run-operation](https://docs.getdbt.com/reference/commands/run-operation)
commands with dbt's built-in `get_columns_in_query` macro.
- This methodology allows the tool to work on
[ephemeral models](https://docs.getdbt.com/docs/building-a-dbt-project/building-models/materializations#ephemeral)
and [analyses](https://docs.getdbt.com/docs/building-a-dbt-project/analyses),
which other approaches, such as those based on listing data warehouse
tables/views, can miss.
## Installation
- If you are using dbt-core version 1.5.0 or later:
```shell
pip install dbt-invoke
```
- If you are using a dbt-core version earlier than 1.5.0:
```shell
pip install dbt-invoke~=0.2
```
## Usage
- You must have previously executed `dbt run`/`dbt seed`/`dbt snapshot` on the
resources for which you wish to create/update property files.
- If you have made updates to your resources, execute the appropriate command
(`dbt run`/`dbt seed`/`dbt snapshot`) before using this tool to
create/update property files.
- Property files will be created, updated, or deleted on a one-to-one basis in
the same paths as the resource files they represent (the only change being a
`.yml` file extension).
- For example, given a resource file in the location
`models/marts/core/users.sql`, this tool will create, update, or delete a
property file in the location `models/marts/core/users.yml`.
- If your dbt project defines properties for multiple resources per `.yml` file, see the
[Migrating to One Resource Per Property File](#migrating-to-one-resource-per-property-file)
section.
- Any newly generated property files are created with the correct resource
type, resource name, and columns. A blank description field will be included
for each column and for the resource itself.
- For example, when generating a new property file for a model `users` with
column names `user_id` and `created_at`, the following yaml will be
generated:
- ```yaml
version: 2
models:
- name: users
description: ''
columns:
- name: user_id
description: ''
- name: created_at
description: ''
```
- When updating an already existing property file, new columns in the resource
will be added, and columns that no longer exist will be removed.
- You may fill in the blank `description` properties and add other properties
(e.g. `tests`). They will remain intact when updating existing property
files as long as the column/resource name to which they belong still exists.
### Creating/Updating Property Files
```shell
dbt-invoke properties.update
# OR, because 'update' is set as the default command:
dbt-invoke properties
```
- The first time you run this command, you should be prompted to add a short
macro called `_log_columns_list` to your dbt project.
- You may accept the prompt to add it automatically.
- Otherwise, copy/paste it into one your dbt project's macro-paths yourself.
- To print the macro, at any time, run `dbt-invoke properties.echo-macro`.
- `` primarily uses the same arguments as the `dbt ls` command to
allow flexibility in selecting the dbt resources for which you wish to
create/update property files (run `dbt ls --help` for details).
- --resource-type
- --models
- --select
- --selector
- --exclude
- --project-dir
- --profiles-dir
- --profile
- --target
- --vars
- --bypass-cache
- --state
- Notes:
- This tool supports only the long flags of `dbt ls` options (for
example: `--models`, and not short flags like `-m`).
- Multiple values for the same argument can be passed as a comma separated
string (Example: `--models modelA,modelB`)
- Keep in mind that dbt may not support multiple values for certain
options.
- Two additional flags are made available.
- `--log-level` to alter the verbosity of logs.
- It accepts one of Python's standard logging levels (debug, info, warning,
error, critical).
- `--threads` to set a maximum number of concurrent threads to use in
collecting resources' column information from the data warehouse and in
creating/updating the corresponding property files. Each thread will run
dbt's get_columns_in_query macro against the data warehouse.
- Some examples:
```shell
# Create/update property files for all supported resource types
dbt-invoke properties
# Create/update property files for all supported resource types, using 4 concurrent threads
dbt-invoke properties --threads 4
# Create/update property files for all models in a models/marts directory
dbt-invoke properties --models marts
# Create/update property files for a 'users' model and an 'orders' models
dbt-invoke properties --models users,orders
# Create/update property files for a 'users' model and all downstream models
dbt-invoke properties --models users+
# Create/update property files for all seeds
dbt-invoke properties --resource-type seed
# Create/update a property file for a snapshot called 'users_snapshot'
dbt-invoke properties --resource-type snapshot --select users_snapshot
# Create/update property files when your working directory is above your dbt project directory
dbt-invoke properties --project-dir path/to/dbt/project/directory
```
### Deleting Property Files
```shell
dbt-invoke properties.delete
```
- `` uses the same arguments as for creating/updating property files,
except for `--threads`.
### Migrating to One Resource Per Property File
- Two conventions for dbt property files are:
1. One resource per property file
2. Multiple resources per property file
- dbt-invoke abides by the "One resource per property file" convention.
- If your dbt project uses the "Multiple resources per property file"
convention, you can migrate to one resource per property file by using
dbt-invoke's `properties.migrate` command as shown here:
```shell
dbt-invoke properties.migrate
```
- Successfully migrated properties will be removed from existing
multiple-resource property files.
- At then end of migration, property files that are newly empty (other than
`version: 2`) will be automatically deleted.
- `` uses the same arguments as for creating/updating property
files, except for `--threads`.
### Help
- To view the list of available commands and their short descriptions, run:
```shell
dbt-invoke --list
```
- To view in depth command descriptions and available options/flags, run:
```shell
dbt-invoke --help
```
### Limitations
- dbt-invoke will try to preserve formatting and comments when updating
existing files. If you want to preserve line-breaks, use `>` or `|` on your
multiline strings, as recommended
[here](https://stackoverflow.com/a/21699210/15202709).
- In order to collect or update the list of columns that should appear in
each property file, dbt's `get_columns_in_query` macro is run for each
matching resource. As of the time of writing, `get_columns_in_query` uses a
SELECT statement [limited to zero rows](https://github.com/fishtown-analytics/dbt/blob/2b48152da66dbd7f07272983bbc261f1b6924f20/core/dbt/include/global_project/macros/adapters/common.sql#L11).
While this is not typically a performance issue for table or incremental
materializations, execution may be slow for complex analyses, views, or
ephemeral materializations.
- This may be partially remedied by increasing the value of the `--threads`
option in `dbt-invoke properties.update`.
- dbt-invoke has not been tested across different types of data warehouses.