{"id":13703891,"url":"https://github.com/Dashlane/dbt-invoke","last_synced_at":"2025-05-05T09:32:12.977Z","repository":{"id":43291612,"uuid":"342590039","full_name":"Dashlane/dbt-invoke","owner":"Dashlane","description":"A CLI for creating, updating, and deleting dbt property files","archived":false,"fork":false,"pushed_at":"2024-07-31T23:09:46.000Z","size":65,"stargazers_count":70,"open_issues_count":10,"forks_count":8,"subscribers_count":10,"default_branch":"main","last_synced_at":"2024-10-03T06:59:23.581Z","etag":null,"topics":["cli","dbt","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Dashlane.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2021-02-26T13:50:52.000Z","updated_at":"2024-09-18T19:55:00.000Z","dependencies_parsed_at":"2024-01-11T10:57:21.028Z","dependency_job_id":null,"html_url":"https://github.com/Dashlane/dbt-invoke","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Dashlane%2Fdbt-invoke","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Dashlane%2Fdbt-invoke/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Dashlane%2Fdbt-invoke/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Dashlane%2Fdbt-invoke/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Dashlane","download_url":"https://codeload.github.com/Dashlane/dbt-invoke/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224439556,"owners_count":17311470,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","dbt","python"],"created_at":"2024-08-02T21:01:01.370Z","updated_at":"2024-11-13T11:30:19.792Z","avatar_url":"https://github.com/Dashlane.png","language":"Python","funding_links":[],"categories":["Packages"],"sub_categories":[],"readme":"# dbt-invoke\n\ndbt-invoke is a CLI (built with [Invoke](http://www.pyinvoke.org/)) for \ncreating, updating, and deleting\n[dbt](https://docs.getdbt.com/docs/introduction) \n[property files](https://docs.getdbt.com/reference/declaring-properties).\n\n\n- Supported dbt resource types:\n  - models\n  - seeds\n  - snapshots\n  - analyses\n  \n\n- Under the hood, this tool works by combining the power of the \n  [dbt ls](https://docs.getdbt.com/reference/commands/list) and \n  [dbt run-operation](https://docs.getdbt.com/reference/commands/run-operation)\n  commands with dbt's built-in `get_columns_in_query` macro.\n  - This methodology allows the tool to work on \n    [ephemeral models](https://docs.getdbt.com/docs/building-a-dbt-project/building-models/materializations#ephemeral) \n    and [analyses](https://docs.getdbt.com/docs/building-a-dbt-project/analyses),\n    which other approaches, such as those based on listing data warehouse \n    tables/views, can miss.\n\n\n## Installation\n\n- If you are using dbt-core version 1.5.0 or later:\n  ```shell\n  pip install dbt-invoke\n  ```\n- If you are using a dbt-core version earlier than 1.5.0:\n  ```shell\n  pip install dbt-invoke~=0.2\n  ```\n\n\n## Usage\n\n- You must have previously executed `dbt run`/`dbt seed`/`dbt snapshot` on the\n  resources for which you wish to create/update property files.\n  - If you have made updates to your resources, execute the appropriate command\n    (`dbt run`/`dbt seed`/`dbt snapshot`) before using this tool to \n    create/update property files.\n\n\n- Property files will be created, updated, or deleted on a one-to-one basis in\n  the same paths as the resource files they represent (the only change being a\n  `.yml` file extension).\n  - For example, given a resource file in the location \n    `models/marts/core/users.sql`, this tool will create, update, or delete a \n    property file in the location `models/marts/core/users.yml`.\n  - If your dbt project defines properties for multiple resources per `.yml` file, see the\n    [Migrating to One Resource Per Property File](#migrating-to-one-resource-per-property-file)\n    section.\n\n    \n- Any newly generated property files are created with the correct resource \n  type, resource name, and columns.  A blank description field will be included\n  for each column and for the resource itself.\n  - For example, when generating a new property file for a model `users` with \n    column names `user_id` and `created_at`, the following yaml will be \n    generated:\n    - ```yaml\n      version: 2\n      models:\n      - name: users\n        description: ''\n        columns:\n        - name: user_id\n          description: ''\n        - name: created_at\n          description: ''\n      ```\n\n  \n- When updating an already existing property file, new columns in the resource\n  will be added, and columns that no longer exist will be removed.\n\n\n- You may fill in the blank `description` properties and add other properties \n  (e.g. `tests`).  They will remain intact when updating existing property \n  files as long as the column/resource name to which they belong still exists.\n\n\n### Creating/Updating Property Files\n\n```shell\ndbt-invoke properties.update \u003coptions\u003e\n\n# OR, because 'update' is set as the default command:\ndbt-invoke properties \u003coptions\u003e\n```\n\n- The first time you run this command, you should be prompted to add a short \n  macro called `_log_columns_list` to your dbt project.\n  - You may accept the prompt to add it automatically.\n  - Otherwise, copy/paste it into one your dbt project's macro-paths yourself.\n  - To print the macro, at any time, run `dbt-invoke properties.echo-macro`.\n\n\n- `\u003coptions\u003e` primarily uses the same arguments as the `dbt ls` command to \n  allow flexibility in selecting the dbt resources for which you wish to \n  create/update property files (run `dbt ls --help` for details).\n  - --resource-type\n  - --models\n  - --select\n  - --selector\n  - --exclude\n  - --project-dir\n  - --profiles-dir\n  - --profile\n  - --target\n  - --vars\n  - --bypass-cache\n  - --state\n\n\n- Notes: \n  - This tool supports only the long flags of `dbt ls` options (for \n  example: `--models`, and not short flags like `-m`).\n  - Multiple values for the same argument can be passed as a comma separated\n  string (Example: `--models modelA,modelB`)\n    - Keep in mind that dbt may not support multiple values for certain \n      options.\n\n\n- Two additional flags are made available.\n  - `--log-level` to alter the verbosity of logs.\n    - It accepts one of Python's standard logging levels (debug, info, warning,\n      error, critical).\n  - `--threads` to set a maximum number of concurrent threads to use in \n    collecting resources' column information from the data warehouse and in \n    creating/updating the corresponding property files. Each thread will run \n    dbt's get_columns_in_query macro against the data warehouse.\n  \n\n- Some examples:\n  ```shell\n  # Create/update property files for all supported resource types\n  dbt-invoke properties\n  \n  # Create/update property files for all supported resource types, using 4 concurrent threads\n  dbt-invoke properties --threads 4\n  \n  # Create/update property files for all models in a models/marts directory\n  dbt-invoke properties --models marts\n  \n  # Create/update property files for a 'users' model and an 'orders' models\n  dbt-invoke properties --models users,orders\n  \n  # Create/update property files for a 'users' model and all downstream models\n  dbt-invoke properties --models users+\n  \n  # Create/update property files for all seeds\n  dbt-invoke properties --resource-type seed\n  \n  # Create/update a property file for a snapshot called 'users_snapshot'\n  dbt-invoke properties --resource-type snapshot --select users_snapshot\n  \n  # Create/update property files when your working directory is above your dbt project directory\n  dbt-invoke properties --project-dir path/to/dbt/project/directory\n  ```\n\n\n### Deleting Property Files\n\n```shell\ndbt-invoke properties.delete \u003coptions\u003e\n```\n- `\u003coptions\u003e` uses the same arguments as for creating/updating property files,\n  except for `--threads`.\n\n\n### Migrating to One Resource Per Property File\n- Two conventions for dbt property files are:\n  1. One resource per property file\n  2. Multiple resources per property file\n\n\n- dbt-invoke abides by the \"One resource per property file\" convention.\n- If your dbt project uses the \"Multiple resources per property file\"\n  convention, you can migrate to one resource per property file by using\n  dbt-invoke's `properties.migrate` command as shown here:\n  ```shell\n  dbt-invoke properties.migrate \u003coptions\u003e\n  ```\n  - Successfully migrated properties will be removed from existing\n    multiple-resource property files.\n    - At then end of migration, property files that are newly empty (other than\n      `version: 2`) will be automatically deleted.\n  - `\u003coptions\u003e` uses the same arguments as for creating/updating property\n    files, except for `--threads`.\n\n### Help\n\n- To view the list of available commands and their short descriptions, run:\n  ```shell\n  dbt-invoke --list\n  ```\n\n- To view in depth command descriptions and available options/flags, run:\n  ```shell\n  dbt-invoke \u003ccommand_name\u003e --help\n  ```\n\n### Limitations\n\n- dbt-invoke will try to preserve formatting and comments when updating\n  existing files. If you want to preserve line-breaks, use `\u003e` or `|` on your\n  multiline strings, as recommended\n  [here](https://stackoverflow.com/a/21699210/15202709).\n- In order to collect or update the list of columns that should appear in \n  each property file, dbt's `get_columns_in_query` macro is run for each\n  matching resource. As of the time of writing, `get_columns_in_query` uses a\n  SELECT statement [limited to zero rows](https://github.com/fishtown-analytics/dbt/blob/2b48152da66dbd7f07272983bbc261f1b6924f20/core/dbt/include/global_project/macros/adapters/common.sql#L11).\n  While this is not typically a performance issue for table or incremental \n  materializations, execution may be slow for complex analyses, views, or \n  ephemeral materializations. \n  - This may be partially remedied by increasing the value of the `--threads` \n    option in `dbt-invoke properties.update`.\n- dbt-invoke has not been tested across different types of data warehouses.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FDashlane%2Fdbt-invoke","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FDashlane%2Fdbt-invoke","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FDashlane%2Fdbt-invoke/lists"}