{"id":22793378,"url":"https://github.com/fivetran/dbt_package_automations","last_synced_at":"2025-04-16T18:51:41.953Z","repository":{"id":40775872,"uuid":"488267480","full_name":"fivetran/dbt_package_automations","owner":"fivetran","description":"Project used by the Fivetran dbt package team to help automate package development and maintenance efforts.","archived":false,"fork":false,"pushed_at":"2025-02-19T19:50:25.000Z","size":112,"stargazers_count":3,"open_issues_count":12,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-29T05:34:05.375Z","etag":null,"topics":["automation","dbt-package","fivetran"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fivetran.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-05-03T15:38:43.000Z","updated_at":"2024-08-06T21:22:18.000Z","dependencies_parsed_at":"2023-12-19T20:03:11.318Z","dependency_job_id":"a3cf4efc-0fa0-40d3-835f-fb93b24f5e46","html_url":"https://github.com/fivetran/dbt_package_automations","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fivetran%2Fdbt_package_automations","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fivetran%2Fdbt_package_automations/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fivetran%2Fdbt_package_automations/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fivetran%2Fdbt_package_automations/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fivetran","download_url":"https://codeload.github.com/fivetran/dbt_package_automations/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249266646,"owners_count":21240792,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","dbt-package","fivetran"],"created_at":"2024-12-12T03:19:40.278Z","updated_at":"2025-04-16T18:51:41.920Z","avatar_url":"https://github.com/fivetran.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n    \u003ca alt=\"License\"\n        href=\"https://github.com/fivetran/dbt_package_automations/blob/main/LICENSE\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/License-Apache%202.0-blue.svg\" /\u003e\u003c/a\u003e\n    \u003ca alt=\"dbt-core\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/dbt_core-version_\u003e=1.0.0_\u003c2.0.0-orange.svg\" /\u003e\u003c/a\u003e\n    \u003ca alt=\"Maintained?\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/Maintained%3F-yes-green.svg\" /\u003e\u003c/a\u003e\n    \u003ca alt=\"PRs\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/Contributions-welcome-blueviolet\" /\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n# Fivetran dbt Package Automations\n# 📣 What does this dbt package do?\nThis package is a combination of macros, bash scripts, and python scripts that are used to help expedite and automate the package development process. See the **Contents** below for the automations available within this package.\n\n# 🤔 Who is the target user of this dbt package?\n- You use dbt\n- You are a member of the Fivetran dbt package team, or would like to expedite the package development process\n\n# 🎯 How do I use the dbt package?\n## Step 1: Installing the Package\nInclude the following dbt_package_automations package version in your `packages.yml`\n\u003e Check [dbt Hub](https://hub.getdbt.com/) for the latest installation instructions, or [read the dbt docs](https://docs.getdbt.com/docs/package-management) for more information on installing packages.\n```yaml\npackages:\n  - git: https://github.com/fivetran/dbt_package_automations.git\n    revision: main\n    warn-unpinned: false\n```\n## Step 2: Using the Automations\nSee the specific details for each macros within the contents below.\n# 📋 Contents\n## Automation Macros\n- [Automation Macros](#automation-macros)\n  - [generate_columns_macro](#generate_columns_macro-source)\n  - [generate_docs](#generate_docs-source)\n  - [get_column_names_only](#get_column_names_only-source)\n  - [get_columns_for_macro](#get_columns_for_macro-source)\n  - [staging_models_automation](#staging_models_automation-source)\n\n## Bash Scripts\n- [Bash Scripts](#bash-scripts)\n  - [generate_columns](#generate_columns-source)\n  - [generate_docs](#generate_docs-source)\n  - [generate_models](#generate_models-source)\n\n## Automation Macros\nThese macros provide the scripts to automate parts of the model creation.\n\n### staging_models_automation ([source](macros/staging_models_automation.sql))\nThis macro is intended to be used as a `run-operation` when generating Fivetran dbt package staging models and all around package framework creation. This macro will receive user input to create the necessary bash commands so they may all be ran at once. The output of this macro within the CLI will then be copied and pasted as a command to generate the staging models/macros.\n\nAdditionally, you can rerun this macro as it will create or replace what currently exists in the macro \u0026 model folders.\n\nThings to note:\n\n- This macro will only work if you have already included your src.yml file.\n- Please double check your outputs as there may be timestamps \u0026 notes that are not relevant to the file.\n\n**Usage:**\n```bash\ndbt run-operation staging_models_automation --args '{package: intercom, display_name: Intercom, source_schema: intercom_schema, source_database: intercom_database, tables: [\"admin\",\"company_history\"]}'\n```\n**CLI Output:**\n```bash\nsource dbt_packages/dbt_package_automations/kick_off_generator.sh  \u0026\u0026 \nsource dbt_packages/dbt_package_automations/generate_source.sh '../dbt_intercom' intercom intercom_database intercom_schema '['admin', 'company_history']'  \u0026\u0026 \nsource dbt_packages/dbt_package_automations/generate_columns.sh '../dbt_intercom' intercom intercom_database intercom_schema admin \u0026\u0026 \nsource dbt_packages/dbt_package_automations/generate_columns.sh '../dbt_intercom' intercom intercom_database intercom_schema company_history \u0026\u0026 \nsource dbt_packages/dbt_package_automations/generate_models.sh '../dbt_intercom' intercom intercom_database intercom_schema admin \u0026\u0026 \nsource dbt_packages/dbt_package_automations/generate_models.sh '../dbt_intercom' intercom intercom_database intercom_schema company_history \u0026\u0026 \nsource dbt_packages/dbt_package_automations/generate_docs_md.sh '../dbt_intercom' intercom intercom_database intercom_schema 'admin\",\"company_history'  \u0026\u0026 \nsource dbt_packages/dbt_package_automations/generate_files.sh intercom 'Intercom'  \u0026\u0026 \nsource dbt_packages/dbt_package_automations/edit_dbt_project_yml.sh intercom 'admin,company_history'  \u0026\u0026 \nsource dbt_packages/dbt_package_automations/edit_integrations_project_yml.sh intercom 'admin,company_history' \n```\n**Args:**\n* `package`         (required): Name of the package for which you are creating staging models/macros.\n* `display_name`    (required): Display name of the package for which you are creating that will be used mainly in the README generation.\n* `source_schema`   (required): Name of the source_schema from which the bash command will query.\n* `source_database` (required): Name of the source_database from which the bash command will query.\n* `tables`          (required): List of the tables for which you want to create staging models/macros.\n\n### generate_columns_macro ([source](macros/generate_columns_macro.sql))\nThis macro is used to generate the macro used as an argument within the [fill_staging_columns](https://github.com/fivetran/dbt_fivetran_utils#fill_staging_columns-source) macro which will list all the expected columns within a respective table. The macro output will contain `name` and `datatype`; however, you may add an optional argument for `alias` if you wish to rename the column within the macro. \n\nThe macro should be run using dbt's `run-operation` functionality, as used below. It will print out the macro text, which can be copied and pasted into the relevant `macro` directory file within the package.\n\n**Usage:**\n```\ndbt run-operation dbt_package_automations.generate_columns_macro --args '{\"table_name\": \"promoted_tweet_report\", \"schema_name\": \"twitter_ads\", \"database_name\": \"intercom_database\"}'\n```\n**Output:**\n```sql\n{% macro get_admin_columns() %}\n\n{% set columns = [\n    {\"name\": \"email\", \"datatype\": dbt_utils.type_string()},\n    {\"name\": \"id\", \"datatype\": dbt_utils.type_string(), \"alias\": \"admin_id\"},\n    {\"name\": \"job_title\", \"datatype\": dbt_utils.type_string()},\n    {\"name\": \"name\", \"datatype\": dbt_utils.type_string()},\n    {\"name\": \"_fivetran_deleted\", \"datatype\": \"boolean\"},\n    {\"name\": \"_fivetran_synced\", \"datatype\": dbt_utils.type_timestamp()}\n] %}\n\n{{ return(columns) }}\n\n{% endmacro %}\n```\n**Args:**\n* `table_name`    (required): Name of the schema which the table you are running the macro for resides in.\n* `schema_name`   (required): Name of the schema which the table you are running the macro for resides in.\n* `database_name` (optional): Name of the database which the table you are running the macro for resides in. If empty, the macro will default this value to `target.database`.\n\n----\n### generate_docs ([source](macros/generate_docs.sql))\nThis macro will generate a `source` command that leverages `generate_docs.sh` to do the following:\n- seeds, runs and creates documentation for integration tests models\n- moves `catalog.json`, `index.html`, `manifest.json` and `run_results.json` into a `\u003cproject_name\u003e/docs` folder. \nWhen the source script is ran, this feature will remove existing files in the `\u003cproject_name\u003e/docs` if any exists. \n\n**Requirements:**\n- This script assumes that you are running in a directory that is adjacent to your project. For example, say you are working on dbt_apple_search_ads. You may run the macro \u0026 the source command from the CLI Output within your dev directory below. \n```bash\n├── apple_search_ads\n│   ├── dbt_apple_search_ads\n│   ├── dbt_apple_search_ads_source\n│   └── dev\n```\n- Make sure your integration_test profiles in `~/.dbt/profiles.yml` is set for the appropriate project name.\n\n**Usage:**\n```bash\ndbt run-operation generate_docs --args '{package: apple_search_ads_source}'\n```\n**CLI Output:**\n```bash\nsource dbt_packages/dbt_package_automations/generate_docs.sh '../dbt_apple_search_ads_source'\n```\n**Args:**\n* `package` (required): Name of the package; include whether package is source or not\n\n----\n----\n### get_column_names_only ([source](macros/get_column_names_only.sql))\nThis macro is used in the `generate_models.sh` script to further the `staging_models_automation` macro. This macro outputs all columns from the specified table, allowing `generate_models.sh` to prefill column fields in the final select statement.\n\nNote this will retain the timestamp from the built-in formatting update from dbt 1.0.0. Therefore in the staging model resulting from `generate_models.sh`, you will need to manually delete the timestamp.\n\n**Usage:**\n```bash\ndbt run-operation get_column_names_only --args '{table_name: log, schema_name: fivetran_log, database_name: database-name' \n\n```\n**CLI Output:**\n```bash\n14:41:40      _fivetran_synced,\n    connector_id,\n    event,\n    id,\n    message_data,\n    message_event,\n    process_id,\n    sequence_number,\n    sync_id,\n    time_stamp,\n    transformation_id\n\n```\n**Args:**\n* `table_name`    (required): Name of the table you are wanting to return column names and datatypes.\n* `schema_name`   (required): Name of the schema where the above mentioned table resides.\n* `database_name` (optional): Name of the database where the above mentioned schema and table reside. By default this will be your target.database.\n\n----\n### get_columns_for_macro ([source](macros/get_columns_for_macro.sql))\nThis macro returns all column names and datatypes for a specified table within a database and is used as part of the [generate_columns_macro](macros/generate_columns_macro.sql).\n\n**Usage:**\n```sql\n{{ fivetran_utils.get_columns_for_macro(table_name=\"team\", schema_name=\"my_teams\", database_name=\"my_database\") }}\n```\n**Args:**\n* `table_name`    (required): Name of the table you are wanting to return column names and datatypes.\n* `schema_name`   (required): Name of the schema where the above mentioned table resides.\n* `database_name` (optional): Name of the database where the above mentioned schema and table reside. By default this will be your target.database.\n\n----\n\n## Bash Scripts\n### generate_columns.sh ([source](generate_columns.sh))\n\nThis bash file can be used to setup or update packages to use the `fill_staging_columns` macro above. The bash script does the following:\n\n* Creates a `.sql` file in the `macros` directory for a source table and fills it with all the columns from the table.\n    * Be sure your `dbt_project.yml` file does not contain any **Warnings** or **Errors**. If warnings or errors are present, the messages from the terminal will be printed above the macro within the `.sql` file in the `macros` directory.\n\nThe usage is as follows, assuming you are executing via a `zsh` terminal and in a dbt project directory that has already imported this repo as a dependency:\n```bash\nsource dbt_packages/dbt_package_automations/generate_columns.sh \"path/to/directory\" file_prefix database_name schema_name table_name\n```\n\nAs an example, assuming we are in a dbt project in an adjacent folder to `dbt_apple_search_ads_source`:\n```bash\nsource dbt_packages/dbt_package_automations/generate_columns.sh '../dbt_apple_search_ads_source' stg_apple_search_ads intercom_database apple_search_ads campaign_history\n```\n\nIn that example, it will:\n* Create a `get_campaign_history_columns.sql` file in the `macros` directory, with the necessary macro within it.\n\n----\n### generate_docs.sh([source](generate_docs.sh))\nThis bash file can be used to create or replace package documentation (`\u003cproject_name\u003e/docs`). \n\n**Requirements:**\n- This script assumes that you are running in a directory that is adjacent to your project. For example, say you are working on dbt_apple_search_ads. You may run the the source command within your dev directory below. \n```bash\n├── apple_search_ads\n│   ├── dbt_apple_search_ads\n│   ├── dbt_apple_search_ads_source\n│   └── dev \n```\n- Make sure your integration_test profiles in `~/.dbt/profiles.yml` is set for the appropriate project name.\n\n**Usage:**\n```bash\nsource dbt_packages/dbt_package_automations/generate_docs.sh '../dbt_apple_search_ads_source'\n```\nThe bash script does the following:\n- seeds, runs and creates documentation for integration tests models\n- moves `catalog.json`, `index.html`, `manifest.json` and `run_results.json` into a `\u003cproject_name\u003e/docs` folder. \n\n----\n### generate_models.sh ([source](generate_models.sh))\n\nThis bash file can be used to setup or update packages to use the `generate_models` macro above. The bash script assumes that there already exists a macro directory with all relevant `get_\u003ctable_name\u003e_columns.sql` files created. The bash script does the following:\n\n* Creates a `..._tmp.sql` file in the `models/tmp` directory and fills it with a `select * from {{ var('table_name') }}` where `table_name` is the name of the source table.\n* Creates or updates a `.sql` file in the `models` directory and fills it with the filled out version of the `fill_staging_columns` macro as shown above. You can then write whatever SQL you want around the macro to finishing off the staging file.\n\n```bash\nsource dbt_packages/dbt_package_automations/generate_models.sh \"path/to/directory\" file_prefix database_name schema_name table_name\n```\n\nAs an example, assuming we are in a dbt project in an adjacent folder to `dbt_apple_search_ads_source`:\n```bash\nsource dbt_packages/dbt_package_automations/generate_models.sh '../dbt_apple_search_ads_source' stg_apple_search_ads intercom_database apple_search_ads campaign_history\n```\n\nWith the above example, the script will:\n* Create a `stg_apple_search_ads__campaign_history_tmp.sql` file in the `models/tmp` directory, with `select * from {{ var('campaign_history') }}` in it.\n* Create or update a `stg_apple_search_ads__campaign_history.sql` file in the `models` directory with the pre-filled out `fill_staging_columns` macro.\n\n# 🙌 How is this package maintained and can I contribute?\n## Package Maintenance\nThe Fivetran team maintaining this package **only** maintains the latest version of the package. We highly recommend you stay consistent with the [latest version](https://hub.getdbt.com/fivetran/jira/latest/) of the package and refer to the [CHANGELOG](https://github.com/fivetran/dbt_jira/blob/main/CHANGELOG.md) and release notes for more information on changes across versions.\n\n## Contributions\nThese dbt packages are developed by a small team of analytics engineers at Fivetran. However, the packages are made better by community contributions! \n\nWe highly encourage and welcome contributions to this package. Check out [this post](https://discourse.getdbt.com/t/contributing-to-a-dbt-package/657) on the best workflow for contributing to a package!\n\n# 🏪 Are there any resources available?\n- If you encounter any questions or want to reach out for help, please refer to the [GitHub Issue](https://github.com/fivetran/dbt_fivetran_utils/issues/new/choose) section to find the right avenue of support for you.\n- If you would like to provide feedback to the dbt package team at Fivetran, or would like to request a future dbt package to be developed, then feel free to fill out our [Feedback Form](https://www.surveymonkey.com/r/DQ7K7WW).\n- Have questions or want to just say hi? Book a time during our office hours [here](https://calendly.com/fivetran-solutions-team/fivetran-solutions-team-office-hours) or send us an email at solutions@fivetran.com.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffivetran%2Fdbt_package_automations","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffivetran%2Fdbt_package_automations","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffivetran%2Fdbt_package_automations/lists"}