{"id":25528953,"url":"https://github.com/meltanolabs/tap-dbt","last_synced_at":"2025-10-19T14:03:53.247Z","repository":{"id":37956219,"uuid":"355455466","full_name":"MeltanoLabs/tap-dbt","owner":"MeltanoLabs","description":"Singer Tap for dbt API v2 built with the Meltano SDK","archived":false,"fork":false,"pushed_at":"2025-03-31T19:09:16.000Z","size":1025,"stargazers_count":12,"open_issues_count":2,"forks_count":7,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-03-31T20:23:50.777Z","etag":null,"topics":["dbt","dbt-cloud","elt","extract-data","meltano-sdk","singer-io","singer-tap"],"latest_commit_sha":null,"homepage":"https://pypi.org/p/tap-dbt","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MeltanoLabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-04-07T07:41:30.000Z","updated_at":"2025-03-31T19:09:03.000Z","dependencies_parsed_at":"2023-02-16T10:01:29.734Z","dependency_job_id":"6fc7021b-7e21-4d94-9169-8d854f89e871","html_url":"https://github.com/MeltanoLabs/tap-dbt","commit_stats":null,"previous_names":["edgarrmondragon/tap-dbt"],"tags_count":28,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MeltanoLabs%2Ftap-dbt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MeltanoLabs%2Ftap-dbt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MeltanoLabs%2Ftap-dbt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MeltanoLabs%2Ftap-dbt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MeltanoLabs","download_url":"https://codeload.github.com/MeltanoLabs/tap-dbt/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248360035,"owners_count":21090636,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dbt","dbt-cloud","elt","extract-data","meltano-sdk","singer-io","singer-tap"],"created_at":"2025-02-19T23:29:38.587Z","updated_at":"2025-10-19T14:03:48.202Z","avatar_url":"https://github.com/MeltanoLabs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# tap-dbt\n\n[![PyPI](https://img.shields.io/pypi/v/tap-dbt.svg?color=blue)](https://pypi.org/project/tap-dbt/)\n[![Python versions](https://img.shields.io/pypi/pyversions/tap-dbt.svg)](https://pypi.org/project/tap-dbt/)\n[![Singer](https://img.shields.io/badge/Singer-Tap-purple.svg)](https://hub.meltano.com/taps/dbt)\n[![Test Tap](https://github.com/edgarrmondragon/tap-dbt/actions/workflows/ci.yml/badge.svg)](https://github.com/MeltanoLabs/tap-dbt/actions/workflows/ci.yml)\n\n`tap-dbt` is a Singer tap for the [dbt Cloud API v2][dbtcloud].\n\nBuilt with the [Singer SDK][sdk].\n\n- [Installation](#Installation)\n- [Features](#Features)\n- [Configuration](#Configuration)\n  - [Inputs](#Inputs)\n  - [JSON example](#JSON-example)\n  - [Environment variables example](#Environment-variables-example)\n  - [Meltano Example](#Meltano-Example)\n- [Usage](#Usage)\n  - [Executing the Tap Directly](#Executing-the-Tap-Directly)\n  - [With Meltano](#With-Meltano)\n\n## Installation\n\n```shell\npip install tap-dbt\n```\n\n## Features\n\n### Streams\n\nFull stream metadata is available in the dbt Labs repository: [openapi_schema]\n\n#### Selected by default\nThe following will be extracted by default if no catalog is used:\n\n- [x] Stream: accounts\n- [x] Stream: jobs\n- [x] Stream: projects\n- [x] Stream: runs\n\n\n#### Configurable\nCan be enabled by setting `selected` in the catalog:\n\n- [x] Stream: connections\n- [x] Stream: environments\n- [x] Stream: repositories\n- [x] Stream: users\n\n\n### Incremental Run Stream\n\nOrdering the query from the Runs endpoint by `-finished_at`, i.e. descending Run Finished Datetime, yields:\n\n|id|finished_at|updated_at|created_at|\n|---|---|---|---|\n|314516|None|2023-05-27 21:05:16.109548+00:00|2023-05-27 21:05:05.664170+00:00|\n|314514|None|2023-05-27 21:00:16.847296+00:00|2023-05-27 21:00:05.458908+00:00|\n|314513|None|2023-05-27 21:00:16.355680+00:00|2023-05-27 21:00:05.427258+00:00|\n|314517|None|2023-05-27 21:05:17.094309+00:00|2023-05-27 21:05:05.696222+00:00|\n|314515|2023-05-27 21:01:28.568431+00:00|2023-05-27 21:01:29.269048+00:00|2023-05-27 21:00:05.488543+00:00|\n|314512|2023-05-27 20:48:59.342035+00:00|2023-05-27 20:48:59.844412+00:00|2023-05-27 20:45:04.509746+00:00|\n|314511|2023-05-27 20:48:46.571106+00:00|2023-05-27 20:48:47.079130+00:00|2023-05-27 20:40:04.257950+00:00|\n|314505|2023-05-27 20:41:35.591976+00:00|2023-05-27 20:41:36.305364+00:00|2023-05-27 20:15:02.808079+00:00|\n|314510|2023-05-27 20:39:27.162437+00:00|2023-05-27 20:39:28.628257+00:00|2023-05-27 20:35:03.939439+00:00|\n|314509|2023-05-27 20:37:39.965974+00:00|2023-05-27 20:37:40.496212+00:00|2023-05-27 20:30:03.802620+00:00|\n\nThe incremental sync has been set up so that it works on `replication_key = \"finished_at\"`, when an INCREMENTAL sync is run:\n\n- If the bookmark is set, the stream is queried in reverse `finished_at` order.\n- If the `finished_at` value is not set, the run is assumed to still be running so the record is included, plus the sort order implies that there should be records with populated `finished_at` appearing later in the stream - *Repeated sync operation will yield the same records if the dbt Job Run is still underway, however this adheres to the 'at least once' delivery promise - https://sdk.meltano.com/en/latest/implementation/at_least_once.html*\n- Once the sync operation reaches records with populated `finished_at`, the values are compared with the bookmark and once the `finished_at` value becomes less than the bookmark the stream finishes syncing.\n\n\n## Configuration\n\nVisit the [API docs][apidocs] for instructions on how to get your API key.\n\nYou can pass configuration using environment variables with the `TAP_DBT_` prefix followed by the uppercased field name\n\n```shell\ntap-dbt --config=ENV\n```\n\nor a JSON file\n\n```shell\ntap-dbt --config=config.json\n```\n\n### Inputs\n\n| Field         | Description                                                     | Type           | Required | Default                                          |\n|---------------|-----------------------------------------------------------------|----------------|----------|--------------------------------------------------|\n| `api_key`     | API key for the dbt Cloud API                                   | `string`       | yes      |                                                  |\n| `account_ids` | dbt Cloud account IDs                                           | `list(string)` | yes      |                                                  |\n| `user_agent`  | User-Agent to make requests with                                | `string`       | no       | `tap-dbt/0.1.0 Singer Tap for the dbt Cloud API` |\n| `base_url`    | Base URL for the dbt Cloud API                                  | `string`       | no       | `https://cloud.getdbt.com/api/v2`                |\n| `page_size`   | Number of records per API call, sets the `limit=` url parameter | `integer`      | no       | 5000                                             |\n\nA full list of supported settings and capabilities for this tap is available by running:\n\n```shell\ntap-dbt --about --format json\n```\n\n### JSON example\n\n```json\n{\n  \"api_key\": \"da39a3ee5e6b4b0d3255bfef95601890afd80709\",\n  \"account_ids\": [\"51341\"],\n  \"user_agent\": \"tap-dbt/0.1.0 Singer Tap for the dbt Cloud API\",\n  \"base_url\": \"https://my-dbt-cloud-api.com\",\n  \"page_size\": 5000\n}\n```\n\n### Environment variables example\n\n```dotenv\nTAP_DBT_API_KEY=da39a3ee5e6b4b0d3255bfef95601890afd80709\nTAP_DBT_ACCOUNT_IDS=51341\nTAP_DBT_USER_AGENT='tap-dbt/0.1.0 Singer Tap for the dbt Cloud API'\nTAP_DBT_BASE_URL=https://my-dbt-cloud-api.com\"\nTAP_DBT_PAGE_SIZE=5000\n```\n\n### Meltano Example\n\n```yaml\nplugins:\n  extractors:\n    - name: tap-dbt\n      logo_url: https://hub.meltano.com/assets/logos/taps/dbt.png\n      label: dbt Cloud\n      docs: https://hub.meltano.com/taps/dbt\n      repo: https://github.com/edgarrmondragon/tap-dbt\n      namespace: dbt\n      pip_url: tap-dbt\n      executable: tap-dbt\n      capabilities:\n        - catalog\n        - discover\n      settings:\n        - name: base_url\n          label: dbt Cloud URL\n          placeholder: \"https://cloud.getdbt.com/api/v2\"\n        - name: api_key\n          kind: password\n          label: API Key\n          docs: \"https://docs.getdbt.com/dbt-cloud/api#section/Authentication\"\n        - name: account_ids\n          kind: array\n          label: Account IDs\n        - name: user_agent\n          label: User-Agent\n          placeholder: \"tap-dbt/0.1.0 Singer Tap for the dbt Cloud API\"\n        - name: page_size\n          kind: integer\n          label: Page Size\n\n```\n\n## Usage\n\nYou can easily run `tap-dbt` with the CLI or using [Meltano][meltano].\n\n### Executing the Tap Directly\n\n```shell\ntap-dbt --version\ntap-dbt --help\ntap-dbt --config .secrets/example.json --discover \u003e ./catalog/json\n```\n\n### With Meltano\n\n```shell\nmeltano elt tap-dbt target-snowflake --job_id dbt_snowflake\n```\n\n[dbtcloud]: https://cloud.getdbt.com\n[sdk]: https://gitlab.com/meltano/singer-sdk\n[apidocs]: https://docs.getdbt.com/dbt-cloud/api#section/Authentication\n[meltano]: https://www.meltano.com\n[openapi_schema]: https://github.com/dbt-labs/dbt-cloud-openapi-spec/blob/master/openapi-v3.yaml\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmeltanolabs%2Ftap-dbt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmeltanolabs%2Ftap-dbt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmeltanolabs%2Ftap-dbt/lists"}