https://github.com/kestra-io/dbt-example

Getting started repository for dbt
https://github.com/kestra-io/dbt-example

Last synced: 11 days ago
JSON representation

Getting started repository for dbt

• Host: GitHub
• URL: https://github.com/kestra-io/dbt-example
• Owner: kestra-io
• Created: 2024-02-17T12:48:58.000Z (9 months ago)
• Default Branch: main
• Last Pushed: 2024-08-05T17:02:40.000Z (4 months ago)
• Last Synced: 2024-08-05T20:02:53.966Z (4 months ago)
• Language: Python
• Size: 337 KB
• Stars: 2
• Watchers: 5
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
## Orchestrate dbt with kestra

A lightweight sample dbt project building dbt models in the same Postgres database used as Kestra backend.

## Getting started

To spin up Kestra in Docker (including Postgres database), use the following commands:

```bash

docker compose up -d

sleep 60 # wait a minute for the containers to start

curl -X POST http://localhost:8080/api/v1/flows/import -F fileUpload=@generate_flow_with_dbt_dag.yaml

curl -X POST http://localhost:8080/api/v1/executions/company.datateam.dbt/generate_flow_with_dbt_dag?wait=true

```

The `curl` commands will deploy and run flows that:

- sync dbt code from this GitHub repository to Kestra 

- dynamically generate Kestra tasks based on dbt models and tests

![kestra](kestra/flow_topology.png)

## Limitations

Kestra runs tasks in a stateless manner. Your dbt models and tests executed with the approach presented here cannot share a working directory. Therefore:

1. You cannot use the DuckDB adapter (unless you use MotherDuck). For local DuckDB, try [this blueprint](https://kestra.io/blueprints/git/50-git-workflow-for-dbt-with-duckdb) instead.

2. If you use some dbt packackages, you need to either use the `Process` task runner or add `dbt deps` before each dbt command.