https://github.com/anelendata/dbt-ksql
dbt ksqlDB adapter
https://github.com/anelendata/dbt-ksql
Last synced: 11 days ago
JSON representation
dbt ksqlDB adapter
- Host: GitHub
- URL: https://github.com/anelendata/dbt-ksql
- Owner: anelendata
- License: apache-2.0
- Created: 2022-09-15T04:54:57.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2022-09-20T06:25:38.000Z (over 3 years ago)
- Last Synced: 2023-08-02T03:35:23.217Z (over 2 years ago)
- Language: Python
- Size: 43 KB
- Stars: 19
- Watchers: 2
- Forks: 3
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
- awesome - anelendata/dbt-ksql - dbt ksqlDB adapter (<a name="Python"></a>Python)
README
# dbt-ksql
A dbt adapter for ksqlDB
Like in many database projects, managing models becomes increasingly challenging.
dbt-ksql helps to modularize the code, manage the model dependency, and facilitate the model deployment.
## Concepts
This project is in a proof-of-concept phase.
We are using part of dbt features to manage ksqlDB by
- Separating production/staging/development environments through dbt's profile and target.
- Modularize the SQL with dbt's variable, reference, macro, etc.
Not all dbt features are supported. ksqlDB transforms streaming data. Some of
dbt features are assuming batch operation (e.g. incremental and snapshot).
### Materialization types
Supported:
- stream
- table
Not-suported:
- ephemeral
- incremental
## Example project
See [example](./example) folder for a dbt project.
It has examples for creating stream, stream as, table.
It also has a example of using macros.
## Pre-release warning
- This project is an experimental phase.
- This has not been published to pypi.
## Pre-release install steps
To install, fetch this repository and run a local install:
```
git clone git@github.com:anelendata/dbt-ksql.git
cd dbt-ksql
python -m venv venv
source venv/bin/activate
pip install -U pip
pip install -e .
```
After the successful instsall, the following command will show
```
dbt --version
```
Something like this:
```
Core:
- installed: 1.1.2
- latest: 1.2.1 - Update available!
Your version of dbt-core is out of date!
You can find instructions for upgrading here:
https://docs.getdbt.com/docs/installation
Plugins:
- ksqldb: 1.1.0 - Could not determine latest version
```
## Example commands
After set up profiles.yml try running the following commands:
- compile: `dbt compile --profiles-dir . --profile prod`
- run: `dbt run --profiles-dir . --profile prod`
## Helper macro
- drop stream: `dbt run-operation drop_stream --args "{relation: some_stream, delete_topic: true}" --profiles-dir . --profile dev`
## About dbt
**[dbt](https://www.getdbt.com/)** enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
dbt is the T in ELT. Organize, cleanse, denormalize, filter, rename, and pre-aggregate the raw data in your warehouse so that it's ready for analysis.
# About this project
This project is developed by
ANELEN and friends. Please check out the ANELEN's
[open innovation philosophy and other projects](https://anelen.co/open-source.html)
---
Copyright © 2022~ Anelen Co., LLC