https://github.com/opensource-observer/dagster-sqlmesh
Dagster SQLMesh Adapter
https://github.com/opensource-observer/dagster-sqlmesh
Last synced: 6 months ago
JSON representation
Dagster SQLMesh Adapter
- Host: GitHub
- URL: https://github.com/opensource-observer/dagster-sqlmesh
- Owner: opensource-observer
- License: apache-2.0
- Created: 2024-08-26T21:54:39.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-08-18T19:42:54.000Z (8 months ago)
- Last Synced: 2025-08-18T21:29:58.751Z (8 months ago)
- Language: Python
- Size: 896 KB
- Stars: 65
- Watchers: 3
- Forks: 8
- Open Issues: 10
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
# dagster-sqlmesh
_WARNING: THIS IS A WORK IN PROGRESS_
SQLMesh library for dagster integration.
## Current features
* A `@sqlmesh_assets` decorator akin to `dagster-dbt`'s `@dbt_assets` decorator.
* A `SQLMeshResource` that allows you to call sqlmesh from inside an asset
(likely one defined by the `@sqlmesh_assets` decorator)
* A `SQLMeshDagsterTranslator` that allows customizing the translation of
sqlmesh models into dagster assets.
## Basic Usage
This dagster sqlmesh adapter is intended to work in a similar pattern to that of
`dagster-dbt` in the most basic case by using the `@sqlmesh_assets`
Assuming that your sqlmesh project is located in a directory `/home/foo/sqlmesh_project`, this is how you'd setup your dagster assets:
```python
from dagster import (
AssetExecutionContext,
Definitions,
)
from dagster_sqlmesh import sqlmesh_assets, SQLMeshContextConfig, SQLMeshResource
sqlmesh_config = SQLMeshContextConfig(path="/home/foo/sqlmesh_project", gateway="name-of-your-gateway")
@sqlmesh_assets(environment="dev", config=sqlmesh_config)
def sqlmesh_project(context: AssetExecutionContext, sqlmesh: SQLMeshResource):
yield from sqlmesh.run(context)
defs = Definitions(
assets=[sqlmesh_project],
resources={
"sqlmesh": SQLMeshResource(config=sqlmesh_config),
},
)
```
## Advanced Usage
### Custom Translator
The translator is centrally configured and ensures consistency across all components. You can customize the translator by specifying a custom class in the config:
```python
from dagster_sqlmesh import SQLMeshDagsterTranslator
class CustomSQLMeshTranslator(SQLMeshDagsterTranslator):
def get_asset_key_str(self, fqn: str) -> str:
# Custom asset key generation logic
return f"custom_prefix__{super().get_asset_key_str(fqn)}"
# Configure with custom translator
sqlmesh_config = SQLMeshContextConfig(
path="/home/foo/sqlmesh_project",
gateway="name-of-your-gateway",
translator_class_name="your_module.CustomSQLMeshTranslator"
)
@sqlmesh_assets(environment="dev", config=sqlmesh_config)
def sqlmesh_project(context: AssetExecutionContext, sqlmesh: SQLMeshResource):
yield from sqlmesh.run(context)
```
This approach ensures that both the `SQLMeshResource` and the `@sqlmesh_assets` decorator use the same translator instance, preventing inconsistencies. The translator is created using `config.get_translator()` and passed to all components that need it, including the `DagsterSQLMeshEventHandler`.
## Contributing
_We are very open to contributions!_
In order to build the project you'll need the following:
* python 3.11 or 3.12
* node 18+
* pnpm 8+
_Note: this is a python project but some of our dependent tools are in typescript. As such all this is needed_
### Installing
The project uses Make commands to simplify the development setup process. To get started:
```bash
make init
```
This will:
- Set up a Python virtual environment with Python 3.12
- Install all Python dependencies
- Install Node.js dependencies via pnpm
_Note: All Make commands automatically use the correct virtual environment - you don't need to activate it manually._
To upgrade dependencies:
```bash
make upgrade-python-deps # Upgrade Python dependencies
make upgrade-node-deps # Upgrade Node.js dependencies
```
### Running tests
We have tests that should work entirely locally. You may see a `db.db` file appear in the root of the repository when these tests are run. It can be safely ignored or deleted.
To run tests:
```bash
make test
```
### Running the "sample" dagster project
In the `sample/dagster_project` directory, is a minimal dagster project with the
accompanying sqlmesh project from `sample/sqlmesh_project` configured as an
asset. To run the sample dagster project deployment with a UI:
```bash
make dagster-dev
```
or
```bash
make dev
```
If you'd like to materialize the dagster assets quickly on the CLI:
```bash
make dagster-materialize
```
_Note: The sqlmesh project that is in the sample folder has a dependency on a
table that doesn't exist by default within the defined duckdb database. You'll
notice there's a `test_source` asset in the dagster project. This asset will
automatically populate that table in duckdb so that the sqlmesh project can be
run properly. Before you run any materializations against the sqlmesh related
assets in dagster, ensure that you've run the `test_source` at least once._
## Future Plans
* Create a new "loader" for sqlmesh and dagster definitions to allow for
automatic creation of administrative jobs for sqlmesh (e.g. migrations).
Additionally, we may want to have this generate assets outside of the
`multi_asset` paradigm within dagster such that assets can have independent
partitions. There is an existing issue for this in [dagster
itself](https://github.com/dagster-io/dagster/issues/14228).