{"id":31777791,"url":"https://github.com/opensource-observer/dagster-sqlmesh","last_synced_at":"2025-10-10T06:20:19.064Z","repository":{"id":254936058,"uuid":"847960284","full_name":"opensource-observer/dagster-sqlmesh","owner":"opensource-observer","description":"Dagster SQLMesh Adapter","archived":false,"fork":false,"pushed_at":"2025-08-18T19:42:54.000Z","size":918,"stargazers_count":65,"open_issues_count":10,"forks_count":8,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-08-18T21:29:58.751Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/opensource-observer.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-08-26T21:54:39.000Z","updated_at":"2025-08-10T09:56:02.000Z","dependencies_parsed_at":"2024-08-27T03:23:41.208Z","dependency_job_id":"21805aa3-36fa-4b0a-875f-1f9135a3a905","html_url":"https://github.com/opensource-observer/dagster-sqlmesh","commit_stats":null,"previous_names":["opensource-observer/dagster-sqlmesh"],"tags_count":11,"template":false,"template_full_name":null,"purl":"pkg:github/opensource-observer/dagster-sqlmesh","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opensource-observer%2Fdagster-sqlmesh","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opensource-observer%2Fdagster-sqlmesh/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opensource-observer%2Fdagster-sqlmesh/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opensource-observer%2Fdagster-sqlmesh/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/opensource-observer","download_url":"https://codeload.github.com/opensource-observer/dagster-sqlmesh/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opensource-observer%2Fdagster-sqlmesh/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279002970,"owners_count":26083488,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-10T02:00:06.843Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-10T06:20:17.654Z","updated_at":"2025-10-10T06:20:19.059Z","avatar_url":"https://github.com/opensource-observer.png","language":"Python","readme":"# dagster-sqlmesh\n\n_WARNING: THIS IS A WORK IN PROGRESS_\n\nSQLMesh library for dagster integration.\n\n## Current features\n\n* A `@sqlmesh_assets` decorator akin to `dagster-dbt`'s `@dbt_assets` decorator.\n* A `SQLMeshResource` that allows you to call sqlmesh from inside an asset\n  (likely one defined by the `@sqlmesh_assets` decorator)\n* A `SQLMeshDagsterTranslator` that allows customizing the translation of\n  sqlmesh models into dagster assets.\n\n## Basic Usage\n\nThis dagster sqlmesh adapter is intended to work in a similar pattern to that of\n`dagster-dbt` in the most basic case by using the `@sqlmesh_assets`\n\nAssuming that your sqlmesh project is located in a directory `/home/foo/sqlmesh_project`, this is how you'd setup your dagster assets:\n\n```python\nfrom dagster import (\n    AssetExecutionContext,\n    Definitions,\n)\nfrom dagster_sqlmesh import sqlmesh_assets, SQLMeshContextConfig, SQLMeshResource\n\nsqlmesh_config = SQLMeshContextConfig(path=\"/home/foo/sqlmesh_project\", gateway=\"name-of-your-gateway\")\n\n@sqlmesh_assets(environment=\"dev\", config=sqlmesh_config)\ndef sqlmesh_project(context: AssetExecutionContext, sqlmesh: SQLMeshResource):\n    yield from sqlmesh.run(context)\n\ndefs = Definitions(\n    assets=[sqlmesh_project],\n    resources={\n        \"sqlmesh\": SQLMeshResource(config=sqlmesh_config),\n    },\n)\n```\n\n## Advanced Usage\n\n### Custom Translator\n\nThe translator is centrally configured and ensures consistency across all components. You can customize the translator by specifying a custom class in the config:\n\n```python\nfrom dagster_sqlmesh import SQLMeshDagsterTranslator\n\nclass CustomSQLMeshTranslator(SQLMeshDagsterTranslator):\n    def get_asset_key_str(self, fqn: str) -\u003e str:\n        # Custom asset key generation logic\n        return f\"custom_prefix__{super().get_asset_key_str(fqn)}\"\n\n# Configure with custom translator\nsqlmesh_config = SQLMeshContextConfig(\n    path=\"/home/foo/sqlmesh_project\", \n    gateway=\"name-of-your-gateway\",\n    translator_class_name=\"your_module.CustomSQLMeshTranslator\"\n)\n\n@sqlmesh_assets(environment=\"dev\", config=sqlmesh_config)\ndef sqlmesh_project(context: AssetExecutionContext, sqlmesh: SQLMeshResource):\n    yield from sqlmesh.run(context)\n```\n\nThis approach ensures that both the `SQLMeshResource` and the `@sqlmesh_assets` decorator use the same translator instance, preventing inconsistencies. The translator is created using `config.get_translator()` and passed to all components that need it, including the `DagsterSQLMeshEventHandler`.\n\n\n## Contributing\n\n_We are very open to contributions!_\n\nIn order to build the project you'll need the following:\n\n* python 3.11 or 3.12\n* node 18+\n* pnpm 8+\n\n_Note: this is a python project but some of our dependent tools are in typescript. As such all this is needed_\n\n### Installing\n\nThe project uses Make commands to simplify the development setup process. To get started:\n\n```bash\nmake init\n```\n\nThis will:\n- Set up a Python virtual environment with Python 3.12\n- Install all Python dependencies\n- Install Node.js dependencies via pnpm\n\n_Note: All Make commands automatically use the correct virtual environment - you don't need to activate it manually._\n\nTo upgrade dependencies:\n```bash\nmake upgrade-python-deps  # Upgrade Python dependencies\nmake upgrade-node-deps   # Upgrade Node.js dependencies\n```\n\n### Running tests\n\nWe have tests that should work entirely locally. You may see a `db.db` file appear in the root of the repository when these tests are run. It can be safely ignored or deleted.\n\nTo run tests:\n\n```bash\nmake test\n```\n\n### Running the \"sample\" dagster project\n\nIn the `sample/dagster_project` directory, is a minimal dagster project with the\naccompanying sqlmesh project from `sample/sqlmesh_project` configured as an\nasset. To run the sample dagster project deployment with a UI:\n\n```bash\nmake dagster-dev \n```\nor \n```bash\nmake dev\n```\n\nIf you'd like to materialize the dagster assets quickly on the CLI:\n\n```bash\nmake dagster-materialize\n```\n\n_Note: The sqlmesh project that is in the sample folder has a dependency on a\ntable that doesn't exist by default within the defined duckdb database. You'll\nnotice there's a `test_source` asset in the dagster project. This asset will\nautomatically populate that table in duckdb so that the sqlmesh project can be\nrun properly. Before you run any materializations against the sqlmesh related\nassets in dagster, ensure that you've run the `test_source` at least once._\n\n## Future Plans\n\n* Create a new \"loader\" for sqlmesh and dagster definitions to allow for\n  automatic creation of administrative jobs for sqlmesh (e.g. migrations).\n  Additionally, we may want to have this generate assets outside of the\n  `multi_asset` paradigm within dagster such that assets can have independent\n  partitions. There is an existing issue for this in [dagster\n  itself](https://github.com/dagster-io/dagster/issues/14228).","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopensource-observer%2Fdagster-sqlmesh","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopensource-observer%2Fdagster-sqlmesh","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopensource-observer%2Fdagster-sqlmesh/lists"}