{"id":48726432,"url":"https://github.com/stateful-y/kedro-dagster","last_synced_at":"2026-04-11T22:58:29.292Z","repository":{"id":262474817,"uuid":"886807141","full_name":"stateful-y/kedro-dagster","owner":"stateful-y","description":"Kedro plugin to support running pipelines on Dagster","archived":false,"fork":false,"pushed_at":"2026-04-04T20:07:47.000Z","size":4580,"stargazers_count":22,"open_issues_count":7,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2026-04-11T22:58:24.787Z","etag":null,"topics":["dagster","data-engineering","data-science","dataops","kedro","kedro-plugin","machine-leaning-engineering","machine-learning","mlops","orchestration"],"latest_commit_sha":null,"homepage":"https://kedro-dagster.readthedocs.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stateful-y.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-11-11T16:33:02.000Z","updated_at":"2026-04-04T20:07:52.000Z","dependencies_parsed_at":"2024-11-12T16:44:09.460Z","dependency_job_id":"64622b2b-65a4-4bcf-a39a-16182d448611","html_url":"https://github.com/stateful-y/kedro-dagster","commit_stats":null,"previous_names":["gtauzin/kedro-dagster","stateful-y/kedro-dagster"],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/stateful-y/kedro-dagster","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stateful-y%2Fkedro-dagster","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stateful-y%2Fkedro-dagster/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stateful-y%2Fkedro-dagster/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stateful-y%2Fkedro-dagster/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stateful-y","download_url":"https://codeload.github.com/stateful-y/kedro-dagster/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stateful-y%2Fkedro-dagster/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31698152,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-11T21:17:31.016Z","status":"ssl_error","status_checked_at":"2026-04-11T21:17:24.556Z","response_time":54,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dagster","data-engineering","data-science","dataops","kedro","kedro-plugin","machine-leaning-engineering","machine-learning","mlops","orchestration"],"created_at":"2026-04-11T22:58:28.612Z","updated_at":"2026-04-11T22:58:29.285Z","avatar_url":"https://github.com/stateful-y.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cpicture\u003e\n    \u003csource media=\"(prefers-color-scheme: light)\" srcset=\"https://raw.githubusercontent.com/stateful-y/kedro-dagster/main/docs/assets/logo_light.png\"\u003e\n    \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"https://raw.githubusercontent.com/stateful-y/kedro-dagster/main/docs/assets/logo_dark.png\"\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/stateful-y/kedro-dagster/main/docs/assets/logo_light.png\" alt=\"Kedro-Dagster\"\u003e\n  \u003c/picture\u003e\n\u003c/p\u003e\n\n[![Python Version](https://img.shields.io/pypi/pyversions/kedro-dagster)](https://pypi.org/project/kedro-dagster/)\n[![License](https://img.shields.io/github/license/stateful-y/kedro-dagster)](https://github.com/stateful-y/kedro-dagster/blob/main/LICENSE)\n[![PyPI Version](https://img.shields.io/pypi/v/kedro-dagster)](https://pypi.org/project/kedro-dagster/)\n[![Conda Version](https://img.shields.io/conda/vn/conda-forge/kedro-dagster.svg)](https://anaconda.org/conda-forge/kedro-dagster)\n[![CodeCov](https://codecov.io/gh/stateful-y/kedro-dagster/branch/main/graph/badge.svg)](https://codecov.io/gh/stateful-y/kedro-dagster/branch/main)\n\n[![Powered by Kedro](https://img.shields.io/badge/powered_by-kedro-ffc900?logo=kedro)](https://kedro.org)\n[![Slack Organisation](https://img.shields.io/badge/slack-chat-blueviolet.svg?label=Kedro%20Slack\u0026logo=slack)](https://slack.kedro.org)\n\n## What is Kedro-Dagster?\n\nThe Kedro-Dagster plugin enables seamless integration between [Kedro](https://kedro.readthedocs.io/), a framework for creating reproducible and maintainable data science code, and [Dagster](https://dagster.io/), a data orchestrator for machine learning and data pipelines. This plugin makes use of Dagster's orchestration capabilities to automate and monitor Kedro pipelines effectively.\n\n\u003cp align=\"center\"\u003e\n  \u003cpicture\u003e\n    \u003csource media=\"(prefers-color-scheme: light)\" srcset=\"https://raw.githubusercontent.com/stateful-y/kedro-dagster/main/docs/assets/example/local_asset_graph_light.png\"\u003e\n    \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"https://raw.githubusercontent.com/stateful-y/kedro-dagster/main/docs/assets/example/local_asset_graph_dark.png\"\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/stateful-y/kedro-dagster/main/docs/assets/example/local_asset_graph_light.png\" alt=\"Kedro-Dagster Asset Graph\"\u003e\n  \u003c/picture\u003e\n\u003c/p\u003e\n\nCurrently, Kedro-Dagster supports Kedro versions 0.19.x and 1.x, and Dagster versions 1.10.x, 1.11.x, and 1.12.x.\n\n## What are the features of Kedro-Dagster?\n\n- **Configuration‑driven workflows:** Centralize orchestration settings in a `dagster.yml` file for each Kedro environment. Define jobs from filtered Kedro pipelines, assign executors, schedules.\n- **Customization:** The core integration lives in the auto‑generated Dagster `definitions.py`. For advanced use cases, you can extend or override these definitions.\n- **Kedro hooks preservation:** Kedro hooks are preserved and called at the appropriate time during pipeline execution, so custom logic (e.g., data validation, logging) continues to work seamlessly.\n- **MLflow compatibility:** Use [Kedro-MLflow](https://github.com/Galileo-Galilei/kedro-mlflow) with Dagster’s [MLflow integration](https://dagster.io/integrations/dagster-mlflow) to track experiments, log models, and register artifacts.\n- **Logger integration:** Unifies Kedro and Dagster logging so logs from Kedro nodes appear in the Dagster UI and are easy to trace and debug. Additionally, provides configuration to customize Dagster run loggers.\n- **(Experimental) Dagster partition support:** Make use of Dagster's partitions to fan-out Kedro nodes acting on partitioned data.\n\n## How to install Kedro-Dagster?\n\nInstall the Kedro-Dagster plugin using `pip`:\n\n```bash\npip install kedro-dagster\n```\n\nor using `uv`:\n\n```bash\nuv pip install kedro-dagster\n```\n\nor using `conda`:\n\n```bash\nconda install -c conda-forge kedro-dagster\n```\n\nor using `mamba`:\n\n```bash\nmamba install -c conda-forge kedro-dagster\n```\n\nor alternatively, add `kedro-dagster` to your `requirements.txt` or `pyproject.toml` file.\n\n## How to get started with Kedro-Dagster?\n\n1. **Initialize the plugin in your Kedro project**\n\nUse the following command to generate a `definitions.py` file, where all translated Kedro objects are available as Dagster objects, and a `dagster.yml` configuration file:\n\n```bash\nkedro dagster init --env \u003cENV_NAME\u003e\n```\n\n2. **Configure jobs, executors, and schedules**\n\nDefine your job executors and schedules in the `dagster.yml` configuration file located in your Kedro project's `conf/\u003cENV_NAME\u003e` directory. This file allows you to filter Kedro pipelines and assign specific executors and schedules to them.\n\n```yaml\n# conf/local/dagster.yml\nschedules:\n  daily: # Schedule name\n    cron_schedule: \"0 0 * * *\" # Schedule parameters\n\nexecutors: # Executor name\n  sequential: # Executor parameters\n    in_process:\n\n  multiprocess:\n    multiprocess:\n      max_concurrent: 2\n\njobs:\n  default: # Job name\n    pipeline: # Pipeline filter parameters\n      pipeline_name: __default__\n    executor: sequential\n\n  parallel_data_processing:\n    pipeline:\n      pipeline_name: data_processing\n      node_names:\n      - preprocess_companies_node\n      - preprocess_shuttles_node\n    schedule: daily\n    executor: multiprocess\n\n  data_science:\n    pipeline:\n      pipeline_name: data_science\n    schedule: daily\n    executor: sequential\n```\n\n3. **Launch the Dagster UI**\n\nStart the Dagster UI to monitor and manage your pipelines using the following command:\n\n```bash\nkedro dagster dev --env \u003cENV_NAME\u003e\n```\n\nThe Dagster UI will be available at http://127.0.0.1:3000.\n\nFor a concrete use-case, see the [Kedro-Dagster example repository](https://github.com/stateful-y/kedro-dagster-example).\n\n## How do I use Kedro-Dagster?\n\nFull documentation is available at [https://kedro-dagster.readthedocs.io/](https://kedro-dagster.readthedocs.io/).\n\n## Can I contribute?\n\nWe welcome contributions, feedback, and questions:\n\n- **Report issues or request features:** [GitHub Issues](https://github.com/stateful-y/kedro-dagster/issues)\n- **Join the discussion:** [GitHub Discussions](https://github.com/stateful-y/kedro-dagster/discussions)\n- **Contributing Guide:** [CONTRIBUTING.md](https://github.com/stateful-y/kedro-dagster/blob/main/CONTRIBUTING.md)\n\nIf you are interested in becoming a maintainer or taking a more active role, please reach out to Guillaume Tauzin on [GitHub Discussions](https://github.com/stateful-y/kedro-dagster/discussions).\n\n## Where can I learn more?\n\n- Full documentation: [https://kedro-dagster.readthedocs.io/](https://kedro-dagster.readthedocs.io/)\n- GitHub Discussions: [https://github.com/stateful-y/kedro-dagster/discussions](https://github.com/stateful-y/kedro-dagster/discussions)\n\nFor questions and discussions, you can also open a [discussion](https://github.com/stateful-y/kedro-dagster/discussions).\n\n## License\n\nThis project is licensed under the terms of the [Apache-2.0 License](https://github.com/stateful-y/kedro-dagster/blob/main/LICENSE).\n\n## Acknowledgements\n\nThis project is maintained by [stateful-y](https://stateful-y.io), an ML consultancy specializing in MLOps and data science \u0026 engineering. If you're interested in collaborating or learning more about our services, please visit our website.\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://stateful-y.io\"\u003e\n    \u003cimg src=\"docs/assets/made_by_stateful-y.png\" alt=\"Made by stateful-y\" width=\"200\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstateful-y%2Fkedro-dagster","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstateful-y%2Fkedro-dagster","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstateful-y%2Fkedro-dagster/lists"}