https://github.com/danielgafni/dagster-ray
Ray integration for Dagster
https://github.com/danielgafni/dagster-ray
dagster dagster-pipes kubernetes orchestration ray
Last synced: 5 months ago
JSON representation
Ray integration for Dagster
- Host: GitHub
- URL: https://github.com/danielgafni/dagster-ray
- Owner: danielgafni
- License: apache-2.0
- Created: 2023-09-01T17:48:01.000Z (almost 3 years ago)
- Default Branch: master
- Last Pushed: 2026-01-23T16:47:36.000Z (5 months ago)
- Last Synced: 2026-01-24T07:24:55.393Z (5 months ago)
- Topics: dagster, dagster-pipes, kubernetes, orchestration, ray
- Language: Python
- Homepage: https://danielgafni.github.io/dagster-ray
- Size: 4.1 MB
- Stars: 61
- Watchers: 3
- Forks: 7
- Open Issues: 40
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
# `dagster-ray`
[](https://pypi.python.org/pypi/dagster-ray)
[](https://pypi.python.org/pypi/dagster-ray)
[](https://pypi.python.org/pypi/dagster-ray)
[](https://github.com/danielgafni/dagster-ray/actions/workflows/CI.yml)
[](https://github.com/pre-commit/pre-commit)
[](https://docs.basedpyright.com)
[](https://github.com/astral-sh/ruff)
---
**Ray integration for Dagster.**
`dagster-ray` enables working with distributed Ray compute from Dagster pipelines, combining Dagster's excellent orchestration capabilities and Ray's distributed computing power together.
> [!NOTE]
> This project is ready for production use, but some APIs may change between minor releases.
Learn more in the [docs](https://danielgafni.github.io/dagster-ray)
## 🚀 Key Features
- **🎯 Run Launchers & Executors**: Submit Dagster runs or individual steps by submitting Ray jobs
- **🔧 Ray Resources**: Automatically create and destroy ephemeral Ray clusters and connect to them in client mode
- **📡 Dagster Pipes Integration**: Submit external scripts as Ray jobs, stream back logs and rich Dagster metadata
- **☸️ KubeRay Support**: Utilize `RayJob` and `RayCluster` custom resources in client or job submission mode ([tutorial](https://danielgafni.github.io/dagster-ray/tutorial/kuberay))
- **🏭 Production Ready**: Tested against a matrix of core dependencies, integrated with Dagster+
- **⚡ Instant Startup**: Leverage `RayCluster` with [Cluster Sharing](https://danielgafni.github.io/dagster-ray/tutorial/kuberay/#cluster-sharing) for lightning-fast development cycles with zero cold start times (intended for development environments)
## Installation
```shell
pip install dagster-ray
```
## 📚 Docs
**📖 [Full Documentation](https://danielgafni.github.io/dagster-ray)**
- **[Tutorial](https://danielgafni.github.io/dagster-ray/tutorial/)**: Step-by-step guide with examples
- **[API Reference](https://danielgafni.github.io/dagster-ray/api/)**: Complete API reference
## 🤝 Contributing
Contributions are very welcome! To get started:
```bash
uv sync --all-extras --all-groups
uv run pre-commit install
```
### 🧪 Testing
```bash
uv run pytest
```
Running KubeRay tests requires the following tools to be present:
- `docker`
- `kubectl`
- `helm`
- `minikube`
❄️ Nix users will find them provided in the dev shell:
```
nix develop
```
### Documentation
To build and serve the documentation locally:
```bash
# Serve documentation locally
uv run --group docs mkdocs serve
```