{"id":21050017,"url":"https://github.com/astronomer/astro-provider-ray","last_synced_at":"2025-07-06T16:33:06.047Z","repository":{"id":252435546,"uuid":"807927399","full_name":"astronomer/astro-provider-ray","owner":"astronomer","description":"This provider contains operators, decorators and triggers to send a ray job from an airflow task","archived":false,"fork":false,"pushed_at":"2025-03-25T15:12:42.000Z","size":642,"stargazers_count":18,"open_issues_count":16,"forks_count":4,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-05-02T09:05:09.721Z","etag":null,"topics":["apache-airflow","dags","mlops-workflow","mlpipelines","workflow"],"latest_commit_sha":null,"homepage":"https://astronomer.github.io/astro-provider-ray/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/astronomer.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.rst","contributing":"docs/CONTRIBUTING.rst","funding":null,"license":"LICENSE","code_of_conduct":"docs/CODE_OF_CONDUCT.rst","threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-30T03:39:25.000Z","updated_at":"2025-04-03T02:40:17.000Z","dependencies_parsed_at":null,"dependency_job_id":"c004edce-2336-4a49-b2db-91934cc750f1","html_url":"https://github.com/astronomer/astro-provider-ray","commit_stats":null,"previous_names":["astronomer/astro-provider-ray"],"tags_count":13,"template":false,"template_full_name":"astronomer/airflow-provider-sample","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astronomer%2Fastro-provider-ray","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astronomer%2Fastro-provider-ray/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astronomer%2Fastro-provider-ray/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astronomer%2Fastro-provider-ray/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/astronomer","download_url":"https://codeload.github.com/astronomer/astro-provider-ray/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254418461,"owners_count":22068070,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-airflow","dags","mlops-workflow","mlpipelines","workflow"],"created_at":"2024-11-19T15:18:47.789Z","updated_at":"2025-05-15T20:31:59.364Z","avatar_url":"https://github.com/astronomer.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e\n  Ray provider\n\u003c/h1\u003e\n\n\u003cdiv align=\"center\"\u003e\n\n:books: [Docs](https://astronomer.github.io/astro-provider-ray/) \u0026nbsp; | \u0026nbsp; :rocket: [Getting Started](https://astronomer.github.io/astro-provider-ray/getting_started/setup.html) \u0026nbsp; | \u0026nbsp; :speech_balloon: [Slack](https://join.slack.com/t/apache-airflow/shared_invite/zt-2nsw28cw1-Lw4qCS0fgme4UI_vWRrwEQ) (``#airflow-ray``)\u0026nbsp; | \u0026nbsp; :fire: [Contribute](https://astronomer.github.io/astro-provider-ray/CONTRIBUTING.html) \u0026nbsp;\n\n\u003c/div\u003e\n\nOrchestrate your Ray jobs using [Apache Airflow®](https://airflow.apache.org/) combining Airflow's workflow management with Ray's distributed computing capabilities.\n\nBenefits of using this provider include:\n- **Integration**: Incorporate Ray jobs into Airflow DAGs for unified workflow management.\n- **Distributed computing**: Use Ray's distributed capabilities within Airflow pipelines for scalable ETL, LLM fine-tuning etc.\n- **Monitoring**: Track Ray job progress through Airflow's user interface.\n- **Dependency management**: Define and manage dependencies between Ray jobs and other tasks in DAGs.\n- **Resource allocation**: Run Ray jobs alongside other task types within a single pipeline.\n\n\n## Table of Contents\n- [Quickstart](#quickstart)\n- [Sample DAGs](#sample-dags)\n- [Getting Involved](#getting-involved)\n- [Changelog](#changelog)\n- [Contributing Guide](#contributing-guide)\n\n## Quickstart\nCheck out the Getting Started guide in our [docs](https://astronomer.github.io/astro-provider-ray/getting_started/setup.html). Sample DAGs are available at [example_dags/](https://github.com/astronomer/astro-provider-ray/tree/main/example_dags).\n\n## Sample DAGs\n\n### Example 1: Using @ray.task for job life cycle\nThe below example showcases how to use the ``@ray.task`` decorator to manage the full lifecycle of a Ray cluster: setup, job execution, and teardown.\n\nThis approach is ideal for jobs that require a dedicated, short-lived cluster, optimizing resource usage by cleaning up after task completion\n\nhttps://github.com/astronomer/astro-provider-ray/blob/bd6d847818be08fae78bc1e4c9bf3334adb1d2ee/example_dags/ray_taskflow_example.py#L1-L57\n\n### Example 2: Using SetupRayCluster, SubmitRayJob \u0026 DeleteRayCluster\nThis example shows how to use separate operators for cluster setup, job submission, and teardown, providing more granular control over the process.\n\nThis approach allows for more complex workflows involving Ray clusters.\n\nKey Points:\n\n- Uses SetupRayCluster, SubmitRayJob, and DeleteRayCluster operators separately.\n- Allows for multiple jobs to be submitted to the same cluster before deletion.\n- Demonstrates how to pass cluster information between tasks using XCom.\n\nThis method is ideal for scenarios where you need fine-grained control over the cluster lifecycle, such as running multiple jobs on the same cluster or keeping the cluster alive for a certain period.\n\nhttps://github.com/astronomer/astro-provider-ray/blob/bd6d847818be08fae78bc1e4c9bf3334adb1d2ee/example_dags/setup-teardown.py#L1-L44\n\n## Getting Involved\n\n| Platform | Purpose | Est. Response time |\n|:---:|:---:|:---:|\n| [Discussion Forum](https://github.com/astronomer/astro-provider-ray/discussions) | General inquiries and discussions | \u003c 3 days |\n| [GitHub Issues](https://github.com/astronomer/astro-provider-ray/issues) | Bug reports and feature requests | \u003c 1-2 days |\n| [Slack](https://join.slack.com/t/apache-airflow/shared_invite/zt-2nsw28cw1-Lw4qCS0fgme4UI_vWRrwEQ) | Quick questions and real-time chat | 12 hrs |\n\n## Changelog\nWe follow [Semantic Versioning](https://semver.org/) for releases. Check [CHANGELOG.rst](https://github.com/astronomer/astro-provider-ray/blob/main/CHANGELOG.rst) for the latest changes.\n\n## Contributing Guide\nAll contributions, bug reports, bug fixes, documentation improvements, enhancements are welcome.\n\nA detailed overview on how to contribute can be found in the [Contributing Guide](https://github.com/astronomer/astro-provider-ray/blob/main/docs/CONTRIBUTING.rst).\n\n## License\n[Apache 2.0 License](https://github.com/astronomer/astro-provider-ray/blob/main/LICENSE)\n\n## Privacy Notice\nThis project follows `Astronomer's Privacy Policy \u003chttps://www.astronomer.io/privacy/\u003e`_\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fastronomer%2Fastro-provider-ray","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fastronomer%2Fastro-provider-ray","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fastronomer%2Fastro-provider-ray/lists"}