{"id":13742233,"url":"https://github.com/PrefectHQ/prefect-ray","last_synced_at":"2025-05-08T23:33:31.897Z","repository":{"id":37054786,"uuid":"473720744","full_name":"PrefectHQ/prefect-ray","owner":"PrefectHQ","description":"Prefect integrations with Ray","archived":true,"fork":false,"pushed_at":"2024-04-26T16:30:16.000Z","size":2071,"stargazers_count":63,"open_issues_count":6,"forks_count":5,"subscribers_count":14,"default_branch":"main","last_synced_at":"2024-05-22T13:31:17.569Z","etag":null,"topics":["parallel","prefect","python","ray"],"latest_commit_sha":null,"homepage":"https://prefecthq.github.io/prefect-ray/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PrefectHQ.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-03-24T18:08:12.000Z","updated_at":"2024-08-03T05:00:54.296Z","dependencies_parsed_at":"2023-12-06T17:26:05.248Z","dependency_job_id":"9af1fc10-b1c7-4494-9992-00df81833125","html_url":"https://github.com/PrefectHQ/prefect-ray","commit_stats":{"total_commits":102,"total_committers":11,"mean_commits":9.272727272727273,"dds":0.607843137254902,"last_synced_commit":"144d668c37db7a63ba2b9aaef01394a13523e361"},"previous_names":[],"tags_count":17,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PrefectHQ%2Fprefect-ray","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PrefectHQ%2Fprefect-ray/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PrefectHQ%2Fprefect-ray/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PrefectHQ%2Fprefect-ray/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PrefectHQ","download_url":"https://codeload.github.com/PrefectHQ/prefect-ray/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224788423,"owners_count":17370132,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["parallel","prefect","python","ray"],"created_at":"2024-08-03T05:00:24.587Z","updated_at":"2024-11-15T13:31:36.982Z","avatar_url":"https://github.com/PrefectHQ.png","language":"Python","funding_links":[],"categories":["Models and Projects"],"sub_categories":["Ray + X (integration)"],"readme":"\u003e [!NOTE]\n\u003e Active development of this project has moved within PrefectHQ/prefect. The code can be found [here](https://github.com/PrefectHQ/prefect/tree/main/src/integrations/prefect-ray) and documentation [here](https://docs.prefect.io/latest/integrations/prefect-ray).\n\u003e Please open issues and PRs against PrefectHQ/prefect instead of this repository.\n\n\n# prefect-ray\n\n\u003cp align=\"center\"\u003e\n    \u003c!--- Insert a cover image here --\u003e\n    \u003c!--- \u003cbr\u003e --\u003e\n    \u003ca href=\"https://pypi.python.org/pypi/prefect-ray/\" alt=\"PyPI version\"\u003e\n        \u003cimg alt=\"PyPI\" src=\"https://img.shields.io/pypi/v/prefect-ray?color=26272B\u0026labelColor=090422\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/PrefectHQ/prefect-ray/\" alt=\"Stars\"\u003e\n        \u003cimg src=\"https://img.shields.io/github/stars/PrefectHQ/prefect-ray?color=26272B\u0026labelColor=090422\" /\u003e\u003c/a\u003e\n    \u003ca href=\"https://pepy.tech/badge/prefect-ray/\" alt=\"Downloads\"\u003e\n        \u003cimg src=\"https://img.shields.io/pypi/dm/prefect-ray?color=26272B\u0026labelColor=090422\" /\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/PrefectHQ/prefect-ray/pulse\" alt=\"Activity\"\u003e\n        \u003cimg src=\"https://img.shields.io/github/commit-activity/m/PrefectHQ/prefect-ray?color=26272B\u0026labelColor=090422\" /\u003e\u003c/a\u003e\n    \u003cbr\u003e\n    \u003ca href=\"https://prefect-community.slack.com\" alt=\"Slack\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/slack-join_community-red.svg?color=26272B\u0026labelColor=0904222\u0026logo=slack\" /\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n## Welcome!\nVisit the full docs [here](https://PrefectHQ.github.io/prefect-ray) to see additional examples and the API reference.\n\n`prefect-ray` contains Prefect integrations with the [Ray](https://www.ray.io/) execution framework, a flexible distributed computing framework for Python.\n\nProvides a `RayTaskRunner` that enables Prefect flows to run tasks execute tasks in parallel using Ray.\n\n## Getting Started\n\n### Python setup\n\nRequires an installation of Python 3.8 or newer.\n\nWe recommend using a Python virtual environment manager such as pipenv, conda, or virtualenv.\n\nThese tasks are designed to work with Prefect 2.0. For more information about how to use Prefect, please refer to the [Prefect documentation](https://docs.prefect.io/).\n\n### Installation\n\nInstall `prefect-ray` with `pip`:\n\n```bash\npip install prefect-ray\n```\n\nUsers running Apple Silicon (such as M1 macs) will need to install ray in a miniforge conda environment.\nClick [here](https://docs.ray.io/en/master/ray-overview/installation.html#m1-mac-apple-silicon-support) for more details.\n\n## Running tasks on Ray\n\nThe `RayTaskRunner` is a [Prefect task runner](https://docs.prefect.io/concepts/task-runners/) that submits tasks to [Ray](https://www.ray.io/) for parallel execution. \n\nBy default, a temporary Ray instance is created for the duration of the flow run.\n\nFor example, this flow counts to 3 in parallel.\n\n```python\nimport time\n\nfrom prefect import flow, task\nfrom prefect_ray import RayTaskRunner\n\n@task\ndef shout(number):\n    time.sleep(0.5)\n    print(f\"#{number}\")\n\n@flow(task_runner=RayTaskRunner)\ndef count_to(highest_number):\n    for number in range(highest_number):\n        shout.submit(number)\n\nif __name__ == \"__main__\":\n    count_to(10)\n\n# outputs\n#3\n#7\n#2\n#6\n#4\n#0\n#1\n#5\n#8\n#9\n```\n\nIf you already have a Ray instance running, you can provide the connection URL via an `address` argument.\n\nTo configure your flow to use the `RayTaskRunner`:\n\n1. Make sure the `prefect-ray` collection is installed as described earlier: `pip install prefect-ray`.\n2. In your flow code, import `RayTaskRunner` from `prefect_ray.task_runners`.\n3. Assign it as the task runner when the flow is defined using the `task_runner=RayTaskRunner` argument.\n\nFor example, this flow uses the `RayTaskRunner` with a local, temporary Ray instance created by Prefect at flow run time.\n\n```python\nfrom prefect import flow\nfrom prefect_ray.task_runners import RayTaskRunner\n\n@flow(task_runner=RayTaskRunner())\ndef my_flow():\n    ... \n```\n\nThis flow uses the `RayTaskRunner` configured to access an existing Ray instance at `ray://192.0.2.255:8786`.\n\n```python\nfrom prefect import flow\nfrom prefect_ray.task_runners import RayTaskRunner\n\n@flow(task_runner=RayTaskRunner(address=\"ray://192.0.2.255:8786\"))\ndef my_flow():\n    ... \n```\n\n`RayTaskRunner` accepts the following optional parameters:\n\n| Parameter | Description |\n| --- | --- |\n| address | Address of a currently running Ray instance, starting with the [ray://](https://docs.ray.io/en/master/cluster/ray-client.html) URI. |\n| init_kwargs | Additional kwargs to use when calling `ray.init`. |\n\nNote that Ray Client uses the [ray://](https://docs.ray.io/en/master/cluster/ray-client.html) URI to indicate the address of a Ray instance. If you don't provide the `address` of a Ray instance, Prefect creates a temporary instance automatically.\n\n!!! warning \"Ray environment limitations\"\n    Ray support for non-x86/64 architectures such as ARM/M1 processors with installation from `pip` alone and will be skipped during installation of Prefect. It is possible to manually install the blocking component with `conda`. See the [Ray documentation](https://docs.ray.io/en/latest/ray-overview/installation.html#m1-mac-apple-silicon-support) for instructions.\n\n    See the [Ray installation documentation](https://docs.ray.io/en/latest/ray-overview/installation.html) for further compatibility information.\n\n## Running tasks on a Ray remote cluster\n\nWhen using the `RayTaskRunner` with a remote Ray cluster, you may run into issues that are not seen when using a local Ray instance. To resolve these issues, we recommend taking the following steps when working with a remote Ray cluster:\n\n1. By default, Prefect will not persist any data to the filesystem of the remote ray worker. However, if you want to take advantage of Prefect's caching ability, you will need to configure a remote result storage to persist results across task runs. \n\nWe recommend using the [Prefect UI to configure a storage block](https://docs.prefect.io/concepts/blocks/) to use for remote results storage.\n\nHere's an example of a flow that uses caching and remote result storage:\n```python\nfrom typing import List\n\nfrom prefect import flow, get_run_logger, task\nfrom prefect.filesystems import S3\nfrom prefect.tasks import task_input_hash\nfrom prefect_ray.task_runners import RayTaskRunner\n\n\n# The result of this task will be cached in the configured result storage\n@task(cache_key_fn=task_input_hash)\ndef say_hello(name: str) -\u003e None:\n    logger = get_run_logger()\n    # This log statement will print only on the first run. Subsequent runs will be cached.\n    logger.info(f\"hello {name}!\")\n    return name\n\n\n@flow(\n    task_runner=RayTaskRunner(\n        address=\"ray://\u003cinstance_public_ip_address\u003e:10001\",\n    ),\n    # Using an S3 block that has already been created via the Prefect UI\n    result_storage=\"s3/my-result-storage\",\n)\ndef greetings(names: List[str]) -\u003e None:\n    for name in names:\n        say_hello.submit(name)\n\n\nif __name__ == \"__main__\":\n    greetings([\"arthur\", \"trillian\", \"ford\", \"marvin\"])\n```\n\n2. If you get an error stating that the module 'prefect' cannot be found, ensure `prefect` is installed on the remote cluster, with:\n```bash\npip install prefect\n```\n\n3. If you get an error with a message similar to \"File system created with scheme 's3' could not be created\", ensure the required Python modules are installed on **both local and remote machines**. The required prerequisite modules can be found in the [Prefect documentation](https://docs.prefect.io/guides/deployment/storage-guide). For example, if using S3 for the remote storage:\n```bash\npip install s3fs\n```\n\n4. If you are seeing timeout or other connection errors, double check the address provided to the `RayTaskRunner`. The address should look similar to: `address='ray://\u003chead_node_ip_address\u003e:10001'`:\n```bash\nRayTaskRunner(address=\"ray://1.23.199.255:10001\")\n```\n\n## Specifying remote options\n\nThe `remote_options` context can be used to control the task’s remote options.\n\nFor example, we can set the number of CPUs and GPUs to use for the `process` task:\n\n```python\nfrom prefect import flow, task\nfrom prefect_ray.task_runners import RayTaskRunner\nfrom prefect_ray.context import remote_options\n\n@task\ndef process(x):\n    return x + 1\n\n\n@flow(task_runner=RayTaskRunner())\ndef my_flow():\n    # equivalent to setting @ray.remote(num_cpus=4, num_gpus=2)\n    with remote_options(num_cpus=4, num_gpus=2):\n        process.submit(42)\n```\n\n## Resources\n\nIf you encounter and bugs while using `prefect-ray`, feel free to open an issue in the [prefect-ray](https://github.com/PrefectHQ/prefect-ray) repository.\n\nIf you have any questions or issues while using `prefect-ray`, you can find help in the [Prefect Slack community](https://prefect.io/slack).\n\nFeel free to star or watch [`prefect-ray`](https://github.com/PrefectHQ/prefect-ray) for updates, too!\n\n## Development\n\n### Contributing\n\nIf you'd like to help contribute to fix an issue or add a feature to `prefect-ray`, please [propose changes through a pull request from a fork of the repository](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork).\n\n1. [Fork the repository](https://docs.github.com/en/get-started/quickstart/fork-a-repo#forking-a-repository)\n2. [Clone the forked repository](https://docs.github.com/en/get-started/quickstart/fork-a-repo#cloning-your-forked-repository)\n3. Install the repository and its dependencies:\n```\n pip install -e \".[dev]\"\n\n```\n4. Make desired changes\n5. Add tests\n6. Insert an entry to [CHANGELOG.md](https://github.com/PrefectHQ/prefect-ray/blob/main/CHANGELOG.md)\n7. Install `pre-commit` to perform quality checks prior to commit:\n```\n pre-commit install\n ```\n8. `git commit`, `git push`, and create a pull request\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPrefectHQ%2Fprefect-ray","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FPrefectHQ%2Fprefect-ray","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPrefectHQ%2Fprefect-ray/lists"}