{"id":13720103,"url":"https://github.com/iterative/tutorial-mnist-dvc-ray","last_synced_at":"2025-06-18T02:38:52.086Z","repository":{"id":226645777,"uuid":"769262336","full_name":"iterative/tutorial-mnist-dvc-ray","owner":"iterative","description":null,"archived":false,"fork":false,"pushed_at":"2024-03-12T15:37:41.000Z","size":1133,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-06-13T23:46:06.041Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/iterative.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-08T17:18:08.000Z","updated_at":"2024-04-28T14:56:00.000Z","dependencies_parsed_at":"2024-11-14T09:31:32.878Z","dependency_job_id":"e37c5fb9-d0ef-432a-b3ef-703f0345f612","html_url":"https://github.com/iterative/tutorial-mnist-dvc-ray","commit_stats":null,"previous_names":["iterative/tutorial-mnist-dvc-ray"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/iterative/tutorial-mnist-dvc-ray","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iterative%2Ftutorial-mnist-dvc-ray","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iterative%2Ftutorial-mnist-dvc-ray/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iterative%2Ftutorial-mnist-dvc-ray/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iterative%2Ftutorial-mnist-dvc-ray/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/iterative","download_url":"https://codeload.github.com/iterative/tutorial-mnist-dvc-ray/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iterative%2Ftutorial-mnist-dvc-ray/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260476171,"owners_count":23014978,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T01:00:59.809Z","updated_at":"2025-06-18T02:38:47.064Z","avatar_url":"https://github.com/iterative.png","language":"Python","funding_links":[],"categories":["Tutorials"],"sub_categories":["Iterative"],"readme":"# 🎓 Tutorial: Automated Distributed ML Pipelines with DVC and Ray\n\nThis tutorial will guide users through creating automated, scalable, and distributed ML pipelines using DVC (Data Version Control) integrated with Ray. \n\n- Run Distributed ML Pipeline with DVC [DVC](https://dvc.org/). \n- Design distributed ML pipeliens with Ray [Ray](https://www.ray.io/). \n- Introduce [Ray Tune](https://docs.ray.io/en/latest/tune/index.html) for hyperparameter optimization\n- Run distributed ML experiment with DVC and Ray on AWS\n\n![DVC + Ray](src/static/preview.png \"DVC + Ray\")\n\n## Branch Structure\n\n| Branch       | Details                                 | Set up instructions                                           |\n|--------------|-----------------------------------------|---------------------------------------------------------------|\n| `main`       | For local Ray cluster setups.           | Follow the [Run DVC pipeline on Ray Cluster (local)](#run-dvc-pipeline-on-ray-cluster-local) section. |\n| `cloud`      | For cloud-based Ray cluster on AWS.     | Follow the [Run DVC pipeline on Ray Cluster (AWS)](#run-dvc-pipeline-on-ray-cluster-aws) section.   |\n\n## Install\n\n```bash\npython -m venv .venv\nsource .venv/bin/activate\npip install -r requirements.txt\n```\n\n\n## Run DVC pipeline on Ray Cluster (local)\n\n### 1 - Start a Ray Cluster (local)\n\nLocally: cpu, single-node\n\n```bash\nexport RAY_ENABLE_WINDOWS_OR_OSX_CLUSTER=1\nray start --head\n```\n\nTo test running cluster you may run a test script\n\n```bash\n# Run a Python script with ray.init() \npython src/test_scripts/hello_cluster.py\n\n# Submit a script as a Job\nray job submit -- python src/test_scripts/hello_cluster.py\n```\n\n### 2 - Run DVC pipeline\n\n```bash\nexport PYTHONPATH=$PWD\ndvc exp run\n```\n\n### 3 - Stop Cluster\n\nRemeber to stop remote cluster after work is done. \n\n```bash\nray stop\n```\n\n## Run DVC pipeline on Ray Cluster (AWS)\n\n### 1 - Start a Ray Cluster on AWS\n\n```bash\nray up cluster.yaml  \n```\n\n### 2 - Open Ray Dashboard\n\n```bash\nray dashboard cluster.yaml\n```\n\n### 3 - Work with Ray Cluster\n\nThere are few options to interact with the Ray Cluster on AWS:\n\n- Submit jobs (Ray scripts)\n- Attach to an interactive screen session on the cluster\n- Execute shell commands on the cluster\n\n\n#### 3.1 - Submit ML trainig job to Ray Cluster\n\n```bash\nexport RAY_ADDRESS='http://localhost:8265'\nray job submit --working-dir $PWD -- python src/test_scripts/mnist.py\n```\n\n#### 3.2 - Attach to an interactive screen session on the cluster (via SSH)\n\nIf you would like to run an application interactively and see the output in real time (for example, during development or debugging), you can кun your script directly on a cluster node (e.g. after SSHing into the node using `ray attach`\n\n```bash\nray attach cluster.yaml\n```\n\n#### 3.3 - Execute shell commands on the cluster\n\nAlso, you may want to execute `shell` commands on a cluster\n\n```bash\nray exec cluster.yaml 'echo \"hello world\"'\n```\n\n### 4 - Run DVC experiment on Ray Cluster\n\n#### 4.1 Run from the Head Node (option 1)\n\nConnect to the session on the cluster\n\n```bash\nray attach cluster.yaml\n```\n\nRun ML pipeline with DVC on the cluster\n\n```bash\n# Navigate to the repo \u0026 run \ncd tutorial-mnist-dvc-ray\n\n# Run DVC experiment\nexport PYTHONPATH=/home/ray/tutorial-mnist-dvc-ray\ndvc exp run\n```\n\n#### 4.2 - Run from laptop (option 2)\n\nExecute `dvc exp run` command on the cluster\n\n```bash\nray exec cluster.yaml \"cd tutorial-mnist-dvc-ray \u0026\u0026 \\\n    export PYTHONPATH=/home/ray/tutorial-mnist-dvc-ray \u0026\u0026 \\\n    \n    dvc exp run\"\n```\n\nDownload (sync) the repository to results\n\n```bash\nray rsync_down cluster.yaml '/home/ray/tutorial-mnist-dvc-ray/' $PWD\n```\n\nCommit results if needed.\n\n### 6 - Stop Cluster\n\nRemember to stop remote cluster after work is done. Save money and the planet!\n\n```bash\nray down cluster.yaml \n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiterative%2Ftutorial-mnist-dvc-ray","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fiterative%2Ftutorial-mnist-dvc-ray","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiterative%2Ftutorial-mnist-dvc-ray/lists"}