{"id":19402671,"url":"https://github.com/cscfi/pytorch-ddp-examples","last_synced_at":"2025-04-24T07:32:19.325Z","repository":{"id":65299389,"uuid":"405058668","full_name":"CSCfi/pytorch-ddp-examples","owner":"CSCfi","description":null,"archived":false,"fork":false,"pushed_at":"2024-06-13T09:23:09.000Z","size":67,"stargazers_count":28,"open_issues_count":0,"forks_count":5,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-06-13T12:22:50.077Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CSCfi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-09-10T11:34:38.000Z","updated_at":"2024-06-13T09:23:13.000Z","dependencies_parsed_at":"2023-01-17T18:10:13.979Z","dependency_job_id":"3652e23f-0931-45a0-9d30-e3ea67e35de1","html_url":"https://github.com/CSCfi/pytorch-ddp-examples","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2Fpytorch-ddp-examples","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2Fpytorch-ddp-examples/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2Fpytorch-ddp-examples/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2Fpytorch-ddp-examples/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CSCfi","download_url":"https://codeload.github.com/CSCfi/pytorch-ddp-examples/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223945280,"owners_count":17229604,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T11:25:03.927Z","updated_at":"2024-11-10T11:25:04.353Z","avatar_url":"https://github.com/CSCfi.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PyTorch multi-GPU and multi-node examples for CSC's supercomputers\n\n[PyTorch distributed][pytorch_dist] and in particular\n`DistributedDataParallel` (DDP), offers a nice way of running\nmulti-GPU and multi-node PyTorch jobs. Unfortunately, the PyTorch\ndocumentation has been a bit lacking in this area, and examples found\nonline can often be out-of-date.\n\nTo make usage of DDP on CSC's Supercomputers easier, we have created a\nset of examples on how to run simple DDP jobs on the cluster. Included\nare also examples with other frameworks, such as [PyTorch\nLightning][lightning] and [DeepSpeed][deepspeed].\n\nAll examples train a simple CNN on MNIST. Scripts have been provided\nfor the Puhti supercomputer, but can be used on other systems with\nminor modifications.\n\nFor larger examples, see also our [Machine learning benchmarks\nrepository](https://github.com/mvsjober/ml-benchmarks).\n\nFinally, you might also be interested in [CSC's machine learning\nguide](https://docs.csc.fi/support/tutorials/ml-guide/) and in\nparticular the section on [Multi-GPU and multi-node\nmachinelearing](https://docs.csc.fi/support/tutorials/ml-multi/).\n\n\n## Multi-GPU, single-node\n\nThe simplest case is using all four GPUs on a single node on Puhti.\n\n```bash\nsbatch run-ddp-gpu4.sh\n```\n\n## Multi-GPU, multi-node\n\nExample using two nodes, four GPUs on each giving a total of 8 GPUs (again, on Puhti):\n\n```bash\nsbatch run-ddp-gpu8.sh\n```\n\n\n## PyTorch Lightning examples\n\nMulti-GPU and multi-node jobs are even easier with [PyTorch\nLightning][lightning]. The [official PyTorch Lightning now has a\nrelatively good Slurm\ndocumentation](https://lightning.ai/docs/pytorch/stable/clouds/cluster_advanced.html?highlight=slurm),\nalthough it has to be modified a bit for Puhti.\n\nFour GPUs on single node on Puhti:\n\n```bash\nsbatch run-lightning-gpu4.sh\n```\n\nTwo nodes, 8 GPUs in total on Puhti:\n\n```bash\nsbatch run-lightning-gpu8.sh\n```\n\n## DeepSpeed examples\n\n[DeepSpeed][deepspeed] should work on Puhti and Mahti with the\n[PyTorch module](https://docs.csc.fi/apps/pytorch/) (from version 1.10\nonwards).\n\nSingle-node with four GPUs (Puhti):\n\n```bash\nsbatch run-deepspeed-gpu4.sh\n```\n\nHere we are using Slurm to launch a single process which uses DeepSpeed's\nlauncher to launch four processes (one for each GPU).\n\nTwo nodes, 8 GPUs in total (Puhti):\n\n```bash\nsbatch run-deepspeed-gpu8.sh\n```\n\nNote that we are using Slurm's `srun` to launch four processess on each node\n(one per GPU), and instead of DeepSpeed's launcher we are relying on MPI to\nprovide it the information it needs to communicate between all the processes.\n\n\n[pytorch_dist]: https://pytorch.org/tutorials/beginner/dist_overview.html\n[ddp]: https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html\n[lightning]: https://www.pytorchlightning.ai/\n[deepspeed]: https://www.deepspeed.ai/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcscfi%2Fpytorch-ddp-examples","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcscfi%2Fpytorch-ddp-examples","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcscfi%2Fpytorch-ddp-examples/lists"}