{"id":15658986,"url":"https://github.com/graykode/horovod-ansible","last_synced_at":"2025-05-05T17:05:02.079Z","repository":{"id":110173057,"uuid":"195538704","full_name":"graykode/horovod-ansible","owner":"graykode","description":"Create Horovod cluster easily using Ansible","archived":false,"fork":false,"pushed_at":"2019-07-08T12:56:41.000Z","size":222,"stargazers_count":22,"open_issues_count":0,"forks_count":5,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-30T22:32:20.899Z","etag":null,"topics":["ansible","deeplearning","distributed-training","horovod","openmpi","pytorch","tensorflow","terraform"],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/graykode.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-07-06T12:47:07.000Z","updated_at":"2024-01-04T16:35:29.000Z","dependencies_parsed_at":"2023-04-13T21:32:18.056Z","dependency_job_id":null,"html_url":"https://github.com/graykode/horovod-ansible","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/graykode%2Fhorovod-ansible","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/graykode%2Fhorovod-ansible/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/graykode%2Fhorovod-ansible/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/graykode%2Fhorovod-ansible/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/graykode","download_url":"https://codeload.github.com/graykode/horovod-ansible/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252542061,"owners_count":21764907,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ansible","deeplearning","distributed-training","horovod","openmpi","pytorch","tensorflow","terraform"],"created_at":"2024-10-03T13:14:35.859Z","updated_at":"2025-05-05T17:05:02.033Z","avatar_url":"https://github.com/graykode.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"## horovod-ansible\n\n\u003cp align=\"center\"\u003e\u003cimg width=\"180\" src=\"https://user-images.githubusercontent.com/16640218/34506318-84d0c06c-efe0-11e7-8831-0425772ed8f2.png\" /\u003e  \u003cimg width=\"120\" src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/2/24/Ansible_logo.svg/512px-Ansible_logo.svg.png\" /\u003e\u003c/p\u003e\n\n[Horovod](https://github.com/horovod/horovod) is a distributed training framework for TensorFlow, Keras, PyTorch, and MXNet. [Ansible](https://github.com/ansible/ansible) is a radically simple IT automation system. We can easily install the horovod on all server through its automatic setup on **AWS or On-premise**\n\n##### Before Start\n\n- All On-premise nodes should be ubuntu\u003e=16.04. **I assumed that all nodes were equipped with Ansible(On-premise)**\n- Until now, only the examples of tensorflow and pyrotorch can used(Not MXNet, Caffe.. etc YET).\n- `AWS` Step : 0 - 1 -3\n- `On-perm` Step : 0 - 2 - 3\n\n\n\n## Usage\n\n### 0. docker setting(both AWS, On-premise)\nAll steps will be conducted under Docker container for beginners.\n\n```bash\n$ docker run -it --name horovod-ansible graykode/horovod-ansible:0.1 /bin/bash\n```\n\n\n\n### 1. AWS\n\nTo create horovod clustering enviorment, start provisioning with `Terraform` code. Change some option [`variables.tf`](https://github.com/graykode/horovod-ansible/blob/master/terraform/variables.tf) which you want. But you should not below `## DO NOT CHANGE BELOW`.\n\nIf I created EC2 with option `number_of_worker` 3, Total architecture is same with below picture.\n\n\u003cp align=\"center\"\u003e\u003cimg width=\"600\" src=\"https://raw.githubusercontent.com/graykode/horovod-ansible/master/images/horovod.png\" /\u003e \u003c/p\u003e\nExport your own AWS Access / Secret keys\n\n```bash\n$ export AWS_ACCESS_KEY_ID=\u003cYour Access Key in AWS\u003e\n$ export AWS_SECRET_ACCESS_KEY=\u003cYour Access Key in Secret\u003e\n```\n\nInitializing terraform and create private key to use.\n\n```bash\n$ cd terraform/ \u0026\u0026 ssh-keygen -t rsa -N \"\" -f horovod\n$ terraform init\n```\n\nprovisioning all resource EC2, VPC(gateway, router, subnet, etc..) \n\n```bash\n$ terraform apply\n```\n\nThen, you can get output :\n\n```bash\nApply complete! Resources: 12 added, 0 changed, 0 destroyed.\n\nOutputs:\n\nhorovod_master_public_ip = \u003cmaster's public IP\u003e\nhorovod_workers_public_ip = \u003cworker0's public IP\u003e,\u003cworker1's public IP\u003e\n```\n\n\n\n### 2. On-premise\n\nAs I said above, assume that all nodes are 'ansible' and network setup is finished. If you want to see install Ansible, Please read [Ansible Install Guide](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html) on document.\n\n\n\n### 3. Setup Horovod Configure using Ansible(both AWS, On-premise)\n\nInstall `ansible` and `jinja2` using pip.\n```bash\n$ ../ansible \u0026\u0026 pip install -r requirements.txt\n```\n\n\n\nSet `inventory.ini` in Ansible Folder.\n\n```ini\nmaster ansible_host=\u003cmaster's public IP\u003e\nworker0 ansible_host=\u003cworker0's public IP\u003e\nworker1 ansible_host=\u003cworker1's public IP\u003e\n....\nworker[n] ansible_host=\n\n[all]\nmaster\nworker0\nworker1\n...\nworker[n]\n\n[master-servers]\nmaster\n\n[worker-servers]\nworker0\nworker1\n...\nworker[n]\n```\n\n\n\nPing Test to all nodes!\n\n```bash\n$ chmod +x ping.sh \u0026\u0026 ./ping.sh\n```\n\nNow ssh configure to using Open MPI, Download Open MPI and build\n\n```bash\n$ chmod +x playbook.sh \u0026\u0026 ./playbook.sh\n```\n\nTest all nodes of mpi that it is fine in master node.\n\n```bash\n$ chmod +x test.sh \u0026\u0026 ./test.sh\n\n# go to master node.\nubuntu@master:~$ mpirun -np 3 -mca btl sm,self,tcp -host master,worker0,worker1 ./test\nProcessor name: master\nmaster (0/3)\nProcessor name: worker0\nslave  (1/3)\nProcessor name: worker1\nslave  (2/3)\n```\n\n\n\n### 4. Install DeepLearning Framework which you want and Horovod(both AWS, On-premise)\n\nI'd like you to change this part fluidly.\n\n- Install Tensorflow on CPU, Horovod and Run Distributed \n\n  ```bash\n  $ chmod +x tensorflow.sh \u0026\u0026 ./tensorflow.sh\n  \n  # go to master node.\n  ubuntu@master:~$ horovodrun -np 3 -H master,worker0,worker1 python3 tensorflow-train.py\n  ```\n\n- Install Pytorch on CPU, Horovod and Run Distributed \n\n  ```bash\n  $ chmod +x pytorch.sh \u0026\u0026 ./pytorch.sh\n  \n  # go to master node.\n  ubuntu@master:~$ horovodrun -np 3 -H master,worker0,worker1 python3 pytorch-train.py\n  ```\n\n- Issue Note : If you want to change framework after install horovod, you reinstall horovod with `HOROVOD_WITH_*` option, '*' is just framework name. please see [horovod issue](https://github.com/horovod/horovod/issues/314). But in my Ansible Script, I 'm not add it yet.\n\n\n\n## Author\n\n- Tae Hwan Jung(Jeff Jung) @graykode\n- Author Email : [nlkey2022@gmail.com](mailto:nlkey2022@gmail.com)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgraykode%2Fhorovod-ansible","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgraykode%2Fhorovod-ansible","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgraykode%2Fhorovod-ansible/lists"}