{"id":13604089,"url":"https://github.com/bytedance/byteps","last_synced_at":"2025-05-14T08:06:59.552Z","repository":{"id":37660165,"uuid":"193652854","full_name":"bytedance/byteps","owner":"bytedance","description":"A high performance and generic framework for distributed DNN training","archived":false,"fork":false,"pushed_at":"2023-10-03T18:02:27.000Z","size":21217,"stargazers_count":3675,"open_issues_count":113,"forks_count":491,"subscribers_count":82,"default_branch":"master","last_synced_at":"2025-04-11T02:51:56.562Z","etag":null,"topics":["deep-learning","distributed-training","keras","machine-learning","mxnet","pytorch","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bytedance.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.rst","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2019-06-25T07:00:13.000Z","updated_at":"2025-04-10T11:46:13.000Z","dependencies_parsed_at":"2022-07-11T23:33:45.110Z","dependency_job_id":"d624724f-0d9e-440e-96d4-dec8f6c9f519","html_url":"https://github.com/bytedance/byteps","commit_stats":{"total_commits":426,"total_committers":24,"mean_commits":17.75,"dds":0.5892018779342723,"last_synced_commit":"f5df52fe2794d4e3d2931a9dcfa5ca56151062a1"},"previous_names":[],"tags_count":20,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bytedance%2Fbyteps","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bytedance%2Fbyteps/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bytedance%2Fbyteps/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bytedance%2Fbyteps/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bytedance","download_url":"https://codeload.github.com/bytedance/byteps/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254101618,"owners_count":22014909,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","distributed-training","keras","machine-learning","mxnet","pytorch","tensorflow"],"created_at":"2024-08-01T19:00:39.704Z","updated_at":"2025-05-14T08:06:54.534Z","avatar_url":"https://github.com/bytedance.png","language":"Python","funding_links":[],"categories":["Paper-Code","\u003ca name=\"Tools\"\u003e\u003c/a\u003e13. Tools","Python","Deep Learning Framework","分布式机器学习","Frameworks"],"sub_categories":["Framework","13.4 Parallel and Distributed computing","Deployment \u0026 Distribution"],"readme":"# BytePS\n\n[![Build Status](https://travis-ci.org/bytedance/byteps.svg?branch=master)](https://travis-ci.org/bytedance/byteps)\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n![Pypi](https://img.shields.io/pypi/v/byteps.svg)\n\nBytePS is a high performance and general distributed training framework. It supports TensorFlow, Keras, PyTorch, and MXNet, and can run on either TCP or RDMA network.\n\nBytePS outperforms existing open-sourced distributed training frameworks by a large margin. For example, on BERT-large training, BytePS can achieve ~90% scaling efficiency with 256 GPUs (see below), which is much higher than [Horovod](https://github.com/horovod/horovod)+[NCCL](https://github.com/NVIDIA/nccl). In certain scenarios, BytePS can double the training speed compared with Horovod+NCCL.\n\n## News\n- [BytePS paper](https://www.usenix.org/conference/osdi20/presentation/jiang) has been accepted to OSDI'20. The code to reproduce the end-to-end evaluation is available [here](https://github.com/byteps/examples).\n- Support [gradient compression](https://github.com/bytedance/byteps/pull/225).\n- [v0.2.4](https://github.com/bytedance/byteps/tree/v0.2.4)\n    * Fix compatibility issue with tf2 + standalone keras\n    * Add support for tensorflow.keras\n    * Improve robustness of broadcast\n- [v0.2.3](https://github.com/bytedance/byteps/tree/v0.2.3)\n    * Add DistributedDataParallel module for PyTorch\n    * Fix the problem of different CPU tensor using the same name\n    * Add skip_synchronize api for PyTorch\n    * Add the option for lazy/non-lazy init\n- [v0.2.0](https://github.com/bytedance/byteps/tree/v0.2)\n    * Largely improve RDMA performance by enforcing page aligned memory.\n    * Add IPC support for RDMA. Now support colocating servers and workers without sacrificing much performance.\n    * Fix a hanging bug in BytePS server.\n    * Fix RDMA-related segmentation fault problem during fork() (e.g., used by PyTorch data loader).\n    * New feature: Enable mixing use of colocate and non-colocate servers, along with a smart tensor allocation strategy.\n    * New feature: Add ``bpslaunch`` as the command to launch tasks.\n    * Add support for pip install: ``pip3 install byteps``\n\n## Performance\n\nWe show our experiment on BERT-large training, which is based on GluonNLP toolkit. The model uses mixed precision.\n\nWe use Tesla V100 32GB GPUs and set batch size equal to 64 per GPU. Each machine has 8 V100 GPUs (32GB memory) with NVLink-enabled. Machines are inter-connected with 100 Gbps RDMA network. This is the same hardware setup you can get on [AWS](https://aws.amazon.com/about-aws/whats-new/2018/12/introducing-amazon-ec2-p3dn-instances-our-most-powerful-gpu-instance-yet/).\n\nBytePS achieves ~90% scaling efficiency for BERT-large with 256 GPUs. The code is available [here](https://github.com/ymjiang/gluon-nlp/tree/bert-byteps/scripts/bert). As a comparison, Horovod+NCCL has only ~70% scaling efficiency even after expert parameter tunning.\n\n![BERT-Large](https://user-images.githubusercontent.com/13852819/69874496-1ca43600-12f6-11ea-997b-b023e4c93360.png)\n\n\nWith slower network, BytePS offers even more performance advantages -- up to 2x of Horovod+NCCL. You can find more evaluation results at [performance.md](docs/performance.md).\n\n## Goodbye MPI, Hello Cloud\n\nHow can BytePS outperform Horovod by so much? One of the main reasons is that BytePS is designed for cloud and shared clusters, and throws away MPI.\n\nMPI was born in the HPC world and is good for a cluster built with homogeneous hardware and for running a single job. However, cloud (or in-house shared clusters) is different.\n\nThis leads us to rethink the best communication strategy, as explained in [here](docs/rationale.md). In short, BytePS only uses NCCL inside a machine, while re-implements the inter-machine communication.\n\nBytePS also incorporates many acceleration techniques such as hierarchical strategy, pipelining, tensor partitioning, NUMA-aware local communication, priority-based scheduling, etc.\n\n## Quick Start\n\nWe provide a [step-by-step tutorial](docs/step-by-step-tutorial.md) for you to run benchmark training tasks. The simplest way to start is to use our [docker images](docker). Refer to [Documentations](docs) for how to [launch distributed jobs](docs/running.md) and more [detailed configurations](docs/env.md). After you can start BytePS, read [best practice](docs/best-practice.md) to get the best performance.\n\nBelow, we explain how to install BytePS by yourself. There are two options.\n\n### Install by pip\n\n```\npip3 install byteps\n```\n\n### Build from source code\n\nYou can try out the latest features by directly installing from master branch:\n\n```\ngit clone --recursive https://github.com/bytedance/byteps\ncd byteps\npython3 setup.py install\n```\n\nNotes for above two options:\n- BytePS assumes that you have already installed one or more of the following frameworks: TensorFlow / PyTorch / MXNet.\n- BytePS depends on CUDA and NCCL. You should specify the NCCL path with `export BYTEPS_NCCL_HOME=/path/to/nccl`. By default it points to `/usr/local/nccl`.\n- The installation requires gcc\u003e=4.9. If you are working on CentOS/Redhat and have gcc\u003c4.9, you can try `yum install devtoolset-7` before everything else. In general, we recommend using gcc 4.9 for best compatibility ([how to pin gcc](https://github.com/bytedance/byteps/blob/3fba75def0d81c1d3225f8f397cc985200f57de7/docker/Dockerfile.mxnet#L72-L80)).\n- RDMA support: During setup, the script will automatically detect the RDMA header file. If you want to use RDMA, make sure your RDMA environment has been properly installed and tested before install ([install on Ubuntu-18.04](https://github.com/bytedance/byteps/blob/3fba75def0d81c1d3225f8f397cc985200f57de7/docker/Dockerfile.mxnet#L29-L33)).\n\n## Examples\n\nBasic examples are provided under the [example](example) folder. \n\nTo reproduce the end-to-end evaluation in our OSDI'20 paper, find the code at this [repo](https://github.com/byteps/examples).\n\n## Use BytePS in Your Code\n\nThough being totally different at its core, BytePS is highly compatible with Horovod interfaces (Thank you, Horovod community!). We chose Horovod interfaces in order to minimize your efforts for testing BytePS.\n\nIf your tasks only rely on Horovod's allreduce and broadcast, you should be able to switch to BytePS in 1 minute. Simply replace `import horovod.tensorflow as hvd` by `import byteps.tensorflow as bps`, and then replace all `hvd` in your code by `bps`. If your code invokes `hvd.allreduce` directly, you should also replace it by `bps.push_pull`.\n\nMany of our examples were copied from Horovod and modified in this way. For instance, compare the MNIST example for [BytePS](https://github.com/bytedance/byteps/blob/master/example/tensorflow/tensorflow_mnist.py) and [Horovod](https://github.com/horovod/horovod/blob/master/examples/tensorflow_mnist.py).\n\nBytePS also supports other native APIs, e.g., PyTorch Distributed Data Parallel and TensorFlow Mirrored Strategy. See [DistributedDataParallel.md](docs/DistributedDataParallel.md) and [MirroredStrategy.md](docs/MirroredStrategy.md) for usage.\n\n## Limitations and Future Plans\nBytePS does not support pure CPU training for now. One reason is that the [cheap PS assumption](docs/rationale.md) of BytePS do not hold for CPU training. Consequently, you need CUDA and NCCL to build and run BytePS.\n\nWe would like to have below features, and there is no fundamental difficulty to implement them in BytePS architecture. However, they are not implemented yet:\n* Sparse model training\n* Fault-tolerance\n* Straggler-mitigation\n\n## Publications\n\n1. [OSDI'20] \"[A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters](https://www.usenix.org/conference/osdi20/presentation/jiang)\". Yimin Jiang, Yibo Zhu, Chang Lan, Bairen Yi, Yong Cui, Chuanxiong Guo. \n\n2. [SOSP'19] \"[A Generic Communication Scheduler for Distributed DNN Training Acceleration](https://i.cs.hku.hk/~cwu/papers/yhpeng-sosp19.pdf)\". Yanghua Peng, Yibo Zhu, Yangrui Chen, Yixin Bao, Bairen Yi, Chang Lan, Chuan Wu, Chuanxiong Guo. (Code is at [bytescheduler branch](https://github.com/bytedance/byteps/tree/bytescheduler/bytescheduler))\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbytedance%2Fbyteps","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbytedance%2Fbyteps","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbytedance%2Fbyteps/lists"}