{"id":13604094,"url":"https://github.com/MachineLearningSystem/Hetu","last_synced_at":"2025-04-11T22:32:38.407Z","repository":{"id":185461796,"uuid":"573001496","full_name":"MachineLearningSystem/Hetu","owner":"MachineLearningSystem","description":"A high-performance distributed deep learning system targeting large-scale and automated distributed training.","archived":false,"fork":true,"pushed_at":"2022-11-30T14:56:15.000Z","size":5015,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2024-11-07T08:42:04.974Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"PKU-DAIR/Hetu","license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MachineLearningSystem.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2022-12-01T13:45:33.000Z","updated_at":"2022-12-01T06:38:23.000Z","dependencies_parsed_at":"2023-08-02T03:17:07.225Z","dependency_job_id":null,"html_url":"https://github.com/MachineLearningSystem/Hetu","commit_stats":null,"previous_names":["machinelearningsystem/hetu"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MachineLearningSystem%2FHetu","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MachineLearningSystem%2FHetu/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MachineLearningSystem%2FHetu/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MachineLearningSystem%2FHetu/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MachineLearningSystem","download_url":"https://codeload.github.com/MachineLearningSystem/Hetu/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248489894,"owners_count":21112658,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T19:00:39.837Z","updated_at":"2025-04-11T22:32:33.390Z","avatar_url":"https://github.com/MachineLearningSystem.png","language":null,"readme":"\u003cdiv align=center\u003e\n\u003cimg src=\"./img/hetu.png\" width=\"300\" /\u003e\n\u003c/div\u003e\n\n# HETU\n\n\u003c!--- [![license](https://img.shields.io/github/license/apache/zookeeper?color=282661)](LICENSE) ---\u003e\n\n[Documentation](https://hetu-doc.readthedocs.io) | [Examples](https://hetu-doc.readthedocs.io/en/latest/Overview/performance.html)\n\nHetu is a high-performance distributed deep learning system targeting trillions of parameters DL model training, developed by \u003ca href=\"https://cuibinpku.github.io\" target=\"_blank\" rel=\"nofollow\"\u003eDAIR Lab\u003c/a\u003e at Peking University. It takes account of both high availability in industry and innovation in academia, which has a number of advanced characteristics:\n\n- Applicability. DL model definition with standard dataflow graph; many basic CPU and GPU operators; efficient implementation of more than plenty of DL models and at least popular 10 ML algorithms.\n\n- Efficiency. Achieve at least 30% speedup compared to TensorFlow on DNN, CNN, RNN benchmarks.\n\n- Flexibility. Supporting various parallel training protocols and distributed communication architectures, such as Data/Model/Pipeline parallel; Parameter server \u0026 AllReduce.\n\n- Scalability. Deployment on more than 100 computation nodes; Training giant models with trillions of model parameters, e.g., Criteo Kaggle, Open Graph Benchmark\n\n- Agility. Automatically ML pipeline: feature engineering, model selection, hyperparameter search.\n\nWe welcome everyone interested in machine learning or graph computing to contribute codes, create issues or pull requests. Please refer to [Contribution Guide](CONTRIBUTING.md) for more details.\n\n## Key Features\n\n\u003cdiv align=center\u003e\n\u003cimg src=\"./img/features.png\" width=\"800\" /\u003e\n\u003c/div\u003e\n\n## Installation\n1. Clone the repository.\n\n2. Prepare the environment. We use Anaconda to manage packages. The following command create the conda environment to be used:`conda env create -f environment.yml`. Please prepare Cuda toolkit and CuDNN in advance.\n\n3. We use CMake to compile Hetu. Please copy the example configuration for compilation by `cp cmake/config.example.cmake cmake/config.cmake`. Users can modify the configuration file to enable/disable the compilation of each module. For advanced users (who not using the provided conda environment), the prerequisites for different modules in Hetu is listed in appendix.\n\n```bash\n# modify paths and configurations in cmake/config.cmake\n\n# generate Makefile\nmkdir build \u0026\u0026 cd build \u0026\u0026 cmake ..\n\n# compile\n# make all\nmake -j 8\n# make hetu, version is specified in cmake/config.cmake\nmake hetu -j 8\n# make allreduce module\nmake allreduce -j 8\n# make ps module\nmake ps -j 8\n# make geometric module\nmake geometric -j 8\n# make hetu-cache module\nmake hetu_cache -j 8\n```\n\n4. Prepare environment for running. Edit the hetu.exp file and set the environment path for python and the path for executable mpirun if necessary (for advanced users not using the provided conda environment). Then execute the command `source hetu.exp` .\n\n\n\n## Basic Usage\n\nTrain logistic regression on gpu:\n\n```bash\nbash examples/cnn/scripts/hetu_1gpu.sh logreg MNIST\n```\n\nTrain a 3-layer mlp on gpu:\n\n```bash\nbash examples/cnn/scripts/hetu_1gpu.sh mlp CIFAR10\n```\n\nTrain a 3-layer cnn with gpu:\n\n```bash\nbash examples/cnn/scripts/hetu_1gpu.sh cnn_3_layers MNIST\n```\n\nTrain a 3-layer mlp with allreduce on 8 gpus (use mpirun):\n\n```bash\nbash examples/cnn/scripts/hetu_8gpu.sh mlp CIFAR10\n```\n\nTrain a 3-layer mlp with PS on 1 server and 2 workers:\n\n```bash\n# in the script we launch the scheduler and server, and two workers\nbash examples/cnn/scripts/hetu_2gpu_ps.sh mlp CIFAR10\n```\n\n## More Examples\nPlease refer to examples directory, which contains CNN, NLP, CTR, MoE, GNN training scripts. If you want more, please join our community and contact with us!\n\n## Community\n* Email: xupeng.miao@pku.edu.cn\n* Click [here](https://join.slack.com/t/hetu-ai/shared_invite/zt-1kpanxc83-9YndNPZYDH9orbR6MeIMbg) to join our Slack community.\n* Hetu homepage: https://hetu-doc.readthedocs.io\n* [Committers \u0026 Contributors](COMMITTERS.md)\n* [Contributing to Hetu](CONTRIBUTING.md)\n* [Development plan](https://hetu-doc.readthedocs.io/en/latest/plan.html)\n\n\n## Enterprise Users\n\nIf you are enterprise users and find Hetu is useful in your work, please let us know, and we are glad to add your company logo here.\n\n* Tencent Inc.\n\n\u003cimg src=\"./img/tencent.png\" width = \"200\"/\u003e\n\n* Alibaba Cloud.\n\n\u003cimg src=\"./img/alibabacloud.png\" width = \"200\"/\u003e\n\n* Kuaishou Tech.\n\n\u003cimg src=\"./img/kuaishou.png\" width = \"200\"/\u003e\n\n\n## License\n\nThe entire codebase is under [license](LICENSE)\n\n## Papers\n  1. Xupeng Miao, Hailin Zhang, Yining Shi, Xiaonan Nie, Zhi Yang, Yangyu Tao, Bin Cui. [HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Framework](https://arxiv.org/abs/2112.07221). VLDB 2022 (Best Scalable Data Science Paper), ChinaSys 2021 Winter\n  2. Xupeng Miao, Yining Shi, Hailin Zhang, Xin Zhang, Xiaonan Nie, Zhi Yang, Bin Cui. [HET-GMP: a Graph-based System Approach to Scaling Large Embedding Model Training](https://dl.acm.org/doi/10.1145/3514221.3517902). SIGMOD 2022, ChinaSys 2022\n  3. Xupeng Miao, Yujie Wang, Youhe Jiang,  Chunan Shi and Xiaonan Nie, Hailin Zhang, Bin Cui. [Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism](https://arxiv.org/abs/2211.13878). VLDB 2023\n  4. Xupeng Miao, Xiaonan Nie, Yingxia Shao, Zhi Yang, Jiawei Jiang, Lingxiao Ma, Bin Cui. [Heterogeneity-Aware Distributed Machine Learning Training via Partial Reduce](https://doi.org/10.1145/3448016.3452773). SIGMOD 2021\n  5. Xupeng Miao, Lingxiao Ma, Zhi Yang, Yingxia Shao, Bin Cui, Lele Yu, Jiawei Jiang. [CuWide: Towards Efficient Flow-based Training for Sparse Wide Models on GPUs](https://ieeexplore.ieee.org/document/9261124). TKDE 2021, ICDE 2021.\n  6. Xiaonan Nie, Xupeng Miao, Zhi Yang, Bin Cui. [TSplit: Fine-grained GPU Memory Management for Efficient DNN Training via Tensor Splitting](https://ieeexplore.ieee.org/document/9835178). ICDE 2022.\n  7. Sicong Dong, Xupeng Miao, Pengkai Liu, Xin Wang, Bin Cui, Jianxin Li. [HET-KG: Communication-Efficient Knowledge Graph Embedding Training via Hotness-Aware Cache](https://ieeexplore.ieee.org/document/9835364). ICDE 2022.\n  8. Xupeng Miao, Yujie Wang, Jia Shen, Yingxia Shao, Bin Cui. Graph Neural Network Training Acceleration over Multi-GPUs. Journal of Software (Chinese).\n  9. Xiaonan Nie, Shijie Cao, Xupeng Miao, Lingxiao Ma, Jilong Xue, Youshan Miao, Zichao Yang, Zhi Yang, Bin Cui. [EvoMoE: An Evolutional Mixture-of-Experts Training Framework via Dense-To-Sparse Gate](https://arxiv.org/abs/2112.14397). arXiv 2021.\n  10. coming soon\n\n## Cite\n\nIf you use Hetu in a scientific publication, we would appreciate citations to the following paper:\n```\n @article{miao2021het,\n   title={HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Framework},\n   author={Miao, Xupeng and Zhang, Hailin and Shi, Yining and Nie, Xiaonan and Yang, Zhi and Tao, Yangyu and Cui, Bin},\n   journal = {Proc. {VLDB} Endow.},\n   volume = {15},\n   number = {2},\n   pages = {312--320},\n   year = {2022},\n   publisher = {VLDB Endowment}\n }\n```\n\n## Acknowledgements\n\nWe learned and borrowed insights from a few open source projects including [TinyFlow](https://github.com/tqchen/tinyflow), [autodist](https://github.com/petuum/autodist), [tf.distribute](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/python/distribute), [FlexFlow](https://github.com/flexflow/FlexFlow) and [Angel](https://github.com/Angel-ML/angel).\n\n## Appendix\nThe prerequisites for different modules in Hetu is listed as follows:\n  ```\n  \"*\" means you should prepare by yourself, while others support auto-download\n  \n  Hetu: OpenMP(*), CMake(*)\n  Hetu (version mkl): MKL 1.6.1\n  Hetu (version gpu): CUDA 10.1(*), CUDNN 7.5(*), CUB 1.12.1(*)\n  Hetu (version all): both\n\n  Hetu-AllReduce: MPI 3.1, NCCL 2.8(*), this module needs GPU version\n\n  Hetu-PS: Protobuf(*), ZeroMQ 4.3.2\n\n  Hetu-Geometric: Pybind11(*), Metis(*)\n\n  Hetu-Cache: Pybind11(*), this module needs PS module\n\n  ##################################################################\n  Tips for preparing the prerequisites\n  \n  Preparing CUDA, CUDNN, CUB, NCCL(NCCl is already in conda environment):\n  1. download from https://developer.nvidia.com \n  2. download CUB from https://github.com/NVIDIA/cub/releases/tag/1.12.1\n  3. install\n  4. modify paths in cmake/config.cmake if necessary\n  \n  Preparing OpenMP:\n  Your just need to ensure your compiler support openmp.\n\n  Preparing CMake, Protobuf, Pybind11, Metis:\n  Install by anaconda: \n  conda install cmake=3.18 libprotobuf pybind11=2.6.0 metis\n\n  Preparing OpenMPI (not necessary):\n  install by anaconda: `conda install -c conda-forge openmpi=4.0.3`\n  or\n  1. download from https://download.open-mpi.org/release/open-mpi/v4.0/openmpi-4.0.3.tar.gz\n  2. build openmpi by `./configure /path/to/build \u0026\u0026 make -j8 \u0026\u0026 make install`\n  3. modify MPI_HOME to /path/to/build in cmake/config.cmake\n\n  Preparing MKL (not necessary):\n  install by anaconda: `conda install -c conda-forge onednn`\n  or\n  1. download from https://github.com/intel/mkl-dnn/archive/v1.6.1.tar.gz\n  2. build mkl by `mkdir /path/to/build \u0026\u0026 cd /path/to/build \u0026\u0026 cmake /path/to/root \u0026\u0026 make -j8` \n  3. modify MKL_ROOT to /path/to/root and MKL_BUILD to /path/to/build in cmake/config.cmake \n\n  Preparing ZeroMQ (not necessary):\n  install by anaconda: `conda install -c anaconda zeromq=4.3.2`\n  or\n  1. download from https://github.com/zeromq/libzmq/releases/download/v4.3.2/zeromq-4.3.2.zip\n  2. build zeromq by 'mkdir /path/to/build \u0026\u0026 cd /path/to/build \u0026\u0026 cmake /path/to/root \u0026\u0026 make -j8`\n  3. modify ZMQ_ROOT to /path/to/build in cmake/config.cmake\n  ```\n","funding_links":[],"categories":["Paper-Code"],"sub_categories":["Framework"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMachineLearningSystem%2FHetu","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FMachineLearningSystem%2FHetu","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMachineLearningSystem%2FHetu/lists"}