{"id":13676034,"url":"https://github.com/DeepRec-AI/HybridBackend","last_synced_at":"2025-04-29T03:30:29.370Z","repository":{"id":37670824,"uuid":"433715906","full_name":"DeepRec-AI/HybridBackend","owner":"DeepRec-AI","description":"A high-performance framework for training wide-and-deep recommender systems on heterogeneous cluster","archived":false,"fork":false,"pushed_at":"2024-04-20T05:31:29.000Z","size":3032,"stargazers_count":157,"open_issues_count":13,"forks_count":31,"subscribers_count":15,"default_branch":"main","last_synced_at":"2025-04-26T03:39:23.030Z","etag":null,"topics":["deep-learning","gpu","hybrid-parallelism","parquet","recommender-system"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DeepRec-AI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":"ROADMAP.md","authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-12-01T06:51:24.000Z","updated_at":"2025-03-17T03:49:45.000Z","dependencies_parsed_at":"2023-02-18T22:30:35.646Z","dependency_job_id":"af5f8d71-c2f3-41f9-91c9-4d8921fad7b4","html_url":"https://github.com/DeepRec-AI/HybridBackend","commit_stats":null,"previous_names":["alibaba/hybridbackend"],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DeepRec-AI%2FHybridBackend","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DeepRec-AI%2FHybridBackend/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DeepRec-AI%2FHybridBackend/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DeepRec-AI%2FHybridBackend/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DeepRec-AI","download_url":"https://codeload.github.com/DeepRec-AI/HybridBackend/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251426690,"owners_count":21587633,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","gpu","hybrid-parallelism","parquet","recommender-system"],"created_at":"2024-08-02T13:00:16.669Z","updated_at":"2025-04-29T03:30:29.308Z","avatar_url":"https://github.com/DeepRec-AI.png","language":"C++","readme":"# HybridBackend\n\n[![cibuild](https://github.com/alibaba/HybridBackend/actions/workflows/cibuild.yaml/badge.svg?branch=main\u0026event=push)](https://github.com/alibaba/HybridBackend/actions/workflows/cibuild.yaml)\n[![readthedocs](https://readthedocs.org/projects/hybridbackend/badge/?version=latest)](https://hybridbackend.readthedocs.io/en/latest/?badge=latest)\n[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](http://makeapullrequest.com)\n[![license](https://img.shields.io/badge/License-Apache%202.0-brightgreen.svg)](https://opensource.org/licenses/Apache-2.0)\n\nHybridBackend is a high-performance framework for training wide-and-deep\nrecommender systems on heterogeneous cluster.\n\n## Features\n\n- Memory-efficient loading of categorical data\n- GPU-efficient orchestration of embedding layers\n- Communication-efficient training and evaluation at scale\n- Easy to use with existing AI workflows\n\n## Usage\n\nA minimal example:\n\n```python\nimport tensorflow as tf\nimport hybridbackend.tensorflow as hb\n\nds = hb.data.Dataset.from_parquet(filenames)\nds = ds.batch(batch_size)\n# ...\n\nwith tf.device('/gpu:0'):\n  embs = tf.nn.embedding_lookup_sparse(weights, input_ids)\n  # ...\n```\n\nPlease see [documentation](https://hybridbackend.readthedocs.io/en/latest/) for\nmore information.\n\n## Install\n\n### Method 1: Install from PyPI\n\n`pip install {PACKAGE}`\n\n| `{PACKAGE}`                                                                             | Dependency                                                              | Python | CUDA | GLIBC  | Data Opt. | Embedding Opt. | Parallelism Opt. |\n| ----------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- | ------ | ---- | ------ | --------- | -------------- | ---------------- |\n| [hybridbackend-tf115-cu121](https://pypi.org/project/hybridbackend-tf115-cu121/)             | [TensorFlow 1.15](https://github.com/NVIDIA/tensorflow)  | 3.8    | 12.1 | \u003e=2.31 | \u0026check;   | \u0026check;        | \u0026check;          |\n| [hybridbackend-tf115-cu100](https://pypi.org/project/hybridbackend-tf115-cu100/)             | [TensorFlow 1.15](https://github.com/tensorflow/tensorflow/tree/r1.15)     | 3.6    | 10.0 | \u003e=2.27 | \u0026check;   | \u0026check;        | \u0026cross;          |\n| [hybridbackend-tf115-cpu](https://pypi.org/project/hybridbackend-tf115-cpu/)                 | [TensorFlow 1.15](https://github.com/tensorflow/tensorflow/tree/r1.15)     | 3.6    | -    | \u003e=2.24 | \u0026check;   | \u0026cross;        | \u0026cross;          |\n\n### Method 2: Build from source\n\nSee [Building Instructions](https://github.com/alibaba/HybridBackend/blob/main/BUILD.md).\n\nWe also provide built docker images for latest [DeepRec](https://github.com/alibaba/DeepRec): \n`registry.cn-shanghai.aliyuncs.com/pai-dlc/hybridbackend:1.0.0-deeprec-py3.6-cu114-ubuntu18.04`\n\n## License\n\nHybridBackend is licensed under the [Apache 2.0 License](LICENSE).\n\n## Community\n\n- Please see [Contributing Guide](https://github.com/alibaba/HybridBackend/blob/main/CONTRIBUTING.md)\n  before your first contribution.\n- Please [register as an adopter](https://github.com/alibaba/HybridBackend/blob/main/ADOPTERS.md)\n  if your organization is interested in adoption. We will discuss\n  [RoadMap](https://github.com/alibaba/HybridBackend/blob/main/ROADMAP.md) with\n  registered adopters in advance.\n- Please cite [HybridBackend](https://ieeexplore.ieee.org/document/9835450) in your publications if it helps:\n\n  ```text\n  @inproceedings{zhang2022picasso,\n    title={PICASSO: Unleashing the Potential of GPU-centric Training for Wide-and-deep Recommender Systems},\n    author={Zhang, Yuanxing and Chen, Langshi and Yang, Siran and Yuan, Man and Yi, Huimin and Zhang, Jie and Wang, Jiamang and Dong, Jianbo and Xu, Yunlong and Song, Yue and others},\n    booktitle={2022 IEEE 38th International Conference on Data Engineering (ICDE)},\n    year={2022},\n    organization={IEEE}\n  }\n  ```\n\n## Contact Us\n\nIf you would like to share your experiences with others, you are welcome to\ncontact us in DingTalk:\n\n[![dingtalk](https://github.com/alibaba/HybridBackend/raw/main/docs/images/dingtalk.png)](https://qr.dingtalk.com/action/joingroup?code=v1,k1,VouhbeuTwXYEgaLzSOE8o6VF2kTHVJ8lw5h93WbZW8o=\u0026_dt_no_comment=1\u0026origin=11)","funding_links":[],"categories":["C++"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FDeepRec-AI%2FHybridBackend","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FDeepRec-AI%2FHybridBackend","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FDeepRec-AI%2FHybridBackend/lists"}