{"id":13751967,"url":"https://github.com/Oneflow-Inc/libai","last_synced_at":"2025-05-09T18:32:58.819Z","repository":{"id":36952561,"uuid":"420927199","full_name":"Oneflow-Inc/libai","owner":"Oneflow-Inc","description":"LiBai(李白): A Toolbox for Large-Scale Distributed Parallel Training","archived":false,"fork":false,"pushed_at":"2025-05-07T10:28:28.000Z","size":36354,"stargazers_count":403,"open_issues_count":81,"forks_count":56,"subscribers_count":41,"default_branch":"main","last_synced_at":"2025-05-07T10:45:03.044Z","etag":null,"topics":["data-parallelism","deep-learning","distributed-training","large-scale","model-parallelism","nlp","oneflow","pipeline-parallelism","self-supervised-learning","transformer","vision-transformer"],"latest_commit_sha":null,"homepage":"https://libai.readthedocs.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Oneflow-Inc.png","metadata":{"files":{"readme":"README.md","changelog":"changelog.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-10-25T07:50:21.000Z","updated_at":"2025-04-27T09:01:16.000Z","dependencies_parsed_at":"2023-02-19T05:01:18.356Z","dependency_job_id":"2f066297-62bc-4aeb-b41f-34eaaee15e71","html_url":"https://github.com/Oneflow-Inc/libai","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Oneflow-Inc%2Flibai","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Oneflow-Inc%2Flibai/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Oneflow-Inc%2Flibai/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Oneflow-Inc%2Flibai/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Oneflow-Inc","download_url":"https://codeload.github.com/Oneflow-Inc/libai/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253303182,"owners_count":21886901,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-parallelism","deep-learning","distributed-training","large-scale","model-parallelism","nlp","oneflow","pipeline-parallelism","self-supervised-learning","transformer","vision-transformer"],"created_at":"2024-08-03T09:00:57.674Z","updated_at":"2025-05-09T18:32:53.803Z","avatar_url":"https://github.com/Oneflow-Inc.png","language":"Python","funding_links":[],"categories":["分布式机器学习"],"sub_categories":[],"readme":"\u003c!-- 配图 --\u003e\n\n\u003ch2 align=\"center\"\u003eLiBai\u003c/h2\u003e\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://libai.readthedocs.io/en/latest/index.html\"\u003e\n        \u003cimg alt=\"docs\" src=\"https://img.shields.io/badge/docs-latest-blue\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://github.com/Oneflow-Inc/libai/blob/master/LICENSE\"\u003e\n        \u003cimg alt=\"GitHub\" src=\"https://img.shields.io/github/license/Oneflow-Inc/libai.svg?color=blue\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://github.com/Oneflow-Inc/libai/releases\"\u003e\n        \u003cimg alt=\"GitHub release\" src=\"https://img.shields.io/github/release/Oneflow-Inc/libai.svg\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://github.com/Oneflow-Inc/libai/issues\"\u003e\n        \u003cimg alt=\"PRs Welcome\" src=\"https://img.shields.io/badge/PRs-welcome-pink.svg\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://github.com/Oneflow-Inc/libai/issues\"\u003e\n        \u003cimg alt=\"Python Checks\" src=\"https://github.com/Oneflow-Inc/libai/workflows/Python checks/badge.svg\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://github.com/Oneflow-Inc/libai/issues\"\u003e\n        \u003cimg alt=\"Docs Release Status\" src=\"https://github.com/Oneflow-Inc/libai/workflows/Document Release/badge.svg\"\u003e\n    \u003c/a\u003e\n\u003c/p\u003e\n\n\n## Introduction\n\n**English** | [简体中文](/README_zh-CN.md)\n\nLiBai is a large-scale open-source model training toolbox based on OneFlow. The main branch works with OneFlow 0.7.0.\n\n\u003cdetails open\u003e\n\u003csummary\u003e \u003cb\u003e Highlights \u003c/b\u003e \u003c/summary\u003e\n\n- **Support a collection of parallel training components**\n\n    LiBai provides multiple parallelisms such as Data Parallelism, Tensor Parallelism, and Pipeline Parallelism. It's also extensible for other new parallelisms.\n\n- **Varied training techniques**\n\n    LiBai provides many out-of-the-box training techniques such as Distributed Training, Mixed Precision Training, Activation Checkpointing, Recomputation, Gradient Accumulation, and Zero Redundancy Optimizer(ZeRO).\n\n- **Support for both CV and NLP tasks**\n\n    LiBai has predefined data process for both CV and NLP datasets such as CIFAR, ImageNet, and BERT Dataset.\n\n- **Easy to use**\n\n    LiBai's components are designed to be modular for easier usage as follows:\n    - LazyConfig system for more flexible syntax and no predefined structures \n    - Friendly trainer and engine\n    - Used as a library to support building research projects on it. See [projects/](/projects) for some projects that are built based on LiBai\n\n- **High Efficiency**\n\n\u003c/details\u003e\n\n## Installation\n\nSee [Installation instructions](https://libai.readthedocs.io/en/latest/tutorials/get_started/Installation.html).\n\n## Getting Started\n\nSee [Quick Run](https://libai.readthedocs.io/en/latest/tutorials/get_started/quick_run.html) for the basic usage of LiBai.\n\n## Documentation\n\nSee LiBai's [documentation](https://libai.readthedocs.io/en/latest/index.html) for full API documentation and tutorials.\n\n## ChangeLog\n\n**Beta 0.3.0** was released in 03/11/2024, the general changes in **0.3.0** version are as follows:\n\n**Features:**\n- Support mock transformers, see [Mock transformers](https://github.com/Oneflow-Inc/libai/tree/main/projects/mock_transformers#readme)\n- Support lm-evaluation-harness for model evaluation\n- User Experience Optimization\n\n**New Supported Models:**\n- These models are natively supported by libai\n\u003ctable class=\"docutils\"\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003cth width=\"130\"\u003e Models \u003c/th\u003e\n      \u003cth valign=\"bottom\" align=\"center\" width=\"140\"\u003e 2D(tp+pp) Inference\u003c/th\u003e\n      \u003cth valign=\"bottom\" align=\"center\" width=\"140\"\u003e 3D Parallel Training \u003c/th\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\"\u003e\u003ca href=\"https://github.com/Oneflow-Inc/libai/tree/main/projects/BLOOM\"\u003e \u003cb\u003e BLOOM \u003c/b\u003e \u003c/td\u003e\n      \u003ctd align=\"center\"\u003e\u0026#10004;\u003c/td\u003e\n      \u003ctd align=\"center\"\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\"\u003e\u003ca href=\"https://github.com/Oneflow-Inc/libai/tree/main/projects/ChatGLM\"\u003e \u003cb\u003e ChatGLM \u003c/b\u003e \u003c/td\u003e\n      \u003ctd align=\"center\"\u003e\u0026#10004;\u003c/td\u003e\n      \u003ctd align=\"center\"\u003e\u0026#10004;\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\"\u003e\u003ca href=\"https://github.com/Oneflow-Inc/libai/tree/main/projects/Couplets\"\u003e \u003cb\u003e Couplets \u003c/b\u003e \u003c/td\u003e\n      \u003ctd align=\"center\"\u003e\u0026#10004;\u003c/td\u003e\n      \u003ctd align=\"center\"\u003e\u0026#10004;\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\"\u003e\u003ca href=\"https://github.com/Oneflow-Inc/libai/tree/main/projects/DALLE2\"\u003e \u003cb\u003e DALLE2 \u003c/b\u003e \u003c/td\u003e\n      \u003ctd align=\"center\"\u003e\u0026#10004;\u003c/td\u003e\n      \u003ctd align=\"center\"\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\"\u003e\u003ca href=\"https://github.com/Oneflow-Inc/libai/tree/main/projects/Llama\"\u003e \u003cb\u003e Llama2 \u003c/b\u003e \u003c/td\u003e\n      \u003ctd align=\"center\"\u003e\u0026#10004;\u003c/td\u003e\n      \u003ctd align=\"center\"\u003e\u0026#10004;\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\"\u003e\u003ca href=\"https://github.com/Oneflow-Inc/libai/tree/main/projects/MAE\"\u003e \u003cb\u003e MAE \u003c/b\u003e \u003c/td\u003e\n      \u003ctd align=\"center\"\u003e\u0026#10004;\u003c/td\u003e\n      \u003ctd align=\"center\"\u003e\u0026#10004;\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\"\u003e\u003ca href=\"https://github.com/Oneflow-Inc/libai/tree/main/projects/Stable_Diffusion\"\u003e \u003cb\u003e Stable_Diffusion \u003c/b\u003e \u003c/td\u003e\n      \u003ctd align=\"center\"\u003e-\u003c/td\u003e\n      \u003ctd align=\"center\"\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\n**New Mock Models:**\n- These models are extended and implemented by libai through mocking transformers.\n\u003ctable class=\"docutils\"\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003cth width=\"130\"\u003e Models \u003c/th\u003e\n      \u003cth valign=\"bottom\" align=\"center\" width=\"140\"\u003eTensor Parallel\u003c/th\u003e\n      \u003cth valign=\"bottom\" align=\"center\" width=\"150\"\u003ePipeline Parallel\u003c/th\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\"\u003e\u003ca href=\"https://huggingface.co/docs/transformers/v4.26.1/en/model_doc/bloom#overview\"\u003e \u003cb\u003e BLOOM \u003c/b\u003e \u003c/td\u003e\n      \u003ctd align=\"center\"\u003e\u0026#10004;\u003c/td\u003e\n      \u003ctd align=\"center\"\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\"\u003e\u003ca href=\"https://github.com/openai/gpt-2/blob/master/model_card.md\"\u003e \u003cb\u003e GPT2 \u003c/b\u003e \u003c/td\u003e\n      \u003ctd align=\"center\"\u003e\u0026#10004;\u003c/td\u003e\n      \u003ctd align=\"center\"\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\"\u003e\u003ca href=\"https://huggingface.co/docs/transformers/v4.28.0/en/model_doc/llama#overview\"\u003e \u003cb\u003e LLAMA \u003c/b\u003e \u003c/td\u003e\n      \u003ctd align=\"center\"\u003e\u0026#10004;\u003c/td\u003e\n      \u003ctd align=\"center\"\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\"\u003e\u003ca href=\"https://huggingface.co/docs/transformers/main/en/model_doc/llama2\"\u003e \u003cb\u003e LLAMA2 \u003c/b\u003e \u003c/td\u003e\n      \u003ctd align=\"center\"\u003e\u0026#10004;\u003c/td\u003e\n      \u003ctd align=\"center\"\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\"\u003e\u003ca href=\"https://huggingface.co/baichuan-inc/Baichuan-7B\"\u003e \u003cb\u003e Baichuan \u003c/b\u003e \u003c/td\u003e\n      \u003ctd align=\"center\"\u003e\u0026#10004;\u003c/td\u003e\n      \u003ctd align=\"center\"\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\"\u003e\u003ca href=\"https://huggingface.co/docs/transformers/v4.26.1/en/model_doc/opt#overview\"\u003e \u003cb\u003e OPT \u003c/b\u003e \u003c/td\u003e\n      \u003ctd align=\"center\"\u003e\u0026#10004;\u003c/td\u003e\n      \u003ctd align=\"center\"\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\nSee [changelog](./changelog.md) for details and release history.\n\n## Contributing\n\nWe appreciate all contributions to improve LiBai. See [CONTRIBUTING](./CONTRIBUTING.md) for the contributing guideline.\n\n## License\n\nThis project is released under the [Apache 2.0 license](LICENSE).\n\n## Citation\n\nIf you find this project useful for your research, consider cite:\n\n```BibTeX\n@misc{of2021libai,\n  author =       {Xingyu Liao and Peng Cheng and Tianhe Ren and Depeng Liang and\n                  Kai Dang and Yi Wang and Xiaoyu Xu},\n  title =        {LiBai},\n  howpublished = {\\url{https://github.com/Oneflow-Inc/libai}},\n  year =         {2021}\n}\n```\n\n## Join the WeChat group\n\n![LiBai_Wechat_QRcode](./docs/source/tutorials/assets/LiBai_Wechat.png)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FOneflow-Inc%2Flibai","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FOneflow-Inc%2Flibai","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FOneflow-Inc%2Flibai/lists"}