{"id":18688248,"url":"https://github.com/flagopen/flagscale","last_synced_at":"2025-04-12T03:52:55.137Z","repository":{"id":199821678,"uuid":"696202301","full_name":"FlagOpen/FlagScale","owner":"FlagOpen","description":"FlagScale is a large model toolkit based on open-sourced projects.","archived":false,"fork":false,"pushed_at":"2025-04-11T03:04:37.000Z","size":41964,"stargazers_count":262,"open_issues_count":24,"forks_count":62,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-04-12T03:52:42.152Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FlagOpen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-25T09:44:13.000Z","updated_at":"2025-04-11T03:04:41.000Z","dependencies_parsed_at":"2023-10-12T20:02:53.467Z","dependency_job_id":"c732fbd3-b88c-42ea-8e4f-ba69c2acb4a7","html_url":"https://github.com/FlagOpen/FlagScale","commit_stats":null,"previous_names":["flagopen/flagscale"],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FlagOpen%2FFlagScale","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FlagOpen%2FFlagScale/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FlagOpen%2FFlagScale/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FlagOpen%2FFlagScale/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FlagOpen","download_url":"https://codeload.github.com/FlagOpen/FlagScale/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248514209,"owners_count":21116899,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T10:36:05.773Z","updated_at":"2025-04-12T03:52:55.118Z","avatar_url":"https://github.com/FlagOpen.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[\u003cimg src=\"flagopen.png\"\u003e](https://flagopen.baai.ac.cn/)\n\n## Latest News\n- **[2025/02]** Released [v0.6.5](https://github.com/FlagOpen/FlagScale/tree/release/v0.6.5):\n  - Added support for DeepSeek-V3 distributed pre-training (beta) and [DeepSeek-V3/R1 serving](#deepseek-r1-serving) across multiple chips.\n  - Introduced an auto-tuning feature for serving and a new CLI feature for one-click deployment.\n  - Enhanced the CI/CD system to support more chips and integrated the workflow of [FlagRelease](https://huggingface.co/FlagRelease).\n- **[2024/11]** Released [v0.6.0](https://github.com/FlagOpen/FlagScale/tree/release/v0.6.0):\n  - Introduced general multi-dimensional heterogeneous parallelism and CPU-based communication between different chips.\n  - Added the full support for LLaVA-OneVision, achieving SOTA results on the [Infinity-MM](https://arxiv.org/abs/2410.18558) dataset.\n  - Open-sourced the optimized CFG implementation and accelerated the generation and understanding tasks for [Emu3](https://arxiv.org/abs/2409.18869).\n  - Implemented the auto-tuning feature and enhanced the CI/CD system.\n- **[2024/4]** Released [v0.3](https://github.com/FlagOpen/FlagScale/tree/release/v0.3): Achieved heterogeneous hybrid training of the Aquila2-70B-Expr model on a cluster using both NVIDIA and Iluvatar chips. Adapted the Aquila2 series to AI chips from six different manufacturers.\n- **[2023/11]** Released [v0.2](https://github.com/FlagOpen/FlagScale/tree/v0.2): Introduced training support for Aquila2-70B-Expr, enabling heterogeneous training across chips with the same or compatible architectures.\n- **[2023/10]** Released [v0.1](https://github.com/FlagOpen/FlagScale/tree/v0.1): Supported Aquila models with optimized training schemes for Aquila2-7B and Aquila2-34B, including parallel strategies, optimizations, and hyper-parameter settings.\n\n## About\n\n[FlagScale](https://github.com/FlagOpen/FlagScale.git) is a comprehensive toolkit designed to support the entire lifecycle of large models, developed with the backing of the Beijing Academy of Artificial Intelligence (BAAI). It builds on the strengths of several prominent open-source projects, including [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) and [vllm](https://github.com/vllm-project/vllm), to provide a robust, end-to-end solution for managing and scaling large models.\n\nThe primary objective of FlagScale is to enable seamless scalability across diverse hardware architectures while maximizing computational resource efficiency and enhancing model performance. By offering essential components for model development, training, and deployment, FlagScale seeks to establish itself as an indispensable toolkit for optimizing both the speed and effectiveness of large model workflows.\n\nFlagScale is also a part of [FlagAI-Open](https://flagopen.baai.ac.cn/), an open-source initiative by BAAI that aims to foster an open-source ecosystem for AI technologies. It serves as a platform where developers, researchers, and AI enthusiasts can collaborate on various AI projects, contribute to the development of cutting-edge AI solutions, and share their work with the global community.\n\n## Quick Start\n\nFlagScale leverages [Hydra](https://github.com/facebookresearch/hydra) for configuration management. The configurations are organized into two levels: an outer experiment-level YAML file and an inner task-level YAML file.\n\n- The experiment-level YAML file defines the experiment directory, backend engine, task type, and other related environmental configurations.\n- The task-level YAML file specifies the model, dataset, and parameters for specific tasks such as training or inference.\n\nAll valid configurations in the task-level YAML file correspond to the arguments used in backend engines such as Megatron-LM and vllm, with hyphens (-) replaced by underscores (_). For a complete list of available configurations, please refer to the backend engine documentation. Simply copy and modify the existing YAML files in the [examples](./examples) folder to get started.\n\n### Setup\nWe recommend using the latest release of [NGC's PyTorch container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch) for setup.\n\n1. Clone the repository:\n    ```sh\n    git clone https://github.com/FlagOpen/FlagScale.git\n    ```\n\n2. Install the requirements:\n    ```sh\n    cd FlagScale/install\n    ./install-requirements.sh --env train\n    ./install-requirements.sh --env inference\n    ```\n    The above instructions create two conda environments: `flagscale-train` and `flagscale-inference`, which contain the dependency environments for training and inference, respectively.\n\n3. Install the packages with customized extensions:\n    ```sh\n    cd vllm\n    pip install .\n\n    pip install -e ./megatron-energon\n    cp -r megatron-energon/src/megatron/energon megatron/megatron\n    ```\n\n### Run a Task\n\nFlagScale provides a unified runner for various tasks, including training，inference and serve. Simply specify the configuration file to run the task with a single command. The runner will automatically load the configurations and execute the task. The following example demonstrates how to run a distributed training task.\n\n#### Train\n\n1. Start the distributed training job:\n    ```sh\n    python run.py --config-path ./examples/aquila/conf --config-name config action=run\n    ```\n    The `data_path` in the demo is the path of the training datasets following the [Megatron-LM format](./megatron/README.md#data-preprocessing). For quickly running the pretraining process, we also provide a small processed data ([bin](https://model.ks3-cn-beijing.ksyuncs.com/nlpdata/pile_wikipedia_demo.bin) and [idx](https://model.ks3-cn-beijing.ksyuncs.com/nlpdata/pile_wikipedia_demo.idx)) from the [Pile](https://pile.eleuther.ai/) dataset.\n\n2. Stop the distributed training job:\n    ```sh\n    python run.py --config-path ./examples/aquila/conf --config-name config action=stop\n    ```\n\n#### Serve\n\n1. Start the server:\n    ```sh\n    python run.py --config-path ./examples/qwen/conf --config-name config_qwen2.5_7b action=run\n    ```\n2. Stop the server:\n    ```sh\n    python run.py --config-path ./examples/qwen/conf --config-name config_qwen2.5_7b action=stop\n    ```\nFor more details, please refer to [Quick Start](./flagscale/serve/README.md).\n\n### DeepSeek-R1 Serving \u003ca name=\"deepseek-r1-serving\"\u003e\u003c/a\u003e\n\nWe support the model serving of DeepSeek R1 and have implemented the `flagscale serve` command for one-click deployment. By configuring just two YAML files, you can easily serve the model using the `flagscale serve` command.\n\n1. **Configure the YAML files:**\n    ```\n    FlagScale/\n    ├── examples/\n    │   └── deepseek_r1/\n    │       └── config/\n    │           └── config_deepseek_r1.yaml\n    |           └── hostfile.txt # Set hostfile (optional)\n    │           └── serve/\n    │               └── deepseek_r1.yaml # Set model parameters and server port\n    ```\n    Note: When task covers multiple nodes, [hostfile.txt](./examples/deepseek/conf/hostfile.txt) is required. The file path should be set in config_deepseek_r1.yaml.\n\n2. **Install FlagScale CLI:**\n    ```sh\n    cd FlagScale\n    pip install .\n    ```\n\n3. **One-click serve:**\n    ```sh\n    flagscale serve deepseek_r1\n    ```\n\n4. **Custom service parameters:**\n    ```sh\n    flagscale serve \u003cMODEL_NAME\u003e \u003cMODEL_CONFIG_YAML\u003e\n    ```\n\nThe configuration files allow you to specify the necessary parameters and settings for your deployment, ensuring a smooth and efficient serving process.\n\n## License\n\nThis project is licensed under the [Apache License (Version 2.0)](https://github.com/FlagOpen/FlagScale/blob/main/LICENSE). This project also contains other third-party components under other open-source licenses. See the [LICENSE](https://github.com/FlagOpen/FlagScale/blob/main/LICENSE) file for more information.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflagopen%2Fflagscale","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fflagopen%2Fflagscale","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflagopen%2Fflagscale/lists"}