{"id":17302123,"url":"https://github.com/tanyuqian/redco","last_synced_at":"2025-04-06T10:10:17.072Z","repository":{"id":163720766,"uuid":"580118287","full_name":"tanyuqian/redco","owner":"tanyuqian","description":"NAACL '24 (Best Demo Paper RunnerUp) / MlSys @ NeurIPS '23 - RedCoast: A Lightweight Tool to Automate Distributed Training and Inference","archived":false,"fork":false,"pushed_at":"2024-10-21T21:18:16.000Z","size":12081,"stargazers_count":60,"open_issues_count":0,"forks_count":7,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-10-22T04:14:22.478Z","etag":null,"topics":["differential-privacy","diffusion-models","distributed-training","fedavg","federated-learning","flan-t5-xxl","gemma","image-captioning","jax","large-language-models","llama","maml","meta-learning","mixed-precision","mlsys","model-parallelism","ppo","reinforcement-learning","seq2seq","stable-diffusion"],"latest_commit_sha":null,"homepage":"https://tanyuqian.github.io/redco/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tanyuqian.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-12-19T19:07:44.000Z","updated_at":"2024-10-21T21:18:20.000Z","dependencies_parsed_at":"2023-10-20T08:31:37.115Z","dependency_job_id":"b606df06-a7da-4aae-b8ef-73db0f305a8b","html_url":"https://github.com/tanyuqian/redco","commit_stats":{"total_commits":667,"total_committers":1,"mean_commits":667.0,"dds":0.0,"last_synced_commit":"306f86c3613911cb007f897a88d3510622c498ac"},"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tanyuqian%2Fredco","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tanyuqian%2Fredco/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tanyuqian%2Fredco/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tanyuqian%2Fredco/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tanyuqian","download_url":"https://codeload.github.com/tanyuqian/redco/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247464196,"owners_count":20942969,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["differential-privacy","diffusion-models","distributed-training","fedavg","federated-learning","flan-t5-xxl","gemma","image-captioning","jax","large-language-models","llama","maml","meta-learning","mixed-precision","mlsys","model-parallelism","ppo","reinforcement-learning","seq2seq","stable-diffusion"],"created_at":"2024-10-15T11:46:38.306Z","updated_at":"2025-04-06T10:10:17.051Z","avatar_url":"https://github.com/tanyuqian.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"![](docs/images/redco_banner.png)\n\n**Red Coast** (redco) is a lightweight and user-friendly tool designed to automate distributed training and inference for large models while simplifying the ML pipeline development process without necessitating MLSys expertise from users.\n\nRedCoast supports *Large Models* + *Complex Algorithms*, in a *lightweight* and *user-friendly* manner: \n* Large Models beyond Transformers, e.g, [Stable Diffusion](https://github.com/tanyuqian/redco/tree/master/examples/text_to_image), etc.\n* Complex algorithms beyond cross entropy, e.g., [Meta Learning](https://github.com/tanyuqian/redco/tree/master/examples/meta_learning), [Differentially-Private (DP) Training](https://github.com/tanyuqian/redco/tree/master/examples/differential_private_training), etc.\n\nCheck out our [Tech Report](https://aclanthology.org/2024.naacl-demo.14/) for more details! \n\n**RedCoast: A Lightweight Tool to Automate Distributed Training of LLMs on Any GPU/TPUs** \\\nBowen Tan, Yun Zhu, Lijuan Liu, Hongyi Wang, Yonghao Zhuang, Jindong Chen, Eric Xing, Zhiting Hu \\\nNAACL 2024, Demo / MLSys Workshop @ NeurIPS 2023 \\\n[[Paper]](https://aclanthology.org/2024.naacl-demo.14/) \n[[Twitter]](https://x.com/BowenTan8/status/1730240627068031295?s=20) \n[[Slides]](https://drive.google.com/file/d/1MmBjxP5gInqhg0ydasby2a5UauLZFxQH/view) \n[[Demo Video]](https://bowentan.bitcron.com/RedCoast_demo.webm) \n[[Documentation]](https://tanyuqian.github.io/redco) \\\n*Best Demo Paper Runner Up @ NAACL 2024*\n\n\n![](docs/images/redco_coding.png)\n\n### Installation\n\n#### Install RedCoast\n```\npip install redco\n```\n\n#### Adjust Jax to GPU/TPU version\nThe command above would automatically install cpu version of jax, so the version of Jax need to be adjusted based on your device. \nFor example, on GPUs,\n```\npip install --upgrade \"jax[cuda12]\"\n```\nIf you are using TPU/CPU/AMD/Apple, see [here](https://github.com/google/jax#installation) for corresponding installation commands.\n\n\n### Examples\n\nExamples across a set of paradigms can be found in [examples/](examples/), including\n\n* [Classification/regression (GLUE \u0026 MNIST)](https://github.com/tanyuqian/redco/tree/master/examples/classification_regression)\n* [Differential-private_training](https://github.com/tanyuqian/redco/tree/master/examples/differential_private_training)\n* [Faderated learning (FedAvg)](https://github.com/tanyuqian/redco/tree/master/examples/faderated_learning)\n* [Image to text (Image captioning)](https://github.com/tanyuqian/redco/tree/master/examples/image_to_text)\n* [Language modeling (Instruction Tuning of LLMs)](https://github.com/tanyuqian/redco/tree/master/examples/language_modeling)\n* [Meta learning (MAML)](https://github.com/tanyuqian/redco/tree/master/examples/meta_learning)\n* [Reinforcement learning (PPO \u0026 DDPG)](https://github.com/tanyuqian/redco/tree/master/examples/reinforcement_learning)\n* [Text to image (StableDiffusion)](https://github.com/tanyuqian/redco/tree/master/examples/text_to_image)\n* [Text to text (Seq2seq)](https://github.com/tanyuqian/redco/tree/master/examples/text_to_text)\n* [LoRA Finetuning](https://github.com/tanyuqian/redco/tree/master/examples/lora)\n\n### Exemplar large model settings\n\nThe table below shows runnable model LLM finetuning on different kinds of servers. Numbers inside the brackets are the maximum length in training. All the settings are with full precision (fp32) and Adam optimizer.\n\n| 2 $\\times$ 1080Ti \u003cbr/\u003e(2 $\\times$ 10G) | 4 $\\times$ A100 \u003cbr/\u003e(4 $\\times$ 40G) | 2 $\\times$ TPU-v4 \u003cbr/\u003e(2 hosts $\\times$ 4 chips $\\times$ 32G) | 16 $\\times$ TPU-v4 \u003cbr/\u003e(16 hosts $\\times$ 4 chips $\\times$ 32G) |\n|-----------------------------------------|---------------------------------------|----------------------------------------------------------------|------------------------------------------------------------------|\n| BART-Large (1024)                       | LLaMA-7B (1024)                       | T5-XL-11B (512)                                                | OPT-66B (512)                                                    |\n| GPT2-Large (512)                        | GPT-J-6B (1024)                       | OPT-13B (1024)                                                 |                                                                  |\n\nGo to [example/language_modeling](https://github.com/tanyuqian/redco/tree/master/examples/language_modeling) and [examples/text_to_text](https://github.com/tanyuqian/redco/tree/master/examples/text_to_text) to try them out!\n\n\n## Reference\n```\n@inproceedings{tan-etal-2024-redcoast,\n    title = \"{R}ed{C}oast: A Lightweight Tool to Automate Distributed Training of {LLM}s on Any {GPU}/{TPU}s\",\n    author = \"Tan, Bowen  and\n      Zhu, Yun  and\n      Liu, Lijuan  and\n      Wang, Hongyi  and\n      Zhuang, Yonghao  and\n      Chen, Jindong  and\n      Xing, Eric  and\n      Hu, Zhiting\",\n    editor = \"Chang, Kai-Wei  and\n      Lee, Annie  and\n      Rajani, Nazneen\",\n    booktitle = \"Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: System Demonstrations)\",\n    month = jun,\n    year = \"2024\",\n    address = \"Mexico City, Mexico\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://aclanthology.org/2024.naacl-demo.14\",\n    pages = \"137--147\",\n    abstract = \"The recent progress of AI can be largely attributed to large language models (LLMs). However, their escalating memory requirements introduce challenges for machine learning (ML) researchers and engineers. Addressing this requires developers to partition a large model to distribute it across multiple GPUs or TPUs. This necessitates considerable coding and intricate configuration efforts with existing model parallel tools, such as Megatron-LM, DeepSpeed, and Alpa. These tools require users{'} expertise in machine learning systems (MLSys), creating a bottleneck in LLM development, particularly for developers without MLSys background. In this work, we present RedCoast (Redco), a lightweight and user-friendly tool crafted to automate distributed training and inference for LLMs, as well as to simplify ML pipeline development. The design of Redco emphasizes two key aspects. Firstly, to automate model parallelism, our study identifies two straightforward rules to generate tensor parallel strategies for any given LLM. Integrating these rules into Redco facilitates effortless distributed LLM training and inference, eliminating the need of additional coding or complex configurations. We demonstrate the effectiveness by applying Redco on a set of LLM architectures, such as GPT-J, LLaMA, T5, and OPT, up to the size of 66B. Secondly, we propose a mechanism that allows for the customization of diverse ML pipelines through the definition of merely three functions, avoiding redundant and formulaic code like multi-host related processing. This mechanism proves adaptable across a spectrum of ML algorithms, from foundational language modeling to complex algorithms like meta-learning and reinforcement learning. As a result, Redco implementations exhibit significantly fewer lines of code compared to their official counterparts. RedCoast (Redco) has been released under Apache 2.0 license at https://github.com/tanyuqian/redco.\",\n}\n```\n\n## Acknowledgement\n\n\nThe name of this package is inspired by *Red Coast Base*, a key location in the story of *Three Body*. From Red Coast Base, humanity broadcasts its first message into the vast universe. We thank Cixin Liu for such a masterpiece!\n\n![](docs/images/red_coast.png)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftanyuqian%2Fredco","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftanyuqian%2Fredco","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftanyuqian%2Fredco/lists"}