{"id":13443409,"url":"https://github.com/hpcaitech/ColossalAI-Examples","last_synced_at":"2025-03-20T16:31:15.361Z","repository":{"id":36950780,"uuid":"449131373","full_name":"hpcaitech/ColossalAI-Examples","owner":"hpcaitech","description":"Examples of training models with hybrid parallelism using ColossalAI","archived":true,"fork":false,"pushed_at":"2023-03-23T09:43:43.000Z","size":2401,"stargazers_count":337,"open_issues_count":31,"forks_count":103,"subscribers_count":20,"default_branch":"main","last_synced_at":"2025-01-13T00:46:07.343Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hpcaitech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-01-18T03:48:13.000Z","updated_at":"2025-01-10T01:35:27.000Z","dependencies_parsed_at":"2024-01-18T14:43:21.963Z","dependency_job_id":"2ea7b1e6-1fc1-4f43-98ea-acd8bf631870","html_url":"https://github.com/hpcaitech/ColossalAI-Examples","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hpcaitech%2FColossalAI-Examples","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hpcaitech%2FColossalAI-Examples/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hpcaitech%2FColossalAI-Examples/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hpcaitech%2FColossalAI-Examples/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hpcaitech","download_url":"https://codeload.github.com/hpcaitech/ColossalAI-Examples/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244649826,"owners_count":20487502,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T03:02:00.309Z","updated_at":"2025-03-20T16:31:14.295Z","avatar_url":"https://github.com/hpcaitech.png","language":"Python","readme":"# ColossalAI-Examples\n\n## 2023.01.05\nThis repo is deprecated. Use our timely maintained example at [ColossalAI/example](https://github.com/hpcaitech/ColossalAI).\n\n## Introduction\n\nThis repository provides various examples for **Colossal-AI**. For each feature of \nColossal-AI, you can find a simple example in the `feature` folder and a corresponding tutorial in feature section of the [**documentation**](https://www.colossalai.org/). For more complex examples for domain-specific models, you can find them in this repository as well. Some of them are covered in the advanced tutorials \nof the [**documentation**](https://www.colossalai.org/).\n\nThis repository is built upon Colossal-AI and Titans.\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/hpcaitech/public_assets/main/examples/images/repo_relation.png\" width=\"300\" title=\"repo-architecture\"\u003e\n\u003c/div\u003e\n\n\n### 🚀 Quick Links\n\n[**Colossal-AI**](https://github.com/hpcaitech/ColossalAI) | \n[**Titans**](https://github.com/hpcaitech/Titans)\n[**Paper**](https://arxiv.org/abs/2110.14883) | \n[**Documentation**](https://www.colossalai.org/) | \n[**Forum**](https://github.com/hpcaitech/ColossalAI/discussions) | \n[**Blog**](https://www.colossalai.org/) \n\n## Setup\n\n1. Install Colossal-AI\n\nYou can download Colossal-AI [here](https://www.colossalai.org/download).\n\n2. Install dependencies\n\n```\npip install -r requirements.txt\n```\n\n## Table of Content\n\nThis repository contains examples of training models with ColossalAI. These examples fall under three categories:\n\n1. Computer Vision\n    - ResNet\n    - SimCLR\n    - Vision Transformer\n        - Data Parallel\n        - Pipeline Parallel\n        - Hybrid Parallel\n    - WideNet\n        - Mixture of experts\n\n2. Natural Language Processing\n    - BERT\n        - Sequence Parallel\n    - GPT-2\n        - Hybrid Parallel\n    - GPT-3\n        - Hybrid Parallel\n    - Knowledge Graph Embedding\n\n3. Features\n    - Mixed Precision Training\n    - Gradient Accumulation\n    - Gradient Clipping\n    - Tensor Parallel\n    - Pipeline Parallel\n    - ZeRO\n\nThe `image` and `language` folders are for complex model applications. The `features` folder is for demonstration of Colossal-AI. The `features` folder aims to be simple so that users can execute in minutes. Each example in the `features` folder relates to a tutorial in the [Official Documentation](https://colossalai.org/).\n\n**If you wish to make contribution to this repository, please read the `Contributing` section below.**\n\n## Discussion\n\nDiscussion about the Colossal-AI project and examples is always welcomed! We would love to exchange ideas with the community to better help this project grow.\nIf you think there is a need to discuss anything, you may jump to our [discussion forum](https://github.com/hpcaitech/ColossalAI/discussions) and create a topic there.\n\nIf you encounter any problem while running these examples, you may want to raise an issue in this repository.\n\n## Contributing\n\nThis project welcomes constructive ideas and implementations from the community. \n\n### Update an Example\n\nIf you find that an example is broken (not working) or not user-friendly, you may put up a pull request to this repository and update this example.\n\n### Add a New Example\n\nIf you wish to add an example for a specific application, please follow the steps below.\n\n1. create a folder in the `image`, `language` or `features` folders. Generally we do not accept new examples for `features` as one example is often enough. **We encourage contribution with hybrid parallel or models of different domains (e.g. GAN, self-supervised, detection, video understanding, text classification, text generation)**\n2. Prepare configuration files and `train.py`\n3. Prepare a detailed readme on environment setup, dataset preparation, code execution, etc. in your example folder\n4. Update the table of content (first section above) in this readme file\n\n\nIf your PR is accepted, we may invite you to put up a tutorial or blog in [ColossalAI Documentation](https://colossalai.org/).\n","funding_links":[],"categories":["Python"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhpcaitech%2FColossalAI-Examples","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhpcaitech%2FColossalAI-Examples","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhpcaitech%2FColossalAI-Examples/lists"}