{"id":26103685,"url":"https://github.com/aleph-alpha/scaling","last_synced_at":"2025-04-12T18:08:39.077Z","repository":{"id":254806542,"uuid":"841991952","full_name":"Aleph-Alpha/scaling","owner":"Aleph-Alpha","description":"Scaling is a distributed training library and installable dependency designed to scale up neural networks, with a dedicated module for training large language models.","archived":false,"fork":false,"pushed_at":"2024-10-30T09:06:10.000Z","size":8041,"stargazers_count":58,"open_issues_count":0,"forks_count":4,"subscribers_count":10,"default_branch":"main","last_synced_at":"2025-04-12T18:07:49.058Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Aleph-Alpha.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-13T13:02:43.000Z","updated_at":"2025-03-17T12:53:00.000Z","dependencies_parsed_at":"2024-10-29T16:50:34.156Z","dependency_job_id":null,"html_url":"https://github.com/Aleph-Alpha/scaling","commit_stats":null,"previous_names":["aleph-alpha/scaling"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aleph-Alpha%2Fscaling","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aleph-Alpha%2Fscaling/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aleph-Alpha%2Fscaling/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aleph-Alpha%2Fscaling/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Aleph-Alpha","download_url":"https://codeload.github.com/Aleph-Alpha/scaling/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248610340,"owners_count":21132921,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-09T20:42:01.970Z","updated_at":"2025-04-12T18:08:39.045Z","avatar_url":"https://github.com/Aleph-Alpha.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"```\nALEPH ALPHA\n┏┓   ┓•\n┗┓┏┏┓┃┓┏┓┏┓\n┗┛┗┗┻┗┗┛┗┗┫\n          ┛\n```\n#\n\nScaling is a distributed training library and installable dependency designed to scale up neural networks, with a dedicated module for training large language models.\n\n# Features\n\nScaling consists of two primary components, a model-agnostic core module (`scaling.core`), which functions as the engine for distributed training workloads, and the `scaling.transformer` suite, specifically designed for LLM training.\nThe Scaling core module features various parallelization and partitioning techniques:\n\n- __Data parallelism__: Distribute training data across multiple devices.\n- __Pipeline parallelism__: Divide model layers into sequential stages across multiple devices.\n- __Tensor parallelism__: Split individual tensors across multiple devices.\n- __3D parallelism__: Seamlessly combine data, pipeline, and tensor parallelism.\n- __ZeRO sharding__: Support for optimizer state partitioning (ZeRO-1) in data parallel training regimens.\n- __Efficient training__: Support for modern performance optimizations such as mixed precision training and activation checkpointing.\n- __Code quality standards__: Rigorous typing, Pydantic classes and extensive tests for ease of development and less potential for bugs.\n\nBuilt upon the Scaling core components, the Transformer module implements a state-of-the-art transformer architecture and training loop.\nAmong the featured architecture options we support:\n\n- Multi-query and grouped-query attention,\n- Different MLP types (e.g., SwiGLU),\n- Rotary positional embeddings,\n- Parameter-efficient fine-tuning methods: Bitfit, Adapters, LoRA.\n\n# Getting Started\n\n## Installation\n\nThe installation requires Linux with Python 3.10 and PyTorch 2.4.0.\nYou will also need the appropriate CUDA dependencies and version installed on your system for GPU support.\nClone this repository and install via [poetry](https://python-poetry.org/docs/):\n\n```bash\npoetry install\n```\n\nSee also the \"Development\" section below for additional, optional steps.\n\n### Flash Attention\n\nTo install Flash Attention, you need to make sure you have PyTorch installed already.\nSimply install the base depenendencies with `pip install .` before installing Flash Attention.\nThen install Flash Attention with:\n\n```bash\npoetry run pip install --no-build-isolation flash-attn==2.4.2\n```\n\nEnsure that your environment variables are set correctly.\nThe `CUDA_HOME` variable should point to the location of your CUDA installation.\nFor additional information or troubleshooting, please refer to [the official documentation](https://github.com/Dao-AILab/flash-attention).\nYou can then use Flash Attention in your Transformer architecture configuration:\n\n```json\n{\n    \"transformer_architecture\": {\n        ...\n        \"masked_softmax\": {\n            \"kernel\": \"flash_attention\",\n            \"softmax_in_fp32\": true,\n            \"deterministic_flash_attn_bwd\": false,\n            \"scale\": 1.0,\n        },\n        ...\n    }\n}\n```\n\n## Quick Start\n\nEverything you need to start a full distributed transformer training is contained in [this example](/examples/transformer_example/).\n\nYou can start a training job by executing:\n\n```bash\npython3 -m examples.transformer_example.run examples/transformer_example/config.yml\n```\n\nFeel free to experiment with the [example config](/examples/transformer_example/config.yml) that controls all relevant training parameters and modify it to suit your needs.\nIn particular, update the topology configuration to reflect the amount of GPU devices available.\nFor instance, if you have a single GPU device available, set the topology parameters accordingly.\n\n```json\n{\n    ...\n    \"topology\": {\n        ...\n        \"model_parallel_size\": 1,\n        \"pipe_parallel_size\": 1,\n        \"data_parallel_size\": 1,\n        ...\n    }\n}\n```\n\nNote: The number of available GPU devices needs to be equal to ```model_parallel_size * pipe_parallel_size * data_parallel_size```. To control this, you can simply set the ```CUDA_VISIBLE_DEVICES``` environment variable to the desired GPU indices.\n\nIf you want to run a large-scale job on a cluster, you can in principle use the same code, but make sure the training script gets executed in parallel. We also provide the tooling to do this. For an in-depth look, check out our more detailed guide on [how to train a model on multiple nodes](/examples/tutorials/multi_node_training.md).\nScaling also features [a basic inference module](/examples/tutorials/inference.md) to generate outputs from model checkpoints.\n\n### A 3D Parallel MNIST MLP Training\n\nIf you are interested to learn more about how to build your own training library using Scaling, check out our [MLP example](/examples/mlp_example/).\nThe MNIST MLP classifier is probably the most used example across myriads of Deep Learning tutorials, and Scaling makes no exception.\nThis is a self-contained codebase built on Scaling that implements a full 3D parallel training loop for MNIST classification that is as simple as it gets while touching upon all important components in ```scaling.core```.\nAt the end of the day, our transformer training suite ```scaling.transformer``` is built in the very same fashion.\nThe MLP example is the best way to start if you want to learn about how to use the building blocks from ```scaling.core``` without getting lost in the details of a complex model architecture.\n\n# Development\n\nPlease install pre-commit hooks:\n\n```bash\npre-commit install\n```\n\n## Mypy\n\nRun mypy to catch typing mistakes:\n\n```\nmypy src\nmypy tests\n```\n\n## Tests\n\nRun tests with:\n\n```\npytest tests/core\npytest tests/transformer\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faleph-alpha%2Fscaling","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faleph-alpha%2Fscaling","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faleph-alpha%2Fscaling/lists"}