{"id":13477956,"url":"https://github.com/pytorch/PiPPy","last_synced_at":"2025-03-27T07:30:33.605Z","repository":{"id":37338636,"uuid":"442014578","full_name":"pytorch/PiPPy","owner":"pytorch","description":"Pipeline Parallelism for PyTorch","archived":false,"fork":false,"pushed_at":"2024-08-21T16:31:25.000Z","size":4208,"stargazers_count":757,"open_issues_count":169,"forks_count":88,"subscribers_count":33,"default_branch":"main","last_synced_at":"2025-03-17T11:54:02.822Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pytorch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":"CITATION","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-12-27T00:54:06.000Z","updated_at":"2025-03-12T10:08:33.000Z","dependencies_parsed_at":"2024-03-30T03:24:26.355Z","dependency_job_id":"64664efd-5eee-4909-a9a8-deb0e2f3c22c","html_url":"https://github.com/pytorch/PiPPy","commit_stats":{"total_commits":696,"total_committers":44,"mean_commits":"15.818181818181818","dds":0.7054597701149425,"last_synced_commit":"1bcb2bfb2d6cc4ac2125c0edb37c35585bb9695f"},"previous_names":["pytorch/tau"],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pytorch%2FPiPPy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pytorch%2FPiPPy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pytorch%2FPiPPy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pytorch%2FPiPPy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pytorch","download_url":"https://codeload.github.com/pytorch/PiPPy/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245802120,"owners_count":20674589,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T16:01:50.326Z","updated_at":"2025-03-27T07:30:33.066Z","avatar_url":"https://github.com/pytorch.png","language":"Python","funding_links":[],"categories":["Python","Open Source Projects"],"sub_categories":["3. Open Source Auto-Parallelism Framework"],"readme":"# PiPPy: Pipeline Parallelism for PyTorch\n\n\u003e [!NOTE]\n\u003e PiPPy has been migrated into [PyTorch](https://github.com/pytorch/pytorch) as a subpackage: [`torch.distributed.pipelining`](https://github.com/pytorch/pytorch/tree/main/torch/distributed/pipelining). You can find the detailed documentation [here](https://pytorch.org/docs/main/distributed.pipelining.html). The current repo mainly serves as a land of [examples](examples/). The PiPPy library code will be removed. Please use the APIs in `torch.distributed.pipelining` instead. Thank you!\n\n[**Why PiPPy?**](#why-pippy)\n| [**Install guide**](#install)\n| [**Examples**](#examples)\n| [**PiPPy Explained**](#pippy-explained)\n\n# Why PiPPy?\n\nOne of the most important techniques for advancing the state of the art in deep learning is scaling. Common techniques for scaling neural networks include _data parallelism_, _tensor/operation parallelism_, and _pipeline parallelism_. In many cases, pipeline parallelism in particular can be an effective technique for scaling, however it is often difficult to implement, requiring intrusive code changes to model code and difficult-to-implement runtime orchestration code. PiPPy aims to provide a toolkit that does said things automatically to allow high-productivity scaling of models.\n\n# What is PiPPy?\n\nThe PiPPy project consists of a compiler and runtime stack for automated parallelism and scaling of PyTorch models. Currently, PiPPy focuses on _pipeline parallelism_, a technique in which the code of the model is partitioned and multiple _micro-batches_ execute different parts of the model code concurrently. To learn more about pipeline parallelism, see [this article](https://www.deepspeed.ai/tutorials/pipeline/).\n\n![pipeline_diagram_web](https://github.com/pytorch/PiPPy/assets/6676466/c93e2fe7-1cd4-49a2-9fd8-231ec9905e0c)\n\nFigure: Pipeline parallel. \"F\", \"B\" and \"U\" denote forward, backward and weight update, respectively. Different colors represent different micro-batches.\n\nPiPPy provides the following features that make pipeline parallelism easier:\n\n* Automatic splitting of model code by tracing the model. The goal is for the user to provide model code as-is to the system for parallelization, without having to make heavyweight modifications to make parallelism work.\n* Related to the last point, PiPPy supports non-trivial topologies, including skip connections and tied weights/layers. PiPPy provides configurable behavior for tied weights, allowing for transmission across pipeline stages or replication and gradient synchronization.\n* First-class support for cross-host pipeline parallelism, as this is where PP is typically used (over slower interconnects). This is currently missing from the torchgpipe-based `torch.distributed.pipeline.sync.Pipe`.\n* Composability with other parallelism schemes such as data parallelism or tensor splitting model parallelism (overall, known as \"3d parallelism\"). Currently, pipelining and data parallelism can be composed. Other compositions will be available in the future.\n* Support for pipeline scheduling paradigms, including schedules like fill-drain (GPipe), 1F1B and interleaved 1F1B. More schedules will be added too.\n\nFor in-depth technical architecture, see [ARCHITECTURE.md](ARCHITECTURE.md).\n\n# Install\n\nPiPPy requires PyTorch version newer than 2.2.0.dev to work. To quickly install, for example, PyTorch nightly, run the following command from the same directory as this README:\n\n```\npip install -r requirements.txt --find-links https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html\n```\n\nYou can also select the CUDA build of PyTorch if your system has NVIDIA GPUs, for example:\n\n```\npip install -r requirements.txt --find-links https://download.pytorch.org/whl/nightly/cu118/torch_nightly.html\n```\n\nTo install PiPPy from source, run the following command in the same directory as this README:\n\n```\npython setup.py install\n```\n\nTo expose PiPPy for development such that changes to this repo are reflected in the imported package, run:\n\n```\npython setup.py develop\n```\n\n# Examples\n\nIn this repo, we provide rich examples based on realistic models. In particular, we show how to apply PiPPy without any code change to the model. Please refer to the [HuggingFace examples directory](examples/huggingface/). Examples include: [BERT](examples/huggingface/pippy_bert.py), [GPT2](examples/huggingface/pippy_gpt2.py), [T5](examples/huggingface/pippy_t5.py), [LLaMA](examples/llama/), etc.\n\n# PiPPy Explained\n\nPiPPy consists of two parts: a _compiler_ and a _runtime_. The compiler takes your model code, splits it up, and transforms it into a `Pipe`, which is a wrapper that describes the model at each pipeline stage and their data-flow relationship. The runtime executes the `PipelineStage`s in parallel, handling things like micro-batch splitting, scheduling, communication, and gradient propagation, etc. We will cover the APIs for these concepts in this section.\n\n## Splitting a Model with Pipe\n\nTo see how we can split a model into a pipeline, let's first take an example trivial neural network:\n\n```python\nimport torch\n\nclass MyNetworkBlock(torch.nn.Module):\n    def __init__(self, in_dim, out_dim):\n        super().__init__()\n        self.lin = torch.nn.Linear(in_dim, out_dim)\n\n    def forward(self, x):\n        x = self.lin(x)\n        x = torch.relu(x)\n        return x\n\n\nclass MyNetwork(torch.nn.Module):\n    def __init__(self, in_dim, layer_dims):\n        super().__init__()\n\n        prev_dim = in_dim\n        for i, dim in enumerate(layer_dims):\n            setattr(self, f'layer{i}', MyNetworkBlock(prev_dim, dim))\n            prev_dim = dim\n\n        self.num_layers = len(layer_dims)\n        # 10 output classes\n        self.output_proj = torch.nn.Linear(layer_dims[-1], 10)\n\n    def forward(self, x):\n        for i in range(self.num_layers):\n            x = getattr(self, f'layer{i}')(x)\n\n        return self.output_proj(x)\n\n\nin_dim = 512\nlayer_dims = [512, 1024, 256]\nmn = MyNetwork(in_dim, layer_dims).to(device)\n```\n\nThis network is written as free-form Python code; it has not been modified for any specific parallelism technique.\n\nLet us see our first usage of the `pippy.Pipe` interface:\n\n```python\nfrom pippy import pipeline, annotate_split_points, Pipe, SplitPoint\n\nannotate_split_points(mn, {'layer0': SplitPoint.END,\n                           'layer1': SplitPoint.END})\n\nbatch_size = 32\nexample_input = torch.randn(batch_size, in_dim, device=device)\nchunks = 4\n\npipe = pipeline(mn, chunks, example_args=(example_input,))\nprint(pipe)\n\n\"\"\"\n************************************* pipe *************************************\nGraphModule(\n  (submod_0): PipeStageModule(\n    (L__self___layer0_mod_lin): Linear(in_features=512, out_features=512, bias=True)\n  )\n  (submod_1): PipeStageModule(\n    (L__self___layer1_mod_lin): Linear(in_features=512, out_features=1024, bias=True)\n  )\n  (submod_2): PipeStageModule(\n    (L__self___layer2_lin): Linear(in_features=1024, out_features=256, bias=True)\n    (L__self___output_proj): Linear(in_features=256, out_features=10, bias=True)\n  )\n)\n\ndef forward(self, arg0):\n    submod_0 = self.submod_0(arg0);  arg0 = None\n    submod_1 = self.submod_1(submod_0);  submod_0 = None\n    submod_2 = self.submod_2(submod_1);  submod_1 = None\n    return [submod_2]\n\"\"\"\n```\n\nSo what's going on here? First, `pipeline` turns our model into a directed acyclic graph (DAG) by tracing the model. Then, it groups together the operations and parameters into _pipeline stages_. Stages are represented as `submod_N` submodules, where `N` is a natural number.\n\nWe used `annotate_split_points` to specify that the code should be split and the end of `layer0` and `layer1`. Our code has thus been split into _three_ pipeline stages. PiPPy also provides `SplitPoint.BEGINNING` if a user wants to split before certain annotation point.\n\nWhile the `annotate_split_points` API gives users a way to specify the split points without modifying the model, PiPPy also provides an API for in-model annotation: `pipe_split()`. For details, you can read [this example](https://github.com/pytorch/PiPPy/blob/main/test/test_pipe.py).\n\nThis covers the basic usage of the `Pipe` API. For more information, see the documentation.\n\n\u003c!-- (TODO: link to docs when live) --\u003e\n\n## Using PipelineStage for Pipelined Execution\n\nGiven the above `Pipe` object, we can use one of the `PipelineStage` classes to execute our model in a pipelined fashion. First off, let us instantiate a `PipelineStage` instance:\n\n```python\n# We are using `torchrun` to run this example with multiple processes.\n# `torchrun` defines two environment variables: `RANK` and `WORLD_SIZE`.\nrank = int(os.environ[\"RANK\"])\nworld_size = int(os.environ[\"WORLD_SIZE\"])\n\n# Initialize distributed environment\nimport torch.distributed as dist\ndist.init_process_group(rank=rank, world_size=world_size)\n\n# Pipeline stage is our main pipeline runtime. It takes in the pipe object,\n# the rank of this process, and the device.\nfrom pippy.PipelineStage import PipelineStage\nstage = PipelineStage(pipe, rank, device)\n```\n\nWe can now run the pipeline by passing input to the first `PipelineStage`:\n\n```python\n# Input data\nx = torch.randn(batch_size, in_dim, device=device)\n\n# Run the pipeline with input `x`. Divide the batch into 4 micro-batches\n# and run them in parallel on the pipeline\nif rank == 0:\n    stage(x)\nelif rank == world_size - 1:\n    output = stage()\nelse:\n    stage()\n```\n\nNote that since we split our model into three stages, we must run this script with three workers. For this example, we will use `torchrun` to run multiple processes within a single machine for demonstration purposes. We can collect up all of the code blocks above into a file named [example.py](examples/basic/example.py) and then run it with `torchrun` like so:\n\n```\ntorchrun --nproc_per_node=3 example.py\n```\n\n## License\nPiPPy is 3-clause BSD licensed, as found in the LICENSE file.\n\n## Citing PiPPy\n\nIf you use PiPPy in your publication, please cite it by using the following BibTeX entry.\n\n```bibtex\n@Misc{pippy2022,\n  author =       {James Reed, Pavel Belevich, Ke Wen, Howard Huang, Will Constable},\n  title =        {PiPPy: Pipeline Parallelism for PyTorch},\n  howpublished = {\\url{https://github.com/pytorch/PiPPy}},\n  year =         {2022}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpytorch%2FPiPPy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpytorch%2FPiPPy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpytorch%2FPiPPy/lists"}