{"id":15157711,"url":"https://github.com/pytorch/torchdistx","last_synced_at":"2025-10-19T07:31:35.403Z","repository":{"id":37551967,"uuid":"460136644","full_name":"pytorch/torchdistx","owner":"pytorch","description":"Torch Distributed Experimental","archived":true,"fork":false,"pushed_at":"2024-08-05T02:54:25.000Z","size":1489,"stargazers_count":115,"open_issues_count":19,"forks_count":32,"subscribers_count":25,"default_branch":"main","last_synced_at":"2025-01-21T10:50:44.454Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pytorch.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-02-16T18:52:43.000Z","updated_at":"2025-01-09T23:21:17.000Z","dependencies_parsed_at":"2024-06-19T02:52:00.037Z","dependency_job_id":"ca5ca980-43a1-4000-b750-751ee149d90c","html_url":"https://github.com/pytorch/torchdistx","commit_stats":{"total_commits":58,"total_committers":7,"mean_commits":8.285714285714286,"dds":"0.18965517241379315","last_synced_commit":"9c1b9f5cb2fa36bfb8b70ec07c40ed42a33cc87a"},"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pytorch%2Ftorchdistx","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pytorch%2Ftorchdistx/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pytorch%2Ftorchdistx/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pytorch%2Ftorchdistx/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pytorch","download_url":"https://codeload.github.com/pytorch/torchdistx/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":237088492,"owners_count":19253565,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-26T20:01:50.735Z","updated_at":"2025-10-19T07:31:30.080Z","avatar_url":"https://github.com/pytorch.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# torchdistX - Torch Distributed Experimental\n\n[**Installation**](#installation) | [**Getting Started**](#getting-started) | [**Documentation**](#documentation)\n\nTorch Distributed Experimental, or in short torchdistX, contains a collection of\nexperimental features for which our team wants to gather feedback from our users\nbefore introducing them in the core PyTorch Distributed package. In a sense\nfeatures included in torchdistX can be considered in an incubation period.\n\nPlease be advised though that all features in torchdistX are subject to change\nand, although our team will make its best effort, we do not guarantee any API\nor ABI compatibility between releases. This means you should exercise caution if\nyou plan to use torchdistX in production.\n\nAs of today the following features are available in torchdistX:\n\n- [Fake Tensor](https://pytorch.org/torchdistx/latest/fake_tensor.html)\n- [Deferred Module Initialization](https://pytorch.org/torchdistx/latest/deferred_init.html)\n\n## Dependencies\ntorchdistX versions corresponding to each PyTorch release:\n\n| `torch`      | `torchdistx` | `python`          |\n| ------------ | ------------ | ----------------- |\n| `main`       | `main`       | `\u003e=3.8`, `\u003c=3.10` |\n| `1.12.0`     | `0.2.0`      | `\u003e=3.7`, `\u003c=3.10` |\n| `1.11.0`     | `0.1.0`      | `\u003e=3.7`, `\u003c=3.9`  |\n\n## Installation\nAs of today only Linux and macOS operating systems are supported. Please note\nthat pre-built Conda and PyPI packages are *only* available for Linux though.\nFor installation on macOS you can follow the instructions in the [From Source](#from-source)\nsection. At this time there are no plans to introduce Windows support.\n\n### Conda\nConda is the recommended way to install torchdistX. Running the following\ncommand in a Conda environment will install torchdistX and all its dependencies.\n\n**Stable**\n\nFor PyTorch CPU:\n```\nconda install -c pytorch -c conda-forge torchdistx cpuonly\n```\n\nFor PyTorch with CUDA 10.2:\n```\nconda install -c pytorch -c conda-forge torchdistx cudatoolkit=10.2\n```\n\nFor PyTorch with CUDA 11.3:\n```\nconda install -c pytorch -c conda-forge torchdistx cudatoolkit=11.3\n```\n\nFor PyTorch with CUDA 11.6:\n```\nconda install -c pytorch -c conda-forge torchdistx cudatoolkit=11.6\n```\n\n**Nightly**\n\nFor PyTorch CPU\n```\nconda install -c pytorch-nightly -c conda-forge torchdistx cpuonly\n```\n\nFor PyTorch with CUDA 10.2\n```\nconda install -c pytorch-nightly -c conda-forge torchdistx cudatoolkit=10.2\n```\n\nFor PyTorch with CUDA 11.3\n```\nconda install -c pytorch-nightly -c conda-forge torchdistx cudatoolkit=11.3\n```\n\nFor PyTorch with CUDA 11.6\n```\nconda install -c pytorch-nightly -c conda-forge torchdistx cudatoolkit=11.6\n```\n\nIn fact torchdistX offers several Conda packages that you can install\nindependently based on your needs:\n\n| Package                                                                 | Description                                      |\n|-------------------------------------------------------------------------|--------------------------------------------------|\n| [torchdistx](https://anaconda.org/pytorch/torchdistx)                   | torchdistX Python Library                        |\n| [torchdistx-cc](https://anaconda.org/pytorch/torchdistx-cc)             | torchdistX C++ Runtime Library                   |\n| [torchdistx-cc-devel](https://anaconda.org/pytorch/torchdistx-cc-devel) | torchdistX C++ Runtime Library Development Files |\n| [torchdistx-cc-debug](https://anaconda.org/pytorch/torchdistx-cc-debug) | torchdistX C++ Runtime Library Debug Symbols     |\n\n### PyPI\n\n**Stable**\n\nFor PyTorch CPU:\n```\npip install torchdistx --extra-index-url https://download.pytorch.org/whl/cpu\n```\n\nFor PyTorch with CUDA 10.2:\n```\npip install torchdistx --extra-index-url https://download.pytorch.org/whl/cu102\n```\n\nFor PyTorch with CUDA 11.3:\n```\npip install torchdistx --extra-index-url https://download.pytorch.org/whl/cu113\n```\n\nFor PyTorch with CUDA 11.6:\n```\npip install torchdistx --extra-index-url https://download.pytorch.org/whl/cu116\n```\n\n**Nightly**\n\nFor PyTorch CPU:\n```\npip install torchdistx --pre --extra-index-url https://download.pytorch.org/whl/nightly/cpu\n```\n\nFor PyTorch with CUDA 10.2:\n```\npip install torchdistx --pre --extra-index-url https://download.pytorch.org/whl/nightly/cu102\n```\n\nFor PyTorch with CUDA 11.3:\n```\npip install torchdistx --pre --extra-index-url https://download.pytorch.org/whl/nightly/cu113\n```\n\nFor PyTorch with CUDA 11.6:\n```\npip install torchdistx --pre --extra-index-url https://download.pytorch.org/whl/nightly/cu116\n```\n\n### From Source\n\n#### Prerequisites\n- After cloning the repository make sure to initialize all submodules by\n  executing `git submodule update --init --recursive`.\n- Create a Python virtual environment and install the build dependencies:\n ```\n# Build against PyTorch CPU\npip install --upgrade -r requirements.txt -r use-cpu.txt\n\n# Build against PyTorch with CUDA 10.2\npip install --upgrade -r requirements.txt -r use-cu102.txt\n\n# Build against PyTorch with CUDA 11.3\npip install --upgrade -r requirements.txt -r use-cu113.txt\n\n# Build against PyTorch with CUDA 11.6\npip install --upgrade -r requirements.txt -r use-cu116.txt\n```\n- The build process requires CMake 3.21 or later. You can install an up-to-date\n  version by executing `pip install cmake`. For other environments please refer\n  to your package manager or [cmake.org](https://cmake.org/download/).\n\nOnce you have all prerequisites run the following commands to install the\ntorchdistX Python package:\n\n```\ncmake -DTORCHDIST_INSTALL_STANDALONE=ON -B build\ncmake --build build\npip install .\n```\n\nFor advanced build options you can check out [CMakeLists.txt](./CMakeLists.txt).\n\n#### Development\nIn case you would like to contribute to the project you can slightly modify the\ncommands listed above:\n\n```\ncmake -B build\ncmake --build build\npip install -e .\n```\n\nWith `pip install -e .` you enable the edit mode (a.k.a. develop mode) that\nallows you to modify the Python files in-place without requiring to repeatedly\ninstall the package. If you are working in C++, whenever you modify a header or\nimplementation file, executing `cmake --build build` alone is sufficient. You do\nnot have to call `pip install` again.\n\nThe project also comes with a [requirements-devel.txt](./requirements-devel.txt)\nto set up a Python virtual environment for development.\n\n```\n# Build against PyTorch CPU\npip install --upgrade -r requirements-devel.txt -r use-cpu.txt\n\n# Build against PyTorch with CUDA 10.2\npip install --upgrade -r requirements-devel.txt -r use-cu102.txt\n\n# Build against PyTorch with CUDA 11.3\npip install --upgrade -r requirements-devel.txt -r use-cu113.txt\n\n# Build against PyTorch with CUDA 11.6\npip install --upgrade -r requirements-devel.txt -r use-cu116.txt\n```\n\n#### Tip\nNote that using the Ninja build system and the ccache tool can significatly\nspeed up your build times. To use them you can replace the initial CMake command\nlisted above with the following version:\n\n```\ncmake -GNinja -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -B build\n```\n\n## Getting Started\n\n### Fake Tensor\nFake tensors, similar to meta tensors, carry no data; however, unlike meta\ntensors which report `meta` as their device, fake tensors act as if they were\nallocated on a real device. In the example below we construct two fake tensors\nwith the `fake_mode` context manager.\n\n```python\n\u003e\u003e\u003e import torch\n\u003e\u003e\u003e from torchdistx import fake\n\u003e\u003e\u003e\n\u003e\u003e\u003e with fake.fake_mode():\n...    a = torch.ones([10])\n...    b = torch.ones([20], device=\"cuda\")\n...\n\u003e\u003e\u003e a\ntensor(..., size=(10,), fake=True)\n\u003e\u003e\u003e b\ntensor(..., size=(20,), device=cuda, fake=True)\n```\n\n### Deferred Module Initialization\nThis feature forces all tensors of a module to be constructed as fake while also\nrecording all operations performed on them. The module, its submodules, and its\ntensors can later be materialized by calling the `materialize_module()` and\n`materialize_tensor()` functions.\n\n```python\n\u003e\u003e\u003e import torch\n\u003e\u003e\u003e from torchdistx import deferred_init\n\u003e\u003e\u003e\n\u003e\u003e\u003e m = deferred_init.deferred_init(torch.nn.Linear, 10, 20)\n\u003e\u003e\u003e m.weight\nParameter containing:\ntensor(..., size=(20, 10), requires_grad=True, fake=True)\n\u003e\u003e\u003e\n\u003e\u003e\u003e deferred_init.materialize_module(m)\n\u003e\u003e\u003e m.weight\nParameter containing:\ntensor([[-0.1838, -0.0080,  0.0747, -0.1663, -0.0936,  0.0587,  0.1988, -0.0977,\n         -0.1433,  0.2620],\n       ..., requires_grad=True)\n```\n\n## Documentation\nFor more documentation, see [our docs website](https://pytorch.org/torchdistx/latest).\n\n## Contributing\nPlease refer to [CONTRIBUTING.md](./CONTRIBUTING.md).\n\n## License\nThis project is BSD licensed, as found in the [LICENSE](LICENSE) file.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpytorch%2Ftorchdistx","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpytorch%2Ftorchdistx","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpytorch%2Ftorchdistx/lists"}