https://github.com/dlzou/computron
Serving distributed deep learning models with model parallel swapping.
https://github.com/dlzou/computron
deep-learning inference-server model-parallelism
Last synced: about 1 year ago
JSON representation
Serving distributed deep learning models with model parallel swapping.
- Host: GitHub
- URL: https://github.com/dlzou/computron
- Owner: dlzou
- License: apache-2.0
- Created: 2023-04-09T21:28:42.000Z (about 3 years ago)
- Default Branch: master
- Last Pushed: 2023-06-19T20:03:59.000Z (about 3 years ago)
- Last Synced: 2025-03-27T17:51:49.532Z (about 1 year ago)
- Topics: deep-learning, inference-server, model-parallelism
- Language: Jupyter Notebook
- Homepage:
- Size: 2.1 MB
- Stars: 5
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Computron
## Abstract
Many of the most performant deep learning models today in fields like language and image understanding are fine-tuned models that contain billions of parameters. In anticipation of workloads that involve serving many of such large models to handle different tasks, we develop Computron, a system that uses memory swapping to serve multiple distributed models on a shared GPU cluster. Computron implements a model parallel swapping design that takes advantage of the aggregate CPU-GPU link bandwidth of a cluster to speed up model parameter transfers. This design makes swapping large models feasible and can improve resource utilization. We demonstrate that Computron successfully parallelizes model swapping on multiple GPUs, and we test it on randomized workloads to show how it can tolerate real world variability factors like burstiness and skewed request rates.
## Installation for Development
Clone this repository and its submodules:
```shell
git clone --recurse-submodules git@github.com:dlzou/computron.git
```
Create an environment, install torch and Colossal-AI from PIP, then install Energon-AI and AlpaServe from the included submodules. Finally, install Computron from source.
```shell
conda create -n computron python=3.10
conda activate computron
pip install torch==1.13 torchvision colossalai transformers
pip install -e energonai/
pip install -e alpa_serve/
pip install -e .
```