https://github.com/substratusai/vllm-docker
https://github.com/substratusai/vllm-docker
Last synced: about 1 month ago
JSON representation
- Host: GitHub
- URL: https://github.com/substratusai/vllm-docker
- Owner: substratusai
- License: apache-2.0
- Created: 2023-10-23T05:31:31.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-01-04T00:09:39.000Z (5 months ago)
- Last Synced: 2025-03-25T10:51:17.243Z (about 2 months ago)
- Language: Shell
- Size: 134 KB
- Stars: 55
- Watchers: 3
- Forks: 8
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# vLLM Docker Container Image
vLLM is a fast and easy-to-use library for LLM inference and serving.
This container image runs the OpenAI API server of vLLM.
The image is only for TPU and CPU inference. For GPU inference, please use the upstream image from vLLM.
Image URLs:
- `substratusai/vllm` (Docker Hub)
- `ghcr.io/substratusai/vllm` (GitHub Container Registry)
This image only publishes the TPU and CPU images for vLLM:
- `substratusai/vllm:main-tpu`
- `substratusai/vllm:main-cpu`
There are also tags available such as `v0.6.3-tpu` and `v0.6.3-cpu`.
Please use the upstream GPU image from vLLM directly:
```
vllm/vllm-openai:latest
```
Support the project by adding a star! ❤️
Join us on Discord:
## Quickstart
Deploy Mistral 7B Instruct using Docker:
```bash
docker run -d -p 8080:8080 --gpus=all \
substratusai/vllm \
--model=mistralai/Mistral-7B-Instruct-v0.1
```
## Building
```
docker build -t ghcr.io/substratusai/vllm .
```