Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/entelecheia/vllm-container

Container images for running vLLM models on the container platforms.
https://github.com/entelecheia/vllm-container

Last synced: 17 days ago
JSON representation

Container images for running vLLM models on the container platforms.

Awesome Lists containing this project

README 

        
# vLLM Container



Container images for running vLLM models on the container platforms.



- GitHub: [https://github.com/entelecheia/vllm-container][repo-url]



LLM is a fast and easy-to-use library for LLM inference and serving.



vLLM is fast with:



- State-of-the-art serving throughput

- Efficient management of attention key and value memory with **PagedAttention**

- Continuous batching of incoming requests

- Optimized CUDA kernels



vLLM is flexible and easy to use with:



- Seamless integration with popular Hugging Face models

- High-throughput serving with various decoding algorithms, including _parallel sampling_, _beam search_, and more

- Tensor parallelism support for distributed inference

- Streaming outputs

- OpenAI-compatible API server



## Prerequisites



- Docker

- Docker Compose

- NVIDIA Docker (for GPU support, Optional)



## Build and Run



1. Build the base Docker image:



   ```bash

   make docker-build-base

   ```



   The base Docker image are used to build other Docker images. It installs all the dependencies required by the project.



   The `docker.base.env` file includes various configuration options and environment variables. The `docker-compose.base.yaml` file uses these variables to customize the behavior of the services. This is a common practice when you want to set different configurations for development, testing, and production environments. The `Dockerfile.base` file uses these variables to customize the Docker build. These files are automatically generated by Copier when you run the `copier copy` command.



2. Build the application Docker image:



   ```bash

   make docker-build-app

   ```



   The application Docker image is used to run the application. It installs the application and its dependencies.



   The `docker.app.env` file includes various configuration options and environment variables. The `docker-compose.app.yaml` file uses these variables to customize the behavior of the services. This is a common practice when you want to set different configurations for development, testing, and production environments. The `Dockerfile.app` file uses these variables to customize the Docker build. These files are automatically generated by Copier when you run the `copier copy` command.



3. Start the Docker container:



   ```bash

   make docker-up-app

   ```



   This will start a Docker container with the image built in the previous step. The container will run a bash lauch script that starts the application.



## Usage



After starting the container, you can access the application at `localhost:`. By default, the port is set to `13084`.



You can also SSH into the container using the SSH port specified in `APP_HOST_SSH_PORT`. By default, the port is set to `2324`.



## Volumes



The `docker-compose.app.yaml` file specifies several volumes that bind mount directories on the host to directories in the container. These include the cache, the workspace directory, and the scripts directory. Changes made in these directories will persist across container restarts.



## Troubleshooting



If you encounter any issues with this setup, please check the following:



- Make sure Docker and Docker Compose are installed correctly.

- Make sure NVIDIA Docker is installed if you're planning to use GPU acceleration.

- Ensure the environment variables in the `docker.app.env` file are correctly set.

- Check the Docker and Docker Compose logs for any error messages.



## Environment Variables



In Docker, environment variables can be used in the `docker-compose.app.yaml` file to customize the behavior of the services.



The `docker-compose` command has an `--env-file` argument, but it's used to set the environment variables for the services defined in the `docker-compose.yaml` file, not for the `docker-compose` command itself. The variables defined in the `--env-file` will not be substituted into the `docker-compose.yaml` file.



However, the environment variables we set in the `.docker/docker.app.env` file are used in the `docker-compose.app.yaml` file. For example, the `$BUILD_FROM` variable is used to set the base image for the Docker build. Therefore, we need to export these variables to the shell environment before running the `docker-compose` command.



This method also allows us to use shell commands in the variable definitions, like `"$(whoami)"` for the `USERNAME` variable, which wouldn't be possible if we used the `--env-file` argument. Shell commands in the `.env` file are not evaluated.



### Files for Environment Variables



- `.docker/docker.common.env`: Common environment variables for all Docker images.

- `.docker/docker.base.env`: Environment variables for the base Docker image.

- `.docker/docker.app.env`: Environment variables for the application Docker image.

- `.env.secret`: Secret environment variables that are not committed to the repository.



## License



This project is released under the [MIT License][license-url].



[license-image]: https://img.shields.io/github/license/entelecheia/vllm-container

[license-url]: https://github.com/entelecheia/vllm-container/blob/main/LICENSE

[version-image]: https://img.shields.io/github/v/release/entelecheia/vllm-container?sort=semver

[release-date-image]: https://img.shields.io/github/release-date/entelecheia/vllm-container

[release-url]: https://github.com/entelecheia/vllm-container/releases

[repo-url]: https://github.com/entelecheia/vllm-container

[changelog]: https://github.com/entelecheia/vllm-container/blob/main/CHANGELOG.md

[contributing guidelines]: https://github.com/entelecheia/vllm-container/blob/main/CONTRIBUTING.md