https://github.com/ivangabriele/docker-llm
Pre-loaded LLMs served as an OpenAI-Compatible API via Docker images.
https://github.com/ivangabriele/docker-llm
api docker docker-image llm llms llong lmsys openai openai-api openorca orca runpod server vast vicuna vllm
Last synced: about 2 months ago
JSON representation
Pre-loaded LLMs served as an OpenAI-Compatible API via Docker images.
- Host: GitHub
- URL: https://github.com/ivangabriele/docker-llm
- Owner: ivangabriele
- License: agpl-3.0
- Created: 2023-10-15T17:07:33.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2023-10-31T07:51:44.000Z (over 1 year ago)
- Last Synced: 2025-02-27T04:11:42.049Z (2 months ago)
- Topics: api, docker, docker-image, llm, llms, llong, lmsys, openai, openai-api, openorca, orca, runpod, server, vast, vicuna, vllm
- Language: Dockerfile
- Homepage:
- Size: 199 KB
- Stars: 5
- Watchers: 1
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.hub.md
- Contributing: CONTRIBUTING.md
- License: LICENSE.md
- Code of conduct: CODE_OF_CONDUCT.md
- Security: SECURITY.md
Awesome Lists containing this project
README
# OpenAI API-Compatible Pre-loaded LLM Server
[![img-github]][link-github]
[![img-docker]][link-docker]
- Model: [lmsys/vicuna-13b-v1.5-16k](https://huggingface.co/lmsys/vicuna-13b-v1.5-16k).
- OpenAI API-Compatible port: `8000`.
- Update `TENSOR_PARALLEL_SIZE` env to match your GPU count if your server has more than 1 GPU. Default: `1`.
- Configuration:
```json
{
"model": "./model", // because this is where this model is pre-loaded
"api_key": "None",
"api_base": "https://[YOUR_POD_ID]-8000.proxy.runpod.net/v1", # RunPod.io
"api_base": "http://[YOUR_INSTANCE_IP]:[YOUR_PUBLIC_8000_BINDING_PORT]/v1", # Vast.ai
"api_type": "open_ai"
}
```
Vast.ai will automatically bind you port `8000` to a random public port. You can find it in your instance details.
You must replace `[YOUR_PUBLIC_8000_BINDING_PORT]` with this random public port.
---
[img-docker]: https://img.shields.io/docker/pulls/ivangabriele/llm?style=for-the-badge
[img-github]: https://img.shields.io/badge/Github-Repo-black?logo=github&style=for-the-badge
[link-docker]: https://hub.docker.com/r/ivangabriele/llm
[link-github]: https://github.com/ivangabriele/docker-llm