Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/ashleykleynhans/llava-docker
Docker image for LLaVA: Large Language and Vision Assistant
https://github.com/ashleykleynhans/llava-docker
ai chatbot chatgpt docker docker-image foundation-models gpt-4 instruction-tuning llama llama-2 llama2 llava llm multimodal runpod vision-language-model visual-language-learning
Last synced: 3 months ago
JSON representation
Docker image for LLaVA: Large Language and Vision Assistant
- Host: GitHub
- URL: https://github.com/ashleykleynhans/llava-docker
- Owner: ashleykleynhans
- License: gpl-3.0
- Created: 2023-10-10T22:54:55.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2024-04-19T14:36:48.000Z (7 months ago)
- Last Synced: 2024-05-02T01:49:37.649Z (6 months ago)
- Topics: ai, chatbot, chatgpt, docker, docker-image, foundation-models, gpt-4, instruction-tuning, llama, llama-2, llama2, llava, llm, multimodal, runpod, vision-language-model, visual-language-learning
- Language: Shell
- Homepage:
- Size: 112 KB
- Stars: 61
- Watchers: 2
- Forks: 13
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- Funding: .github/FUNDING.yml
- License: LICENSE
Awesome Lists containing this project
README
# Docker image for LLaVA: Large Language and Vision Assistant
## Installs
* Ubuntu 22.04 LTS
* CUDA 12.1
* Python 3.10.12
* [LLaVA](
https://github.com/haotian-liu/llava) v1.2.0 (LLaVA 1.6)
* Torch 2.1.2
* xformers 0.0.23.post1
* [Jupyter Lab](https://github.com/jupyterlab/jupyterlab)
* [code-server](https://github.com/coder/code-server)
* [runpodctl](https://github.com/runpod/runpodctl)
* [OhMyRunPod](https://github.com/kodxana/OhMyRunPod)
* [RunPod File Uploader](https://github.com/kodxana/RunPod-FilleUploader)
* [croc](https://github.com/schollz/croc)
* [rclone](https://rclone.org/)
* speedtest-cli
* screen
* tmux
* llava-v1.6-mistral-7b model
## Available on RunPod
This image is designed to work on [RunPod](https://runpod.io?ref=2xxro4sy).
You can use my custom [RunPod template](
https://runpod.io/gsc?template=g7wd33iuwv&ref=2xxro4sy)
to launch it on RunPod.
## Building the Docker image
> [!NOTE]
> You will need to edit the `docker-bake.hcl` file and update `USERNAME`,
> and `RELEASE`. You can obviously edit the other values too, but these
> are the most important ones.
```bash
# Clone the repo
git clone https://github.com/ashleykleynhans/llava-docker.git
# Log in to Docker Hub
docker login
# Build the image, tag the image, and push the image to Docker Hub
cd llava-docker
docker buildx bake -f docker-bake.hcl --push
```
## Running Locally
### Install Nvidia CUDA Driver
- [Linux](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html)
- [Windows](https://docs.nvidia.com/cuda/cuda-installation-guide-microsoft-windows/index.html)
### Start the Docker container
```bash
docker run -d \
--gpus all \
-v /workspace \
-p 3000:3001 \
-p 7777:7777 \
-p 8888:8888 \
-p 2999:2999 \
ashleykza/llava:latest
```
You can obviously substitute the image name and tag with your own.
## Models
> [!IMPORTANT]
> If you select a 13B or larger model, CUDA will result in OOM errors
> with a GPU that has less than 48GB of VRAM, so A6000 or higher is
> recommended for 13B.
You can add an environment called `MODEL` to your Docker container to
specify the model that should be downloaded. If the `MODEL` environment
variable is not set, the model will default to `liuhaotian/llava-v1.6-mistral-7b`.
### LLaVA-v1.6
| Model | Environment Variable Value | Version | LLM | Default |
|----------------------------------------------------------------------------------|----------------------------------|------------|---------------|---------|
| [llava-v1.6-vicuna-7b](https://huggingface.co/liuhaotian/llava-v1.6-vicuna-7b) | liuhaotian/llava-v1.6-vicuna-7b | LLaVA-1.6 | Vicuna-7B | no |
| [llava-v1.6-vicuna-13b](https://huggingface.co/liuhaotian/llava-v1.6-vicuna-13b) | liuhaotian/llava-v1.6-vicuna-13b | LLaVA-1.6 | Vicuna-13B | no |
| [llava-v1.6-mistral-7b](https://huggingface.co/liuhaotian/llava-v1.6-mistral-7b) | liuhaotian/llava-v1.6-mistral-7b | LLaVA-1.6 | Mistral-7B | yes |
| [llava-v1.6-34b](https://huggingface.co/liuhaotian/llava-v1.6-34b) | liuhaotian/llava-v1.6-34b | LLaVA-1.6 | Hermes-Yi-34B | no |
### LLaVA-v1.5
| Model | Environment Variable Value | Version | Size | Default |
|--------------------------------------------------------------------|----------------------------|-----------|------|---------|
| [llava-v1.5-7b](https://huggingface.co/liuhaotian/llava-v1.5-7b) | liuhaotian/llava-v1.5-7b | LLaVA-1.5 | 7B | no |
| [llava-v1.5-13b](https://huggingface.co/liuhaotian/llava-v1.5-13b) | liuhaotian/llava-v1.5-13b | LLaVA-1.5 | 13B | no |
| [BakLLaVA-1](https://huggingface.co/SkunkworksAI/BakLLaVA-1) | SkunkworksAI/BakLLaVA-1 | LLaVA-1.5 | 7B | no |
## Ports
| Connect Port | Internal Port | Description |
|--------------|---------------|----------------------|
| 3000 | 3001 | LLaVA |
| 7777 | 7777 | Code Server |
| 8888 | 8888 | Jupyter Lab |
| 2999 | 2999 | RunPod File Uploader |
## Environment Variables
| Variable | Description | Default |
|----------------------|--------------------------------------------------|----------------------------------|
| JUPYTER_LAB_PASSWORD | Set a password for Jupyter lab | not set - no password |
| DISABLE_AUTOLAUNCH | Disable LLaVA from launching automatically | (not set) |
| DISABLE_SYNC | Disable syncing if using a RunPod network volume | (not set) |
| MODEL | The path of the Huggingface model | liuhaotian/llava-v1.6-mistral-7b |
## Logs
LLaVA creates log files, and you can tail the log files
instead of killing the services to view the logs.
| Application | Log file |
|---------------|----------------------------------|
| Controller | /workspace/logs/controller.log |
| Webserver | /workspace/logs/webserver.log |
| Model Worker | /workspace/logs/model-worker.log |
For example:
```bash
tail -f /workspace/logs/webserver.log
```
## Flask API
### Add port 5000
If you are running the RunPod template, edit your pod and add HTTP port 5000.
If you are running locally, add a port mapping for port 5000.
### Starting the Flask API
```bash
# Stop model worker and controller to free up VRAM
fuser -k 10000/tcp 40000/tcp
# Install dependencies
source /venv/bin/activate
pip3 install flask protobuf
cd /workspace/LLaVA
export HF_HOME="/workspace"
python -m llava.serve.api -H 0.0.0.0 -p 5000
```
### Sending requests to the Flask API
You can use the [test script](
https://github.com/ashleykleynhans/LLaVA/blob/main/llava/serve/test_api.py)
to test your API.
## Acknowledgements
1. Matthew Berman for giving me a demo on LLaVA, as well as his amazing
[YouTube videos](https://www.youtube.com/@matthew_berman/videos]).
## Community and Contributing
Pull requests and issues on [GitHub](https://github.com/ashleykleynhans/llava-docker)
are welcome. Bug fixes and new features are encouraged.
## Appreciate my work?
