Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/noco-ai/spellbook-docker

AI stack for interacting with LLMs, Stable Diffusion, Whisper, xTTS and many other AI models
https://github.com/noco-ai/spellbook-docker

automatic-speech-recognition bark llama2 llm-inference mixtral musicgeneration stable-diffusion text-to-speech whisper xttsv2

Last synced: 2 months ago
JSON representation

AI stack for interacting with LLMs, Stable Diffusion, Whisper, xTTS and many other AI models

Host: GitHub
URL: https://github.com/noco-ai/spellbook-docker
Owner: noco-ai
License: osl-3.0
Created: 2023-11-06T19:01:58.000Z (about 1 year ago)
Default Branch: master
Last Pushed: 2024-05-01T15:46:58.000Z (9 months ago)
Last Synced: 2024-05-02T10:51:47.560Z (9 months ago)
Topics: automatic-speech-recognition, bark, llama2, llm-inference, mixtral, musicgeneration, stable-diffusion, text-to-speech, whisper, xttsv2
Language: Shell
Homepage: https://github.com/noco-ai/spellbook-docker/wiki
Size: 2.39 MB
Stars: 105
Watchers: 2
Forks: 5
Open Issues: 2
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

README

# Spellbook Docker Compose

The repository contains the Docker Compose files for running the Spellbook AI Assistant stack. The function calling features
require ExLlama and a Nvidia Ampere or better GPU for real-time results.

![UI demo](https://github.com/noco-ai/spellbook-docker/blob/master/ui-demo.gif)

## Stack Architecture

![Software stack diagram](https://github.com/noco-ai/spellbook-docker/blob/master/stack.png)

## Ubuntu 22 Install Instructions

These instructions should work to get the SpellBook framework up and running on Ubuntu 22. A Nvidia video card supported by ExLlama is required for routing.

- Default username: admin
- Default password: admin

### Docker Installation

```bash
# add Dockers official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

# add the repository to apt sources:
echo \
"deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
"$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

# install docker, create user and let current user access docker
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo groupadd docker
sudo usermod -aG docker $USER
sudo newgrp docker
sudo shutdown -r now
```

### Build and Start Containers (No GPU)

The docker-compose-nogpu.yml is useful for running the UI and middleware in a situation where you want another backend handling you GPUs and LLMs. For example
if you are also using Text Generation UI and do not want to mess with it settings this compose file can be used to just run the UI, allowing you then to then connect it to the endpoint provided by Oobabooga or any other OpenAI compatible backend.

```bash
docker compose -f docker-compose-nogpu.yml build
docker compose -f docker-compose-nogpu.yml up
```

### Build and Start Additional Workers (No GPU)

If you have more than one server you can run additional Elemental Golem workers to give the UI access to more resources. A few steps need to be taken on the
primary Spellbook server that is running the UI, middleware and other resources like Vault.

- Run these command on the primary server that is running the UI and middleware software in Docker.
- Copy the read token to a temp file to copy it to the worker server.
- Make note of the LAN IP address of the primary server. It is needed for the GOLEM_VAULT_HOST and GOLEM_VAULT_HOST variables.
- Make sure ports for RabbitMQ and Vault are open.

```bash
sudo more /var/lib/docker/volumes/spellbook-docker_vault_share/_data/read-token
ip address
sudo ufw allow 5671
sudo ufw allow 5672
sudo ufw allow 8200
```

- Run these command on the server running the worker. The GOLEM_ID needs to be unique for every server and golem1 is used by the primary.
- The first time you run the container it will timeout, git CTL + C or wait then copy the Vault token.
```bash
docker compose -f docker-compose-worker-nogpu.yml build
GOLEM_VAULT_HOST=10.10.10.X GOLEM_AMQP_HOST=10.10.10.X GOLEM_ID=golem2 docker compose -f docker-compose-worker-nogpu.yml up
sudo su
echo "TOKEN FROM PRIMARY SERVER" > /var/lib/docker/volumes/spellbook-docker_vault_share/_data/read-token
exit
GOLEM_VAULT_HOST=10.10.10.X GOLEM_AMQP_HOST=10.10.10.X GOLEM_ID=golem2 docker compose -f docker-compose-worker-nogpu.yml up
```

### Nvidia Driver Installation

```bash
# make sure system see's the Nvidia graphic(s) card
lspci | grep -e VGA

# check available drivers
ubuntu-drivers devices

# install the latest driver
sudo apt install nvidia-driver-535

# restart the server
sudo shutdown -h now

# confirm driver was installed
nvidia-smi

# install the Nvidia docker toolkit
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
&& curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list \
&& \
sudo apt-get update

sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# verify see the output of nvidia-smi for inside a container
sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi
```

### Build and Start Containers (Nvidia GPU)

```bash
docker compose build
docker compose up
```

### Build and Start Additional Workers (Nvidia GPU)

Follow the directions under the *Build and Start Additional Workers (No GPU)* section substituting the build and up lines for the ones found below.
```bash
docker compose -f docker-compose-worker.yml build
GOLEM_VAULT_HOST=10.10.10.X GOLEM_AMQP_HOST=10.10.10.X GOLEM_ID=golem2 docker compose -f docker-compose-worker.yml up
```

### Port Forwarding

This repository assumes you are running the docker containers on your local system if this is not the case make sure ports **3000** and **4200** are forwarded to the host running the docker containers.

### Fresh Install

For a fresh install of the stack run the following commands, this will remove all downloaded models and all conversation and configuration records.

```bash
cd spellbook-docker
docker compose down
docker volume rm spellbook-docker_models_share
docker volume rm spellbook-docker_vault_share
git pull origin master
docker compose build
docker compose up
```