https://github.com/vaniisgh/tinny_chat

A tiny rag chat bot, that runs completely locally with no API keys etc! Can be configured to use any ollama model, current config uses tinyllama hence, tinnychat.
https://github.com/vaniisgh/tinny_chat

Last synced: 12 days ago
JSON representation

A tiny rag chat bot, that runs completely locally with no API keys etc! Can be configured to use any ollama model, current config uses tinyllama hence, tinnychat.

• Host: GitHub
• URL: https://github.com/vaniisgh/tinny_chat
• Owner: vaniisgh
• Created: 2024-11-06T19:26:25.000Z (3 months ago)
• Default Branch: main
• Last Pushed: 2024-11-06T19:44:37.000Z (3 months ago)
• Last Synced: 2024-11-06T20:36:05.778Z (3 months ago)
• Language: Python
• Size: 8.79 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
## Tinny QA bot



  



A completely local solution to querying your doccuments, it has a single endpoint

    - /query_document

that helps you query a doccument using a llm ( by default tinyllama), you can modify

the llm and prompt from the main.py file.

use the /docs endpoint of fastapi to see the required input and output parameters.

Use any PDF file, or a JSON in the example format in [sample_data.json](./sample_data.json) as the input document, and send the questions in the [sample_questions.json](./sample_questions.json) format.

Note: every query runs a separate `invoke` for the model, more queries 

slower response and pulling the model also takes some time.

### Run the dockerised application

```

docker compose up --build -d

```

for more instructions refer to [screenshots](./additional_screenshots.md)

## Running without docker

You need the following services to run the QA bot:

- Chroma DB

- Ollama Docker container

- This service

You  can use the steps below to start the server

```shell

cd path/to/this/repo

pip install -r requirements.txt

```

reanme [./confing_no_docker.ini] to ./config.ini to run wihtout docker compose

run chromadb, if you change the port please chage in the config.ini[./config.ini]

file as well.

```shell

chroma run --host localhost --port 8000 --path ./my_chroma_data

```

run the current app

```

fastapi dev main.py --port 8080 --reload

```

start your docker desktop and then run the follwoing to

get your ollama ready !

```shell

docker pull ollama/ollama

docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

docker ps

```

copy the name of the docker container

```shell

docker exec -it [container-name] bash

ollama pull tinyllama && exit

```