Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/aidatatools/llm_sentinel
A project (LLM Sentinel) that showcases NVIDIA's NeMo-Guardrails and LangChain for improving LLM safety
https://github.com/aidatatools/llm_sentinel
llm-inference llms safety
Last synced: 13 days ago
JSON representation
A project (LLM Sentinel) that showcases NVIDIA's NeMo-Guardrails and LangChain for improving LLM safety
- Host: GitHub
- URL: https://github.com/aidatatools/llm_sentinel
- Owner: aidatatools
- License: mit
- Created: 2024-06-15T07:27:56.000Z (7 months ago)
- Default Branch: main
- Last Pushed: 2024-09-03T01:16:58.000Z (4 months ago)
- Last Synced: 2024-11-05T13:26:55.188Z (2 months ago)
- Topics: llm-inference, llms, safety
- Language: Python
- Homepage:
- Size: 316 KB
- Stars: 6
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# LLM Sentinel (NeMoLangChainOllamaGradio)
**LLM Sentinel** (NeMo LangChain Ollama Gradio) for NVIDIA GenAI Contest
## Desired and Required Libraries
- nemoguardrails==0.9.0
- langchain-community==0.0.38
- ollama==0.2.1
- gradio==4.42.0
- python-dotenv==1.0.1
It's tested on Python 3.9 and above on macOS, and Ubuntu Linux
## Step 1: Set Up Your Environment
1. **Hardware Requirements**: Ensure you have access to NVIDIA GPUs, ideally A100 80GB VRAM, to run the model (Llama3:70b) efficiently. In my case I rent A100 GPU from Digital Ocean Paperspace. Please see the screenshot. OS: Ubuntu 22.04 Disk Size: At least 200 GB (llama3:70b)-> 40GB, (llama3:8b)-> 5GB
```bash
ssh [email protected]
```
1. First git clone the repository
```bash
cd ~
git clone https://github.com/aidatatools/LLM_Sentinel.git
cd LLM_Sentinel
```
1. **venv**:
Ensure you have Python 3.10 or later installed.
```bash
cd ~/LLM_Sentinel
python3.10 -m venv venv
source venv/bin/activate
```
1. Install requirements.txt
```bash
pip install -r requirements.txt
```
1. Check the backend **ollama** service is running, and the model (llama3:8b)(for DEV) or (llama3:70b)(for Production) exists. If you are not familiar with ollama, please visit
```bash
ollama list
curl http://127.0.0.1:11434
```
1. Copy .env.example to .env and set the variable(ENV_PROD) to True or False
```bash
echo 'ENV_PROD=False' > .env
```
## Step 2: Start the Web UI to Interact with Chatbot
1. Start the WebUI in terminal:
```bash
python chatbot3.py
```
1. Open a browser, and visit the site with port number:
## Reference
- [Safeguarding LLMs with Guardrails](https://towardsdatascience.com/safeguarding-llms-with-guardrails-4f5d9f57cff2)
- [LlamaGuard-based Moderation Rails Performance](https://github.com/NVIDIA/NeMo-Guardrails/blob/develop/docs/evaluation/README.md#llamaguard-based-moderation-rails-performance)
-
- [Project description for LLM Sentinel](https://medium.com/aidatatools/llm-sentinel-a-project-which-can-make-the-llm-chatbot-safer-250e40b110fe)