{"id":18286609,"url":"https://github.com/aidatatools/llm_sentinel","last_synced_at":"2025-04-12T08:16:37.923Z","repository":{"id":244518387,"uuid":"815447507","full_name":"aidatatools/LLM_Sentinel","owner":"aidatatools","description":"A project (LLM Sentinel) that showcases NVIDIA's NeMo-Guardrails and LangChain for improving LLM safety","archived":false,"fork":false,"pushed_at":"2025-01-22T15:23:57.000Z","size":325,"stargazers_count":7,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-12T08:16:31.749Z","etag":null,"topics":["llm-inference","llms","safety"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aidatatools.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-15T07:27:56.000Z","updated_at":"2025-03-30T16:06:24.000Z","dependencies_parsed_at":"2024-06-16T03:35:45.036Z","dependency_job_id":null,"html_url":"https://github.com/aidatatools/LLM_Sentinel","commit_stats":null,"previous_names":["aidatatools/nemolangchainollamagradio","aidatatools/llm_sentinel"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aidatatools%2FLLM_Sentinel","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aidatatools%2FLLM_Sentinel/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aidatatools%2FLLM_Sentinel/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aidatatools%2FLLM_Sentinel/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aidatatools","download_url":"https://codeload.github.com/aidatatools/LLM_Sentinel/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248537193,"owners_count":21120711,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["llm-inference","llms","safety"],"created_at":"2024-11-05T13:21:14.083Z","updated_at":"2025-04-12T08:16:37.905Z","avatar_url":"https://github.com/aidatatools.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LLM Sentinel (NeMoLangChainOllamaGradio)\n\n**LLM Sentinel** (NeMo LangChain Ollama Gradio) for NVIDIA GenAI Contest\n\n\u003chttps://www.nvidia.com/en-us/ai-data-science/generative-ai/developer-contest-with-langchain/terms-and-conditions/\u003e\n\n## Desired and Required Libraries\n\n- nemoguardrails==0.9.0\n- langchain-community==0.0.38\n- ollama==0.2.1\n- gradio==4.42.0\n- python-dotenv==1.0.1\n\nIt's tested on Python 3.9 and above on macOS, and Ubuntu Linux\n\n## Step 1: Set Up Your Environment\n\n1. **Hardware Requirements**: Ensure you have access to NVIDIA GPUs, ideally A100 80GB VRAM, to run the model (Llama3:70b) efficiently. In my case I rent A100 GPU from Digital Ocean Paperspace. Please see the screenshot. OS: Ubuntu 22.04 Disk Size: At least 200 GB (llama3:70b)-\u003e 40GB, (llama3:8b)-\u003e 5GB\n\n   ![create_a_new_machine](img/create_a_new_machine.png \"create_a_new_machine\")\n\n   ```bash\n   ssh paperspace@XXX.XXX.XXX.XXX\n   ```\n\n1. First git clone the repository\n\n   ```bash\n   cd ~\n   git clone https://github.com/aidatatools/LLM_Sentinel.git\n   cd LLM_Sentinel\n   ```\n\n1. **venv**:\n\n   Ensure you have Python 3.10 or later installed.\n\n   ```bash\n   cd ~/LLM_Sentinel\n   python3.10 -m venv venv\n   source venv/bin/activate\n   ```\n\n1. Install requirements.txt\n\n   ```bash\n   pip install -r requirements.txt\n   ```\n\n1. Check the backend **ollama** service is running, and the model (llama3:8b)(for DEV) or (llama3:70b)(for Production) exists. If you are not familiar with ollama, please visit \u003chttps://ollama.com\u003e\n\n   ```bash\n   ollama list\n   curl http://127.0.0.1:11434\n   ```\n\n1. Copy .env.example to .env and set the variable(ENV_PROD) to True or False\n\n   ```bash\n   echo 'ENV_PROD=False' \u003e .env\n   ```\n\n## Step 2: Start the Web UI to Interact with Chatbot\n\n1. Start the WebUI in terminal:\n\n   ```bash\n   python chatbot3.py\n   ```\n\n1. Open a browser, and visit the site with port number:\n\n   \u003chttp://127.0.0.1:7860\u003e\n\n## Reference\n\n- [Safeguarding LLMs with Guardrails](https://towardsdatascience.com/safeguarding-llms-with-guardrails-4f5d9f57cff2)\n- [LlamaGuard-based Moderation Rails Performance](https://github.com/NVIDIA/NeMo-Guardrails/blob/develop/docs/evaluation/README.md#llamaguard-based-moderation-rails-performance)\n- \u003chttps://github.com/NVIDIA/NeMo-Guardrails\u003e\n- [Project description for LLM Sentinel](https://medium.com/aidatatools/llm-sentinel-a-project-which-can-make-the-llm-chatbot-safer-250e40b110fe)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faidatatools%2Fllm_sentinel","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faidatatools%2Fllm_sentinel","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faidatatools%2Fllm_sentinel/lists"}