{"id":20685404,"url":"https://github.com/yas-sim/openvino-llm-chatbot-rag","last_synced_at":"2025-04-22T13:39:26.028Z","repository":{"id":210648201,"uuid":"727110268","full_name":"yas-sim/openvino-llm-chatbot-rag","owner":"yas-sim","description":"LLM chatbot example using OpenVINO with RAG (Retrieval Augmented Generation).","archived":false,"fork":false,"pushed_at":"2024-01-25T13:54:05.000Z","size":167,"stargazers_count":5,"open_issues_count":1,"forks_count":6,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-29T15:34:42.341Z","etag":null,"topics":["chatbot","cloud-free","dolly2","edge-computing","edge-inference","huggingface","intel","langchain","llama2","llm","local-inference","natural-language-processing","neural-chat","offline","openvino","rag","retrieval-augmented-generation"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yas-sim.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-04T07:52:48.000Z","updated_at":"2025-01-24T17:05:49.000Z","dependencies_parsed_at":"2024-01-25T15:02:18.390Z","dependency_job_id":null,"html_url":"https://github.com/yas-sim/openvino-llm-chatbot-rag","commit_stats":{"total_commits":23,"total_committers":1,"mean_commits":23.0,"dds":0.0,"last_synced_commit":"4fb35695aba28d03b89c8dbaf2703f0bea807165"},"previous_names":["yas-sim/openvino-llm-chatbot-rag"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yas-sim%2Fopenvino-llm-chatbot-rag","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yas-sim%2Fopenvino-llm-chatbot-rag/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yas-sim%2Fopenvino-llm-chatbot-rag/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yas-sim%2Fopenvino-llm-chatbot-rag/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yas-sim","download_url":"https://codeload.github.com/yas-sim/openvino-llm-chatbot-rag/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250250001,"owners_count":21399566,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chatbot","cloud-free","dolly2","edge-computing","edge-inference","huggingface","intel","langchain","llama2","llm","local-inference","natural-language-processing","neural-chat","offline","openvino","rag","retrieval-augmented-generation"],"created_at":"2024-11-16T22:27:14.267Z","updated_at":"2025-04-22T13:39:26.003Z","avatar_url":"https://github.com/yas-sim.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Q\u0026A Chatbot for OpenVINO web documentation by OpenVINO\r\n\r\nThis is an example of an LLM based Q\u0026A chatbot that can refer to external documents using RAG (Retrieval Augmented Genration) technique. The program uses OpenVINO as the inferencing acceleration library.\r\n\r\nThe program can answer your questions by referring the OpenVINO technical documentation from the OpenVINO official web site.\r\n\r\nThis program doesn't rely on any cloud services or webAPIs for inferencing. The program downloads all the data, including reference documents and DL models, and **can perform inference offline**. You don't need any cloud services once you prepare the data locally. \r\n\r\n## Programs / Files\r\n\r\n|#|Program/File|Description|\r\n|---|---|---|\r\n|1|`llm-model-downloader.py`|Download databrics/dolly-2 and meta-llama/llama2-7b-chat models, and convert them into OpenVINO IR models.|\r\n|2|`openvino-doc-specific-extractor.py`|Convert OpenVINO HTML documents into vector store (DB).\u003cbr\u003eReads HTML documents, extracts text, generates embeddings, and store it into vector store.\u003cbr\u003eYou need to download an archived (zipped) HTML document from OpenVINO document web site.|\r\n|3|`openvino-rag-server.py`|OpenVINO Q\u0026A demo server|\r\n|4|`openvino-rag-client.py`|OpenVION Q\u0026A demo client|\r\n|5|`.env`|Configurations (no secrets nor credentials ncluded. just a configuration file)|\r\n|6|`requirements.txt`|Python module requirements file|\r\n|7|`huggingface_login.py`|(optional) A Python script to login to HuggingFace hub.|\r\n\r\n## How to run\r\n\r\n0. Install Python prerequisites\r\n\r\nInstall steps for Windows.\r\n```sh\r\npython -m venv venv\r\nvenv/Scripts/activate\r\npython -m pip install -U pip\r\npip install -U setuptools wheel\r\npip install -r requirements.txt\r\n\r\n# Install en_core_web_sm, a Spacy pipeline for English\r\npython -m spacy download en_core_web_sm\r\n```\r\n\r\n1. Downloading OpenVINO Documents\r\n- Go to [OpenVINO web document page](https://docs.openvino.ai/2023.2/get_started.html) and download the archived document file from 'Download Docs' link on the right.\r\n- https://docs.openvino.ai/2023.2/get_started.html\r\n- Extract the contents of downloaded zip file into '`openvino_html_doc`' folder\r\n\r\n2. Generate vector store from the OpenVINO documents\r\n- Run '`openvino-doc-specific-extractor.py`'.\r\n- The program will store the document object in a pickle file (`doc_obj.pickle`) and use it if it exists the next time.\r\n```sh\r\npython openvino-doc-specific-extractor.py\r\n```\r\n- `.vectorstore_300_0` directory will be created.\r\n\t- '_300_0' means the chunk size is 300 and chunk overlap is 0.\r\n\t- You can generate the vector store with different chunk configurations by modifying the last few lines of Python code.\r\n\t- You can modify the `.env` file to specify which vector store file to use in the client and server programs. \r\n\r\n3. Download LLM models and convert them into OpenVINO IR models\r\n- `llm-model-downloader.py` will download 'dolly2-3b', 'llama2-7b-chat', and 'Intel/neural-chat-7b-v3-1' models as default.\r\n\t- You can specify the LLM model to use by modifying `.env` file.\r\n- You need to have account and access token to download the 'llama2-7b-chat' model. Go to HuggingFace web site and register yourself to get the access token. Also, you need to request the access to the llama2 models at llama2 project page.\r\n- The downloader will generate FP16, INT8 and INT4 models by default. You can use one of them. Please modify `.env` file to specify which model of data type to use.\r\n```sh\r\npython llm-model-downloader.py\r\n```\r\n\r\n4. Run the demo\r\n- Run the server\r\n- Note: The '`--host 0.0.0.0`' option is to accept external connection. '`--port xx`' option is also available.\r\n```sh\r\nuvicorn openvino-rag-server:app --host 0.0.0.0\r\n```\r\n- Run the client\r\n- Note: You can change the server URL (or IP address) and port number by editing `.env` file.\r\n```sh\r\nstreamlit run openvino-rag-client.py\r\n``` \r\nNote: You can start the server and client in arbitrary order.\r\n\r\n## Examples\r\n![pic1](./resources/screenshot1.png)\r\n\r\n## Tested environment\r\n- OS: Windows 11\r\n- OpenVINO: OpenVINO 2023.2.0\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyas-sim%2Fopenvino-llm-chatbot-rag","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyas-sim%2Fopenvino-llm-chatbot-rag","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyas-sim%2Fopenvino-llm-chatbot-rag/lists"}