{"id":13585050,"url":"https://github.com/junruxiong/IncarnaMind","last_synced_at":"2025-04-07T06:32:26.273Z","repository":{"id":194845942,"uuid":"683066449","full_name":"junruxiong/IncarnaMind","owner":"junruxiong","description":"Connect and chat with your multiple documents (pdf and txt) through GPT 3.5, GPT-4 Turbo, Claude and Local Open-Source LLMs","archived":false,"fork":false,"pushed_at":"2024-06-15T22:29:55.000Z","size":52857,"stargazers_count":783,"open_issues_count":8,"forks_count":50,"subscribers_count":16,"default_branch":"main","last_synced_at":"2024-11-05T05:33:20.989Z","etag":null,"topics":["ai","chatbot","generative-ai","gpt","langchain","llm","nlp","openai","pdf"],"latest_commit_sha":null,"homepage":"https://www.incarnamind.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/junruxiong.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-08-25T14:13:15.000Z","updated_at":"2024-11-02T15:54:31.000Z","dependencies_parsed_at":null,"dependency_job_id":"4352449c-ccef-43cc-a961-0a1465ba9e2a","html_url":"https://github.com/junruxiong/IncarnaMind","commit_stats":null,"previous_names":["junruxiong/incarnamind"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/junruxiong%2FIncarnaMind","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/junruxiong%2FIncarnaMind/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/junruxiong%2FIncarnaMind/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/junruxiong%2FIncarnaMind/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/junruxiong","download_url":"https://codeload.github.com/junruxiong/IncarnaMind/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223273408,"owners_count":17117799,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","chatbot","generative-ai","gpt","langchain","llm","nlp","openai","pdf"],"created_at":"2024-08-01T15:04:42.558Z","updated_at":"2025-04-07T06:32:26.265Z","avatar_url":"https://github.com/junruxiong.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# 🧠 IncarnaMind\n\n## 👀 In a Nutshell\n\nIncarnaMind enables you to chat with your personal documents 📁 (PDF, TXT) using Large Language Models (LLMs) like GPT ([architecture overview](#high-level-architecture)). While OpenAI has recently launched a fine-tuning API for GPT models, it doesn't enable the base pretrained models to learn new data, and the responses can be prone to factual hallucinations. Utilize our [Sliding Window Chunking](#sliding-window-chunking) mechanism and Ensemble Retriever enables efficient querying of both fine-grained and coarse-grained information within your ground truth documents to augment the LLMs.\n\nFeel free to use it and we welcome any feedback and new feature suggestions 🙌.\n\n## ✨ New Updates\n\n### Open-Source and Local LLMs Support\n\n- **Recommended Model:** We've primarily tested with the Llama2 series models and recommend using [llama2-70b-chat](https://huggingface.co/TheBloke/Llama-2-70B-chat-GGUF) (either full or GGUF version) for optimal performance. Feel free to experiment with other LLMs.\n- **System Requirements:** It requires more than 35GB of GPU RAM to run the GGUF quantized version.\n\n### Alternative Open-Source LLMs Options\n\n- **Insufficient RAM:** If you're limited by GPU RAM, consider using the [Together.ai](https://api.together.xyz/playground) API. It supports llama2-70b-chat and most other open-source LLMs. Plus, you get $25 in free usage.\n- **Upcoming:** Smaller and cost-effecitive, fine-tuned models will be released in the future.\n\n### How to use GGUF models\n\n- For instructions on acquiring and using quantized GGUF LLM (similar to GGML), please refer to this [video](https://www.youtube.com/watch?v=lbFmceo4D5E) (from 10:45 to 12:30)..\n\nHere is a comparison table of the different models I tested, for reference only:\n\n| Metrics   | GPT-4  | GPT-3.5 | Claude 2.0 | Llama2-70b | Llama2-70b-gguf | Llama2-70b-api |\n|-----------|--------|---------|------------|------------|-----------------|----------------|\n| Reasoning | High   | Medium  | High       | Medium     | Medium          | Medium         |\n| Speed     | Medium | High    | Medium     | Very Low   | Low             | Medium         |\n| GPU RAM   | N/A    | N/A     | N/A        | Very High  | High            | N/A            |\n| Safety    | Low    | Low     | Low        | High       | High            | Low            |\n\n## 💻 Demo\n\nhttps://github.com/junruxiong/IncarnaMind/assets/44308338/89d479fb-de90-4f7c-b166-e54f7bc7344c\n\n## 💡 Challenges Addressed\n\n- **Fixed Chunking**: Traditional RAG tools rely on fixed chunk sizes, limiting their adaptability in handling varying data complexity and context.\n\n- **Precision vs. Semantics**:  Current retrieval methods usually focus either on semantic understanding or precise retrieval, but rarely both.\n\n- **Single-Document Limitation**: Many solutions can only query one document at a time, restricting multi-document information retrieval.\n\n- **Stability**: IncarnaMind is compatible with OpenAI GPT, Anthropic Claude, Llama2, and other open-source LLMs, ensuring stable parsing.\n\n## 🎯 Key Features\n\n- **Adaptive Chunking**: Our Sliding Window Chunking technique dynamically adjusts window size and position for RAG, balancing fine-grained and coarse-grained data access based on data complexity and context.\n\n- **Multi-Document Conversational QA**: Supports simple and multi-hop queries across multiple documents simultaneously, breaking the single-document limitation.\n\n- **File Compatibility**: Supports both PDF and TXT file formats.\n\n- **LLM Model Compatibility**: Supports OpenAI GPT, Anthropic Claude, Llama2 and other open-source LLMs.\n\n## 🏗 Architecture\n\n### High Level Architecture\n\n![image](figs/High_Level_Architecture.png)\n\n### Sliding Window Chunking\n\n![image](figs/Sliding_Window_Chunking.png)\n\n## 🚀 Getting Started\n\n### 1. Installation\n\nThe installation is simple, you just need to run few commands.\n\n#### 1.0. Prerequisites\n\n- 3.8 ≤ Python \u003c 3.11 with [Conda](https://www.anaconda.com/download)\n- One/All of [OpenAI API Key](https://beta.openai.com/signup), [Anthropic Claude API Key](https://console.anthropic.com/account/keys), [Together.ai API KEY](https://api.together.xyz/settings/api-keys) or [HuggingFace toekn for Meta Llama models](https://huggingface.co/settings/tokens)\n- And of course, your own documents.\n\n#### 1.1. Clone the repository\n\n```shell\ngit clone https://github.com/junruxiong/IncarnaMind\ncd IncarnaMind\n```\n\n#### 1.2. Setup\n\nCreate Conda virtual environment:\n\n```shell\nconda create -n IncarnaMind python=3.10\n```\n\nActivate:\n\n```shell\nconda activate IncarnaMind\n```\n\nInstall all requirements:\n\n```shell\npip install -r requirements.txt\n```\n\nInstall [llama-cpp](https://github.com/abetlen/llama-cpp-python) seperatly if you want to run quantized local LLMs:\n\n- For `NVIDIA` GPUs support, use `cuBLAS`\n\n```shell\nCMAKE_ARGS=\"-DLLAMA_CUBLAS=on\" FORCE_CMAKE=1 pip install llama-cpp-python==0.1.83 --no-cache-dir\n```\n\n- For Apple Metal (`M1/M2`) support, use\n\n```shell\nCMAKE_ARGS=\"-DLLAMA_METAL=on\"  FORCE_CMAKE=1 pip install llama-cpp-python==0.1.83 --no-cache-dir\n```\n\nSetup your one/all of API keys in **configparser.ini** file:\n\n```shell\n[tokens]\nOPENAI_API_KEY = (replace_me)\nANTHROPIC_API_KEY = (replace_me)\nTOGETHER_API_KEY = (replace_me)\n# if you use full Meta-Llama models, you may need Huggingface token to access.\nHUGGINGFACE_TOKEN = (replace_me)\n```\n\n(Optional) Setup your custom parameters in **configparser.ini** file:\n\n```shell\n[parameters]\nPARAMETERS 1 = (replace_me)\nPARAMETERS 2 = (replace_me)\n...\nPARAMETERS n = (replace_me)\n```\n\n### 2. Usage\n\n#### 2.1. Upload and process your files\n\nPut all your files (please name each file correctly to maximize the performance) into the **/data** directory and run the following command to ingest all data:\n(You can delete example files in the **/data** directory before running the command)\n\n```shell\npython docs2db.py\n```\n\n#### 2.2. Run\n\nIn order to start the conversation, run a command like:\n\n```shell\npython main.py\n```\n\n#### 2.3. Chat and ask any questions\n\nWait for the script to require your input like the below.\n\n```shell\nHuman:\n```\n\n#### 2.4. Others\n\nWhen you start a chat, the system will automatically generate a **IncarnaMind.log** file.\nIf you want to edit the logging, please edit in the **configparser.ini** file.\n\n```shell\n[logging]\nenabled = True\nlevel = INFO\nfilename = IncarnaMind.log\nformat = %(asctime)s [%(levelname)s] %(name)s: %(message)s\n```\n\n## 🚫 Limitations\n\n- Citation is not supported for current version, but will release soon.\n- Limited asynchronous capabilities.\n\n## 📝 Upcoming Features\n\n- Frontend UI interface\n- Fine-tuned small size open-source LLMs\n- OCR support\n- Asynchronous optimization\n- Support more document formats\n\n## 🙌 Acknowledgements\n\nSpecial thanks to [Langchain](https://github.com/langchain-ai/langchain), [Chroma DB](https://github.com/chroma-core/chroma), [LocalGPT](https://github.com/PromtEngineer/localGPT), [Llama-cpp](https://github.com/abetlen/llama-cpp-python) for their invaluable contributions to the open-source community. Their work has been instrumental in making the IncarnaMind project a reality.\n\n## 🖋 Citation\n\nIf you want to cite our work, please use the following bibtex entry:\n\n```bibtex\n@misc{IncarnaMind2023,\n  author = {Junru Xiong},\n  title = {IncarnaMind},\n  year = {2023},\n  publisher = {GitHub},\n  journal = {GitHub Repository},\n  howpublished = {\\url{https://github.com/junruxiong/IncarnaMind}}\n}\n```\n\n## 📑 License\n\n[Apache 2.0 License](LICENSE)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjunruxiong%2FIncarnaMind","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjunruxiong%2FIncarnaMind","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjunruxiong%2FIncarnaMind/lists"}