{"id":14964800,"url":"https://github.com/su77ungr/casalioy","last_synced_at":"2025-04-08T03:11:52.036Z","repository":{"id":163302178,"uuid":"638783342","full_name":"su77ungr/CASALIOY","owner":"su77ungr","description":" ♾️ toolkit for air-gapped LLMs on consumer-grade hardware","archived":false,"fork":false,"pushed_at":"2023-10-27T15:49:48.000Z","size":3370,"stargazers_count":233,"open_issues_count":18,"forks_count":31,"subscribers_count":12,"default_branch":"main","last_synced_at":"2025-04-08T03:11:45.133Z","etag":null,"topics":["langchain","llamacpp","llm","qdrant","question-answering"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/su77ungr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"su77ungr","custom":"buymeacoffee.com/cassowary"}},"created_at":"2023-05-10T05:15:16.000Z","updated_at":"2025-03-18T23:45:29.000Z","dependencies_parsed_at":"2023-08-31T10:49:48.552Z","dependency_job_id":"87f4cd82-8c61-4f68-88ec-fe5b9b7320a1","html_url":"https://github.com/su77ungr/CASALIOY","commit_stats":{"total_commits":202,"total_committers":10,"mean_commits":20.2,"dds":0.2524752475247525,"last_synced_commit":"f360b5c0c49a7685fd23b6165f1058431b9e1b73"},"previous_names":["casalioy/casalioy","su77ungr/casalioy"],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/su77ungr%2FCASALIOY","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/su77ungr%2FCASALIOY/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/su77ungr%2FCASALIOY/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/su77ungr%2FCASALIOY/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/su77ungr","download_url":"https://codeload.github.com/su77ungr/CASALIOY/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247767236,"owners_count":20992548,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["langchain","llamacpp","llm","qdrant","question-answering"],"created_at":"2024-09-24T13:33:48.079Z","updated_at":"2025-04-08T03:11:52.011Z","avatar_url":"https://github.com/su77ungr.png","language":"Python","funding_links":["https://github.com/sponsors/su77ungr","buymeacoffee.com/cassowary","https://www.buymeacoffee.com/cassowary"],"categories":[],"sub_categories":[],"readme":"\u003c!--suppress HtmlDeprecatedAttribute --\u003e\n\n\u003e  Critical Notice: GGUF was introduced by the llama.cpp team on August 21st 2023. It is a replacement for GGML, which is no longer supported by llama.cpp. The main branch will keep using the deprecated variant. Switch to the [gguf-fix](https://github.com/su77ungr/CASALIOY/tree/gguf-fix) branch for GGUF support. \n\n\u003cdiv align=\"center\"\u003e\n      \u003ch4\u003e\n      Helpful? Keep it running \u003ca href=\"https://www.buymeacoffee.com/cassowary\" target=\"_blank\"\u003e\u003cimg src=\"https://github.com/su77ungr/CASALIOY/assets/69374354/1de2e6a1-d4ee-44a8-baf1-db4f2765b833\" alt=\"Buy Me A Coffee\" height=\"20\" width=\"20\"\u003e\u003c/a\u003e  \n      \u003c/h4\u003e\n      \n#   \n      \n\u003ch2\u003e\n\u003cp\u003e\n\u003cimg height=\"400\" src=\"https://github.com/su77ungr/CASALIOY/assets/69374354/41c1edcb-af2f-4327-bdcd-5248ffe6ecf9\" alt=\"Qdrant\"\u003e\u003cbr\u003e\n      \n\u003ca href=\"https://github.com/su77ungr/CASALIOY/issues/8\"\u003e\u003cimg src=\"https://img.shields.io/badge/Feature-Requests-bc1439.svg\" alt=\"Roadmap 2023\"\u003e [![Docker Pulls](https://badgen.net/docker/pulls/su77ungr/casalioy?icon=docker\u0026label=pulls)](https://hub.docker.com/r/su77ungr/casalioy/)\u003c/a\u003e\n![example workflow](https://github.com/su77ungr/CASALIOY/actions/workflows/docker-image.yml/badge.svg)\n\u003cbr\u003e\n\u003c/p\u003e\n      \nThe fastest toolkit for air-gapped LLMs with \u003ca  href=\"#chat-inside-gui-new-feature\"\u003e\u003cimg src=\"https://img.shields.io/badge/GUI-blue.svg\" alt=\"Roadmap 2023\"\u003e\n      \n[LangChain](https://github.com/hwchase17/langchain) + [LlamaCpp](https://pypi.org/project/llama-cpp-python/) + [qdrant](https://qdrant.tech/)\n      \n\u003c/div\u003e\n         \n\u003cbr\u003e\n     \n      \n# Setup\n\n### Docker ( 🚰 under construction. tested on Ubuntu LTS) \n\n```bash\ndocker pull su77ungr/casalioy:stable\n```\n\n```bash\ndocker run -it -p 8501:8501 --shm-size=16gb su77ungr/casalioy:stable /bin/bash\n```\nfor GPU support of stable use `casalioy:gpu` (unstable)\n\n\u003e All set! Proceed with ingesting your [dataset](#ingesting-your-own-dataset)\n\n### Build it from source\n\n\u003e First install all requirements:\n\n```shell\npython -m pip install poetry\npython -m poetry config virtualenvs.in-project true\npython -m poetry install\n. .venv/bin/activate\npython -m pip install --force streamlit sentence_transformers  # Temporary bandaid fix, waiting for streamlit \u003e=1.23\npre-commit install\n```\n\nIf you want GPU support for llama-ccp:\n\n```shell\npip uninstall -y llama-cpp-python\nCMAKE_ARGS=\"-DLLAMA_CUBLAS=on\" FORCE_CMAKE=1 pip install --force llama-cpp-python\n```\n\n\u003e Edit the example.env to fit your models and rename it to .env\n\n```env\n# Generic\nMODEL_N_CTX=1024\nTEXT_EMBEDDINGS_MODEL=sentence-transformers/all-MiniLM-L6-v2\nTEXT_EMBEDDINGS_MODEL_TYPE=HF  # LlamaCpp or HF\nUSE_MLOCK=true\n\n# Ingestion\nPERSIST_DIRECTORY=db\nDOCUMENTS_DIRECTORY=source_documents\nINGEST_CHUNK_SIZE=500\nINGEST_CHUNK_OVERLAP=50\n\n# Generation\nMODEL_TYPE=LlamaCpp # GPT4All or LlamaCpp\nMODEL_PATH=TheBloke/TinyLlama-1.1B-Chat-v0.3-GGUF/tinyllama-1.1b-chat-v0.3.Q6_K.gguf\nMODEL_TEMP=0.8\nMODEL_STOP=[STOP]\nCHAIN_TYPE=stuff\nN_RETRIEVE_DOCUMENTS=100 # How many documents to retrieve from the db\nN_FORWARD_DOCUMENTS=6 # How many documents to forward to the LLM, chosen among those retrieved\n```\n\nThis should look like this\n\n```\n└── repo\n      ├── startLLM.py\n      ├── casalioy\n      │   └── ingest.py, load_env.py, startLLM.py, gui.py, ...\n      │   └── misc/ \n      ├── source_documents\n      │   └── sample.csv\n      │   └── ...\n      ├── models\n      │   ├── ggml-vic7b-q5_1.bin\n      │   └── ...\n      └── .env, Dockerfile, ...\n```\n\n\n\u003e 👇 Update your installation!\n\n\n      git pull \u0026\u0026 poetry install\n\n\n\n## Ingesting your own dataset\n\nTo automatically ingest different data types (.txt, .pdf, .csv, .epub, .html, .docx, .pptx, .eml, .msg)\n\n\u003e This repo includes dummy [files](https://github.com/su77ungr/CASALIOY/tree/main/source_documents)\n\u003e inside `source_documents` to run tests with.\n\n```shell\npython casalioy/ingest.py # optional \u003cpath_to_your_data_directory\u003e\n```\n\nOptional: use `y` flag to purge existing vectorstore and initialize fresh instance\n\n```shell\npython casalioy/ingest.py # optional \u003cpath_to_your_data_directory\u003e y\n```\n\nThis spins up a local qdrant namespace inside the `db` folder containing the local vectorstore. Will take time,\ndepending on the size of your document.\nYou can ingest as many documents as you want by running `ingest`, and all will be accumulated in the local embeddings\ndatabase. To remove dataset simply remove `db` folder.\n\n## Ask questions to your documents, locally!\n\nIn order to ask a question, run a command like:\n\n```shell\npython casalioy/startLLM.py\n```\n\nAnd wait for the script to require your input.\n\n```shell\n\u003e Enter a query:\n```\n\nHit enter. You'll need to wait 20-30 seconds (depending on your machine) while the LLM model consumes the prompt and\nprepares the answer. Once done, it will print the answer and the 4 sources it used as context from your documents; you\ncan then ask another question without re-running the script, just wait for the prompt again.\n\nNote: you could turn off your internet connection, and the script inference would still work. No data gets out of your\nlocal environment.\n\nType `exit` to finish the script.\n\n## Chat inside GUI (new feature)\n\nIntroduced by [@alxspiker](https://github.com/alxspiker) -\u003e see [#21](https://github.com/su77ungr/CASALIOY/pull/21)\n\n```shell\nstreamlit run casalioy/gui.py\n```\n\n# LLM options       \n   \n### Leaderboard \n      \nList of available open LLMs [HuggingFace](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)\n### models outside of the GPT-J ecosphere  (work out of the box)\n\n\u003e 🪢 avoid using non-v3 models when using other quantization than q5 (LlamaCpp introduced [v3](https://github.com/ggerganov/llama.cpp/pull/1508) for ggml)\n      \n| Model                                                                                                                                            | BoolQ | PIQA | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA | Avg. |\n|:-------------------------------------------------------------------------------------------------------------------------------------------------|:-----:|:----:|:---------:|:----------:|:-----:|:-----:|:----:|:----:|\n| [GPT4All-13b-snoozy GGMLv3](https://huggingface.co/TheBloke/GPT4All-13B-snoozy-GGML/blob/main/GPT4All-13B-snoozy.ggmlv3.q4_0.bin) \n  [GPT4All-13b-snoozy (deprecated)](https://gpt4all.io/models/ggml-gpt4all-l13b-snoozy.bin)                | 83.3  | 79.2 |   75.0    |    71.3    | 60.9  | 44.2  | 43.4 | 65.3 | \n      \n      \n      \n### models inside of the GPT-J ecosphere\n\n| Model                                                                             | BoolQ | PIQA | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA | Avg. |\n|:----------------------------------------------------------------------------------|:-----:|:----:|:---------:|:----------:|:-----:|:-----:|:----:|:----:|\n| [GPT4All-J vanilla](https://gpt4all.io/models/ggml-gpt4all-j.bin)                                                                 | 73.4  | 74.8 |   63.4    |    64.7    | 54.9  | 36.0  | 40.2 | 58.2 |\n| [GPT4All-J v1.1-breezy](https://gpt4all.io/models/ggml-gpt4all-j-v1.1-breezy.bin) | 74.0  | 75.1 |   63.2    |    63.6    | 55.4  | 34.9  | 38.4 | 57.8 |\n| [GPT4All-J v1.2-jazzy](https://gpt4all.io/models/ggml-gpt4all-j-v1.2-jazzy.bin)   | 74.8  | 74.9 |   63.6    |    63.8    | 56.6  | 35.3  | 41.0 | 58.6 |\n| [GPT4All-J v1.3-groovy](https://gpt4all.io/models/ggml-gpt4all-j-v1.3-groovy.bin) | 73.6  | 74.3 |   63.8    |    63.5    | 57.7  | 35.0  | 38.8 | 58.1 |\n| [GPT4All-J Lora 6B](https://gpt4all.io/models/)                                   | 68.6  | 75.8 |   66.2    |    63.5    | 56.4  | 35.7  | 40.2 | 58.1 |\n\nall the supported models from [here](https://huggingface.co/nomic-ai/gpt4all-13b-snoozy) (custom LLMs in Pipeline)\n\n### Convert GGML model to GGJT-ready model v1 (for truncation error or not supported models)\n\n1. Download ready-to-use models\n\n\u003e Browse Hugging Face for [models](https://huggingface.co/)\n\n2. Convert locally\n\n\u003e ``` python casalioy/misc/convert.py``` [see discussion](https://github.com/su77ungr/CASALIOY/issues/10#issue-1706854398)\n\n# Pipeline\n\n\u003cbr\u003e\u003cbr\u003e\n\n\u003cimg src=\"https://qdrant.tech/articles_data/langchain-integration/flow-diagram.png\"\u003e\u003c/img\u003e\n\u003cbr\u003e\u003cbr\u003e\n\nSelecting the right local models and the power of `LangChain` you can run the entire pipeline locally, without any data\nleaving your environment, and with reasonable performance.\n\n- `ingest.py` uses `LangChain` tools to parse the document and create embeddings locally using `LlamaCppEmbeddings`. It\n  then stores the result in a local vector database using `Qdrant` vector store.\n\n- `startLLM.py` can handle every LLM that is llamacpp compatible (default `GPT4All-J`). The context for the answers is\n  extracted from the local vector store using a similarity search to locate the right piece of context from the docs.\n\n\u003cbr\u003e\u003cbr\u003e\n\n\n# Disclaimer\n\nThe contents of this repository are provided \"as is\" and without warranties of any kind, whether express or implied. We\ndo not warrant or represent that the information contained in this repository is accurate, complete, or up-to-date. We\nexpressly disclaim any and all liability for any errors or omissions in the content of this repository.\n\nBy using this repository, you are agreeing to comply with and be bound by the above disclaimer. If you do not agree with\nany part of this disclaimer, please do not use this repository.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsu77ungr%2Fcasalioy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsu77ungr%2Fcasalioy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsu77ungr%2Fcasalioy/lists"}