{"id":22814808,"url":"https://github.com/seonglae/llama2gptq","last_synced_at":"2025-04-22T18:49:48.280Z","repository":{"id":174011061,"uuid":"648632685","full_name":"seonglae/llama2gptq","owner":"seonglae","description":"Chat to LLaMa 2 that also provides responses with reference documents over vector database. Locally available model using GPTQ 4bit quantization.","archived":false,"fork":false,"pushed_at":"2023-11-25T09:23:11.000Z","size":9937,"stargazers_count":29,"open_issues_count":5,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-19T11:53:58.211Z","etag":null,"topics":["chatai","chatbot","chatgpt","cuda","gpt","langchain","llama-2","llama2","model-quantization","quantization","question-answering","rye","streamlit-chat","transformers"],"latest_commit_sha":null,"homepage":"https://llama2gptq.nuxt.space/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/seonglae.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["seonglae"],"custom":["https://paypal.me/seonglae","https://www.buymeacoffee.com/seongland"]}},"created_at":"2023-06-02T12:31:29.000Z","updated_at":"2025-01-18T13:00:50.000Z","dependencies_parsed_at":"2023-07-20T12:07:42.331Z","dependency_job_id":"2c646f28-9d2d-46a7-ba02-97840e278a0c","html_url":"https://github.com/seonglae/llama2gptq","commit_stats":{"total_commits":63,"total_committers":3,"mean_commits":21.0,"dds":"0.19047619047619047","last_synced_commit":"58c6311ce7bf1770a594625dee58ae39e27a5805"},"previous_names":["seonglae/angryface","seonglae/llama2gptq"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/seonglae%2Fllama2gptq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/seonglae%2Fllama2gptq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/seonglae%2Fllama2gptq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/seonglae%2Fllama2gptq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/seonglae","download_url":"https://codeload.github.com/seonglae/llama2gptq/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250302600,"owners_count":21408410,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chatai","chatbot","chatgpt","cuda","gpt","langchain","llama-2","llama2","model-quantization","quantization","question-answering","rye","streamlit-chat","transformers"],"created_at":"2024-12-12T13:10:43.738Z","updated_at":"2025-04-22T18:49:48.253Z","avatar_url":"https://github.com/seonglae.png","language":"Python","funding_links":["https://github.com/sponsors/seonglae","https://paypal.me/seonglae","https://www.buymeacoffee.com/seongland"],"categories":[],"sub_categories":[],"readme":"# LLaMa2 GPTQ\n\nChat AI which can provide responses with reference documents by Prompt engineering over vector database. It suggests related web pages provided through the integration with my previous product, Texonom.\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"img/angryface.png\" style=\"width: 150px\"/\u003e\n\u003c/p\u003e\n\nPursuing local, private and personal AI without requesting external API attained by optimizing inference performance with GPTQ model quantization. This project was inspired by the [langchain](https://github.com/hwchase17/langchain) projects like [notion-qa](https://github.com/hwchase17/notion-qa), [localGPT](https://github.com/PromtEngineer/localGPT).\n\n# Demos\n\n### CLI Demo\n\nhttps://github.com/seonglae/llama2gptq/assets/27716524/dba5cd39-ea5c-44d9-bf29-2e8f04039413\n\n### Chat Demo\n\nhttps://github.com/seonglae/llama2gptq/assets/27716524/258de629-0b61-4670-b76b-9f2357adf4c7\n\n\u003cbr/\u003e\n\n## Install\n\nThis project is using [rye](https://mitsuhiko.github.io/rye/) as package manager\nCurrently only available with [CUDA](https://texonom.com/a9e934a523d346c5a984d95e3d0676e3)\n\n```\nrye sync\n```\n\nor using pip\n\n```\nCUDA_VERSION=cu118\nTORCH_VERSION=2.0.1\npip install torch==$TORCH_VERSION --index-url https://download.pytorch.org/whl/$CUDA_VERSION --force\npip install torch==$TORCH_VERSION --index-url https://download.pytorch.org/whl/$CUDA_VERSION\npip install .\n```\n\n## QA\n\n### 1. Chat with Web UI\n\n```zsh\nstreamlit run chat.py\n```\n\n### 2. Chat with CLI\n\n```zsh\npython main.py chat\n```\n\n## Ingest Documents\n\nCurrently code structure is mainly focussed on Notion's csv exported data\n\n### Custom source documents\n\n```zsh\n# Put document files to ./knowledge folder\npython main.py process\n# Or use provided Texonom DB\ngit clone https://huggingface.co/datasets/texonom/md-chroma-instructor-xl db\n```\n\n## Quantize Model\n\nDefault model is orca 3b for now\n\n```zsh\npython main quantize --source_model facebook/opt-125m --output opt-125m-4bit-gptq --push\n```\n\n## Future Plan\n\n- [ ] [MPS](https://texonom.com/8d71e4de36e4416c83f65ee7bdaa412b) support using dynamic model selecting\n- [ ] Stateful Web App support like [chat-langchain](https://chat.langchain.dev/)\n\n## App Stack\n\n### LLM Stack\n\n- [Langchain](https://texonom.com/945567c597364cbb98336ca08c059856) for Prompt Engineering\n- [ChromaDB](https://texonom.com/8af886db7d684e03911a86b652620816) for storing embeddings\n- [Transformers](https://texonom.com/f5101287cc9249ab812e281e374e5629) for LLM engine\n- [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ) for Quantization \u0026 Inference\n\n### Python Stack\n\n- [Rye](https://texonom.com/rye-429b5d5f3d7f4026ab5d1abd61facc73) for package management\n- [Mypy](https://texonom.com/8a894731430f4138ac0fdd522cd74772) for type checking\n- [Fire](https://github.com/google/python-fire) for CLI implementation\n- [Streamlit](https://texonom.com/9e295c64d27e4999878a022b1c538964) for Web UI implementation\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fseonglae%2Fllama2gptq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fseonglae%2Fllama2gptq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fseonglae%2Fllama2gptq/lists"}