{"id":19961349,"url":"https://github.com/damo-nlp-sg/multipurpose-chatbot","last_synced_at":"2025-05-03T22:30:48.128Z","repository":{"id":226547943,"uuid":"766226274","full_name":"DAMO-NLP-SG/Multipurpose-Chatbot","owner":"DAMO-NLP-SG","description":"A chatbot UI for RAG, multimodal, text completion. (support Transformers, llama.cpp, MLX, vLLM)","archived":false,"fork":false,"pushed_at":"2024-04-18T09:55:23.000Z","size":24310,"stargazers_count":10,"open_issues_count":0,"forks_count":0,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-04-18T11:40:52.036Z","etag":null,"topics":["chatbot-application","gradio-interface","gradio-python-llm","llm-inference"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DAMO-NLP-SG.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-03-02T17:19:40.000Z","updated_at":"2024-04-18T04:51:50.000Z","dependencies_parsed_at":"2024-03-11T08:51:08.211Z","dependency_job_id":null,"html_url":"https://github.com/DAMO-NLP-SG/Multipurpose-Chatbot","commit_stats":null,"previous_names":["damo-nlp-sg/multipurpose-chatbot"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DAMO-NLP-SG%2FMultipurpose-Chatbot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DAMO-NLP-SG%2FMultipurpose-Chatbot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DAMO-NLP-SG%2FMultipurpose-Chatbot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DAMO-NLP-SG%2FMultipurpose-Chatbot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DAMO-NLP-SG","download_url":"https://codeload.github.com/DAMO-NLP-SG/Multipurpose-Chatbot/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224374778,"owners_count":17300691,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chatbot-application","gradio-interface","gradio-python-llm","llm-inference"],"created_at":"2024-11-13T02:07:17.059Z","updated_at":"2024-11-13T02:07:18.035Z","avatar_url":"https://github.com/DAMO-NLP-SG.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Multi-purpose Chatbot (Local, Remote and HF spaces)\n\nA Chatbot UI that support Chatbot, RAG, Text completion, Multi-modal across [HF Transformers](https://github.com/huggingface/transformers), [llama.cppp](https://github.com/ggerganov/llama.cpp), [Apple MLX](https://github.com/ml-explore/mlx) and [vLLM](https://github.com/vllm-project/vllm).\n\nDesigned support both locally, remote and huggingface spaces.\n\n![image](assets/image_doc_rag.gif)\n\n---\n\n**Checkout cool demos using Multi-purpose chatbot.**\n- [MultiModal SeaLLMs/SeaLLM-7B](https://huggingface.co/spaces/SeaLLMs/SeaLLM-7B)\n- [MultiPurpose-Chatbot-DEMO test](https://huggingface.co/spaces/nxphi47/MultiPurpose-Chatbot-DEMO) - This DEMO test the UI without LLM.\n\n**Supported features**\n- Vanilla chat interface - [ChatInterfaceDemo](multipurpose_chatbot/demos/chat_interface.py)\n- Chat with short document (full context) - [DocChatInterfaceDemo](multipurpose_chatbot/demos/multimodal_chat_interface.py)\n- Chat with visual image - [VisionChatInterfaceDemo](multipurpose_chatbot/demos/multimodal_chat_interface.py)\n- Chat with visual image and short document - [VisionDocChatInterfaceDemo](multipurpose_chatbot/demos/multimodal_chat_interface.py)\n- Chat with long document via RAG - [RagChatInterfaceDemo](multipurpose_chatbot/demos/rag_chat_interface.py)\n- Text completion (free form prompting) - [TextCompletionDemo](multipurpose_chatbot/demos/text_completion.py)\n- Batch inference (via file upload with vLLM) - [BatchInferenceDemo](multipurpose_chatbot/demos/batch_inference.py)\n\n**Support backend**\n- [GPU Transformers](https://github.com/huggingface/transformers) with full support MultiModal, document QA, RAG, completion.\n- [llama.cppp](https://github.com/ggerganov/llama.cpp) like Transformers, except pending MultiModal. PR welcome.\n- [Apple MLX](https://github.com/ml-explore/mlx) like Transformers, except pending MultiModal. PR welcome.\n- [vLLM](https://github.com/vllm-project/vllm) like Transformers + **Batch inference via file upload**, pending MultiModal. PR welcome.\n\nMulti-purpose Chatbot use `ENVIRONMENT VARIABLE` instead of `argparse` to set hyperparmeters to support seamless integration with HF space, which requires us to set params via environment vars. The app is launch only with `python app.py`\n\n## Installation\n\n```bash\npip install -r requirements.txt\n```\n\n#### Transformers\n```bash\npip install -r transformers_requirements.txt\n```\n\n\n#### VLLM\n```bash\npip install -r vllm_requirements.txt\n```\n\n\n#### llama.cpp\nFollow [Llama-cpp-python](https://llama-cpp-python.readthedocs.io/en/latest/#installation) to install `llama.cpp`\n\ne.g: On Macos\n```bash\nCMAKE_ARGS=\"-DLLAMA_METAL=on\" pip install llama-cpp-python\n```\n\n\n#### MLX\n\nOnly on MacOS, remember to install [**NATIVE** python environment](https://ml-explore.github.io/mlx/build/html/install.html).\n\n```bash\npython -c \"import platform; print(platform.processor())\"\n# should output \"arm\", if not reinstall python with native\n```\n\nInstall requirements\n```bash\npip install -r mlx_requirements.txt\n```\n\n\n## Usage\n\nWe use bash environment to define model variables\n\n#### Transformers\n\n`MODEL_PATH` must be a model with chat_template with system prompt (e.g Mistral-7B-Instruct-v0.2 does not have system prompt)\n\n```bash\nexport BACKEND=transformers\nexport MODEL_PATH=teknium/OpenHermes-2.5-Mistral-7B\nexport RAG_EMBED_MODEL_NAME=sentence-transformers/all-MiniLM-L6-v2\nexport DEMOS=DocChatInterfaceDemo,ChatInterfaceDemo,RagChatInterfaceDemo,TextCompletionDemo\npython app.py\n```\n\n#### Llava-1.5 Transformers\n\n\n```bash\nexport CUDA_VISIBLE_DEVICES=0\nexport TEMPERATURE=0.7\nexport MAX_TOKENS=512\nexport MODEL_PATH=llava-hf/llava-1.5-7b-hf\nexport IMAGE_TOKEN=\"\u003cimage\u003e\"\nexport BACKEND=llava15_transformers\nexport DEMOS=VisionChatInterfaceDemo,VisionDocChatInterfaceDemo,TextCompletionDemo\npython app.py\n\n```\n\n#### VLLM\n\n```bash\nexport BACKEND=vllm\nexport MODEL_PATH=teknium/OpenHermes-2.5-Mistral-7B\nexport RAG_EMBED_MODEL_NAME=sentence-transformers/all-MiniLM-L6-v2\nexport DEMOS=DocChatInterfaceDemo,ChatInterfaceDemo,RagChatInterfaceDemo,TextCompletionDemo\npython app.py\n```\n\n\n#### llama.cpp\n\n```bash\nexport BACKEND=llama_cpp\nexport MODEL_PATH=/path/to/model.gguf\nexport RAG_EMBED_MODEL_NAME=sentence-transformers/all-MiniLM-L6-v2\nexport DEMOS=DocChatInterfaceDemo,ChatInterfaceDemo,RagChatInterfaceDemo,TextCompletionDemo\npython app.py\n```\n\n\n#### MLX\n\n```bash\nexport BACKEND=mlx\nexport MODEL_PATH=mlx-community/Nous-Hermes-2-Mistral-7B-DPO-4bit-MLX\nexport RAG_EMBED_MODEL_NAME=sentence-transformers/all-MiniLM-L6-v2\nexport DEMOS=DocChatInterfaceDemo,ChatInterfaceDemo,RagChatInterfaceDemo,TextCompletionDemo\npython app.py\n```\n\n\n## Customization\n\n#### Configs:\n\n* [configs.py](multipurpose_chatbot/configs.py) where you can find customize UI markdowns and settings global variables\n\n#### Backend and engines\n* [base_engine](multipurpose_chatbot/engines/base_engine.py),  [transformers_engine](multipurpose_chatbot/engines/transformers_engine.py) and [llama_cpp_engine](multipurpose_chatbot/engines/llama_cpp_engine.py) to find how different model backend works. Feel free to extend and implement new features.\n* [llava15_transformers_engine](multipurpose_chatbot/engines/llava15_transformers_engine.py) describe how to implement Llava-1.5\n\n\n#### Gradio Demo tabs\n* Checkout [chat_interface](multipurpose_chatbot/demos/chat_interface.py), [multimodal_chat_interface](multipurpose_chatbot/demos/multimodal_chat_interface.py) and other interface demo under [multipurpose_chatbot/demos](multipurpose_chatbot/demos) to find out how the demo works.\n\n\n#### Enableing demos\n\nSetting comma-separated demo class names (e.g `ChatInterfaceDemo` to enable demo).\n\n```bash\nexport DEMOS=VisionDocChatInterfaceDemo,VisionChatInterfaceDemo,DocChatInterfaceDemo,ChatInterfaceDemo,RagChatInterfaceDemo,TextCompletionDemo\n```\n\n\n## Contributing\n\nWe welcome and value any contributions and collaborations. Feel free to open a PR\n\n\n## Citation\n\nIf you find our project useful, hope you can star our repo and cite our repo as follows:\n```\n@article{multipurpose_chatbot_2024,\n  author = {Xuan-Phi Nguyen, },\n  title = {Multipurpose Chatbot},\n  year = 2024,\n}\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdamo-nlp-sg%2Fmultipurpose-chatbot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdamo-nlp-sg%2Fmultipurpose-chatbot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdamo-nlp-sg%2Fmultipurpose-chatbot/lists"}