Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/sachs7/multi-modal-langchain-chatbot
A Multi-modal chatbot with LangChain, that supports RAG, Paperswithcode, and Image generation using Dall-E-3
https://github.com/sachs7/multi-modal-langchain-chatbot
chatbot dall-e-3 langchain langchain-agent multi-modal openai-chatbot paperswithcode rag
Last synced: about 1 month ago
JSON representation
A Multi-modal chatbot with LangChain, that supports RAG, Paperswithcode, and Image generation using Dall-E-3
- Host: GitHub
- URL: https://github.com/sachs7/multi-modal-langchain-chatbot
- Owner: sachs7
- License: mit
- Created: 2024-02-22T14:50:04.000Z (11 months ago)
- Default Branch: main
- Last Pushed: 2024-02-28T16:10:27.000Z (11 months ago)
- Last Synced: 2024-02-28T17:30:51.309Z (11 months ago)
- Topics: chatbot, dall-e-3, langchain, langchain-agent, multi-modal, openai-chatbot, paperswithcode, rag
- Language: Python
- Homepage:
- Size: 2.69 MB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Multi-Modal LangChain Chatbot
A Multi-Modal Chatbot with LangChain that also supports the agentic RAG
The ChatBot supports:
1. Chat with your documents using basic RAG
2. Query "Paperswithcode" site to get a gist of recent papers on any topic
3. Generate images using DallE-3
The code base has been organized into the following folders:
1. `docs`: Store your PDF documents in this folder
2. `services`: Add all your agent-specific services here
3. `tools`: Create a tool for your services
The `main.py` consists of the actual System Prompt and the registration of the tools.
Make sure to register your new tools here. The UI can be created using both, Gradio, and Streamlit.
Note: When using image generation, I was finding it difficult to represent images in Gradio, hence I went with Streamlit at the end. But feel free to play around.
# How to run:
1. `pip install -r requirements.txt` to install all the libraries
2. From your terminal run, `export OPENAI_API_KEY=sk-xxxx`
3. Store your PDFs in the `docs` folder
4. If using Gradio, `python3 main.py`
5. If using Streamlit, `streamlit run main.py`
6. Once you start the app, a `chroma_persists` folder will be created along with a `logs` folder
### Improvements:
1. This is a basic RAG, one can create CRAG, Rank based RAG, etc
2. Showcase Chat History (currently, not showing it)
3. Add more agents specific to your needs (don't forget to register it in `main.py`)
4. Use the `config` file to store model details, etc that are needed for RAG as well as other agents
5. ~~Currently, the chromaDB gets recreated for every query to RAG. Need to handle this to make ChromaDB persist (instantiate once) once the service is up.~~ Fixed ✔️
# Sample Output:
1. 
2. 
3. 
4. 