{"id":26874046,"url":"https://github.com/mohith202/ema-chatbot","last_synced_at":"2025-03-31T09:33:22.680Z","repository":{"id":244811530,"uuid":"816341082","full_name":"Mohith202/Ema-Chatbot","owner":"Mohith202","description":"A public chatbot can use PDF from user or uses preloaded dataset and answer Query from user while displaying source.","archived":false,"fork":false,"pushed_at":"2025-03-22T06:55:48.000Z","size":42969,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-22T07:27:14.643Z","etag":null,"topics":["embeddings-word2vec","lama","llm","pdf-document-processor","python3","voice-to-text"],"latest_commit_sha":null,"homepage":"https://ema-model-89t7szg5bruabesgecwsqu.streamlit.app/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Mohith202.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-17T14:43:46.000Z","updated_at":"2025-03-22T06:55:51.000Z","dependencies_parsed_at":"2025-03-22T07:24:20.766Z","dependency_job_id":"ffd59ed5-6b35-4ff7-b320-0e7f3505cf7d","html_url":"https://github.com/Mohith202/Ema-Chatbot","commit_stats":null,"previous_names":["mohith202/ema-chatbot"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Mohith202%2FEma-Chatbot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Mohith202%2FEma-Chatbot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Mohith202%2FEma-Chatbot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Mohith202%2FEma-Chatbot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Mohith202","download_url":"https://codeload.github.com/Mohith202/Ema-Chatbot/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246447302,"owners_count":20778949,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["embeddings-word2vec","lama","llm","pdf-document-processor","python3","voice-to-text"],"created_at":"2025-03-31T09:33:22.115Z","updated_at":"2025-03-31T09:33:22.668Z","avatar_url":"https://github.com/Mohith202.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"**PROCESS PLANNING OF CHATBOT**\n\n**More detail of the process can be found in \"Process_planning_of_chatbot.docx: https://docs.google.com/document/d/1TT4Uw04-QSFhzyTF74MQCTh00UnZ14x_/edit?usp=sharing\u0026ouid=106890396692261549451\u0026rtpof=true\u0026sd=true \" and live project link is :https://ema-chatbot.streamlit.app/**\n \n**Step-by-Step Process**\n\n**1. Requirements:**\n\n•\tData collection: gone through many articles to finalizing a best article and converted to pdf using AI tool.\n\n•\tAPI selection: search for various model like GPT-3 by openAI ,transformer from hugging face and langchain. Finalized langchain for its modular and flexible framework for building advanced NLP applications with a focus on chaining operations and managing conversational context.\n\n•\tModel selection: Together offers many model in which meta-llama/Llama-3-70b-chat-hf is collection of pretrained and instruction tuned generative text models in  70B sizes from hugging face. The cost is also average ($0.9) per 1M token.\n\n•\tFinalized LangChain for document loading, text splitting, and embeddings.\n\n**2. Data Preparation**\n\nData preparation involves collected PDF, loading, and processing the documents to make them suitable for query-answering tasks.\n\nLoading PDF Documents:\n\n•\tUsed PyPDFLoader from langchain_community.document_loaders to load documents from both files and folders.\n\n**3. Text Splitting**\n\nSplitting Documents:\n\n•\tUsed RecursiveCharacterTextSplitter to split documents into manageable chunks (2242) for processing.\n\n**4. Embeddings and Vector Store**\n\nGenerating Embeddings:\n\n•\tUsed SentenceTransformerEmbeddings to convert document chunks into embeddings.\n\n•\tAn embedding function was used to converts text data into numerical vectors (embeddings) that can capture the semantic meaning of the text. So that it make easy to integrate with vector stores.\n\nVector Store:\n\n•\tUsed Chromadb for storing and retrieving document embeddings.\n\n**5. Language Model Integration**\n\nUsing Together API:\n\n•\tTogether LLM was used for generating responses based on queries.\n\n**6. Building the Query Agent**\n\nPrompt Template:\n\n•\tDefined a prompt template for the LLM to generate responses.\n\nRetrievalQA Chain:\n\n•\tUsed the RetrievalQA chain from LangChain to create a query-answering pipeline.\n\nConversational Agent:\n\n•\tImplemented a class to handle the conversational context and history.\n\n**7. Streamlit User Interface**\n\nBuilding the UI:\n\n•\tUsed Streamlit to create a simple user interface for uploading files and entering queries so that easy testing/query input can be done.\n\n**Future Enhancements:**\n\n•\tCan add a list for a topic related search. So that only relevant topics are used.\n\n•\tOptimize the data preparation process so that a faster query can be generated.\n\n•\tCan add logs so that if its crash, it should be easy to debug.\n\n**Language Used:**\n\n![Python](https://img.shields.io/badge/Python-3776AB?style=for-the-badge\u0026logo=python\u0026logoColor=white)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmohith202%2Fema-chatbot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmohith202%2Fema-chatbot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmohith202%2Fema-chatbot/lists"}