{"id":21296087,"url":"https://github.com/olonok69/rag-speak-with-your-unstructured-data","last_synced_at":"2025-10-06T01:38:10.940Z","repository":{"id":262501791,"uuid":"885314160","full_name":"olonok69/RAG-Speak-with-your-Unstructured-Data","owner":"olonok69","description":"RAG demo with Langchain , Gemini and NVidia NIM Models","archived":false,"fork":false,"pushed_at":"2024-11-21T12:53:51.000Z","size":23857,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-07T06:39:51.447Z","etag":null,"topics":["chromadb","elasticsearch","faiss","gemini","langchain","milvus","nvidia-nim","python","streamlit","vertex-ai"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/olonok69.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-11-08T10:52:18.000Z","updated_at":"2025-03-15T18:12:59.000Z","dependencies_parsed_at":"2025-05-07T06:42:38.180Z","dependency_job_id":null,"html_url":"https://github.com/olonok69/RAG-Speak-with-your-Unstructured-Data","commit_stats":null,"previous_names":["olonok69/rag-demo","olonok69/rag-speak-with-your-unstructured-data"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/olonok69/RAG-Speak-with-your-Unstructured-Data","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olonok69%2FRAG-Speak-with-your-Unstructured-Data","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olonok69%2FRAG-Speak-with-your-Unstructured-Data/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olonok69%2FRAG-Speak-with-your-Unstructured-Data/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olonok69%2FRAG-Speak-with-your-Unstructured-Data/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/olonok69","download_url":"https://codeload.github.com/olonok69/RAG-Speak-with-your-Unstructured-Data/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olonok69%2FRAG-Speak-with-your-Unstructured-Data/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278547772,"owners_count":26004772,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-05T02:00:06.059Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chromadb","elasticsearch","faiss","gemini","langchain","milvus","nvidia-nim","python","streamlit","vertex-ai"],"created_at":"2024-11-21T14:19:46.583Z","updated_at":"2025-10-06T01:38:10.923Z","avatar_url":"https://github.com/olonok69.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# RAG: Speak with your Unstructured Data\nThe explanation in Video is available in my youtube Channel https://www.youtube.com/@themachinelearningengineer\n\n### Subscribe\nhttps://www.youtube.com/@themachinelearningengineer?sub_confirmation=1\n\n### Videos in English:\n- Part 1 Use cases  https://youtu.be/UZd-Sb0qmZI\n- Part 2 Review Code https://youtu.be/5c526BAxyU4\n- Part 3 Docker and Docker-compose https://youtu.be/yUk9jHY2QCY\n\n### Videos in Spanish\n- Part 1 Use cases  https://youtu.be/qjgQUs4eJ34\n- Part 2 Review Code https://youtu.be/cIf-ITLsYBk\n- Part 3 Docker and Docker-compose  https://youtu.be/J2vQapnbFGU\n\n\n# Introduction \nThis repo constains an E2E Rag application which showcases diferent use and Integrations of Langchain with other technologies to \"chat\" with your Unstructured Data (audio, video, images, pdf, excel, csv and html).\n\nThe main technology we will use will be Langchain, as LLMs we will use Gemini Pro Fast, Microsoft Phi3.5 Mini and LLama 3.2 3B, the last two hosted on Nvidia NIM. \n\nVectorStores Elastic, Choma, Faiss and Vilmus. \n\nEmbeddings Models: Gemini and Nvidia embeddings\n\nAs a UX tool and application server we will use Streamlit\n\nDocker and Docker compose to create containers\n\n![alt text](images/intro.png)\n\n# High Level Description of Functionalities\n\n### Speak with your pdfs\nHere we load a large pdf and extract text with langchain Dataloaders. Then create embeddings with Gemini embedding models (Ex. models/embedding-001) and add them to Elastic or Faiss vector Stores using their Integration with langchain. Finally we can query/chat with the document using an LLM ,default gemini-1.5-flash-002, but you can use any multimodal LLM in Gemini API.\n\n![alt text](images/1image.png)\n### Extract pdfs\nHere we load a pdf and extract information based in a Data model created with Pydantic. we are using the integration of Langchain with that library and simply use a multi-modal LLM and a prompt in Natural language to find all the information related to a company in this case. The information obtained from the llm then it is parsed and sanitized using the pydantic clase. See EntityDataExtraction class [here](src/data_models.py). As LLM ,default gemini-1.5-flash-002, but you can use any text LLM in Gemini API.\n\n![alt text](images/2image.png)\n\n### Extract excel\nHere we use the integration of Langchain with Pandas Library to manipulate excel or csv files. We send an information to extract data from a file and then the LLM will generate python code using pandas to get what it is requested. The code it is executed internally in lanchain/pandas class and the output it is returned to the LLM to evaluate if it has all the need it information or need refinement and create more pandas code. One the LLM consider it has all the information to build an answer, this is returned to the user. As LLM ,default gemini-1.5-flash-002, but you can use any text LLM in Gemini API. Example [here](docs/excel.md)\n\n![alt text](images/3image.png)\n\n### Extract Image\nHere we load an Image to a multimodal model and send a prompt asking for some information. We can use as well a similar approach that we did on extract from pdf using a pydantic data model. As LLM,default gemini-1.5-flash-002, but you can use any multimodal LLM in Gemini API.\n\n![alt text](images/4image.png)\n\n### Extract Audio\nHere we load an audio to a multimodal model and send a prompt asking for some information. We can use as well a similar approach that we did on extract from pdf using a pydantic data model. As LLM,default gemini-1.5-flash-002, but you can use any multimodal LLM in Gemini API.\n\n![alt text](images/5image.png)\n\n### Extract Video\nHere we load an video to a multimodal model and send a prompt asking for some information. We can use as well a similar approach that we did on extract from pdf using a pydantic data model. As LLM,default gemini-1.5-flash-002, but you can use any multimodal LLM in Gemini API.\n\n![alt text](images/6image.png)\n\n### Extract html\nHere we load a number of html web pages and extract text and tables using BeautifulSoup. Here we use the infrastructure of NVIDIA NIM and its SDK to call models for embeddings and generate text:\n- NVIDIA_EMBEDDINGS=NV-Embed-QA\n- NVIDIA_MODEL=microsoft/phi-3-small-128k-instruct (here we can use as well LLama3.2 3b Instruct or any other llm text able).\n\nTo split the text in this case we use the following model\nTEXT_SPLITTER_MODEL=intfloat/e5-large-v2\n\nAs vector store we use Milvus or Elastic\n\n![alt text](images/7image.png)\n\n### Speak pdf multi\nHere we use a tool like pymupdf or pymupdf4llm to extract text images and tables from a pdf. One we have these elements, we use a Multimodal LLM, in this case gemini-1.5-flash-002, to create descriptive summaries of them. Then we use a embeddings model to to create embeddings of those summaries. \nOne we have the embeddings we create id,s for all those summaries and embeddings and we upload them to a vector store (Chroma in this case) with metadata related to the document and page where the origen of the summary it is located. Last step we upload in binary format(images here) or the original text to any specific summary to the Chroma DocStore and link embedding, summary and original source using the id. \n\nWhen we query or ask the model, we will create an embedding of the query , get the top 5 more relevant vectors similar to the query and using the ID,s of those documents, Chroma will return the binary data source of those ID,s. \n\nThen the LLM will produce an anwer using the binary data (images or other unstructured format), original text returned from the docStore and the query introduced by the user\n\n![alt text](images/8image.png)\n\n# Getting Started\nDocumentation Elastic --\u003e https://www.elastic.co/blog/getting-started-with-the-elastic-stack-and-docker-compose\n\n1.\tInstallation process\n    - create docker : docker build --pull --rm --build-arg user_app=app -f \"Dockerfile\" -t demo-ai-rag:latest \".\"\n    - create compose : docker compose -f \"docker/docker-compose.yml\" up --build\n    - create .env file. example [here](docs/env.md) and place in the keys folder\n    - copy your Goocle cloude service account .json file to keys folder\n    - create user for streamlit and place config.yaml in keys folder\n2.\tSoftware dependencies\n    - python 3.11\n    - requiremrents file [here](requirements.txt)\n3.\tLatest releases\n4.\tAPI references\n\n# Project Structure\n\n![alt text](images/project.png)\n\n# How to create Users and Passwords to this app\n\ncreate a yaml file with name config.yaml in the keys folder\nhere you have and example with 2 users user1 and nvidiademo\n```\ncredentials:\n  usernames:\n    user1:\n      email: test@gmail.com\n      name: user1 \n      password: $2b$12$ZoeMqMyYQZqFGM7iF2bugeYG3E1kL7RA8jLkTvPcpdIlpIz46Yz/i # To be replaced with hashed password\n    nvidiademo:\n      email: demo@nvidia.com\n      name: nvidiademo\n      password: $2b$12$vL2J2PauSXISet6wZIM0j.DFZdq7cyrJ/2TQEn8UT.NyxZuKGfEZa # To be replaced with hashed password\ncookie:\n  expiry_days: 0\n  key: laPalma50 # Must be string\n  name: thecokkei\n  ```\n\n\n  ###### To generate the hashes\n  - Go to the folder hasher and edit the file ex.yaml\n  - Fill email , name and password with your values\n  - Run hasher.py\n  - Replace the password in config.yaml with its hash value\n\n  ```\n  credentials:\n  usernames:\n    user1:\n      email: test@gmail.com\n      name: user1 \n      password:  text#me\n    nvidiademo:\n      email: demo@nvidia.com\n      name: nvidiademo\n      password:  laPalma50 \ncookie:\n  expiry_days: 0\n  key: laPalma50 # Must be string\n  name: thecokkei\n  ```\n\n  OUTPUT\n\n  ```\n  {'usernames': {'user1': {'email': 'test@gmail.com', 'name': 'user1', 'password': '$2b$12$ZoeMqMyYQZqFGM7iF2bugeYG3E1kL7RA8jLkTvPcpdIlpIz46Yz/i'}, 'nvidiademo': {'email': 'demo@nvidia.com', 'name': 'nvidiademo', 'password': '$2b$12$vL2J2PauSXISet6wZIM0j.DFZdq7cyrJ/2TQEn8UT.NyxZuKGfEZa'}}}\n  ```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Folonok69%2Frag-speak-with-your-unstructured-data","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Folonok69%2Frag-speak-with-your-unstructured-data","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Folonok69%2Frag-speak-with-your-unstructured-data/lists"}