Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/farhaj499/rag_with_pineconedb


https://github.com/farhaj499/rag_with_pineconedb

Last synced: 21 days ago
JSON representation

Awesome Lists containing this project

README

        

# Agentic AI RAG System with Pinecone and Gemini (Local PDFs)

This project implements a Retrieval Augmented Generation (RAG) system that answers questions based on two local PDF documents stored in Google Drive. It uses Pinecone as a vector database for efficient retrieval of relevant information and Gemini (or another LLM) to generate natural language responses.

## Overview

This RAG system operates as follows:

1. **Data Loading:** Two PDF files are loaded from a designated Google Drive folder using `PyPDFDirectoryLoader`.
2. **Text Chunking:** The extracted text from both PDFs is divided into smaller chunks to optimize retrieval and manage context for the LLM.
3. **Embedding Generation:** Sentence embeddings are created for each text chunk using the "sentence-transformers/all-mpnet-base-v2" Hugging Face model.
4. **Vector Database Storage:** The text chunks and their corresponding embeddings are stored in a Pinecone vector database.
5. **Retrieval and Question Answering:** When a user asks a question, the system generates an embedding for the query, retrieves the most similar text chunks from Pinecone, and uses Gemini (or an alternative LLM) to generate a natural language answer based on the retrieved context.

## Technologies Used

* **Pinecone:** Vector database for storing and retrieving embeddings.
* **Hugging Face Transformers:** For generating sentence embeddings using "sentence-transformers/all-mpnet-base-v2".
* **Gemini (or alternative LLM):** Large Language Model for generating natural language responses.
* **Python:** Programming language for implementation.
* **LangChain (Optional but highly recommended):** For streamlining the RAG pipeline.
* **PyPDFDirectoryLoader (LangChain):** For loading PDF documents from a directory.