Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/benitomartin/rag-aws-qdrant
Serverless Application with AWS Lambda and Qdrant for Semantic Search
https://github.com/benitomartin/rag-aws-qdrant
aws aws-api-gateway aws-lambda langchain openai python qdrant ruff
Last synced: 7 days ago
JSON representation
Serverless Application with AWS Lambda and Qdrant for Semantic Search
- Host: GitHub
- URL: https://github.com/benitomartin/rag-aws-qdrant
- Owner: benitomartin
- Created: 2024-06-16T10:13:36.000Z (5 months ago)
- Default Branch: main
- Last Pushed: 2024-06-29T19:22:23.000Z (5 months ago)
- Last Synced: 2024-06-29T20:32:56.889Z (5 months ago)
- Topics: aws, aws-api-gateway, aws-lambda, langchain, openai, python, qdrant, ruff
- Language: Python
- Homepage:
- Size: 1.09 MB
- Stars: 9
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# RAG AWS API QDRANT 🚀
This repository contains a full Q&A pipeline using LangChain framework, Qdrant as vector database and AWS Lambda Function and API Gateway. The data used are research papers that can be loaded into the vector database, and the AWS Lambda Function processes the request using the retrieval and generation logic. Therefore it can use any other research paper from Arxiv.
This [Medium article](https://medium.com/@bmartinc80/building-a-serverless-application-with-aws-lambda-and-qdrant-for-semantic-search-ddb7646d4c2f) contains the complete instructions for the project set up.
The main steps taken to build the RAG pipeline can be summarize as follows:
* **Data Ingestion**: load data from https://arxiv.org
* **Indexing**: RecursiveCharacterTextSplitter for indexing in chunks
* **Vector Store**: Qdrant inserting metadata
* **QA Chain Retrieval**: RetrievalQA
* **AWS Lambda and API**: Process the request
* **Streamlit**: UI
Feel free to ⭐ and clone this repo 😉
## Tech Stack
## Project Structure
The project has been structured with the following files:
- `Dockerfile:` Dockerfile
- `.env_sample`: sample environmental variables
- `.gitattributes`: gitattributes
- `Makefile`: install requirements, formating, linting, and clean up
- `pyproject.toml`: linting and formatting using ruff
- `requirements.txt:` project requirements
- `create_vector_store.py:` script to create the collection in Qdrant
- `rag_app.py:` RAG Logic
- `lambda_function.py:` lambda function
- `test_lambda.py:` script to test the lambda function
- `build_and_deploy.sh:` script to create and push the Docker image and deploy it as an AWS Lambda function
## Project Set Up
The Python version used for this project is Python 3.10. You can follow along the medium article.
1. Clone the repo (or download it as a zip file):
```bash
git clone https://github.com/benitomartin/rag-aws-qdrant.git
```
2. Create the virtual environment named `main-env` using Conda with Python version 3.10:
```bash
conda create -n main-env python=3.10
conda activate main-env
```
3. Execute the `Makefile` script and install the project dependencies included in the requirements.txt:
```bash
pip install -r requirements.txt
or
make install
```
4. Creat **AWS Account**, credentials, and proper policies with full access to ECR and Lambda for the project to function correctly. Make sure to configure the appropriate credentials to interact with AWS services.
5. Make sure the `.env` file is complete and run the `build_and_deploy.sh script`
```bash
chmod +x build_and_deploy.sh
./build_and_deploy.sh
```
6. If you get timeout and/or memory error you can increase them:
```bash
aws lambda update-function-configuration \
--function-name rag-llm-lambda \
--timeout 300 \
--memory-size 1024
```
7. Create an API Endpoint as per medium article description
8. Run the streamlit app:
```bash
streamlit run streamlit_app.py
```
## Streamlit UI
