https://github.com/maylad31/vector_sqlite

Faiss with sqlite
https://github.com/maylad31/vector_sqlite

faiss faiss-vector-database genai generative-ai rag retrieval-augmented-generation sqlite vector-database vector-search

Last synced: about 1 month ago
JSON representation

Faiss with sqlite

• Host: GitHub
• URL: https://github.com/maylad31/vector_sqlite
• Owner: maylad31
• Created: 2024-09-07T12:41:07.000Z (2 months ago)
• Default Branch: main
• Last Pushed: 2024-09-09T13:39:17.000Z (2 months ago)
• Last Synced: 2024-09-30T02:04:09.736Z (about 2 months ago)
• Topics: faiss, faiss-vector-database, genai, generative-ai, rag, retrieval-augmented-generation, sqlite, vector-database, vector-search
• Language: Python
• Homepage:
• Size: 25.4 KB
• Stars: 2
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# FAISS with SQLite

## Overview

Connect faiss with sqlite. Keep vectors in faiss, data in sqlite. Planning to add support for fts and may be alternatives for faiss. Why use this? For RAG, libraries like langchain are overcomplicated, going through docs and changing stuff is hard, and on top of that they change frequently. I prefer to have much more control on my pipeline and if you feel the same, this code might be a good starting point.



## Detailed Solution

### FAISS Index Setup

1. **Creating the Index**: Use `IndexIDMap2`over flat(IndexFlatL2) to create and manage vectors.

2. **Serializing the Index**: Serialize the FAISS index to save it locally.

3. **Deserializing the Index**: Load the FAISS index from the serialized file when needed.

### SQLite Metadata Management

1. **Creating the Database**: Initialize a SQLite database with a table to store metadata.

2. **Inserting Metadata**: Add records with IDs that correspond to FAISS vector IDs.

3. **Querying Metadata**: Retrieve metadata based on vector IDs obtained from FAISS searches.

### Example Workflow

**Perform Search**:

    - Query FAISS to find nearest neighbors for a given vector.

    - Use the resulting IDs to query SQLite and retrieve metadata.

## Installation

For faiss, check its github repo for instructions on how to install it. Sqlite comes with Python.

## Contact

If you have an interesting project, you may connect with me on https://www.linkedin.com/in/mayankladdha31/

Please star this repo if you found it useful.