https://github.com/mdarshad1000/vectorlite

VectorLite is a lightweight, easy-to-use, vector database with various indexing techniques. Primarily built to study how vector DBs work and how they can be used in practice.
https://github.com/mdarshad1000/vectorlite

Last synced: about 6 hours ago
JSON representation

VectorLite is a lightweight, easy-to-use, vector database with various indexing techniques. Primarily built to study how vector DBs work and how they can be used in practice.

• Host: GitHub
• URL: https://github.com/mdarshad1000/vectorlite
• Owner: mdarshad1000
• Created: 2024-11-03T23:36:34.000Z (16 days ago)
• Default Branch: master
• Last Pushed: 2024-11-18T22:24:06.000Z (1 day ago)
• Last Synced: 2024-11-18T23:21:51.149Z (1 day ago)
• Language: Python
• Size: 22.5 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
> This is a work in progress.

# VectorLite

VectorLite is a lightweight, easy-to-use, and scalable vector database. It's primarily built to explore the workings of vector databases and their practical applications.

## Components of VectorLite 

1. **Database**: The core container for tables and indexes.

2. **Table**: Stores and manages vectors and their metadata.

3. **Index**: Facilitates efficient vector search.

   - **Types of Indexes**:

     - Flat (Brute Force)

     - IVF (Inverted File Index)

     - HNSW (Hierarchical Navigable Small World)

     - PQ (Product Quantization)

     - SQ (Scalar Quantization)

4. **Embedder**: Converts text to vectors using various models.

   - Sentence Transformer

   - OpenAI Embeddings

   - Cohere

### Methods in Index Class

1. **Construct the Index**: Initializes a new index.

2. **Search Vectors in the Index**: Retrieves vectors closest to the query vector.

### Methods in Table Class

1. **Initialise the Table**: Sets up a new table for storing vectors.

2. **Add Vectors to the Table**: Inserts new vectors into the table.

3. **Search Vectors in the Table**: Finds vectors within the table based on a query.

4. **Query the Index**: Searches the index using table data.

5. **Delete Vectors from the Table**: Removes vectors from the table.

### TODO:

- [x] Implement Flat Index

- [x] Implement IVF Index

- [ ] Implement HNSW Index

- [ ] Implement PQ Index

- [ ] Implement SQ Index

- [ ] **Persist to Disk**: Implement serialization and deserialization for data persistence.

- [ ] **Add Metadata Filtering**: Allow filtering search results based on metadata.

- [ ] **Search Across Multiple Tables**: Introduce join functionality for querying multiple tables. 

- [ ]