https://github.com/dnth/postgresql-multimodal-retrieval
Vector/Hybrid Search & Retrieval on PostgreSQL database using Vision Language Model.
https://github.com/dnth/postgresql-multimodal-retrieval
huggingface-datasets multimodal-deep-learning pgvector postgresql retrieval search-engine vector-database vision-language-model
Last synced: about 1 month ago
JSON representation
Vector/Hybrid Search & Retrieval on PostgreSQL database using Vision Language Model.
- Host: GitHub
- URL: https://github.com/dnth/postgresql-multimodal-retrieval
- Owner: dnth
- License: apache-2.0
- Created: 2024-07-05T13:07:22.000Z (11 months ago)
- Default Branch: main
- Last Pushed: 2024-07-29T15:18:40.000Z (10 months ago)
- Last Synced: 2024-12-02T07:45:55.614Z (6 months ago)
- Topics: huggingface-datasets, multimodal-deep-learning, pgvector, postgresql, retrieval, search-engine, vector-database, vision-language-model
- Language: Jupyter Notebook
- Homepage:
- Size: 16.5 MB
- Stars: 3
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# postgresql-multimodal-retrieval
Multimodal retrieval using Vision Language Model with PostgreSQL database - A full stack implementation.
+ Database: PostgreSQL
+ Vision Language Model: OpenAI CLIP (`transformers` implementation)
+ Dataset: Hugging Face Datasets
+ Frontend: Flet / Gradio
+ Deployment: Docker
+ Infrastructure: Hugging Face Spaces
Features:
+ Image to image search
+ Text to image search
+ Hybrid search
## Setting up
### Create a conda environment
```bash
conda create -n postgresql-multimodal python=3.10
conda activate postgresql-multimodal
```
### Install PostgreSQL
```bash
conda install -c conda-forge postgresql
psql --version
```
### Install pgvector
```bash
conda install -c conda-forge pgvector
```
### Initialize and start PostgreSQL
```bash
initdb -D mylocal_db
pg_ctl -D mylocal_db -l logfile start
```
### Create a database
```bash
createuser retrieval_user
createdb retrieval_db -O retrieval_user
```
### Install packages
```bash
pip install -r requirements.txt
```
Install the pgmmr package
```
pip install -e .
```
## Usage
### Compute embeddings
Run
```
python compute_embeddings.py
```
This will compute embeddings for all the images in the database.
### Query
Run hybrid search (vector search and keyword search). The results are ranked using RRF.
```
python query.py "a cat and flower" --num_results 12
```
### Gradio App
Run
```
python gradio_app.py
```
### Flet App
```
flet run flet_app.py
```
## References
+ https://minimaxir.com/2024/06/pokemon-embeddings/
+ https://github.com/minimaxir/pokemon-embeddings/tree/main
+ https://towardsdatascience.com/quick-fire-guide-to-multi-modal-ml-with-openais-clip-2dad7e398ac0
+ https://github.com/pgvector/pgvector
+ https://huggingface.co/nomic-ai
+ https://huggingface.co/datasets/wanghaofan/pokemon-wiki-captions
https://github.com/jamescalam/clip-demos/blob/main/tutorials/00-get-started/00-get-started.ipynb