https://github.com/weaviate-tutorials/demo-multimodal-search

https://github.com/weaviate-tutorials/demo-multimodal-search

hacktoberfest

Last synced: about 1 year ago
JSON representation

• Host: GitHub
• URL: https://github.com/weaviate-tutorials/demo-multimodal-search
• Owner: weaviate-tutorials
• Created: 2023-09-25T18:31:10.000Z (over 2 years ago)
• Default Branch: main
• Last Pushed: 2023-12-29T20:30:49.000Z (over 2 years ago)
• Last Synced: 2025-03-25T05:27:26.163Z (about 1 year ago)
• Topics: hacktoberfest
• Language: Python
• Homepage:
• Size: 5.52 MB
• Stars: 6
• Watchers: 1
• Forks: 5
• Open Issues: 1
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# Multimodal (and multi-lingual) search demo

This app demonstrates Weaviate's multi-modal capabilities.

It uses a CLIP model to encode images and text into the same vector space.

## Prerequisites

To run this example, you need:

- Docker (to run Weaviate)

- Python 3.8 or higher

## Setup instructions

1. Install the Python dependencies.

    ```bash

    pip install -r requirements.txt

    ```

2. Run docker-compose to spin up an Weaviate instance and the CLIP inference container.

    ```bash

    docker compose up -d

    ```

3. Create the collection definition and import data, as well as some pre-prepared queries.

    ```bash

    python add_data.py

    ```

4. Start the Streamlit app.

    ```bash

    streamlit run app.py

    ```

## Usage instructions

Input a search query into the text box, or upload an image.

![mm_demo_screen](./mm_demo_screen.png)

This will return the top 6 results from the Weaviate instance. The results are sorted by the cosine similarity between the query and the vector representation of the object.

Example search results for an image query:

![mm_demo_by_img](./mm_demo_by_img.png)

Example search results for a text query:

![mm_demo_by_text](./mm_demo_by_txt.png)

Note - The model used is multi-lingual! That means it can understand queries in multiple languages. Try a search with an image, and then try inputting a description for that image in different languages!

## Dataset license

[Universe image](data/images/universe1.jpg) from [Unsplash](https://unsplash.com/photos/two-stars-in-the-middle-of-a-black-sky-fsH1KjbdjE8)

[Forest image](data/images/forest2.jpg) from [Unsplash](https://unsplash.com/photos/green-pine-trees-d6kSvT2xZQo)