https://github.com/myscale/vector-db-benchmark

Framework for benchmarking fully-managed vector databases
https://github.com/myscale/vector-db-benchmark

benchmark milvus myscale pinecone qdrant vector-database weaviate

Last synced: 2 days ago
JSON representation

Framework for benchmarking fully-managed vector databases

• Host: GitHub
• URL: https://github.com/myscale/vector-db-benchmark
• Owner: myscale
• License: apache-2.0
• Created: 2023-05-12T02:13:03.000Z (over 1 year ago)
• Default Branch: master
• Last Pushed: 2024-05-23T10:33:18.000Z (6 months ago)
• Last Synced: 2024-06-11T09:14:21.664Z (5 months ago)
• Topics: benchmark, milvus, myscale, pinecone, qdrant, vector-database, weaviate
• Language: Python
• Homepage: https://myscale.github.io/benchmark
• Size: 4.97 MB
• Stars: 58
• Watchers: 4
• Forks: 13
• Open Issues: 2
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-vector-database - MyScale Vector Database Benchmark 🚀

README

        
# MyScale Vector Database Benchmark 🚀

> [!IMPORTANT]

> Visit  for results.

This benchmark assesses the performance of **fully-managed** vector databases with typical workloads.

- For the setup, datasets, and detailed results of the benchmark, please visit .

- A summary of the benchmark results is available in our [blog post](https://blog.myscale.com/myscale-outperform-specialized-vectordb/).

Here's a preview of the results:

1. **Queries Per Second (QPS):** Higher QPS is preferable as it signifies greater throughput.

   - Throughput for Vector Search

   ![Throughput](images/qps.png)

   - Throughput for Filtered Vector Search

   ![Throughput](images/qps-filtered-search.png)

2. The **cost-performance ratio** is calculated by dividing the monthly cost by the QPS of the services per one hundred units. A lower ratio suggests better cost effectiveness.

   - Cost-performance ratio for Vector Search

   ![Monthly Cost ($) Per 100 QPS](images/cost-per-100-qps.png)

   - Cost-performance ratio for Filtered Vector Search

   ![Monthly Cost ($) Per 100 QPS](images/cost-per-100-qps-filtered-search.png)

## Run the Benchmark

First, install the necessary libraries on the client used for the benchmark.

```shell

pip install -r requirements.txt

```

Afterwards, follow the [Step-by-Step Guide for Benchmark](docs/step-by-step-guide-for-benchmark.md) to execute the benchmark for each cloud service. You can refer to [Results Visualization](docs/results-visualization.md) for visualizing the test results.

## Special Thanks

This repository is a fork of [qdrant/vector-db-benchmark](https://github.com/qdrant/vector-db-benchmark/), specifically tailored for fully-managed vector databases.