{"id":13596977,"url":"https://github.com/myscale/vector-db-benchmark","last_synced_at":"2026-01-28T00:54:46.032Z","repository":{"id":165405588,"uuid":"639686040","full_name":"myscale/vector-db-benchmark","owner":"myscale","description":"Framework for benchmarking fully-managed vector databases","archived":false,"fork":false,"pushed_at":"2024-09-24T02:57:10.000Z","size":5245,"stargazers_count":68,"open_issues_count":2,"forks_count":14,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-11-06T21:43:45.283Z","etag":null,"topics":["benchmark","milvus","myscale","pinecone","qdrant","vector-database","weaviate"],"latest_commit_sha":null,"homepage":"https://myscale.github.io/benchmark","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/myscale.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-12T02:13:03.000Z","updated_at":"2024-11-01T10:18:37.000Z","dependencies_parsed_at":null,"dependency_job_id":"df7b8c6f-227f-4dfc-b0e7-c905aa5dc8a3","html_url":"https://github.com/myscale/vector-db-benchmark","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/myscale%2Fvector-db-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/myscale%2Fvector-db-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/myscale%2Fvector-db-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/myscale%2Fvector-db-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/myscale","download_url":"https://codeload.github.com/myscale/vector-db-benchmark/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248135219,"owners_count":21053642,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","milvus","myscale","pinecone","qdrant","vector-database","weaviate"],"created_at":"2024-08-01T17:00:16.846Z","updated_at":"2026-01-28T00:54:41.001Z","avatar_url":"https://github.com/myscale.png","language":"Python","funding_links":[],"categories":["Benchmarks \u0026 Evaluation","Comparisons"],"sub_categories":[],"readme":"# MyScale Vector Database Benchmark 🚀\n\n\u003e [!IMPORTANT]\n\u003e Visit \u003chttps://myscale.github.io/benchmark/#/benchmark\u003e for results.\n\nThis benchmark assesses the performance of **fully-managed** vector databases with typical workloads.\n\n- For the setup, datasets, and detailed results of the benchmark, please visit \u003chttps://myscale.github.io/benchmark\u003e.\n- A summary of the benchmark results is available in our [blog post](https://blog.myscale.com/myscale-outperform-specialized-vectordb/).\n\nHere's a preview of the results:\n\n1. **Queries Per Second (QPS):** Higher QPS is preferable as it signifies greater throughput.\n   - Throughput for Vector Search\n   ![Throughput](images/qps.png)\n   - Throughput for Filtered Vector Search\n   ![Throughput](images/qps-filtered-search.png)\n2. The **cost-performance ratio** is calculated by dividing the monthly cost by the QPS of the services per one hundred units. A lower ratio suggests better cost effectiveness.\n   - Cost-performance ratio for Vector Search\n   ![Monthly Cost ($) Per 100 QPS](images/cost-per-100-qps.png)\n   - Cost-performance ratio for Filtered Vector Search\n   ![Monthly Cost ($) Per 100 QPS](images/cost-per-100-qps-filtered-search.png)\n\n## Run the Benchmark\n\nFirst, install the necessary libraries on the client used for the benchmark.\n\n```shell\npip install -r requirements.txt\n```\n\nAfterwards, follow the [Step-by-Step Guide for Benchmark](docs/step-by-step-guide-for-benchmark.md) to execute the benchmark for each cloud service. You can refer to [Results Visualization](docs/results-visualization.md) for visualizing the test results.\n\n## Special Thanks\n\nThis repository is a fork of [qdrant/vector-db-benchmark](https://github.com/qdrant/vector-db-benchmark/), specifically tailored for fully-managed vector databases.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmyscale%2Fvector-db-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmyscale%2Fvector-db-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmyscale%2Fvector-db-benchmark/lists"}