{"id":28591041,"url":"https://github.com/valkey-io/valkey-search","last_synced_at":"2026-03-11T03:00:56.722Z","repository":{"id":273344372,"uuid":"895239212","full_name":"valkey-io/valkey-search","owner":"valkey-io","description":"valkey-search is a C++ module which extends valkey with vector search and secondary indexing capabilities. It enables users to index and query data stored in Valkey using complex queries with filters while maintaining high performance and scalability. ","archived":false,"fork":false,"pushed_at":"2026-03-07T07:09:05.000Z","size":94209,"stargazers_count":109,"open_issues_count":125,"forks_count":62,"subscribers_count":13,"default_branch":"main","last_synced_at":"2026-03-07T14:50:20.107Z","etag":null,"topics":["search-engine","valkey"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/valkey-io.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-11-27T20:27:49.000Z","updated_at":"2026-03-07T07:09:09.000Z","dependencies_parsed_at":"2025-03-12T07:24:36.748Z","dependency_job_id":"c2bb1f7b-70bc-4175-9a85-9c9ac984cb6d","html_url":"https://github.com/valkey-io/valkey-search","commit_stats":null,"previous_names":["valkey-io/valkey-search"],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/valkey-io/valkey-search","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/valkey-io%2Fvalkey-search","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/valkey-io%2Fvalkey-search/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/valkey-io%2Fvalkey-search/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/valkey-io%2Fvalkey-search/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/valkey-io","download_url":"https://codeload.github.com/valkey-io/valkey-search/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/valkey-io%2Fvalkey-search/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30362807,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-10T21:41:54.280Z","status":"ssl_error","status_checked_at":"2026-03-10T21:40:59.357Z","response_time":106,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["search-engine","valkey"],"created_at":"2025-06-11T09:02:02.180Z","updated_at":"2026-03-11T03:00:56.715Z","avatar_url":"https://github.com/valkey-io.png","language":"C++","funding_links":[],"categories":["C++"],"sub_categories":[],"readme":"# valkey-search\n\n**Valkey-Search** (BSD-3-Clause), provided as a Valkey module, is a high-performance Search engine\noptimized for AI-driven / Search / Analytics / Recommendation System related workloads. It delivers single-digit millisecond\nlatency and high QPS, capable of handling billions of vectors with over 99% recall as part of vector searches. It also provides\nsupport for hybrid / pure non vector workloads including Numeric, Tag, and Full-text searches.\n\nValkey-Search allows users to create indexes and perform searches, incorporating complex filters.\nUsers can index data using either **[Valkey Hash](hashes.md)** or **[Valkey-JSON](valkey-json.md)** data types.\nThe vector queries support Approximate Nearest Neighbor (ANN) search with HNSW and exact matching using K-Nearest Neighbors (KNN).\n\n## Supported Commands\n\n```plaintext\nFT.CREATE\nFT.DROPINDEX\nFT.INFO\nFT._LIST\nFT.SEARCH\nFT.AGGREGATE\n```\n\nFor a detailed description of the supported commands and configuration options, see the [Command Reference](https://valkey.io/commands/#search).\n\nFor comprehensive examples, refer to the [Quick Start Guide](https://github.com/valkey-io/valkey-search/blob/main/QUICK_START.md).\n\n## Scaling\n\nvalkey-search supports both **Standalone** and **Cluster** modes. Query processing and ingestion scale linearly with CPU cores in both modes. For large storage requirements, users can leverage Cluster mode for horizontal scaling of the keyspace.\n\nIf replica lag is acceptable, users can achieve horizontal query scaling by directing clients to read from replicas.\n\n## Performance\n\nvalkey-search achieves high performance by storing vectors in-memory and applying optimizations throughout the stack to efficiently utilize the host resources, such as:\n\n- **Parallelism:**  Threading model that enables lock-free execution in the read path.\n- **CPU Cache Efficiency:** Designed to promote efficient use of CPU cache.\n- **SIMD Optimizations:** Leveraging SIMD (Single Instruction, Multiple Data) for enhanced vector processing.\n\n## Hybrid Queries\n\nvalkey-search supports hybrid queries, combining Vector Similarity Search with filtering on indexed fields, such as **Numeric**, **Tag**, and **Text indexes**.\n\nThere are two primary approaches to hybrid queries:\n\n- **Pre-filtering:** Begin by filtering the dataset and then perform an exact similarity search. This works well when the filtered result set is small but can be costly with larger datasets.\n- **Post-filtering:** Perform the similarity search first, then filter the results. This is suitable when the filter-qualified result set is large but may lead to empty or lower than expected amount of results.\n\nvalkey-search uses a **hybrid approach** with a query planner that selects the most efficient query execution path between:\n\n- **Pre-filtering**\n- **Inline-filtering:** Filters results during the similarity search process.\n\n## Skip Index on RDB Load\n\nvalkey-search includes an operational feature to mitigate issues in indexes. If set to true, index schema is loaded from rdb, but indexes are not loaded, they are created empty and will be re-filled by the backfill process.\n\n### Configuration\n\nThe config is non-modifiable and should be provided to search module on load.\n```sh\nvalkey-server --loadmodule /path/to/libsearch.so --skip-rdb-load yes\n```\n\n### Benefits\n\n- **Index consistency**: Recover from corrupted or inconsistent vector indexes\n- **Deleted Memory reclaim**: If index has many deleted vectors rebuilding the index during startup will free this memory\n\nWhen enabled, vector indexes are rebuilt via the backfill process while non-vector indexes (TAG, NUMERIC) continue to work immediately. \n\n## Build Instructions\n\n### Install basic tools\n\n#### Ubuntu / Debian\n\n```sh\nsudo apt update\nsudo apt install -y clangd          \\\n                    build-essential \\\n                    g++             \\\n                    cmake           \\\n                    libgtest-dev    \\\n                    ninja-build     \\\n                    libssl-dev      \\\n                    clang-tidy      \\\n                    clang-format    \\\n                    libsystemd-dev\n```\n\n\n**IMPORTANT**: building valkey-search requires GCC version 12 or higher, or Clang version 16 or higher. For Debian/Ubuntu, in case a lower version of GCC is installed, you may upgrade to gcc/g++ 12 with:\n\n```sh\nsudo apt update\nsudo apt install -y gcc-12 g++-12\nsudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-12 1000\nsudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 1000\n```\n\n#### Red Hat / CentOS / Amazon Linux\n\n```sh\nsudo yum update\nsudo yum install -y gcc             \\\n                    gcc-c++         \\\n                    cmake           \\\n                    gtest           \\\n                    gtest-devel     \\\n                    ninja-build     \\\n                    openssl-devel   \\\n                    clang-tidy      \\\n                    clang-format    \\\n                    systemd-devel\n```\n\n### Build the module\n\nvalkey-search uses **CMake** for its build system. To simplify, a build script is provided. To build the module, run:\n\n```sh\n./build.sh\n```\n\nTo view the available arguments, use:\n\n```sh\n./build.sh --help\n```\n\nRun unit tests with:\n\n```sh\n./build.sh --run-tests\n```\n\n#### Integration Tests\n\nInstall required dependencies (Ubuntu / Debian):\n\n```sh\nsudo apt update\nsudo apt install -y lsb-release      \\\n                     curl            \\\n                     coreutils       \\\n                     libsystemd-dev  \\\n                     python3-pip     \\\n                     python3.12-venv \\\n                     locales-all     \\\n                     locales         \\\n                     gpg\n\ncurl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg\necho \"deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main\" | sudo tee /etc/apt/sources.list.d/redis.list\nsudo apt update\nsudo apt-get install -y memtier-benchmark\n```\n\nRun the integration tests with:\n\n```sh\n./build.sh --run-integration-tests\n```\n\n## Load the Module\n\nTo start Valkey with the module, use the `--loadmodule` option:\n\n```sh\nvalkey-server --loadmodule /path/to/libsearch.so\n```\n\nTo enable JSON support, load the JSON module as well:\n\n```sh\nvalkey-server --loadmodule /path/to/libsearch.so --loadmodule /path/to/libjson.so\n```\n\nFor optimal performance, valkey-search will match the number of worker threads to the number of CPU cores on the host. You can override this with:\n\n```sh\nvalkey-server \"--loadmodule /path/to/libsearch.so --reader-threads 64 --writer-threads 64\"\n```\n\n## Development Environment\n\nFor development purposes, it is recommended to use \u003cb\u003eVSCode\u003c/b\u003e, which is already configured to run within a Docker container and is integrated with clang-tidy and clang-format. Follow these steps to set up your environment:\n\n1. \u003cb\u003eInstall VSCode Extensions:\u003c/b\u003e\n    - Install the `Dev Containers` extension by Microsoft in VSCode.\n    - Note: Building the code may take some time, and it is important to use a host with decent CPU capacity. If you prefer, you can use a remote host. In that case, also install the following extensions:\n      - `Remote - SSH` by Microsoft\n      - `Remote Explorer` by Microsoft\n\n2. \u003cb\u003eRun the dev container setup script\u003c/b\u003e\n    - Issue the following command from the cloned repo root directory:\n        ```sh\n        .devcontainer/setup.sh\n        ```\n\n3. \u003cb\u003eOpen the Repository in VSCode:\u003c/b\u003e\n    - On your local machine, open the root directory of the cloned valkey-search repository in VSCode.\n    - If the repository is located on a remote host:\n      1. Press Ctrl+Shift+P (Windows/Linux) or Cmd+Shift+P (macOS) to open the Command Palette.\n      2. Type Remote-SSH: Connect to Host and select it.\n      3. Choose the appropriate host and provide any necessary authentication details.\n\n       Once connected, VSCode will open the repository in the context of the remote host.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvalkey-io%2Fvalkey-search","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvalkey-io%2Fvalkey-search","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvalkey-io%2Fvalkey-search/lists"}