{"id":13410064,"url":"https://github.com/matsui528/rii","last_synced_at":"2026-04-10T10:03:08.434Z","repository":{"id":32854924,"uuid":"142134229","full_name":"matsui528/rii","owner":"matsui528","description":"Fast and memory-efficient ANN with a subset-search functionality","archived":false,"fork":false,"pushed_at":"2025-07-01T05:29:27.000Z","size":291,"stargazers_count":153,"open_issues_count":3,"forks_count":21,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-12-16T14:00:04.731Z","etag":null,"topics":["approximate-nearest-neighbor-search","data-compression","nearest-neighbor-search","product-quantization","pybind11"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/matsui528.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-07-24T09:12:54.000Z","updated_at":"2025-12-10T07:05:49.000Z","dependencies_parsed_at":"2022-08-07T18:15:38.041Z","dependency_job_id":"ea6cd952-92dd-4d3e-8dbe-21315a328d39","html_url":"https://github.com/matsui528/rii","commit_stats":{"total_commits":157,"total_committers":6,"mean_commits":"26.166666666666668","dds":0.6114649681528662,"last_synced_commit":"99910515d3a562d3050d9428c27e19476a47d8ba"},"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"purl":"pkg:github/matsui528/rii","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matsui528%2Frii","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matsui528%2Frii/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matsui528%2Frii/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matsui528%2Frii/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/matsui528","download_url":"https://codeload.github.com/matsui528/rii/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matsui528%2Frii/sbom","scorecard":{"id":625325,"data":{"date":"2025-08-11","repo":{"name":"github.com/matsui528/rii","commit":"7fb5d59655cd1ef022d005b5951bd8346ea38fd3"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":4.1,"checks":[{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Maintained","score":2,"reason":"3 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 2","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Code-Review","score":1,"reason":"Found 1/7 approved changesets -- score normalized to 1","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/build.yml:1","Warn: no topLevel permission defined: .github/workflows/publish.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/build.yml:40: update your workflow using https://app.stepsecurity.io/secureworkflow/matsui528/rii/build.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/build.yml:42: update your workflow using https://app.stepsecurity.io/secureworkflow/matsui528/rii/build.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/build.yml:52: update your workflow using https://app.stepsecurity.io/secureworkflow/matsui528/rii/build.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/publish.yml:19: update your workflow using https://app.stepsecurity.io/secureworkflow/matsui528/rii/publish.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/publish.yml:22: update your workflow using https://app.stepsecurity.io/secureworkflow/matsui528/rii/publish.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/publish.yml:38: update your workflow using https://app.stepsecurity.io/secureworkflow/matsui528/rii/publish.yml/main?enable=pin","Warn: pipCommand not pinned by hash: .github/workflows/build.yml:50","Warn: pipCommand not pinned by hash: .github/workflows/build.yml:51","Warn: pipCommand not pinned by hash: .github/workflows/publish.yml:29","Warn: pipCommand not pinned by hash: .github/workflows/publish.yml:30","Warn: pipCommand not pinned by hash: .github/workflows/publish.yml:31","Info:   0 out of   4 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   2 third-party GitHubAction dependencies pinned","Info:   0 out of   5 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Packaging","score":10,"reason":"packaging workflow detected","details":["Info: Project packages its releases by way of GitHub Actions.: .github/workflows/publish.yml:9"],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'main'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 30 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-21T06:14:34.125Z","repository_id":32854924,"created_at":"2025-08-21T06:14:34.125Z","updated_at":"2025-08-21T06:14:34.125Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31637748,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-10T07:40:12.752Z","status":"ssl_error","status_checked_at":"2026-04-10T07:40:11.664Z","response_time":98,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["approximate-nearest-neighbor-search","data-compression","nearest-neighbor-search","product-quantization","pybind11"],"created_at":"2024-07-30T20:01:04.784Z","updated_at":"2026-04-10T10:03:08.416Z","avatar_url":"https://github.com/matsui528.png","language":"C++","funding_links":[],"categories":["C++"],"sub_categories":[],"readme":"\u003cimg src=\"http://yusukematsui.me/project/rii/img/logotype97.png\" width=\"300\"\u003e\n\n![Build](https://github.com/matsui528/rii/workflows/Build/badge.svg)\n[![Documentation Status](https://readthedocs.org/projects/rii/badge/?version=latest)](https://rii.readthedocs.io/en/latest/?badge=latest)\n[![PyPI version](https://badge.fury.io/py/rii.svg)](https://badge.fury.io/py/rii)\n[![Downloads](https://pepy.tech/badge/rii)](https://pepy.tech/project/rii)\n\n\n\n\n\n\nReconfigurable Inverted Index (Rii): IVFPQ-based fast and memory efficient approximate nearest neighbor search method\nwith a subset-search functionality.\n\nReference:\n- [Y. Matsui](http://yusukematsui.me/), [R. Hinami](http://www.satoh-lab.nii.ac.jp/member/hinami/), and [S. Satoh](http://research.nii.ac.jp/~satoh/index.html), \"**Reconfigurable Inverted Index**\", ACM Multimedia 2018 (oral). [**[paper](https://dl.acm.org/ft_gateway.cfm?id=3240630)**] [**[project](http://yusukematsui.me/project/rii/rii.html)**]\n\n## Summary of features\n![](http://yusukematsui.me/project/rii/img/teaser1.png)  |  ![](http://yusukematsui.me/project/rii/img/teaser2.png)\n:---:|:---:\nThe search can be operated for a subset of a database. | Rii remains fast even after many new items are added.\n- Fast and memory efficient ANN. Rii enables you to run billion-scale search in less than 10 ms.\n- You can run the search over a **subset** of the whole database\n- Rii Remains fast even after many vectors are newly added (i.e., the data structure can be **reconfigured**)\n\n\n## Installing\nYou can install the package via pip. This library works with Python 3.6+ on linux/mac/wsl/Windows10\n\n```\npip install rii\n```\n\n\n\u003cdetails\u003e\n  \u003csummary\u003eFor windows (maintained by @ashleyabraham)\u003c/summary\u003e\n\n  ### Installing in Windows 10 via `pip install`\n  Requires MS Visual Studio Build tools C++ 14.0 or 14.1 toolset or above to compile and install via pip install\n\n  ### Pre-compiled binary for Windows 10 \n  Pre-compiled binaries doesn't require MS Visual Studio Build tools\n\n  ```\n  #Python 3.8\n  pip install https://github.com/ashleyabraham/rii/releases/download/v0.2.8/rii-0.2.8-cp38-cp38-win_amd64.whl\n  ```\n  ```\n  #Python 3.7\n  pip install https://github.com/ashleyabraham/rii/releases/download/v0.2.8/rii-0.2.8-cp37-cp37m-win_amd64.whl\n```\n\n  #### OpenMP\n  OpenMP requires libomp140_x86_64.dll to compile in windows, which is part of MS Visual Studio Build tools and it is not redistributable.\n  \n  In order to use OpenMP 3.0 /openmp:llvm flag is used which causes warnings of multiple libs loading, use at your descretion when used with other parallel processing library loadings. To supress use\n\n  `SET KMP_DUPLICATE_LIB_OK=TRUE`\n\n  #### SIMD\n  The /arch:AVX2 flag is used in MSVC to set appropriate SIMD preprocessors and compiler intrinsics\n\n\u003c/details\u003e\n\n\n\n\n## [Documentation](https://rii.readthedocs.io/en/latest/index.html)\n- [Tutorial](https://rii.readthedocs.io/en/latest/source/tutorial.html)\n- [Tips](https://rii.readthedocs.io/en/latest/source/tips.html)\n- [API](https://rii.readthedocs.io/en/latest/source/api.html)\n\n\n## Usage\n\n### Basic ANN\n\n```python\nimport rii\nimport nanopq\nimport numpy as np\n\nN, Nt, D = 10000, 1000, 128\nX = np.random.random((N, D)).astype(np.float32)  # 10,000 128-dim vectors to be searched\nXt = np.random.random((Nt, D)).astype(np.float32)  # 1,000 128-dim vectors for training\nq = np.random.random((D,)).astype(np.float32)  # a 128-dim vector\n\n# Prepare a PQ/OPQ codec with M=32 sub spaces\ncodec = nanopq.PQ(M=32).fit(vecs=Xt)  # Trained using Xt\n\n# Instantiate a Rii class with the codec\ne = rii.Rii(fine_quantizer=codec)\n\n# Add vectors\ne.add_configure(vecs=X)\n\n# Search\nids, dists = e.query(q=q, topk=3)\nprint(ids, dists)  # e.g., [7484 8173 1556] [15.06257439 15.38533878 16.16935158]\n```\nNote that you can construct a PQ codec and instantiate the Rii class at the same time if you want.\n```python\ne = rii.Rii(fine_quantizer=nanopq.PQ(M=32).fit(vecs=Xt))\ne.add_configure(vecs=X)\n```\nFurthermore, you can even write them in one line by chaining a function.\n```python\ne = rii.Rii(fine_quantizer=nanopq.PQ(M=32).fit(vecs=Xt)).add_configure(vecs=X)\n```\n\n### Subset search\n\n```python\n# The search can be conducted over a subset of the database\ntarget_ids = np.array([85, 132, 236, 551, 694, 728, 992, 1234]) # Specified by IDs\n# For windows, you must specify dtype=np.int64 as follows.\n# target_ids = np.array([85, 132, 236, 551, 694, 728, 992, 1234], dtype=np.int64)  \n\nids, dists = e.query(q=q, topk=3, target_ids=target_ids)\nprint(ids, dists)  # e.g., [728  85 132] [14.80522156 15.92787838 16.28690338]\n```\n\n### Data addition and reconfiguration\n\n```python\n# Add new vectors\nX2 = np.random.random((1000, D)).astype(np.float32)\ne.add(vecs=X2)  # Now N is 11000\ne.query(q=q)  # Ok. (0.12 msec / query)\n\n# However, if you add quite a lot of vectors, the search might become slower\n# because the data structure has been optimized for the initial item size (N=10000)\nX3 = np.random.random((1000000, D)).astype(np.float32) \ne.add(vecs=X3)  # A lot. Now N is 1011000\ne.query(q=q)  # Slower (0.96 msec/query)\n\n# In such case, run the reconfigure function. That updates the data structure\ne.reconfigure()\ne.query(q=q)  # Ok. (0.21 msec / query)\n```\n\n### I/O by pickling\n```python\nimport pickle\nwith open('rii.pkl', 'wb') as f:\n    pickle.dump(e, f)\nwith open('rii.pkl', 'rb') as f:\n    e_dumped = pickle.load(f)  # e_dumped is identical to e\n```\n\n### Util functions\n```python\n# Print the current parameters\ne.print_params()\n\n# Delete all PQ-codes and posting lists. fine_quantizer is kept.\ne.clear()\n\n# You can switch the verbose flag\ne.verbose = False\n\n# You can merge two Rii instances if they have the same fine_quantizer\ne1 = rii.Rii(fine_quantizer=codec)\ne2 = rii.Rii(fine_quantizer=codec)\ne1.add_configure(vecs=X1)\ne2.add_configure(vecs=X2)\ne1.merge(e2)  # Now e1 contains both X1 and X2\n\n```\n\n## [Examples](./examples)\n- [Simple tag search](./examples/tag_search/simple_tag_search.ipynb)\n- [Benchmark](./examples/benchmark/)\n\n## Author\n- [Yusuke Matsui](http://yusukematsui.me)\n\n## Credits\n- The logo is designed by [@richardbmx](https://github.com/richardbmx) ([#4](https://github.com/matsui528/rii/issues/4))\n- The windows implementation is by [@ashleyabraham](https://github.com/ashleyabraham) ([#42](https://github.com/matsui528/rii/pull/42))\n- Fixing the build ([#50](https://github.com/matsui528/rii/pull/50)) and supporting for ARM CPUs ([#58](https://github.com/matsui528/rii/pull/58)) by [@timzag](https://github.com/timzag)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatsui528%2Frii","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmatsui528%2Frii","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatsui528%2Frii/lists"}