{"id":29154414,"url":"https://github.com/matsui528/nanopq","last_synced_at":"2025-12-24T22:48:30.940Z","repository":{"id":32767841,"uuid":"141382694","full_name":"matsui528/nanopq","owner":"matsui528","description":"Pure python implementation of product quantization for nearest neighbor search","archived":false,"fork":false,"pushed_at":"2025-06-07T06:53:08.000Z","size":117,"stargazers_count":344,"open_issues_count":2,"forks_count":48,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-06-07T07:02:49.130Z","etag":null,"topics":["approximate-nearest-neighbor-search","data-compression","nearest-neighbor-search","product-quantization"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/matsui528.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-07-18T04:53:45.000Z","updated_at":"2025-06-07T06:47:08.000Z","dependencies_parsed_at":"2023-01-16T22:31:11.717Z","dependency_job_id":"0aee7b12-57a3-409a-b4e6-74e9e4590747","html_url":"https://github.com/matsui528/nanopq","commit_stats":{"total_commits":75,"total_committers":6,"mean_commits":12.5,"dds":0.5066666666666666,"last_synced_commit":"5c9e138f86a3a9bf33290f427883b0c405164716"},"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/matsui528/nanopq","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matsui528%2Fnanopq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matsui528%2Fnanopq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matsui528%2Fnanopq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matsui528%2Fnanopq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/matsui528","download_url":"https://codeload.github.com/matsui528/nanopq/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matsui528%2Fnanopq/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262884174,"owners_count":23379345,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["approximate-nearest-neighbor-search","data-compression","nearest-neighbor-search","product-quantization"],"created_at":"2025-07-01T02:33:57.456Z","updated_at":"2025-12-24T22:48:30.905Z","avatar_url":"https://github.com/matsui528.png","language":"Python","funding_links":[],"categories":["Open Sources"],"sub_categories":[],"readme":"# nanopq\n\n[![Build Status](https://github.com/matsui528/nanopq/actions/workflows/build.yml/badge.svg)](https://github.com/matsui528/nanopq/actions)\n[![Documentation Status](https://readthedocs.org/projects/nanopq/badge/?version=latest)](https://nanopq.readthedocs.io/en/latest/?badge=latest)\n[![PyPI version](https://badge.fury.io/py/nanopq.svg)](https://badge.fury.io/py/nanopq)\n[![Downloads](https://pepy.tech/badge/nanopq)](https://pepy.tech/project/nanopq)\n\nNano Product Quantization (nanopq): a vanilla implementation of Product Quantization (PQ) and Optimized Product Quantization (OPQ) written in pure python without any third party dependencies.\n\n\n\n## Installing\nYou can install the package via pip. This library works with Python 3.5+ on linux.\n```\npip install nanopq\n```\n\n## [Documentation](https://nanopq.readthedocs.io/en/latest/index.html)\n- [Tutorial](https://nanopq.readthedocs.io/en/latest/source/tutorial.html)\n- [API](https://nanopq.readthedocs.io/en/latest/source/api.html)\n\n## Example\n\n```python\nimport nanopq\nimport numpy as np\n\nN, Nt, D = 10000, 2000, 128\nX = np.random.random((N, D)).astype(np.float32)  # 10,000 128-dim vectors to be indexed\nXt = np.random.random((Nt, D)).astype(np.float32)  # 2,000 128-dim vectors for training\nquery = np.random.random((D,)).astype(np.float32)  # a 128-dim query vector\n\n# Instantiate with M=8 sub-spaces\npq = nanopq.PQ(M=8)\n\n# Train codewords\npq.fit(Xt)\n\n# Encode to PQ-codes\nX_code = pq.encode(X)  # (10000, 8) with dtype=np.uint8\n\n# Results: create a distance table online, and compute Asymmetric Distance to each PQ-code \ndists = pq.dtable(query).adist(X_code)  # (10000, ) \n```\n\n## Author\n- [Yusuke Matsui](http://yusukematsui.me)\n\n## Contributors\n- [@Hiroshiba](https://github.com/Hiroshiba) fixed a bug of importlib ([#3](https://github.com/matsui528/nanopq/pull/3))\n- [@calvinmccarter](https://github.com/calvinmccarter) implemented parametric initialization for OPQ ([#14](https://github.com/matsui528/nanopq/pull/14))\n- [@de9uch1](https://github.com/de9uch1) exntended the interface to the faiss so that OPQ can be handled ([#19](https://github.com/matsui528/nanopq/pull/19))\n- [@mpskex](https://github.com/mpskex) implemented (1) initialization of clustering and (2) dot-product for computation ([#24](https://github.com/matsui528/nanopq/pull/24))\n- [@lsb](https://github.com/lsb) fixed a typo ([#26](https://github.com/matsui528/nanopq/pull/26))\n- [@asukaminato0721](https://github.com/asukaminato0721) used Literal for string inputs ([#42](https://github.com/matsui528/nanopq/pull/42))\n\n## Reference\n- [H. Jegou, M. Douze, and C. Schmid, \"Product Quantization for Nearest Neighbor Search\", IEEE TPAMI 2011](https://ieeexplore.ieee.org/document/5432202/) (the original paper of PQ)\n- [T. Ge, K. He, Q. Ke, and J. Sun, \"Optimized Product Quantization\", IEEE TPAMI 2014](https://ieeexplore.ieee.org/document/6678503/) (the original paper of OPQ)\n- [Y. Matsui, Y. Uchida, H. Jegou, and S. Satoh, \"A Survey of Product Quantization\", ITE MTA 2018](https://www.jstage.jst.go.jp/article/mta/6/1/6_2/_pdf/) (a survey paper of PQ) \n- [PQ in faiss](https://github.com/facebookresearch/faiss/wiki/Faiss-building-blocks:-clustering,-PCA,-quantization#pq-encoding--decoding) (Faiss contains an optimized implementation of PQ. [See the difference to ours here](https://nanopq.readthedocs.io/en/latest/source/tutorial.html#difference-from-pq-in-faiss))\n- [Rayuela.jl](https://github.com/una-dinosauria/Rayuela.jl) (Julia implementation of several encoding algorithms including PQ and OPQ)\n- [PQk-means](https://github.com/DwangoMediaVillage/pqkmeans) (clustering on PQ-codes. The implementation of nanopq is compatible to [that of PQk-means](https://github.com/DwangoMediaVillage/pqkmeans/blob/master/tutorial/1_pqkmeans.ipynb))\n- [Rii](https://github.com/matsui528/rii) (IVFPQ-based ANN algorithm using nanopq)\n- [Product quantization in Faiss and from scratch](https://www.youtube.com/watch?v=PNVJvZEkuXo) (Related tutorial)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatsui528%2Fnanopq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmatsui528%2Fnanopq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatsui528%2Fnanopq/lists"}