{"id":16742451,"url":"https://github.com/jvdd/pqknn","last_synced_at":"2025-04-10T13:32:47.629Z","repository":{"id":53123660,"uuid":"308881997","full_name":"jvdd/pqknn","owner":"jvdd","description":"Product Quantization k-Nearest Neighbors","archived":false,"fork":false,"pushed_at":"2021-06-24T22:46:38.000Z","size":29,"stargazers_count":20,"open_issues_count":3,"forks_count":4,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-24T12:12:25.867Z","etag":null,"topics":["approximate-nearest-neighbor-search","clustering","clustering-algorithm","k-nearest-neighbors","nearest-neighbor-search","pqknn-algorithm","product-quantization"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jvdd.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-10-31T12:56:54.000Z","updated_at":"2024-12-14T10:12:00.000Z","dependencies_parsed_at":"2022-09-10T15:20:57.648Z","dependency_job_id":null,"html_url":"https://github.com/jvdd/pqknn","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jvdd%2Fpqknn","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jvdd%2Fpqknn/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jvdd%2Fpqknn/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jvdd%2Fpqknn/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jvdd","download_url":"https://codeload.github.com/jvdd/pqknn/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248225710,"owners_count":21068078,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["approximate-nearest-neighbor-search","clustering","clustering-algorithm","k-nearest-neighbors","nearest-neighbor-search","pqknn-algorithm","product-quantization"],"created_at":"2024-10-13T01:23:47.665Z","updated_at":"2025-04-10T13:32:47.609Z","avatar_url":"https://github.com/jvdd.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Product Quantization k-Nearest Neighbors\n\nImplementation of Product Quantization k-Nearest Neighbors ([original paper](https://lear.inrialpes.fr/pubs/2011/JDS11/jegou_searching_with_quantization.pdf)).  \n\n\u003e Product quantization is a vector compression algorithm that summarizes partitions of the train data efficiently by means of their k-means centroids.  \n\u003e The PQKNN algorithm performs a k-nearest neighbors search on the compressed data in a smart manner using lookups on pre-computed tables. \n\n## Benefit (\u0026 explaination) of PQKNN\n\nThe bottleneck of classic k-nearest neighbors is that its performance degrades on large datasets.  \nEach query results in a full pass over the data to compute all distances!\n\n**By using product quantization, the nearest neighbor search is accelerated by greatly simplifying the distance calculations.** \n\n### Training\n\n*At train time* the data is compressed by means of the product quantization algorithm.  \nThis compression happens as follows;\n1) The original train data vectors are partitioned into *n* disjoint subvectors.\n2) A *c*-bit code is mapped to each partition using k-means clustering (with k = 2^c).\n\n### Predicting\n*At prediction time* the k-nearest neighbors search is performed.  \nFor each test sample, a table containing the distance of the sample to each of the centroids is calculated. Doing table lookups to the shared centroids (per partition) speeds up the distance computation to a great extent.\n\n## Disadvantage of PQKNN\n\nPQKNN is an approximate nearest neighbors algorithm, meaning that an approximate distance (based on the centroids) is used to classify training samples. This might be less exact than the classic k-nearest neighbors.\n\n## Hyperparameters of PQKNN\n\nWhen creating a ProductQuantizationKNN instance, one has to assign 2 hyperparameters;\n* *n*: the amount of subvectors, i.e., the number of partitions.\n* *c*: the number of bits to encode each subvector with. This determines the amount of clusters for k-means (used the compression step), i.e., *k* = 2^*c*.\n\nWhen training, i.e., compressing the train data, no other hyperparameters are required.\n\nWhen predicting, one has to assign 1 hyperparameter;\n* *k*: the k of k-nearest neighbors.\n\n### Example usage\n\n\n```py\nfrom product_quantization import ProductQuantizationKNN\n\n# Fetch the train data \u0026 train labels + the test data\ntrain_data, train_labels = ...\ntest_data = ...\n\n# Create PQKNN object that partitions each train sample in 7 subvectors and encodes each subvector in 4 bits.\npqknn = ProductQuantizationKNN(n=7, c=4)\n# Perform the compression\npqknn.compress(train_data, train_labels)\n\n# Classify the test data using k-Nearest Neighbor search (with k = 10) on the compressed training\npreds = pqknn.predict(test_data, nearest_neighbors=10)\n```\n\nFor more elaborate example look at the Notebook `Example.ipynb`.  \nYou should be able to rerun this notebook as the data is downloaded in the notebook itself.\n\n## Benchmark PQKNN implementation on MNIST\n\nIn the Notebook `Example.ipynb` this implementation of PQKNN is benchmarked on the famous MNIST dataset.\nThe train-test-split is identical as [the one provided by Yann LeCun](http://yann.lecun.com/exdb/mnist/);\n* Train data; 60,000 28x28 images that are flattened into an array with shape (60000, 784).\n* Test data; 10,000 28x28 images that are flattened into an array with shape (10000, 784).\n\n\n| Algorithm                    | Train time (s) | Predict time (s) | Accuracy (%) |\n|------------------------------|----------------|------------------|--------------|\n| `PQKNN` *(n=7, c=4)*             | 25.09          | 7.44             | 92.35        |\n| `PQKNN` *(n=7, c=7)*             | 132.72         | 15.73            | 96.6         |\n| `PQKNN` *(n=4, c=8)*             | 58.24          | \u003cb\u003e7.14\u003c/b\u003e            | 96.08        |\n| `sklearn.neighbors.KNeighborsClassifier` | \u003cb\u003e12.97\u003c/b\u003e      | 209.55    | \u003cb\u003e96.65\u003c/b\u003e   |\n\n\n**Discussion**: We observe that train time is significantly faster for sklearn its `KNeighborsClassifier` (2x-10x faster). This does not wheigh up against the enormous boost gained at prediction time for the PQKNN algorithm (15x-30x faster), at the cost of a small loss in accuracy. This significantly faster lookup (predictions) enable a trained PQKNN algorithm to have higher performance in production.\n\n*Minor remark*: as the PQKNN implementation works multithreaded. The displayed results for `sklearn.neighbors.KNeighborsClassifier` are with *n_jobs*=-1. When n_jobs=None, the performance at predict time is even 60x slower than the PQKNN implementation\n\n---\n\n## References\n\nOriginal paper: https://lear.inrialpes.fr/pubs/2011/JDS11/jegou_searching_with_quantization.pdf  \nJegou, Herve, Matthijs Douze, and Cordelia Schmid. \"Product quantization for nearest neighbor search.\" IEEE transactions on pattern analysis and machine intelligence 33.1 (2010): 117-128.\n\nBlogpost explaining very well the underlying concepts; https://mccormickml.com/2017/10/13/product-quantizer-tutorial-part-1/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjvdd%2Fpqknn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjvdd%2Fpqknn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjvdd%2Fpqknn/lists"}