{"id":13436235,"url":"https://github.com/technicolor-research/pq-fast-scan","last_synced_at":"2025-03-18T20:31:06.547Z","repository":{"id":83200409,"uuid":"42242916","full_name":"technicolor-research/pq-fast-scan","owner":"technicolor-research","description":"PQ Fast Scan","archived":false,"fork":false,"pushed_at":"2019-05-31T07:41:10.000Z","size":57,"stargazers_count":55,"open_issues_count":1,"forks_count":14,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-04-14T20:32:04.272Z","etag":null,"topics":["nearest-neighbor-search","product-quantization","simd"],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause-clear","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/technicolor-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2015-09-10T12:24:35.000Z","updated_at":"2024-03-17T13:57:21.000Z","dependencies_parsed_at":"2023-05-05T10:20:17.080Z","dependency_job_id":null,"html_url":"https://github.com/technicolor-research/pq-fast-scan","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/technicolor-research%2Fpq-fast-scan","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/technicolor-research%2Fpq-fast-scan/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/technicolor-research%2Fpq-fast-scan/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/technicolor-research%2Fpq-fast-scan/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/technicolor-research","download_url":"https://codeload.github.com/technicolor-research/pq-fast-scan/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244301370,"owners_count":20430931,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["nearest-neighbor-search","product-quantization","simd"],"created_at":"2024-07-31T03:00:45.759Z","updated_at":"2025-03-18T20:31:06.246Z","avatar_url":"https://github.com/technicolor-research.png","language":"C++","funding_links":[],"categories":["Uncategorized","ANN search"],"sub_categories":["Uncategorized"],"readme":"# PQ Fast Scan #\n\n## Description ##\n\nThe PQ Fast Scan project is a C++11 implementation of fast vector scanning \ntechniques for nearest neighbor search in large databases of high-dimensional\nvectors.\n\nPQ Fast Scan builds on Product Quantization (PQ), a widely used solution for \nnearest neighbor search. To find the nearest neighbor of a query vector, PQ \ncomputes the distance between the query vector and database vectors using \n*lookup tables* stored in the *L1 cache*. This procedure, named *PQ Scan*, \nperforms many cache accesses, which limit its performance. L1 cache accesses do \nnot parallelize well (maximum 2 concurrent accesses) and have a high latency \n(about 4 cycles). To avoid these issues, *PQ Fast Scan* uses lookup tables \nstored in *SIMD registers*, which can be queried in parallel (16 concurrent \naccesses) and with a low latency (1 cycle).\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://storage.sbg1.cloud.ovh.net/v1/AUTH_cb25a15f19bd41a5b3bdd2b60b718b72/pq-fast-scan/pq-fast-scan-overview.png\"\u003e\n\u003c/p\u003e\n\nThis novel approach allows PQ Fast Scan to perform 4-6x faster than PQ Scan, \nwhile returning the exact same results.\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://storage.sbg1.cloud.ovh.net/v1/AUTH_cb25a15f19bd41a5b3bdd2b60b718b72/pq-fast-scan/pq-fast-scan-performance.png\"\u003e\n\u003c/p\u003e\n\n**Contact:**  \nNicolas Le Souarnec: nicolas.le-scouarnec technicolor.com  \nFabien André: fabien.andre technicolor.com  \nPlease replace the space by an at sign.\n\n## Publication ##\n\nF. Andre, A.-M. Kermarrec, and N. Le Scouarnec. [Cache locality is not enough: High-Performance Nearest Neighbor Search with Product Quantization.](http://www.vldb.org/pvldb/vol9/p288-andre.pdf) *[PVLDB](http://www.vldb.org/pvldb/vol9.html)*, 9(4), 2015  \nPublication presented at VLDB'16: [Slides](http://assets.xion345.info/vldb16/slides.pdf), [Poster](http://assets.xion345.info/vldb16/poster.pdf)\n\n## License ##\n\nThe PQ Fast Scan project is made available under the Clear BSD license \nterms (See LICENSE file).\n\nCopyright (c) 2015 – Thomson Licensing, SAS\n\n## Building ##\n\n### Requirements ###\n\nHardware:\n\n* Processor supporting SSE2, SSE3, SSSE3 and POPCNT\n\nSoftware:\n\n* Reasonably recent Linux distribution\n* [g++](https://gcc.gnu.org/) (**4.9.x recommended**)\n* [CMake](http://www.cmake.org/) (2.8 or higher)\n* [libpfm4](http://perfmon2.sourceforge.net/)\n\nOn Debian-based distributions, you can install these dependencies with the \nfollowing command:\n\n    $ sudo apt-get install build-essential gcc g++ libpfm4-dev cmake\n\nPQ Fast Scan can be built with other g++ versions than g++ 4.9, but we \nbenchmarked it with this version only. Obtaining best results with other g++ \nversions may require some adjustments to the source code. Therefore, if your \ndistribution provides g++ 4.9, we recommend installing it:\n\n    $ sudo apt-get install g++-4.9 gcc-4.9\n\nOtherwise, you may either build PQ Fast Scan with another g++ version, or \ninstall g++ 4.9 from sources.\nInstalling g++ 4.9 from sources takes about 20 minutes on a reasonably recent \nsystem. You can do so with the following commands:\n\n    $ sudo apt-get install libgmp-dev libmpfr-dev libmpc-dev\n    $ wget ftp://ftp.irisa.fr/pub/mirrors/gcc.gnu.org/gcc/\\\n    releases/gcc-4.9.3/gcc-4.9.3.tar.bz2\n    $ tar xvf gcc-4.9.3.tar.bz2\n    $ mdkir build-gcc-4.9; cd build-gcc-4.9\n    $ ../gcc-4.9.3/configure --enable-languages=c,c++ --program-suffix=-4.9\n    $ make -j8\n    $ sudo make install\n    $ which g++-4.9\n    \nSome quick tests with g++ 5.2 seemed to indicate that PQ Fast Scan performs \nwell when compiled with this version. You may also be able to compile PQ Fast \nScan with clang++ but we have not tested it.\n\n### Building PQ Fast Scan ###\n\nOnce you have gathered all the requirements, building PQ Fast Scan is \nstraightforward:\n\n    $ git clone https://github.com/technicolor-research/pq-fast-scan.git\n    $ mkdir build-pq-fast-scan\n    $ cd build-pq-fast-scan\n\nIf you use g++ 4.9:\n\n    $ CC=gcc-4.9 CXX=g++-4.9 cmake ../pq-fast-scan\n    $ make\n\nTo use the default g++ version of your distribution:\n\n    $ cmake ../pq-fast-scan\n    $ make\n\n## Usage ##\n\nThis project provides two executables :\n    \n* **pqscan**, to benchmark different PQ Scan implementations on *synthetic* data\n* **pqfastscan**, to benchmark PQ Fast Scan against PQ Scan on *real* data\n\n**pqscan** does not take any command-line argument. It outputs run times and \nperformance counters for different PQ Scan implementations. You may need to run \nit as root to access the performance counters kernel API.\n\n    $ sudo ./pqscan\n    Scan: time=74758,cycles=269918483,instructions=800272344,L1-dcache-loads=\n    400231701,L1-dcache-load-misses=3139139 [...]\n    Scan +sse +prefetch: time=91214,cycles=329937021,instructions=787927217,\n    L1-dcache-loads=462852107,L1-dcache-load-misses=3139943 [...]\n    Scan +avx +prefetch: time=68490,cycles=231844613,instructions=618770051,\n    L1-dcache-loads=409382281,L1-dcache-load-misses=3141246 [...]\n    Scan +avx +vgather +prefetch: time=97841,cycles=331413774,\n    instructions=188461772,L1-dcache-loads=228194484, [...]\n\n**pqfastscan** requires four input files: a *partition* to scan, a *list*\nof IDs of query vectors, a set of *query vectors* and a set of *distance*\n*tables*.\n\nWe provide the following input files:\n\n* *partitions* and *lists*: `100M1-8-partitions.tar.xz`\n    [Link 1](https://storage.sbg1.cloud.ovh.net/v1/AUTH_cb25a15f19bd41a5b3bdd2b60b718b72/pq-fast-scan/100M1-8-partitions.tar.xz) [Link 2](http://assets.xion345.info/datasets/100M1-8-partitions.tar.xz)  \n    `SHA256: ef7db5883a4fb80eb9163295b8326d5088915f8b9748912ccb5c2ff755d9801e`\n* *query vectors*: `bigann_query.bvecs.gz`  \n    [Corpus Texmex](http://corpus-texmex.irisa.fr/), download *ANN_SIFT1B Query Set*\n* *distance tables*: `bigann_distance_tables.fvecs.xz`\n    [Link 1](https://storage.sbg1.cloud.ovh.net/v1/AUTH_cb25a15f19bd41a5b3bdd2b60b718b72/pq-fast-scan/bigann_distance_tables.fvecs.xz) [Link 2](http://assets.xion345.info/datasets/bigann_distance_tables.fvecs.xz)  \n    `SHA256: 3c4c766c7092dbd5c7251b5c1bec934a0759b7b0c7ba6694735b7fbc76060e63`\n\nFor more information about these datasets, see the Datasets section.\n\nDecompress the datasets with the following commands:\n\n    $ tar xvf 100M1-8-partitions.tar.xz\n    $ gzip -d bigann_query.bvecs.gz\n    $ xz -d bigann_distance_tables.fvecs.xz\n\nYou can then run pqfastscan, e.g. to scan partition 0:\n\n    $ ./pqfastscan 100M1-8-partition/100M1-partition-0.dat bigann_query.bvecs\\\n    bigann_distance_tables.fvecs 100M1-8-partition/100M1-list-0.txt\n    \n    vec_id,partition_id,partition_n,bh_size,keep,pq_us,fast_pq_us, [...]\n    1,0,25159451,100,125797,75392,12272,24404,93728.6\n    4,0,25159451,100,125797,74754,17480,435694,96652.2\n    [...]\n\nYou may replace 0 by another partition number [0-7]. Make sure you use the list \nfile corresponding to the partition, e.g., `100M1-list-6.txt` with \n`100M1-partition-6.dat`.\n\npqfastscan outputs one line (comma separated values) for each query vector of\nthe list with the scan time for PQ Scan (`pq_us`) and PQ Fast Scan (`fast_pq_us`).\n\nIn addition to run times, you may also want to output performance counters.\nTo do so, use the `-p` option. Like pqscan, you may also need to run pqfastscan\nas root when using performance counters:\n\n    $ sudo ./pqfastscan -p 100M1-8-partition/100M1-partition-0.dat\\\n    bigann_query.bvecs bigann_distance_tables.fvecs\\\n    100M1-8-partition/100M1-list-0.txt\n    \n    vec_id,partition_id,partition_n,bh_size,keep,pq_us,pq_cycles, [...]\n    1,0,25159451,100,125797,76247,272870815,805615127,402679306,12027, [...]\n    4,0,25159451,100,125797,76000,273304611,805588015,402670181,18040, [...]\n    [...]\n\nFor more information about pqfastscan options, see `./pqfastscan -h`.\n\n## Datasets ##\n\nTo test pqfastscan, we provide:\n\n* *partitions* and *lists*: `100M1-8-partitions.tar.xz`\n    [Link 1](https://storage.sbg1.cloud.ovh.net/v1/AUTH_cb25a15f19bd41a5b3bdd2b60b718b72/pq-fast-scan/100M1-8-partitions.tar.xz) [Link 2](http://assets.xion345.info/datasets/100M1-8-partitions.tar.xz)  \n    `SHA256: ef7db5883a4fb80eb9163295b8326d5088915f8b9748912ccb5c2ff755d9801e`\n* *distance tables*: `bigann_distance_tables.fvecs.xz`\n    [Link 1](https://storage.sbg1.cloud.ovh.net/v1/AUTH_cb25a15f19bd41a5b3bdd2b60b718b72/pq-fast-scan/bigann_distance_tables.fvecs.xz) [Link 2](http://assets.xion345.info/datasets/bigann_distance_tables.fvecs.xz)  \n    `SHA256: 3c4c766c7092dbd5c7251b5c1bec934a0759b7b0c7ba6694735b7fbc76060e63`\n\nBesides, we use the query set from the well-known ANN_SIFT1B dataset;\n\n* *query vectors*: `bigann_query.bvecs.gz`  \n    [Corpus Texmex](http://corpus-texmex.irisa.fr/), \n    download *ANN_SIFT1B Query Set*\n\nWe generated the *partitions*, *lists* and *distance tables* files we use from\nthe ANN_SIFT1B dataset.\n\nYou may re-generate these files yourself. To generate these files from the ANN_SIFT1B dataset,\nyou need an implementation of product quantization. The authors of the \n[seminal paper on product quantization](http://www.computer.org/csdl/trans/tp/2011/01/ttp2011010117-abs.html)\ndistribute \n[two implementations](http://people.rennes.inria.fr/Herve.Jegou/projects/ann.html)\nof product quantization: a C implementation, and a matlab implementation.\nWe used the C implementation (libpq), which we obtained under a commercial\nlicence. You can obtain similar files using the matlab implementation,\nfreely available.\n\nOnce you have an implementation of product quantization, you can generate\n*partitions*, *lists* and *distance tables* following these steps:\n\n1. Extract the first 100 million vectors from the ANN_SIFT1B Base Set\n(`bigann_base.bvecs`). We name this extracted dataset ANN_SIFT100M1.\n2. Build an IVFADC (database) using the ANN_SIFT1B Learning Set \n(`bigann_learn.bvecs`). We used a *coarse quantizer* with *8 centroids* and a\n*product quantizer* with *8 sub-quantizers* of *256 centroids* each.\n3. Modify the assignement of sub-quantizer centroid indexes. To do so, use a\n    [k-means variant which forces groups of same sizes](http://elki.dbs.ifi.lmu.de/wiki/Tutorial/SameSizeKMeans).\n4. Add all vectors of ANN_SIFT100M1 to the IVFADC.\n5. We name *partition* each Voronoi cell of the coarse quantizer. Save each \npartition [0-7] to a different file. See below for file format.\n6. Using the coarse quantizer, map each query vector to its corresponding\npartition (no multiple assignement, i.e. ma=1). Save the query vectors IDs \ncorresponding to each partition in a different *list*. See below for file format.\n7. Compute distance tables for each vector. Save *distance tables* sequentially\nin a single file. See below for file format.\n\n**partitions file format:** [Binary file]  \nThe number of pqcodes in the partition (32-bit int), followed by all pqcodes of \nthe partition (8x8 bits per pqcode).  \n*Filename:* The last number before the extension is parsed as the partition\nnumber.\n\n**distance tables file format:**    [Binary file]  \nEach query vector has 8 distance tables of 256 floats associated with it.\nThe 8 distance tables of each query vector are stored as a vector of 2048\nfloats (8 x 256 floats). These vectors of 2048 floats are stored as an\n[fvecs file](http://corpus-texmex.irisa.fr/).  \n*Filename:* No restrictions.\n\n**lists file format:**              [Text file]  \nOne vector ID per line.  \n*Filename:* The last number before the extension is parsed as the partition\nnumber. If the vector ID 42 is in the file xxxx-list-6.txt, it means that the\nvector number 42 from the query set was mapped to the partition 6 by the coarse\nquantizer.\n\n## License\nThis project is licensed under the terms of BSD 3-clause Clear license.\nby downloading this program, you commit to comply with the license as stated in the LICENSE.md file.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftechnicolor-research%2Fpq-fast-scan","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftechnicolor-research%2Fpq-fast-scan","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftechnicolor-research%2Fpq-fast-scan/lists"}