{"id":17994897,"url":"https://github.com/una-dinosauria/local-search-quantization","last_synced_at":"2025-08-02T08:37:45.869Z","repository":{"id":77457997,"uuid":"70345888","full_name":"una-dinosauria/local-search-quantization","owner":"una-dinosauria","description":"State-of-the-art method for large-scale ANN search as of Oct 2016. Presented at ECCV 16.","archived":false,"fork":false,"pushed_at":"2018-06-14T01:46:23.000Z","size":75,"stargazers_count":75,"open_issues_count":0,"forks_count":11,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-06-26T16:09:29.465Z","etag":null,"topics":["computer-vision","cuda","eccv-16","gpu","julia","multi-codebook","quantization"],"latest_commit_sha":null,"homepage":"","language":"Julia","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/una-dinosauria.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-10-08T17:07:40.000Z","updated_at":"2025-03-23T02:41:57.000Z","dependencies_parsed_at":null,"dependency_job_id":"bd127d22-aebe-4a81-b688-cadc9689f537","html_url":"https://github.com/una-dinosauria/local-search-quantization","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/una-dinosauria/local-search-quantization","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/una-dinosauria%2Flocal-search-quantization","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/una-dinosauria%2Flocal-search-quantization/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/una-dinosauria%2Flocal-search-quantization/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/una-dinosauria%2Flocal-search-quantization/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/una-dinosauria","download_url":"https://codeload.github.com/una-dinosauria/local-search-quantization/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/una-dinosauria%2Flocal-search-quantization/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268355797,"owners_count":24237371,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-02T02:00:12.353Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","cuda","eccv-16","gpu","julia","multi-codebook","quantization"],"created_at":"2024-10-29T20:17:06.910Z","updated_at":"2025-08-02T08:37:45.838Z","avatar_url":"https://github.com/una-dinosauria.png","language":"Julia","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Local-search Quantization\n\nThis is the code for the papers\n\n* Martinez, J., Clement, J., Hoos H. H. and Little, J. J.:\n[*Revisiting additive quantization*](https://www.cs.ubc.ca/~julm/papers/eccv16.pdf),\nfrom ECCV 2016, and\n* Martinez, J., Hoos H. H. and Little, J. J.:\n[*Solving multi-codebook quantization in the GPU*](https://www.cs.ubc.ca/~julm/papers/eccvw16.pdf),\nfrom VSM (ECCV workshops) 2016.\n\nThe code in this repository was mostly written by [Julieta Martinez](http://www.cs.ubc.ca/~julm/) and [Joris Clement](https://github.com/flyingdutchman23).\n\n## Dependencies\n\nOur code is mostly written in [Julia](http://julialang.org/), and should run\nunder version 0.6 or later. To get Julia, go to the\n[Julia downloads page](http://julialang.org/downloads/) and install the latest\nstable release.\n\nWe use a number of dependencies that you have to install using\n`Pkg.install( \"package_name\" )`, where `package_name` is\n\n* [HDF5](https://github.com/JuliaIO/HDF5.jl) -- for reading/writing data\n* [Distributions](https://github.com/JuliaStats/Distributions.jl) -- for random inits\n* [Distances](https://github.com/JuliaStats/Distances.jl) -- for quick distance computation\n* [IterativeSolvers](https://github.com/JuliaLang/IterativeSolvers.jl) -- for [LSQR](https://github.com/JuliaMath/IterativeSolvers.jl/blob/master/src/lsqr.jl)\n* [Clustering](https://github.com/JuliaStats/Clustering.jl) -- for k-means\n\nTo run encoding in a GPU, you have to compile Julia from source (I know this sucks! but it will no longer be necessary with Julia 1.0). You will also need to install\n\n* [CUDAdrv](https://github.com/JuliaGPU/CUDAdrv.jl) -- the CUDA driver API\n* [CUBLAS](https://github.com/JuliaGPU/CUBLAS.jl) -- for fast matrix multiplication in the GPU\n* A CUDA-enabled GPU with compute capability 3.5 or higher. We have tested our code on K40 and Titan X GPUs\n\nFinally, to run the sparse encoding demo you will need Matlab to run the\n[SPGL1](https://github.com/mpf/spgl1) solver by van den Berg and Friedlander, as\nwell as the [MATLAB.jl](https://github.com/JuliaInterop/MATLAB.jl) package to\ncall Matlab functions from Julia.\n\n## Demos\n\nFirst, clone this repository and download the [SIFT1M](http://corpus-texmex.irisa.fr/)\ndataset. To do so run the following commands:\n\n```bash\ngit clone git@github.com:jltmtz/local-search-quantization.git\ncd local-search-quantization\nmkdir data\ncd data\nwget ftp://ftp.irisa.fr/local/texmex/corpus/sift.tar.gz\ntar -xvzf sift.tar.gz\nrm sift.tar.gz\ncd ..\n```\n\nAlso, compile the auxiliary search cpp code:\n```bash\ncd src/linscan/cpp/\n./compile.sh\ncd ../../../\n```\n\nFor expedience, the following demos train on the first 10K vectors of the\nSIFT1M dataset. To reproduce the paper results you will have to use the full\ntraining set with 100K vectors.\n\nThere are 3 main functionalities showcased in this code:\n\n### 1) Baselines and LSQ demo with encoding in the CPU\nSimply run\n```bash\njulia demos/demo_pq.jl\njulia demos/demo_opq.jl\njulia demos/demo_lsq.jl\n```\nThis will train PQ, OPQ, and LSQ on a subset of SIFT1M, encode the base set and\ncompute a recall@N curve. To get better speed in LSQ, you can also run the code\non parallel in multiple cores using\n```bash\njulia -p n demos/demo_lsq.jl\n```\nWhere `n` is the number of CPU cores on your machine.\n\n### 2) LSQ demo with encoding in the GPU\nIf you have a CUDA-enabled GPU, you might want to try out encoding in the GPU.\n\nFirst, compile the CUDA code:\n\n```bash\ncd src/encodings/cuda\n./compile.sh\ncd ../../../\n```\nand then run\n```bash\njulia demos/demo_lsq_gpu.jl\n```\n\nor\n\n```bash\njulia -p n demos/demo_lsq_gpu.jl\n```\nWhere `n` is the number of CPU cores on your machine.\n\n### 3) LSQ demo with sparse encoding\n\nThis is very similar to demo #1, but the learned codebooks will be sparse.\n\nFirst of all, you have to download the [SPGL1](https://github.com/mpf/spgl1)\nsolver by van den Berg and Friedlander, and add the function that implements\nExpression 8 to the package\n\n```bash\ncd matlab\ngit clone git@github.com:mpf/spgl1.git\nmv sparse_lsq_fun.m spgl1/\nmv splitarray.m spgl1/\ncd ..\n```\n\nNow you should be able to run the demo\n\n```bash\njulia -p n demos/demo_lsq_sparse.jl\n```\nWhere `n` is the number of CPU cores on your machine.\n\nNote that you need MATLAB installed on your computer to run this demo, as well\nas well as the [MATLAB.jl](https://github.com/JuliaInterop/MATLAB.jl) package to\ncall Matlab functions from Julia. Granted, getting all this to work can be a bit\nof a pain -- if at this point you (like me) love Julia more than any other\nlanguage, please consider porting [SPGL1](https://github.com/mpf/spgl1) to Julia.\n\n## Citing\n\nThank for your interest in our research! If you find this code useful, please\nconsider citing our paper\n\n```\nJulieta Martinez, Joris Clement, Holger H. Hoos, James J. Little. \"Revisiting\nadditive quantization\", ECCV 2016.\n```\n\nIf you use our GPU implementation please consider citing\n\n```\nJulieta Martinez, Holger H. Hoos, James J. Little. \"Solving multi-codebook\nquantization in the GPU\", 4th Workshop on Web-scale Vision and Social Media\n(VSM), at ECCV 2016.\n```\n\n## FAQ\n\n* **Q:** *What is ChainQ?*\n\n  **A:** ChainQ is a quantization method inspired by [optimized tree quantization (OTQ)](http://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Babenko_Tree_Quantization_for_2015_CVPR_paper.pdf). Instead of learning the\n  dimension splitting and sharing among codebooks (which OTQ finds using Gurobi),\n  we simply take the natural splitting and sharing given by contiguous dimensions.\n  Therefore, our codebooks form a chain, not a general tree. This means we can\n  solve encoding optimally using the Viterbi algorithm.\n\n* **Q:** *LSQ is very slow...?*\n\n  **A:** Compared to PQ and OPQ yes, but (a) it gives much better compression rates,\n  and (b) it is much better in quality and speed compared to\n  [additive quantization (AQ)](http://www.cv-foundation.org/openaccess/content_cvpr_2014/papers/Babenko_Additive_Quantization_for_2014_CVPR_paper.pdf) (our most similar baseline). The authors have made the [AQ code available](https://github.com/arbabenko/Quantizations), so you can compare yourself :)\n\n\n* **Q:** *The code does not reproduce the results of the paper...?*\n\n  **A:** The demos train on 10K vectors and for 10 iterations. To reproduce the\n  results of the paper, train with the whole 100K vectors and do it for 100\n  iterations. You can also control the number of ILS iterations to use\n  for database encoding in the LSQ demos; which corresponds to LSQ-16 and LSQ-32\n  in the paper.\n\n* **Q:** *Why do I see all those warnings when I run your code?*\n\n  **A:** Julia 0.5 issues a warning when a method is redefined more than once in\n  the Main scope. This is annoying for many people and will disappear in Julia\n  0.6 (see https://github.com/JuliaLang/julia/issues/18725)\n\n## Acknowledgments\n\nSome of our evaluation code and our OPQ implementation has been adapted from\n[Cartesian k-means](https://github.com/norouzi/ckmeans) by [Mohamad Norouzi](https://github.com/norouzi)\nand [optimized product quantization](http://kaiminghe.com/cvpr13/index.html) by [Kaiming He](http://kaiminghe.com/).\n\n## License\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Funa-dinosauria%2Flocal-search-quantization","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Funa-dinosauria%2Flocal-search-quantization","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Funa-dinosauria%2Flocal-search-quantization/lists"}