{"id":16876944,"url":"https://github.com/jlmelville/rcpphnsw","last_synced_at":"2025-09-10T04:17:09.122Z","repository":{"id":45806212,"uuid":"138253991","full_name":"jlmelville/rcpphnsw","owner":"jlmelville","description":"Rcpp bindings for the approximate nearest neighbors library hnswlib","archived":false,"fork":false,"pushed_at":"2024-12-25T21:26:00.000Z","size":400,"stargazers_count":36,"open_issues_count":3,"forks_count":11,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-03-29T01:06:11.676Z","etag":null,"topics":["approximate-nearest-neighbor-search","hnsw","k-nearest-neighbors","knn","nearest-neighbor-search","nmslib","r","rcpp"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jlmelville.png","metadata":{"files":{"readme":"README.md","changelog":"NEWS.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-06-22T04:19:10.000Z","updated_at":"2025-01-28T16:12:45.000Z","dependencies_parsed_at":"2024-01-27T22:23:57.212Z","dependency_job_id":"10505d16-fee9-4219-9820-59ae527a793f","html_url":"https://github.com/jlmelville/rcpphnsw","commit_stats":{"total_commits":228,"total_committers":6,"mean_commits":38.0,"dds":0.08333333333333337,"last_synced_commit":"ecd0f096a1cc7da0363bc0891bb2761e8d0394ce"},"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jlmelville%2Frcpphnsw","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jlmelville%2Frcpphnsw/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jlmelville%2Frcpphnsw/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jlmelville%2Frcpphnsw/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jlmelville","download_url":"https://codeload.github.com/jlmelville/rcpphnsw/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247276163,"owners_count":20912288,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["approximate-nearest-neighbor-search","hnsw","k-nearest-neighbors","knn","nearest-neighbor-search","nmslib","r","rcpp"],"created_at":"2024-10-13T15:41:02.230Z","updated_at":"2025-04-05T02:07:38.087Z","avatar_url":"https://github.com/jlmelville.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# RcppHNSW\n\n[![AppVeyor Build Status](https://ci.appveyor.com/api/projects/status/github/jlmelville/rcpphnsw?branch=master\u0026svg=true)](https://ci.appveyor.com/project/jlmelville/rcpphnsw)\n[![R-CMD-check](https://github.com/jlmelville/rcpphnsw/workflows/R-CMD-check/badge.svg)](https://github.com/jlmelville/rcpphnsw/actions)\n[![Coverage Status](https://img.shields.io/codecov/c/github/jlmelville/rcpphnsw/master.svg)](https://codecov.io/github/jlmelville/rcpphnsw?branch=master)\n[![CRAN Status Badge](https://www.r-pkg.org/badges/version/RcppHNSW)](https://cran.r-project.org/package=RcppHNSW)\n[![Dependencies](https://tinyverse.netlify.app/badge/RcppHNSW)](https://cran.r-project.org/package=RcppHNSW)\n[![CRAN Monthly Downloads](https://cranlogs.r-pkg.org/badges/RcppHNSW)](https://cran.r-project.org/package=RcppHNSW)\n![CRAN Downloads](https://cranlogs.r-pkg.org/badges/grand-total/RcppHNSW)\n[![Last Commit](https://img.shields.io/github/last-commit/jlmelville/rcpphnsw)](https://github.com/jlmelville/rcpphnsw)\n\nRcpp bindings for [hnswlib](https://github.com/nmslib/hnswlib).\n\n## Status\n\n*February 4 2024* RcppHNSW 0.6.0 is released to CRAN, supporting\n[hnswlib version 0.8.0](https://github.com/nmslib/hnswlib/releases/tag/v0.8.0).\n\n*September 19 2023* RcppHNSW 0.5.0 is released to CRAN, supporting\n[hnswlib version 0.7.0](https://github.com/nmslib/hnswlib/releases/tag/v0.7.0),\na `getItems` method for returning the items used to build the index and some\nperformance improvements if your data is already column-stored. Also, a small\nroxygen problem with the package documentation was fixed.\n\n*July 18 2022* RcppHNSW 0.4.1 is released. Unfortunately, there are valgrind\nproblems with the version of hnswlib used in RcppHNSW 0.4.0, so that has been\nrolled back.\n\n*July 16 2022* RcppHNSW 0.4.0 is released. This release matches [hnswlib version\n0.6.2](https://github.com/nmslib/hnswlib/releases/tag/v0.6.2), but otherwise\nadds no new features. Some minor CRAN check NOTEs are fixed and there is also a\nminor license change: previously the license was GPLv3. From this version, it\nnow supports GPLv3 or later.\n\n*September 6 2020* RcppHNSW 0.3.0 is now available on CRAN, with multi-threading\nsupport.\n\n*August 30 2020*. Although not yet on CRAN, support for building and searching\nan index in parallel (via the `n_threads` function argument and `setNumThreads`\nobject method) has been added to the current development version (available via\n`devtools::install_github`). Thanks to \n[Dmitriy Selivanov](https://github.com/dselivanov) for a lot of the work on\nthis.\n\n*September 20 2019*. RcppHNSW 0.2.0 is now available on CRAN, up to date with\nhnswlib at \u003chttps://github.com/nmslib/hnswlib/commit/c5c38f0\u003e, with new methods:\n`size`, `resizeIndex` and `markDeleted`. Also, a bug that prevented searching\nwith datasets smaller than `k` has been fixed. Thanks to\n[Yuxing Liao](https://github.com/yxngl) for spotting that.\n\n*January 21 2019*. RcppHNSW is now available on CRAN.\n\n*October 20 2018*. By inserting some preprocessor symbols into hnswlib, these\nbindings no longer require a non-portable compiler flag and hence will pass `R\nCMD CHECK` without any warnings: previously you would be warned about\n`-march=native`. The price paid is not using specialized functions for the\ndistance calculations that are architecture-specific. I have not checked how bad\nthe performance hit is. The old settings remain in `src/Makevars` and\n`src/Makevars.win` (commented out), if you want to build the project from\nsource directly. Otherwise, [Release\n0.0.0.9000](https://github.com/jlmelville/rcpphnsw/releases/tag/v0.0.0.9000) is\nthe last version with the old behavior, which can be installed with something\nlike:\n\n```R\ndevtools::install_github(\"jlmelville/rcpphnsw@v0.0.0.9000\")\n```\n\n## hnswlib\n\nhnswlib is a header-only C++ library for finding approximate nearest neighbors\n(ANN) via Hierarchical Navigable Small Worlds\n[(Yashunin and Malkov, 2016)](https://arxiv.org/abs/1603.09320).\nIt is part of the [nmslib](https://github.com/nmslib/nmslib]) project.\n\n## The RcppHNSW Package\n\nAn R package that interfaces with hnswlib, taking enormous amounts of inspiration\nfrom [Dirk Eddelbuettel](https://github.com/eddelbuettel)'s\n[RcppAnnoy](https://github.com/eddelbuettel/rcppannoy) package which did the\nsame for the [Annoy](https://github.com/spotify/annoy) ANN C++ library.\n\nOne difference is that I use\n[roxygen2](https://cran.r-project.org/package=roxygen2) to generate the man\npages. The `NAMESPACE` is still built manually, however (I don't believe you can\n`export` the classes currently).\n\n## Installing\n\nFrom CRAN:\n\n```R\ninstall.packages(\"RcppHNSW\")\n```\n\nDevelopment versions from github:\n\n```R\ndevtools::install_github(\"jlmelville/RcppHNSW\")\n```\n\n## Function example\n\n```R\nirism \u003c- as.matrix(iris[, -5])\n\n# function interface returns results for all rows in nr x k matrices\nall_knn \u003c- RcppHNSW::hnsw_knn(irism, k = 4, distance = \"l2\")\n# other distance options: \"euclidean\", \"cosine\" and \"ip\" (inner product distance)\n\n# for high-dimensional data you may see a speed-up if you store the data\n# where each *column* is an item to be indexed and searched. Set byrow = TRUE\n# for this.\n# Admittedly, the iris dataset is *not* high-dimensional\niris_by_col \u003c- t(irism)\nall_knn \u003c- RcppHNSW::hnsw_knn(iris_by_col, k = 4, distance = \"l2\", byrow = FALSE)\n\n# process can be split into two steps, so you can build with one set of data\n# and search with another\nann \u003c- hnsw_build(irism[1:100, ])\niris_nn \u003c- hnsw_search(irism[101:150, ], ann, k = 5)\n```\n\n## Class Example\n\nAs noted in the \"Do not use named parameters\" section below, you should avoid\nusing named parameters when using class methods. But I do use them in a few\nplaces below to document the name of the parameters the positional arguments\nrefer to.\n\n```R\nlibrary(RcppHNSW)\ndata \u003c- as.matrix(iris[, -5])\n\n# Create a new index using the L2 (squared Euclidean) distance\n# nr and nc are the number of rows and columns of the data to be added, respectively\n# ef and M determines speed vs accuracy trade off\n# You must specify the maximum number of items to add to the index when it\n# is created. But you can increase this number: see the next example\nM \u003c- 16\nef \u003c- 200\ndim \u003c- ncol(data)\nnitems \u003c- nrow(data)\nann \u003c- new(HnswL2, dim, nitems, M, ef)\n\n# Add items to index\nfor (i in 1:nitems) {\n  ann$addItem(data[i, ])\n}\n\n# Find 4 nearest neighbors of row 1\n# indexes are in res$item, distances in res$distance\n# set include_distances = TRUE to get distances as well as index\nres \u003c- ann$getNNsList(data[1, ], k = 4, include_distances = TRUE)\n\n# It's more efficient to use the batch methods if you have all the data you\n# need at once\nann2 \u003c- new(HnswL2, dim, nitems, M, ef)\nann2$addItems(data)\n# Retrieve the 4 nearest neighbors for every item in data\nres2 \u003c- ann2$getAllNNsList(data, 4, TRUE)\n# labels of the data are in res$item, distances in res$distance\n\n# If you are able to store your data column-wise, then the overhead of copying\n# the data into a form usable by hnsw can be noticeably reduced\ndata_by_col \u003c- t(data)\nann3 \u003c- new(HnswL2, dim, nitems, M, ef)\nann3$addItemsCol(data_by_col)\n# Retrieve the 4 nearest neighbors for every item in data_by_col\nres3 \u003c- ann3$getAllNNsListCol(data_by_col, 4, TRUE)\n# The returned neared neighbor data matrices are also returned column-wise\nall(res2$item == t(res3$item) \u0026 res2$distance == t(res3$distance))\n\n# Save the index\nann$save(\"iris.hnsw\")\n\n# load it back in: you do need to know the dimension of the original data\nann4 \u003c- new(HnswL2, dim, \"iris.hnsw\")\n# new index should behave like the original\nall(ann$getNNs(data[1, ], 4) == ann4$getNNs(data[1, ], 4))\n\n# other distance classes:\n# Cosine: HnswCosine\n# Inner Product: HnswIP\n# Euclidean: HnswEuclidean\n```\n\nHere's a rough equivalent of the serialization/deserialization example from\nthe\n[hnswlib README](https://github.com/nmslib/hnswlib#python-bindings-examples),\nbut using the recently-added `resizeIndex` method to increase the size of the\nindex after its initial specification, avoiding having to read from or write\nto disk:\n\n```R\nlibrary(\"RcppHNSW\")\nset.seed(12345)\n\ndim \u003c- 16\nnum_elements \u003c- 100000\n\n# Generate sample data\ndata \u003c- matrix(stats::runif(num_elements * dim), nrow = num_elements)\n\n# Split data into two batches\ndata1 \u003c- data[1:(num_elements / 2), ]\ndata2 \u003c- data[(num_elements / 2 + 1):num_elements, ]\n\n# Create index\nM \u003c- 16\nef \u003c- 10\n# Set the initial index size to the size of the first batch\np \u003c- new(HnswL2, dim, num_elements / 2, M, ef)\n\nmessage(\"Adding first batch of \", nrow(data1), \" elements\")\np$addItems(data1)\n\n# Query the elements for themselves and measure recall:\nidx \u003c- p$getAllNNs(data1, k = 1)\nmessage(\"Recall for the first batch: \", formatC(mean(idx == 1:nrow(data1))))\n\n# Increase the total capacity, so that it will handle the new data\np$resizeIndex(num_elements)\n\nmessage(\"Adding the second batch of \", nrow(data2), \" elements\")\np$addItems(data2)\n\n# Query the elements for themselves and measure recall:\nidx \u003c- p$getAllNNs(data, k = 1)\n# You can get distances with:\n# res \u003c- p$getAllNNsList(data, k = 1, include_distances = TRUE)\n# res$dist contains the distance matrix, res$item stores the indexes\n\nmessage(\"Recall for two batches: \", formatC(mean(idx == 1:num_elements)))\n```\n\nAlthough there's no longer any need for this, for completeness, here's how you\nwould use `save` and `new` to achieve the same effect without `resizeIndex`:\n\n```R\nfilename \u003c- \"first_half.bin\"\n# Serialize index\np$save(filename)\n\n# Reinitialize and load the index\nrm(p)\nmessage(\"Loading index from \", filename)\n# Increase the total capacity, so that it will handle the new data\np \u003c- new(HnswL2, dim, filename, num_elements)\nunlink(filename)\n```\n\n## API\n\n### **DO NOT USE NAMED PARAMETERS**\n\nBecause these are wrappers around C++ code, you **cannot** use named\nparameters in the calling R code. Arguments are parsed by position. This is\nmost annoying in constructors, which take multiple integer arguments, e.g.\n\n```R\n### DO THIS ###\ndim \u003c- 10\nnum_elements \u003c- 100\nM \u003c- 200\nef_construction \u003c- 16\nindex \u003c- new(HnswL2, dim, num_elements, M, ef_construction)\n\n### DON'T DO THIS ###\nindex \u003c- new(HnswL2, dim, ef_construction = 16, M = 200, num_elements = 100)\n# treated as if you wrote:\nindex \u003c- new(HnswL2, dim, 16, 200, 100)\n```\n\n### OK onto the API\n\n* `new(HnswL2, dim, max_elements, M = 16, ef_contruction = 200)` creates a new\nindex using the squared L2 distance (i.e. square of the Euclidean distance),\nwith `dim` dimensions and a maximum size of `max_elements` items. `ef` and `M`\ndetermine the speed vs accuracy trade off. Other classes for different distances\nare: `HnswCosine` for the cosine distance and `HnswIp` for the \"Inner Product\"\ndistance (like the cosine distance without normalizing).\n* `new(HnswL2, dim, max_elements, M, ef_contruction, random_seed)` same as the\nprevious constructor, but with a specified random seed.\n* `new(HnswL2, dim, filename)` load a previously saved index (see `save` below)\nwith `dim` dimensions from the specified `filename`.\n* `new(HnswL2, dim, filename, max_elements)` load a previously saved index (see\n`save` below) with `dim` dimensions from the specified `filename`, and a new\nmaximum capacity of `max_elements`. This is a way to increase the capacity of\nthe index without a complete rebuild.\n* `setEf(ef)` set search parameter `ef`.\n* `setNumThreads(num_threads)` Use (at most) this number of threads when adding\nitems (via `addItems`) and searching the index (via `getAllNNs` and\n`getAllNNsList`). See also the `setGrainSize` parameter.\n* `setGrainSize(grain_size)` The minimum amount of work to do (adding or\nsearching items) per thread. If you don't have enough work for all the threads\nspecified by `setNumThreads` to process `grain_size` items per thread, then\nfewer threads will be used. This is useful for cases where the cost of context\nswitching between larger number of threads would outweigh the performance gain\nfrom parallelism. For example, if you have 100 items to process and asked for\nfour threads, then 25 items will be processed per thread. However, setting the \n`grain_size` to 50 will result in 50 items being processed per thread, and \ntherefore only two threads being used.\n* `addItem(v)` add vector `v` to the index. Internally, each vector gets an\nincreasing integer label, with the first vector added getting the label `1`, the\nsecond `2` and so on. These labels are returned in `getNNs` and related methods\nto identify which vector in the index are neighbors.\n* `addItems(m)` add the row vectors of the matrix `m` to the index. Internally,\neach row vector gets an increasing integer label, with the first row added\ngetting the label `1`, the second `2` and so on. These labels are returned in\n`getNNs` and related methods to identify which vector in the index are\nneighbors. The number of threads specified by `setNumThreads` is used for\nbuilding the index and may be non-deterministic.\n* `addItemsCol(m)` Like `addItems` but adds the *column* vectors of `m` to the\nindex. Storing data column-wise makes copying the data for use by `hnsw` more\nefficient.\n* `save(filename)` saves an index to the specified `filename`. To load an index,\nuse the `new(HnswL2, dim, filename)` constructor (see above).\n* `getItems(ids)` returns a matrix where each row is the data vector from the\nindex associated with integer indices in the vector of `ids`. For cosine\nsimilarity, the l2 row-normalized vectors are returned. `ids` are one-indexed,\ni.e. to get the first and tenth vectors that were added to the index, use\n`getItems(c(1, 10))`, not `getItems(c(0, 9))`.\n* `getNNs(v, k)` return a vector of the labels of the `k`-nearest neighbors of\nthe vector `v`. Labels are integers numbered from one, representing the\ninsertion order into the index, e.g. the label `1` represents the first item\nadded to the index. If `k` neighbors can't be found, an error will be thrown.\nThis normally means that `ef` or `M` have been set too small, but also bear in\nmind that you can't return more items than were put into the index.\n* `getNNsList(v, k, include_distances = FALSE)` return a list containing a\nvector named `item` with the labels of the `k`-nearest neighbors of the vector\n`v`. Labels are integers numbered from one, representing the insertion order\ninto the index, e.g. the label `1` represents the first item added to the index.\nIf `include_distances = TRUE` then also return a vector `distance` containing\nthe distances. If `k` neighbors can't be found, an error is thrown.\n* `getAllNNs(m, k)` return a matrix of the labels of the `k`-nearest neighbors\nof each row vector in `m`. Labels are integers numbered from one, representing\nthe insertion order into the index, e.g. the label `1` represents the first item\nadded to the index. If `k` neighbors can't be found, an error is thrown. The \nnumber  of threads specified by `setNumThreads` is used for searching.\n* `getAllNNsList(m, k, include_distances = FALSE)` return a list containing a\nmatrix named `item` with the labels of the `k`-nearest neighbors of each row\nvector in `m`. Labels are integers numbered from one, representing the insertion\norder into the index, e.g. the label `1` represents the first item added to the\nindex. If `include_distances = TRUE` then also return a matrix `distance`\ncontaining the distances. If `k` neighbors can't be found, an error is thrown.\nThe number  of threads specified by `setNumThreads` is used for searching.\n* `getAllNNsCol(m, k)` like `getAllNNs` but each item to be searched in `m` is\nstored by *column*, not row. In addition the returned matrix of `k`-nearest\nneighbors is also stored column-wise: i.e. the dimension of the return value\nmatrix is `k x n` where `n` is the number of items (columns) in `m`. By passing\nthe data column-wise, some overhead associated with copying data to and from\n`hnsw` can be reduced.\n* `getAllNNsListCol(m, k)` like `getAllNNsList` but each item to be searched in\n`m` is stored by *column*, not row. In addition, the matrices in the returned \nlist are also stored column-wise: i.e. the dimension of the return value matrix\nis `k x n` where `n` is the number of items (columns) in `m`. By passing the \ndata column-wise, some overhead associated with copying data to and from `hnsw`\ncan be reduced.\n* `size()` returns the number of items in the index. This is an upper limit on\nthe number of neighbors you can expect to return from `getNNs` and the other\nsearch methods.\n* `markDeleted(i)` marks the item with label `i` (the `i`th item added to the\nindex) as deleted. This means that the item will not be returned in any further\nsearches of the index. It does not reduce the memory used by the index. Calls to\n`size()` do *not* reflect the number of marked deleted items.\n* `resize(max_elements)` changes the maximum capacity of the index to\n`max_elements`.\n\n## Differences from Python Bindings\n\n* Arbitrary integer labeling is not supported. Where labels are used, e.g. in\nthe return value of `getNNsList` or as input in `markDeleted` or `getItems`, the \nlabels represent the order in which the items were added to the index, using \n1-indexing to be consistent with R. So in the Python bindings, the first item in\nthe index has a default of label `0`, but here it will have label `1`.\n* The interface roughly follows the Python one but deviates with naming and also\nrolls the declaration and initialization of the index into one call. And as\nnoted above, you must pass arguments by position, not keyword.\n\n## License\n\n[GPL-3 or later](https://www.gnu.org/licenses/gpl-3.0.en.html).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjlmelville%2Frcpphnsw","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjlmelville%2Frcpphnsw","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjlmelville%2Frcpphnsw/lists"}