Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/Nosto/spartann
Hyper performant kNN using Annoy for Apache Spark.
https://github.com/Nosto/spartann
ann annoy apache-spark k-nearest-neighbors k-nearest-neighbours knn ml spark
Last synced: 3 months ago
JSON representation
Hyper performant kNN using Annoy for Apache Spark.
- Host: GitHub
- URL: https://github.com/Nosto/spartann
- Owner: Nosto
- License: apache-2.0
- Created: 2021-05-25T14:51:55.000Z (over 3 years ago)
- Default Branch: master
- Last Pushed: 2022-10-25T12:38:25.000Z (over 2 years ago)
- Last Synced: 2024-08-01T13:29:18.358Z (6 months ago)
- Topics: ann, annoy, apache-spark, k-nearest-neighbors, k-nearest-neighbours, knn, ml, spark
- Language: Scala
- Homepage:
- Size: 10.9 MB
- Stars: 7
- Watchers: 14
- Forks: 3
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Spartann
Hyper performant kNN using Annoy for Apache Spark. For some backgrorund reading, I recommend this good-read by Mridang Agarwalla https://medium.com/@mridang.agarwalla/f4757b2b10eb
## Installation
Unfortunately, Spartann is not available in any public Maven repositories except the GitHub Package Registry. For more information on how to install packages
from the GitHub Package
Registry, [https://docs.github.com/en/packages/guides/configuring-gradle-for-use-with-github-packages#installing-a-package][see the GitHub docs]
## Usage
The usage of Annoy is simple. Spartann accepts a dataset of identifier and vector pairs. To take the simplest object.
```
case class Book(isbn: String, features: Array[Double]) extends Embeddings[String]
```
All classes must implement the `Embeddings[T]` interface where `T` is the identifer of the object. `T` in this case is a `String` but may be anything. You must also specify the Annoy configuration.
```
final val annoyConfig: AnnoyConfig = AnnoyConfig(50, 256, annoy4s.Euclidean)
```
`AnnoyConfig` accepts the number of trees, the dimensionality of the the data and the distance algorithm. An increase in the dimensionality or the number of tress will also degrade performance.
Annoy indexes all items with a non-negative integer value and allocates memory for max(i)+1 items. As shown in both examples, the iterators are zipped so that each item is indexed. The order of the items is not relevant.
### In Dataframes
Assume you have a dataset of `Dataset[Book]` and you need to find the similar books published by a publisher.
```
val records: List[Book] = List(...)
sqlContext.createDataset(records)(Encoders.kryo[Book])
.mapPartitions((books: Iterator[Book]) => {
val vectors: Iterator[(Long, Book)] = books.zipWithIndex.map(_.swap)
.map(item => (item._1.toLong, item._2))
Annoyer.create(vectors, annoyConfig = annoyConfig, verbose = false, maxResults = 50)
})(Encoders.kryo[Neighbour[String]])
.show()
```
⚠ SpartAnn has been tested with `mapPartitions` and the code currently lacks comprehensive tests against `mapGroups`.
### In RDDs
Assume you have a dataset of `RDD[Book]` and you need to find the similar books published by a publisher.
```
sc.parallelize(..)
.mapPartitions((books: Iterator[Book]) => {
val vectors: Iterator[(Long, Book)] = books.zipWithIndex.map(_.swap)
.map(item => (item._1.toLong, item._2))
Annoyer.create(vectors, annoyConfig = annoyConfig, verbose = false, maxResults = 50)
})
```
## Caveats
The current implementation doesn't allow on-disk Annoy indexes. Any upstream pull-requests that allow persisting indexes onto HDFS are welcome.
The current implemenntation doesn't allow querying the nearest neighbours by vectors as represented by the API method `get_nns_by_vector`.
The current implementation doesn't allow querying the distances between two items as represented by the API method `get_distance`.
Methods such as setting the seed using `set_seed(int)`, querying the number of items using `get_n_items()`, querying the number of trees using `get_n_trees()` were deemed superflous and removed from SpartAnn. In the event that these are needed, upstream pull-requests are welcomed.
## Authors
* Mridang Agarwalla
## License
Apache-2.0 License
[see the GitHub docs]: https://docs.github.com/en/packages/guides/configuring-gradle-for-use-with-github-packages#installing-a-package