{"id":15145764,"url":"https://github.com/soundcloud/cosine-lsh-join-spark","last_synced_at":"2025-05-06T18:05:59.391Z","repository":{"id":48442351,"uuid":"42043766","full_name":"soundcloud/cosine-lsh-join-spark","owner":"soundcloud","description":"Approximate Nearest Neighbors in Spark","archived":false,"fork":false,"pushed_at":"2021-07-26T08:48:50.000Z","size":259,"stargazers_count":174,"open_issues_count":7,"forks_count":43,"subscribers_count":123,"default_branch":"master","last_synced_at":"2025-01-02T22:09:18.408Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/soundcloud.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-09-07T09:25:11.000Z","updated_at":"2024-11-06T10:37:32.000Z","dependencies_parsed_at":"2022-09-26T16:30:58.817Z","dependency_job_id":null,"html_url":"https://github.com/soundcloud/cosine-lsh-join-spark","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soundcloud%2Fcosine-lsh-join-spark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soundcloud%2Fcosine-lsh-join-spark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soundcloud%2Fcosine-lsh-join-spark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soundcloud%2Fcosine-lsh-join-spark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/soundcloud","download_url":"https://codeload.github.com/soundcloud/cosine-lsh-join-spark/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":233297391,"owners_count":18654809,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-26T11:42:39.384Z","updated_at":"2025-01-10T05:15:08.269Z","avatar_url":"https://github.com/soundcloud.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Cosine LSH Join Spark\n\nA spark library for approximate nearest neighbours (ANN).\n\n# Background\n\nIn many computational problems such as NLP, Recommendation Systems and Search,\nitems (e.g. words) are represented as vectors in a multidimensional space.\nThen given a specific item it's nearest neighbours need to be find e.g. given\na query find the most similar ones. A naive liner scan over the data set might\nbe too slow for most data sets.\n\nHence, more efficient algorithms are needed. One of the most widely used\napproaches is Locality Sensitive Hashing (LSH). This family of algorithms are\nvery fast but might not give the exact solution and are hence called\napproximate nearest neighbours (ANN). The trade off between accuracy and speed\nis generally set via parameters of the algorithm.\n\n# Joiner Interface\n\nThis is an interface to find the k nearest neighbors from a data set for every other object in the\n   same data set. Implementations may be either exact or approximate.\n\n    trait Joiner {\n        def join(matrix: IndexedRowMatrix): CoordinateMatrix\n    }\n\nmatrix is a row oriented matrix. Each row in the matrix represents\nan item in the dataset. Items are identified by their\nmatrix index.\nReturns a similarity matrix with MatrixEntry(itemA, itemB, similarity).\n\n# Example\n    // item_a --\u003e (1.0, 1.0, 1.0)\n\t// item_b --\u003e (2.0, 2.0, 2.0)\n\t// item_c --\u003e (6.0, 3.0, 2.0)\n\n\tval rows = Seq(\n      IndexedRow(1, Vectors.dense(1.0, 1.0, 1.0)),\n      IndexedRow(2, Vectors.dense(2.0, 2.0, 2.0)),\n      IndexedRow(5, Vectors.dense(6.0, 3.0, 2.0))\n    )\n    val matrix = new IndexedRowMatrix(sc.parallelize(rows))\n    val similariyMatrix = joiner.join(matrix)\n\n    val results = similariyMatrix.entries.map {\n          entry =\u003e\n            \"item:%d item:%d cosine:%.2f\".format(entry.i, entry.j, entry.value)\n        }\n\n    results.foreach(println)\n\n    // above will print:\n    // item:2 item:3 cosine:0,87\n    // item:1 item:3 cosine:0,87\n    // item:1 item:2 cosine:1,00\n\nPlease see included **Main.scala** file for a more detailed example.\n\n## Implementations of the joiner interface\n\n### LSH\nThis is an implementation of the following paper for Spark:\n\n[Randomized Algorithms and NLP: Using Locality Sensitive Hash Function for High Speed Noun Clustering](http://dl.acm.org/citation.cfm?id=1219917)\n\n-- \u003ccite\u003eRavichandran et al.\u003c/cite\u003e\n\nThe algorithm determines a set of candidate items in the first stage and only computes the exact cosine similarity for those candidates. It has been succesfully used in production with typical run times of a couple of minutes for millions of items.\n\nNote that candidates are ranked by their exact cosine similarity. Hence, this algorithm will not return any false positives (items that the system thinks are nearby but are actually not). Most real world applications require this e.g. in recommendation systems it is ok to return similar items which are almost as good as the exact nearest neighbours but showing false positives would result in senseless recommendations.\n\n    val lsh = new Lsh(\n      minCosineSimilarity = 0.5,\n      dimensions = 2,\n      numNeighbours = 3,\n      numPermutations = 1,\n      partitions = 1,\n      storageLevel = StorageLevel.MEMORY_ONLY\n    )\n\nPlease see the original publication for a detailed description of the parameters.\n\n### NearestNeighbours\nBrute force method to compute exact nearest neighbours.\nAs this is a very expensive computation O(n^2) an additional sample parameter may be passed such\nthat neighbours are just computed for a random fraction.\nThis interface may be used to tune parameters for approximate solutions\non a small subset of data.\n\n# QueryJoiner Interface\nAn interface to find the nearest neighbours in a catalog matrix for each entry in a query matrix.\nImplementations may be either exact or approximate.\n\n    trait QueryJoiner {\n      def join(queryMatrix: IndexedRowMatrix, catalogMatrix: IndexedRowMatrix): CoordinateMatrix\n    }\n\n## Implementations of the QueryJoiner Interface\n\n### QueryLsh\nStandard Lsh implementation. A query matrix is hashed multiple times and exact hash matches are searched for in a catalog Matrix. These candidates are used to compute the exact cosine distance.\n\n### QueryHamming\n\nImplementation based on approximated cosine distances. The cosine distances are\napproximated using hamming distances which are way faster to compute.\nThe catalog matrix is broadcasted. This implementation is therefore suited for\ntasks where the catalog matrix is very small compared to the query matrix.\n\n### QueryNearestNeighbours\nBrute force O(size(query) * size(catalog)) method to compute exact nearest neighbours for rows in the query matrix. As this is a very expensive computation additional sample parameters may be passed such that neighbours are just computed for a random fraction of the data set. This interface may be used to tune parameters for approximate solutions on a small subset of data.\n\n# Maven\nThe artifacts are hosted on Maven Central. For Spark 1.x add the following line to your build.sbt file:\n\n\tlibraryDependencies += \"com.soundcloud\" % \"cosine-lsh-join-spark_2.10\" % \"0.0.5\"\n\nFor Spark 2.x use:\n\n    libraryDependencies += \"com.soundcloud\" % \"cosine-lsh-join-spark_2.10\" % \"1.0.1\"\n\nor if you're on scala 2.11.x use:\n\n    libraryDependencies += \"com.soundcloud\" % \"cosine-lsh-join-spark_2.11\" % \"1.0.1\"\n\n## Releasing (maintainers only)\n\nIn order to release the library using the release plugin `sbt release`, you need to set up the following:\n * A PGP key in order to sign the package.\n * Register an account on sonatype's JIRA: https://issues.sonatype.org/secure/Signup!default.jspa\n * Request the right permissions by filing an issue\n   (see for example https://issues.sonatype.org/browse/OSSRH-39877)\n * Add your credentials to `$HOME/.ivy2/.credentials`\n   as explained here: https://www.scala-sbt.org/1.x/docs/Publishing.html#Credentials\n * Run the release command\n * Close and release the staging repository as explained here: http://central.sonatype.org/pages/releasing-the-deployment.html\n\n# Contributors\n\n[Özgür Demir](https://github.com/ozgurdemir)\n\n[Rany Keddo](https://github.com/purzelrakete/)\n\n[Alexey Rodriguez Yakushev](https://github.com/alexeyrodriguez)\n\n[Aaron Levin](https://github.com/aaronlevin)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoundcloud%2Fcosine-lsh-join-spark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsoundcloud%2Fcosine-lsh-join-spark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoundcloud%2Fcosine-lsh-join-spark/lists"}