{"id":17175680,"url":"https://github.com/saurfang/spark-knn","last_synced_at":"2025-04-06T00:10:48.530Z","repository":{"id":45516037,"uuid":"42156716","full_name":"saurfang/spark-knn","owner":"saurfang","description":"k-Nearest Neighbors algorithm on Spark","archived":false,"fork":false,"pushed_at":"2023-11-14T14:03:01.000Z","size":567,"stargazers_count":240,"open_issues_count":22,"forks_count":111,"subscribers_count":23,"default_branch":"master","last_synced_at":"2025-03-29T23:11:04.742Z","etag":null,"topics":["knn","spark"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/saurfang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-09-09T04:49:45.000Z","updated_at":"2025-03-29T18:44:36.000Z","dependencies_parsed_at":"2022-07-19T14:42:08.134Z","dependency_job_id":"1de9c3ed-a1f4-4762-b4e8-2eb1f7d51381","html_url":"https://github.com/saurfang/spark-knn","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saurfang%2Fspark-knn","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saurfang%2Fspark-knn/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saurfang%2Fspark-knn/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saurfang%2Fspark-knn/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/saurfang","download_url":"https://codeload.github.com/saurfang/spark-knn/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247415973,"owners_count":20935387,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["knn","spark"],"created_at":"2024-10-14T23:57:20.702Z","updated_at":"2025-04-06T00:10:48.506Z","avatar_url":"https://github.com/saurfang.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# spark-knn\n\n[![Join the chat at https://gitter.im/saurfang/spark-knn](https://badges.gitter.im/saurfang/spark-knn.svg)](https://gitter.im/saurfang/spark-knn?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge)\n\n[![Build Status](https://travis-ci.org/saurfang/spark-knn.svg)](https://travis-ci.org/saurfang/spark-knn)\n[![codecov.io](http://codecov.io/github/saurfang/spark-knn/coverage.svg?branch=master)](http://codecov.io/github/saurfang/spark-knn?branch=master)\n\nWIP...\n\nk-Nearest Neighbors algorithm (k-NN) implemented on Apache Spark. This uses a hybrid spill tree approach to\nachieve high accuracy and search efficiency. The simplicity of k-NN and lack of tuning parameters makes k-NN\na useful baseline model for many machine learning problems.\n\n## How to Use\n\nThis package is published using [sbt-spark-package](https://github.com/databricks/sbt-spark-package) and\nlinking information can be found at http://spark-packages.org/package/saurfang/spark-knn\n\nk-NN can be used for both classification and regression, which are exposed using the new [Spark ML](http://spark.apache.org/docs/latest/ml-guide.html)\nAPI based on DataFrame. Both models accept a weight column so predictions can be optionally weighted.\n\n### KNNClassifier\n\n```scala\n//read in raw label and features\nval training = MLUtils.loadLibSVMFile(sc, \"data/mllib/sample_libsvm_data.txt\").toDF()\n\nval knn = new KNNClassifier()\n  .setTopTreeSize(training.count().toInt / 500)\n  .setK(10)\n\nval knnModel = knn.fit(training)\n\nval predicted = knnModel.transform(training)\n```\n\n### KNNRegression\n\n```scala\n//read in raw label and features\nval training = MLUtils.loadLibSVMFile(sc, \"data/mllib/sample_libsvm_data.txt\").toDF()\n\nval knn = new KNNRegression()\n  .setTopTreeSize(training.count().toInt / 500)\n  .setK(10)\n\nval knnModel = knn.fit(training)\n\nval predicted = knnModel.transform(training)\n```\n\nFurthermore, KNN itself is also exposed for advanced usage which returns arbitrary columns associated with found neighbors.\nFor example, this can power clustering use case described in the reference Google paper.\n\nWhen the model is trained, data points are repartitioned and within each partition a search tree is built to support\n efficient querying. When model is used in prediction, the prediction vectors are repartitioned, searched, collected and\n joined back to the search DataFrame. Assuming the training set is much larger, subsequent prediction can be much quicker\n than training. Overall the algorithm displays a `O(m log n)` runtime much better than the naive `O(m n)`\n runtime (for n training points, m prediction points and k = 1). See [benchmark](#benchmark) section for more details.\n\nThe number of neighbors can be set before and after training. Other parameters must be set before training and they control\nthe number of partitions and trade off between accuracy and efficiency of individual search tree.\nPlease refer to Scala doc for more information.\n\n## Using the Python interface with spark-submit\n\nTo run a Spark script in Python with `spark-submit`, use:\n\n```\ncd python\npython setup.py bdist_egg\ncd ..\nsbt package\n\nspark-submit --py-files python/dist/pyspark_knn-*.egg --driver-class-path spark-knn-core/target/scala-2.11/spark-knn_*.jar --jars spark-knn-core/target/scala-2.11/spark-knn_*.jar YOUR_SCRIPT\n```\n\n## Benchmark\n\nPreliminary benchmark results can be found at [here](data/mnist/benchmark.md).\n\nWe have benchmarked our implementation against MNIST dataset. For the canonical 60k training dataset, our implementation\n is able to get a reasonable cross validated F1 score of 0.97 comparing to brute force exact algorithm's *to be computed*.\n\nWhile the implementation is approximate, it doesn't suffer much even when the dimension is high (as many as the full MNIST\nraw dimension: 784). This can be a huge advantage over other approximate implementation such as KD-tree and LSH. *further\nbenchmark is required*\n\nThe implementation also exhibits sub-linear runtime which can lead to huge savings for large datasets.\n\n![](data/mnist/benchmark.png)\n\nNote: the duration in the above plot is total runtime thus brute-force exhibits polynomial runtime while SpillTree shows\nclose to linearithmic runtime.\n\nFinally the implementation scales horizontally and has been successfully applied on datasets with low hundreds millions of\nobservations and low hundreds dimensions. We have no reason to say why it can't scale to billions of observations as described\nin the original Google paper.\n\n\n## Progress\n\n- [x] implementation of MetricTree, SpillTree, HybridSpillTree\n- [x] distributed KNN based on HybridSpillTree\n- [x] \\(weighted\\) Classifier and Regression on ml API\n- [ ] benchmark against Brute Force and LSH based kNN in terms of model and runtime performance\n- [ ] benchmark against LSH based kNN\n- [ ] refactoring of Tree related code\n\n      NB: currently tree are recursively constructed and contain some duplicated code. The data structure is also questionable.\n      However preliminary empirical testing shows each tree can comfortably handle tens to hundreds thousands of high dimensional data points.\n- [ ] upgrade ml implementation to use DataSet API (pending Spark 1.6)\n\n      NB: the largest cost of this implementation is disk I/O of repartition and distance calculation. While distance calculation\n      has no good way to optimize, with DataSet API, we might be able to drastically reduce the shuffle size during training\n      and prediction.\n- [ ] explore use of random projection for dimension reduction\n\n## Credits\n\n- Liu, Ting, et al.\n\"An investigation of practical approximate nearest neighbor algorithms.\"\nAdvances in neural information processing systems. 2004.\n\n- Liu, Ting, Charles Rosenberg, and Henry Rowley.\n\"Clustering billions of images with large scale nearest neighbor search.\"\nApplications of Computer Vision, 2007. WACV'07. IEEE Workshop on. IEEE, 2007.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsaurfang%2Fspark-knn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsaurfang%2Fspark-knn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsaurfang%2Fspark-knn/lists"}