{"id":13795147,"url":"https://github.com/irvingc/dbscan-on-spark","last_synced_at":"2025-05-12T21:33:34.602Z","repository":{"id":28717987,"uuid":"32238933","full_name":"irvingc/dbscan-on-spark","owner":"irvingc","description":"An implementation of DBSCAN runing on top of Apache Spark","archived":false,"fork":false,"pushed_at":"2018-01-10T01:29:42.000Z","size":114,"stargazers_count":183,"open_issues_count":11,"forks_count":58,"subscribers_count":19,"default_branch":"master","last_synced_at":"2024-08-03T23:04:42.251Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/irvingc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-03-15T00:45:16.000Z","updated_at":"2024-07-03T02:57:16.000Z","dependencies_parsed_at":"2022-08-18T22:31:44.271Z","dependency_job_id":null,"html_url":"https://github.com/irvingc/dbscan-on-spark","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/irvingc%2Fdbscan-on-spark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/irvingc%2Fdbscan-on-spark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/irvingc%2Fdbscan-on-spark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/irvingc%2Fdbscan-on-spark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/irvingc","download_url":"https://codeload.github.com/irvingc/dbscan-on-spark/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225157000,"owners_count":17429698,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T23:00:52.646Z","updated_at":"2024-11-18T09:31:42.874Z","avatar_url":"https://github.com/irvingc.png","language":"Scala","funding_links":[],"categories":["人工智能"],"sub_categories":["机器学习"],"readme":"# DBSCAN on Spark\n\n### Overview\n\nThis is an implementation of the [DBSCAN clustering algorithm](http://en.wikipedia.org/wiki/DBSCAN) \non top of [Apache Spark](http://spark.apache.org/). It is loosely based on the paper from He, Yaobin, et al.\n[\"MR-DBSCAN: a scalable MapReduce-based DBSCAN algorithm for heavily skewed data\"](http://www.researchgate.net/profile/Yaobin_He/publication/260523383_MR-DBSCAN_a_scalable_MapReduce-based_DBSCAN_algorithm_for_heavily_skewed_data/links/0046353a1763ee2bdf000000.pdf). \n\n\nI have also created a [visual guide](http://www.irvingc.com/visualizing-dbscan) that explains how the algorithm works.\n### Current vesion of DBSCAN is dbscan-on-spark_2.10:0.2.0-SNAPSHOT\nBe aware that current version of DBSCAN in this repo  is  :\n\t\u003cgroupId\u003ecom.irvingc.spark\u003c/groupId\u003e\n\t\u003cartifactId\u003e**dbscan-on-spark_2.10**\u003c/artifactId\u003e\n\t\u003cversion\u003e**0.2.0-SNAPSHOT**\u003c/version\u003e\nIt is not present in any  official repository and to make it work, you need to build it yourself.\n### Getting DBSCAN on Spark\n\nVersion 0.1.0 of DBSCAN on Spark is published to [bintray](https://bintray.com/). If you use SBT you\ncan include SBT in your application adding the following to your build.sbt:\n\n```\nresolvers += \"bintray/irvingc\" at \"http://dl.bintray.com/irvingc/maven\"\n\nlibraryDependencies += \"com.irvingc.spark\" %% \"dbscan\" % \"0.1.0\"\n```\n\n\n\nIf you use Maven or Ivy you can use a similar resolver, but you just\nneed to account for the scala version (the example is for Scala 2.10):\n\n```\n...\n\n\t\u003crepositories\u003e\n\t\t\u003crepository\u003e\n\t\t\t\u003cid\u003edbscan-on-spark-repo\u003c/id\u003e\n\t\t\t\u003cname\u003eRepo for DBSCAN on Spark\u003c/name\u003e\n\t\t\t\u003curl\u003ehttp://dl.bintray.com/irvingc/maven\u003c/url\u003e\n\t\t\u003c/repository\u003e\n\t\u003c/repositories\u003e\n...\n\n\t\u003cdependency\u003e\n\t\t\u003cgroupId\u003ecom.irvingc.spark\u003c/groupId\u003e\n\t\t\u003cartifactId\u003edbscan_2.10\u003c/artifactId\u003e\n\t\t\u003cversion\u003e0.1.0\u003c/version\u003e\n\t\u003c/dependency\u003e\n\n\n```\nDBSCAN on Spark is built against Scala 2.10.\n\n\n### Example usage \n\n\nI have created a [sample project](https://github.com/irvingc/dbscan-on-spark-example) \nshowing how DBSCAN on Spark can be used. The following however should give you a\ngood idea of how it should be included in your application.\n\n```scala\nimport org.apache.spark.mllib.clustering.dbscan.DBSCAN\n\nobject DBSCANSample {\n\n  def main(args: Array[String]) {\n\n    val conf = new SparkConf().setAppName(\"DBSCAN Sample\")\n    val sc = new SparkContext(conf)\n\n    val data = sc.textFile(src)\n\n    val parsedData = data.map(s =\u003e Vectors.dense(s.split(',').map(_.toDouble))).cache()\n\n    log.info(s\"EPS: $eps minPoints: $minPoints\")\n\n    val model = DBSCAN.train(\n      parsedData,\n      eps = eps,\n      minPoints = minPoints,\n      maxPointsPerPartition = maxPointsPerPartition)\n\n    model.labeledPoints.map(p =\u003e  s\"${p.x},${p.y},${p.cluster}\").saveAsTextFile(dest)\n\n    sc.stop()\n  }\n}\n```\n\n### License\n\nDBSCAN on Spark is available under the Apache 2.0 license. \nSee the [LICENSE](LICENSE) file for details.\n\n\n### Credits\n\nDBSCAN on Spark is maintained by Irving Cordova (irving@irvingc.com).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Firvingc%2Fdbscan-on-spark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Firvingc%2Fdbscan-on-spark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Firvingc%2Fdbscan-on-spark/lists"}