{"id":21458532,"url":"https://github.com/hibayesian/spark-lof","last_synced_at":"2025-07-30T18:07:57.874Z","repository":{"id":21089832,"uuid":"92004006","full_name":"hibayesian/spark-lof","owner":"hibayesian","description":"A parallel implementation of local outlier factor based on Spark","archived":false,"fork":false,"pushed_at":"2022-01-26T11:04:23.000Z","size":49,"stargazers_count":17,"open_issues_count":2,"forks_count":13,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-08T19:33:25.044Z","etag":null,"topics":["local-outlier-factor","machine-learning","outlier-detection","spark"],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hibayesian.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-05-22T02:36:47.000Z","updated_at":"2023-08-31T21:52:42.000Z","dependencies_parsed_at":"2022-08-25T14:01:28.402Z","dependency_job_id":null,"html_url":"https://github.com/hibayesian/spark-lof","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/hibayesian/spark-lof","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hibayesian%2Fspark-lof","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hibayesian%2Fspark-lof/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hibayesian%2Fspark-lof/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hibayesian%2Fspark-lof/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hibayesian","download_url":"https://codeload.github.com/hibayesian/spark-lof/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hibayesian%2Fspark-lof/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264951611,"owners_count":23687974,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["local-outlier-factor","machine-learning","outlier-detection","spark"],"created_at":"2024-11-23T06:23:08.796Z","updated_at":"2025-07-12T06:37:25.568Z","avatar_url":"https://github.com/hibayesian.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Spark-LOF\nIn anomaly detection, the local outlier factor(LOF) algorithm is based on a concept of a local density, where locality is given by k nearest neighbors, whose distance is used to estimate the density. By comparing the local density of an object to the local densities of its neighbors, one can identify regions of similar density, and points that have a substantially lower density than their neighbors. Due to the local approach, LOF is able to identify outliers in a data set that would not be outliers in another area of the data set. Spark-LOF is a parallel implementation of local outlier factor based on Spark.\n\n# Examples\n## Scala API\n```scala\nval spark = SparkSession\n  .builder()\n  .appName(\"LOFExample\")\n  .master(\"local[4]\")\n  .getOrCreate()\n\nval schema = new StructType(Array(\n  new StructField(\"col1\", DataTypes.DoubleType),\n  new StructField(\"col2\", DataTypes.DoubleType)))\nval df = spark.read.schema(schema).csv(\"data/outlier.csv\")\n\nval assembler = new VectorAssembler()\n  .setInputCols(df.columns)\n  .setOutputCol(\"features\")\nval data = assembler.transform(df).repartition(4)\n\nval startTime = System.currentTimeMillis()\nval result = new LOF()\n  .setMinPts(5)\n  .transform(data)\nval endTime = System.currentTimeMillis()\nresult.count()\n    \n// Outliers have much higher LOF value than normal data\nresult.sort(desc(\"lof\")).head(10).foreach { row =\u003e\n  println(row.get(0) + \" | \" + row.get(1) + \" | \" + row.get(2))\n}\nprintln(\"Total time = \" + (endTime - startTime) / 1000.0 + \"s\")\n```\n\n# Requirements\nSpark-LOF is built against Spark 3.1.1.\n\n# Build From Source\n```scala\nsbt assembly\n```\n\n# Licenses\nSpark-LOF is available under Apache Licenses 2.0.\n\n# Contact \u0026 Feedback\nIf you encounter bugs, feel free to submit an issue or pull request. Also you can mail to:\n+ hibayesian (hibayesian@gmail.com).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhibayesian%2Fspark-lof","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhibayesian%2Fspark-lof","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhibayesian%2Fspark-lof/lists"}