{"id":16599720,"url":"https://github.com/tupol/spark-xkmeans","last_synced_at":"2026-04-16T11:02:37.520Z","repository":{"id":57741032,"uuid":"181126687","full_name":"tupol/spark-xkmeans","owner":"tupol","description":"Extension to the standard K-Means implementation of Spark ML library","archived":false,"fork":false,"pushed_at":"2019-05-23T11:49:32.000Z","size":27,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-10-05T05:58:38.864Z","etag":null,"topics":["clustering","kmeans","kmeans-clustering","library","machine-learning","scala","spark"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tupol.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-04-13T05:44:08.000Z","updated_at":"2019-08-16T08:22:59.000Z","dependencies_parsed_at":"2022-09-06T23:31:03.607Z","dependency_job_id":null,"html_url":"https://github.com/tupol/spark-xkmeans","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/tupol/spark-xkmeans","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tupol%2Fspark-xkmeans","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tupol%2Fspark-xkmeans/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tupol%2Fspark-xkmeans/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tupol%2Fspark-xkmeans/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tupol","download_url":"https://codeload.github.com/tupol/spark-xkmeans/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tupol%2Fspark-xkmeans/sbom","scorecard":{"id":902488,"data":{"date":"2025-08-11","repo":{"name":"github.com/tupol/spark-xkmeans","commit":"4a297b19f7783cf113d7a89fd0a2cb1821cfb881"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3,"checks":[{"name":"Code-Review","score":0,"reason":"Found 0/7 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}}]},"last_synced_at":"2025-08-24T16:08:18.074Z","repository_id":57741032,"created_at":"2025-08-24T16:08:18.074Z","updated_at":"2025-08-24T16:08:18.074Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31882886,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-16T09:23:21.276Z","status":"ssl_error","status_checked_at":"2026-04-16T09:23:15.028Z","response_time":69,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clustering","kmeans","kmeans-clustering","library","machine-learning","scala","spark"],"created_at":"2024-10-12T00:12:23.618Z","updated_at":"2026-04-16T11:02:37.502Z","avatar_url":"https://github.com/tupol.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Spark XKmeans\n\n[![Maven Central](https://img.shields.io/maven-central/v/org.tupol/spark-xkmeans_2.11.svg)](https://mvnrepository.com/artifact/org.tupol/spark-xkmeans) \u0026nbsp;\n[![GitHub](https://img.shields.io/github/license/tupol/spark-xkmeans.svg)](https://github.com/tupol/spark-xkmeans/blob/master/LICENSE) \u0026nbsp; \n[![Travis (.org)](https://img.shields.io/travis/tupol/spark-xkmeans.svg)](https://travis-ci.com/tupol/spark-xkmeans) \u0026nbsp; \n[![Codecov](https://img.shields.io/codecov/c/github/tupol/spark-xkmeans.svg)](https://codecov.io/gh/tupol/spark-xkmeans) \u0026nbsp;\n[![Javadocs](https://www.javadoc.io/badge/org.tupol/spark-xkmeans_2.11.svg)](https://www.javadoc.io/doc/org.tupol/spark-xkmeans_2.11) \u0026nbsp;\n[![Gitter](https://badges.gitter.im/spark-xkmeans/community.svg)](https://gitter.im/spark-xkmeans/community?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge) \u0026nbsp; \n[![Twitter](https://img.shields.io/twitter/url/https/_tupol.svg?color=%2317A2F2)](https://twitter.com/_tupol) \u0026nbsp; \n\n## Motivation\nWhile working on PoC for fraud detection in the banking sector we experimented with the K-Means \nalgorithm for detecting anomalies. Interestingly enough, one of the first questions that came up \nwere “why is this an anomaly?”. \n\nThe question makes perfect sense when the amount of features used in the building the scoring\nmodel can easily go to hundreds. \n\n## Problem\nThe K-Means implementation in Spark provides a single output column containing the predicted \ncluster. We need to find a way to collect additional information so we can explain why a data point\nis an anomaly and explain also each cluster.\n\n## Proposed Solution\nTo solve the problem, I am using the premise of a gaussian distribution of the data points inside \nthe cluster, collect the statistics corresponding to each cluster, overall and by feature. The \nstatistical data collected includes count, minimum, maximum, average, standard deviation, skewness \nand kurtosis. With the statistical data, for each data point we can we can use the probability \ndensity function to compute the probability that a data point belongs to a certain cluster. \n\nTo distinguish anomalies we use a probability based threshold. For anomalies we can examine closer \nthe probabilities that the point belongs to the cluster by each feature, and those with the lowest\nprobabilities are most likely the features that drove the data point to be labeled as an anomaly. \n\nA similar approach can help explaining each cluster in a human understandable manner, as besides\nthe cluster centers, which are averages, we have also additional statistical information, like \nvariance, skewness and kurtosis.\n\n## Other Remarks\nThere are some similarities between the proposed XKmeans and the Gaussian Mixture algorithm, \nin the sense that both compute some statistics about the clusters, but XKmeans is not changing the \nKMeans algorithm, but merely collecting ome statistical data on the side in order to produce the \nfeature by feature probabilities.\n\n## Usage\nThe `XKMeans` can be used instead of the traditional `KMeans` algorithm.\n\n```scala\nimport org.apache.spark.ml.clustering.tupol.XKMeans\nimport org.apache.spark.ml.evaluation.ClusteringEvaluator\nimport org.apache.spark.sql.DataFrame\n\n// Loads data.\nval dataset: DataFrame = ???\n\n// Trains a k-means model.\nval xkmeans = new XKMeans().setK(2).setSeed(1L)\nval model = xkmeans.fit(dataset)\n\n// Make predictions\nval predictions = model.transform(dataset)\n\n// Evaluate clustering by computing Silhouette score\nval evaluator = new ClusteringEvaluator()\n\nval silhouette = evaluator.evaluate(predictions)\nprintln(s\"Silhouette with squared euclidean distance = $silhouette\")\n\n// Shows the result.\nprintln(\"Cluster Centers: \")\nmodel.clusterCenters.foreach(println)\n\n```\n\n## Input Parameters\n\nStandard `KMeans` parameters:\n- `k` is the number of desired clusters. Note that it is possible for fewer than k clusters to be returned, for example, if there are fewer than k distinct points to cluster.\n- `maxIterations` is the maximum number of iterations to run.\n- `initializationMode` specifies either random initialization or initialization via `k-means||`.\n- `runs` This param has no effect since Spark 2.0.0.\n- `initializationSteps` determines the number of steps in the `k-means||` algorithm.\n- `epsilon` determines the distance threshold within which we consider k-means to have converged.\n- `initialModel` is an optional set of cluster centers used for initialization. If this parameter is supplied, only one run is performed.\n\nSpecific `XKMeans` parameters:\n- `featuresCol` is an optional list of feature names that can be used to express better the probability by feature.\n\n\n## Input Columns\n\n| Param name              | Type(s)    | Default                | Description                      |\n| ----------------------- | ---------- | ---------------------- | -------------------------------- |\n| featuresCol             | Vector     | \"features\"             | Feature vector                   |\n\n## Output Columns\n\n| Param name              | Type(s)    | Default                | Description                      |\n| ----------------------- | ---------- | ---------------------- | -------------------------------- |\n| predictionCol           | Int        | \"prediction\"           | Predicted cluster center         |\n| distanceToCentroid      | Double     | \"distanceToCentroid\"   | Distance to cluster center       |\n| probabilityCol          | Double     | \"probability\"          | Probability of belonging to cluster |\n| probabilityByFeatureCol | Vector     | \"probabilityByFeature\" | Probability by each feature / dimension |\n\n\n## Conclusion\nThe approach exemplified through the Spark ML K-Means extension can help understanding the \nclusters and the predictions better. Understanding the clusters can help with all K-Means based \nuse-cases, weather they are classification problem or anomaly detection. \nUnderstanding the prediction results is very valuable for anomaly detection use cases.\n\nThe proposed solution has it limitations. \nFirst, the solution assumes a normal distribution of data inside the cluster so understanding the \ntype of data distribution is important. \nSecond, understanding each feature is crucial in understanding both the clusters themselves as\nwell as why some data points were labeled anomalies. \nThere are also lessons learned from trying to extend the Spark ML library and what are the\nlimitations of it.\n\n## Audience\nData scientists that are using Spark ML library and are interested in anomaly detection cases as \nwell as developers looking into how to extends the existing Spark ML framework or statistical data \ncomposition.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftupol%2Fspark-xkmeans","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftupol%2Fspark-xkmeans","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftupol%2Fspark-xkmeans/lists"}