{"id":13995553,"url":"https://github.com/intel-spark/SparseML","last_synced_at":"2025-07-22T22:31:07.561Z","repository":{"id":87341152,"uuid":"55931876","full_name":"intel-spark/SparseML","owner":"intel-spark","description":"Spark MLlib code optimized to efficiently support sparse data ","archived":false,"fork":false,"pushed_at":"2016-12-22T00:59:36.000Z","size":122,"stargazers_count":50,"open_issues_count":1,"forks_count":30,"subscribers_count":24,"default_branch":"master","last_synced_at":"2024-11-29T18:40:27.573Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/intel-spark.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2016-04-11T01:17:54.000Z","updated_at":"2022-05-12T09:29:22.000Z","dependencies_parsed_at":"2023-03-13T18:55:35.327Z","dependency_job_id":null,"html_url":"https://github.com/intel-spark/SparseML","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/intel-spark/SparseML","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/intel-spark%2FSparseML","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/intel-spark%2FSparseML/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/intel-spark%2FSparseML/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/intel-spark%2FSparseML/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/intel-spark","download_url":"https://codeload.github.com/intel-spark/SparseML/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/intel-spark%2FSparseML/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266585677,"owners_count":23952163,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-22T02:00:09.085Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-09T14:03:28.858Z","updated_at":"2025-07-22T22:31:07.284Z","avatar_url":"https://github.com/intel-spark.png","language":"Scala","funding_links":[],"categories":["Scala","人工智能"],"sub_categories":["机器学习"],"readme":"# SparseML\n\nYuhao Yang (yuhao.yang@intel.com)\n\u003cbr\u003e\u003cbr\u003e\nFrom purchase history to movie ratings, data sparsity has always been one of the primary\ncharacteristics of big data. Powerful as Spark is on parallel processing for the partitioned\ndata, many of the algorithms in MLlib are implemented based on the assumption of certain degree\nof data density, such like the gradients of logistic regression, or cluster centers of KMeans.\nYet during collaboration with some Spark users, we often find their feature number at the\ndimension of millions or even billions, which far exceeds the capacity of some important algorithms\nin MLlib, or become impractical due to enormous memory consumption even with great sparsity in the\ntraining data. To fill the gap, we present a Spark package containing some major improvements we\nhave conducted to support the sparse data at large scope. Through optimization on data structure,\nnetwork communication and arithmetic operation, we can extensively compress the memory consumption\nand reduce computation cost for sparse data, thus to enable the algorithms on larger feature\ndimensions and scope. Two of the examples are the successful support of our implementation on\nlogistic regression with 1 billion features and KMeans with 10M features and hundreds of clusters.\nWe’ll also share some work we are contributing to Spark and some best practices we have accumulated\nin the context of sparse data support on Spark MLlib.\n\n\n## Usage:\nThe class/function signature remains the same as in Spark MLlib. Please refer to the examples folder\n\n## Performance:\nAlthough the concrete performance improvements depends on the sparsity of the dataset. The algorithms\nin SparseSpark generally significantly reduce the time and memory consumption compared with the original\nSpark implementation.\n\n\n## Accuracy\nThe optimization does not affect the accuracy. It yields the same result with the Spark version,\nyet with less computation resources.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fintel-spark%2FSparseML","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fintel-spark%2FSparseML","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fintel-spark%2FSparseML/lists"}