{"id":14982317,"url":"https://github.com/endymecy/spark-ml-source-analysis","last_synced_at":"2025-10-20T10:26:33.656Z","repository":{"id":38417115,"uuid":"49274059","full_name":"endymecy/spark-ml-source-analysis","owner":"endymecy","description":"spark ml 算法原理剖析以及具体的源码实现分析","archived":false,"fork":false,"pushed_at":"2019-03-25T13:24:10.000Z","size":13831,"stargazers_count":1957,"open_issues_count":0,"forks_count":826,"subscribers_count":194,"default_branch":"master","last_synced_at":"2025-05-22T20:06:22.216Z","etag":null,"topics":["machine-learning","source-analysis","spark"],"latest_commit_sha":null,"homepage":"https://github.com/endymecy/spark-ml-source-analysis","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/endymecy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-01-08T13:39:03.000Z","updated_at":"2025-05-22T11:29:26.000Z","dependencies_parsed_at":"2022-07-11T19:50:18.967Z","dependency_job_id":null,"html_url":"https://github.com/endymecy/spark-ml-source-analysis","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/endymecy/spark-ml-source-analysis","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/endymecy%2Fspark-ml-source-analysis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/endymecy%2Fspark-ml-source-analysis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/endymecy%2Fspark-ml-source-analysis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/endymecy%2Fspark-ml-source-analysis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/endymecy","download_url":"https://codeload.github.com/endymecy/spark-ml-source-analysis/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/endymecy%2Fspark-ml-source-analysis/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":280075540,"owners_count":26267573,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-20T02:00:06.978Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["machine-learning","source-analysis","spark"],"created_at":"2024-09-24T14:05:10.889Z","updated_at":"2025-10-20T10:26:33.621Z","avatar_url":"https://github.com/endymecy.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"\u0026emsp;\u0026emsp;\u0026emsp;\u0026emsp;\u0026emsp;\u0026emsp;\u0026emsp;\u0026emsp;\u0026emsp;\u0026emsp;\u0026emsp;\u0026emsp;\u0026emsp;\u0026emsp;![spark](spark-logo-trademark.png)\n\n# spark机器学习算法研究和源码分析\n\n\u0026emsp;\u0026emsp;本项目对`spark ml`包中各种算法的原理加以介绍并且对算法的代码实现进行详细分析，旨在加深自己对机器学习算法的理解，熟悉这些算法的分布式实现方式。\n\n## 本系列文章支持的spark版本\n\n\u0026emsp;\u0026emsp;本系列文章大部分的算法基于spark 1.6.1，少部分基于spark 2.x。\n\n## 本系列的目录结构\n\n\u0026emsp;\u0026emsp;本系列目录如下：\n* [数据类型](数据类型/data-type.md)\n* [基本统计](基本统计/summary-statistics.md)\n    * [summary statistics（概括统计）](基本统计/summary-statistics.md)\n    * [correlations（相关性系数）](基本统计/correlations.md)\n    * [tratified sampling（分层取样）](基本统计/tratified-sampling.md)\n    * [hypothesis testing（假设检验）](基本统计/hypothesis-testing.md)\n    * [random data generation（随机数生成）](基本统计/random-data-generation.md)\n    * [Kernel density estimation（核密度估计）](基本统计/kernel-density-estimation.md)\n* [协同过滤](推荐/ALS.md)\n    * [交换最小二乘](推荐/ALS.md)\n* [分类和回归](分类和回归/readme.md)\n    * [线性模型](分类和回归/线性模型/readme.md)\n        * [SVMs(支持向量机)](分类和回归/线性模型/支持向量机/lsvm.md)\n        * [逻辑回归](分类和回归/线性模型/逻辑回归/logic-regression.md)\n        * [线性回归](分类和回归/线性模型/回归/regression.md)\n        * [广义线性回归](分类和回归/线性模型/广义线性回归/glr.md)\n    * [朴素贝叶斯](分类和回归/朴素贝叶斯/nb.md)\n    * [决策树](分类和回归/决策树/decision-tree.md)\n    * [组合树](分类和回归/组合树/readme.md)\n        * [随机森林](分类和回归/组合树/随机森林/random-forests.md)\n        * [梯度提升树](分类和回归/组合树/梯度提升树/gbts.md)\n    * [生存回归](分类和回归/生存回归/survival-regression.md)\n    * [保序回归](分类和回归/保序回归/isotonic-regression.md)\n* [聚类](聚类/readme.md)\n    * [k-means||算法](聚类/k-means/k-means.md)\n    * [GMM（高斯混合模型）](聚类/gaussian-mixture/gaussian-mixture.md)\n    * [PIC（快速迭代聚类）](聚类/PIC/pic.md)\n    * [LDA（隐式狄利克雷分布)](聚类/LDA/lda.md)\n    * [二分k-means算法](聚类/bis-k-means/bisecting-k-means.md)\n    * [流式k-means算法](聚类/streaming-k-means/streaming-k-means.md)\n* [最优化算法](最优化算法/梯度下降/gradient-descent.md)\n    * [梯度下降算法](最优化算法/梯度下降/gradient-descent.md)\n    * [拟牛顿法](最优化算法/L-BFGS/lbfgs.md)\n    * [NNLS(非负最小二乘)](最优化算法/非负最小二乘/NNLS.md)\n    * [带权最小二乘](最优化算法/WeightsLeastSquares.md)\n    * [迭代再加权最小二乘](最优化算法/IRLS.md)\n* [降维](降维/SVD/svd.md)\n    * [EVD（特征值分解）](降维/EVD/evd.md)\n    * [SVD（奇异值分解）](降维/SVD/svd.md)\n    * [PCA（主成分分析）](降维/PCA/pca.md)\n* [特征抽取和转换](特征抽取和转换/TF-IDF.md)\n    * [特征抽取](特征抽取和转换/TF-IDF.md)\n       * [TF-IDF](特征抽取和转换/TF-IDF.md)\n       * [Word2Vec](特征抽取和转换/Word2Vector.md)\n       * [CountVectorizer](特征抽取和转换/CountVectorizer.md)\n    * [特征转换](特征抽取和转换/normalizer.md)\n       * [Tokenizer](特征抽取和转换/Tokenizer.md)\n       * [StopWordsRemover](特征抽取和转换/StopWordsRemover.md)\n       * [n-gram](特征抽取和转换/n_gram.md)\n       * [Binarizer](特征抽取和转换/Binarizer.md)\n       * [PolynomialExpansion](特征抽取和转换/PolynomialExpansion.md)\n       * [Discrete Cosine Transform (DCT)](特征抽取和转换/DCT.md)\n       * [StringIndexer](特征抽取和转换/StringIndexer.md)\n       * [IndexToString](特征抽取和转换/IndexToString.md)\n       * [OneHotEncoder](特征抽取和转换/OneHotEncoder.md)\n       * [VectorIndexer](特征抽取和转换/VectorIndexer.md)\n       * [Normalizer(规则化)](特征抽取和转换/normalizer.md)\n       * [StandardScaler（特征缩放）](特征抽取和转换/StandardScaler.md)\n       * [MinMaxScaler](特征抽取和转换/MinMaxScaler.md)\n       * [MaxAbsScaler](特征抽取和转换/MaxAbsScaler.md)\n       * [Bucketizer](特征抽取和转换/Bucketizer.md)\n       * [ElementwiseProduct(元素智能乘积)](特征抽取和转换/element-wise-product.md)\n       * [SQLTransformer](特征抽取和转换/SQLTransformer.md)\n       * [VectorAssembler](特征抽取和转换/VectorAssembler.md)\n       * [QuantileDiscretizer](特征抽取和转换/QuantileDiscretizer.md)\n    * [特征选择](特征抽取和转换/VectorSlicer.md)\n       * [VectorSlicer](特征抽取和转换/VectorSlicer.md)\n       * [RFormula](特征抽取和转换/RFormula.md)\n       * [ChiSqSelector(卡方选择器)](特征抽取和转换/chi-square-selector.md)\n\n    \n## 说明\n\n\u0026emsp;\u0026emsp;本专题的大部分内容来自[spark源码](https://github.com/apache/spark)、[spark官方文档](https://spark.apache.org/docs/latest)，并不用于商业用途。转载请注明本专题地址。\n本专题引用他人的内容均列出了参考文献，如有侵权，请务必邮件通知作者。邮箱地址：`endymecy@sina.cn`。\n\n\u0026emsp;\u0026emsp;本专题的部分文章中用到了latex来写数学公式,可以在浏览器中安装`MathJax`插件用来展示这些公式。\n\n\u0026emsp;\u0026emsp;本人水平有限，分析中难免有错误和误解的地方，请大家不吝指教，万分感激。\n    \n## License\n\n\u0026emsp;\u0026emsp;本文使用的许可见 [LICENSE](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fendymecy%2Fspark-ml-source-analysis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fendymecy%2Fspark-ml-source-analysis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fendymecy%2Fspark-ml-source-analysis/lists"}