{"id":15348775,"url":"https://github.com/zuston/sparkfe","last_synced_at":"2026-03-19T02:39:24.255Z","repository":{"id":93117633,"uuid":"358140473","full_name":"zuston/SparkFE","owner":"zuston","description":"The native Spark execution engine for AI and Feature Engineering.","archived":false,"fork":false,"pushed_at":"2021-04-14T11:57:12.000Z","size":10339,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-01T14:42:46.435Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://docs.fedb.io/sparkfe","language":null,"has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zuston.png","metadata":{"files":{"readme":"README-CN.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-04-15T05:40:35.000Z","updated_at":"2023-01-13T04:44:09.000Z","dependencies_parsed_at":"2023-03-24T06:02:57.370Z","dependency_job_id":null,"html_url":"https://github.com/zuston/SparkFE","commit_stats":{"total_commits":42,"total_committers":3,"mean_commits":14.0,"dds":0.1428571428571429,"last_synced_commit":"e5177f18a42634c5470038c454867723a607927b"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/zuston/SparkFE","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zuston%2FSparkFE","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zuston%2FSparkFE/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zuston%2FSparkFE/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zuston%2FSparkFE/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zuston","download_url":"https://codeload.github.com/zuston/SparkFE/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zuston%2FSparkFE/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29413388,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-13T06:24:03.484Z","status":"ssl_error","status_checked_at":"2026-02-13T06:23:12.830Z","response_time":78,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-01T11:52:08.183Z","updated_at":"2026-02-13T17:31:55.552Z","avatar_url":"https://github.com/zuston.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"\n\u003cdiv align=center\u003e\u003cimg src=\"./images/sparkfe_logo.png\"/\u003e\u003c/div\u003e\n\n* [**Slack频道**](https://hybridsql-ws.slack.com/archives/C01TMST8AE7)\n* [**项目讨论**](https://github.com/4paradigm/SparkFE/discussions)\n* [**README in English**](./README.md)\n\n## 介绍\n\nSparkFE是面向特征工程场景的基于LLVM优化的高性能Spark原生执行引擎。\n\nSpark发展迅速逐渐成为大数据处理领域的事实标准，然而它并非为机器学习场景设计，在AI场景有越来越多的限制。SparkFE使用C++重写了底层执行引擎，在特征抽取上获得超过6倍的性能提升，并且能够保证离线在线一致性，让AI场景落地更加容易。想了解更多细节，请参考[SparkFE中文文档](https://docs.fedb.io/v/zh-hans/sparkfe)。\n\n![Architecture](./images/sparkfe_architecture.png)\n\n## 特性\n\n* **高性能**\n\n    基于LLVM优化，在相同硬件下部分场景性能比Spark快6倍以上，同一个应用的运行时间更短且TCO成本更低。\n    \n* **零迁移成本**\n\n    Scala、Java、Python或R语言实现的SparkSQL应用，不需要修改源码和重新编译，替换SPARK_HOME即可享受原生执行引擎加速。\n    \n* **针对机器学习优化**\n  \n    提供机器学习场景常用的特殊拼表操作以及定制化UDF/UDAF支持，基本满足生产环境下机器学习场景特征工程的研发需求。\n\n* **离线在线一致性**\n  \n    结合[FEDB](https://github.com/4paradigm/fedb)，使用SparkFE开发的机器学习应用支持一键上线，保证离线在线一致性，大大降低机器学习场景的落地成本。\n\n* **Upstream同步** \n  \n    兼容Spark 3.0及后续版本，保证Spark基础功能与社区上游同步，特殊场景也可以回退使用Spark本身的优化。\n\n## 性能\n\nSparkFE在AI计算场景上比开源Spark有明显的性能提升，部分性能测试结果如下。\n\n![Benchmark](./images/sparkfe_benchmark.png)\n\n本地可进行性能测试复现结果，步骤如下。\n\n```bash\ndocker run -it 4pdosc/sparkfe bash\n\ngit clone https://github.com/4paradigm/SparkFE.git \ncd ./SparkFE/benchmark/taxi_tour_multiple_window/\n\nwget http://103.3.60.66:8001/sparkfe_resources/taxi_tour_parquet.tar.gz\ntar xzvf ./taxi_tour_parquet.tar.gz\n\nexport SPARK_HOME=/spark-3.0.0-bin-hadoop2.7/\n./submit_spark_job.sh\n\nexport SPARK_HOME=/spark-3.0.0-bin-sparkfe/\n./submit_spark_job.sh\n```\n\n## 快速开始\n\n### 使用Docker镜像\n\n运行官方的[SparkFE容器镜像](https://hub.docker.com/r/4pdosc/sparkfe)。\n\n```bash\ndocker run -it 4pdosc/sparkfe bash\n```\n\n直接执行Spark命令即可，默认使用SparkFE进行SQL优化加速。\n\n```bash\n$SPARK_HOME/bin/spark-submit \\\n  --master local \\\n  --class org.apache.spark.examples.sql.SparkSQLExample \\\n  $SPARK_HOME/examples/jars/spark-examples*.jar\n```\n\n### 使用SparkFE发行版\n\n从[Releases页面](https://github.com/4paradigm/SparkFE/releases)下载预编译包，解压后可执行Spark命令。\n\n```bash\ntar xzvf ./spark-3.0.0-bin-sparkfe.tgz\n\nexport SPARK_HOME=`pwd`/spark-3.0.0-bin-sparkfe/\n\n$SPARK_HOME/bin/spark-submit \\\n  --master local \\\n  --class org.apache.spark.examples.sql.SparkSQLExample \\\n  $SPARK_HOME/examples/jars/spark-examples*.jar\n```\n\n## 参与贡献\n\n使用官方镜像参与项目开发。\n\n```\ndocker run -it 4pdosc/sparkfe bash\n\ngit clone --recurse-submodules git@github.com:4paradigm/SparkFE.git\ncd ./SparkFE/sparkfe/\n```\n\n从源码编译sparkfe模块。\n\n| 操作系统 | 编译命令 | 备注 |\n| ------- | ------ | ---- |\n| Linux\t  | mvn clean package| 支持CentOS 6、Ubuntu等Linux发行版 |\n| MacOS   | mvn clean package -Pmacos | 支持macOS Big Sur以及后续版本 |\n| All in one | mvn clean package -Pallinone | 同时支持Linux、MacOS操作系统 |\n\n从源码编译SparkFE发行版。\n\n```bash\ncd ../spark/\n\n./dev/make-distribution.sh --name sparkfe --pip --tgz -Phadoop-2.7 -Pyarn\n```\n\n## 未来规划\n\n### SQL兼容\n\nSparkFE兼容大部分SparkSQL应用，未来将继续完善与ANSI SQL语法的兼容性，从而降低开发者的迁移成本。\n\n* [2021 H1\u0026H2] 完善多种Window语法的支持，支持带复杂表达式的Where、GroupBy等语法。\n* [2021 H1\u0026H2] 针对AI场景扩展特征工程所需要的语法特性和UDF/UDAF函数。\n\n### 性能优化\n\nSparkFE基于C++和LLVM优化后性能提升明显，未来将进一步优化减少跨语言调用开销和支持异构计算硬件。\n\n* [2021 H1] 支持多种行编码格式，兼容Spark UnsafeRow内存布局。\n* [2021 H1] 自动优化窗口数据倾斜和拼表数据倾斜场景。\n* [2021 H1] 集成针对机器学习场景的Native LastJoin优化过程。\n* [2021 H2] 全流程列式存储格式支持，降低OLAP系统读写文件开销以及支持CPU向量化计算。\n* [2021 H2] 异构计算硬件支持。\n\n### 生态集成\n\nSparkFE目前兼容Spark应用生态，未来将与更多开源系统对接集成，满足真实场景落地需求。\n\n* [2021 H2] Spark多版本集成和提供预编译包下载。\n\n## 许可证\n\n[Apache License 2.0](./LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzuston%2Fsparkfe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzuston%2Fsparkfe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzuston%2Fsparkfe/lists"}