{"id":13570074,"url":"https://github.com/melin/spark-jobserver","last_synced_at":"2025-04-04T06:31:45.592Z","repository":{"id":58350868,"uuid":"531090699","full_name":"melin/spark-jobserver","owner":"melin","description":"REST job server for Apache Spark","archived":false,"fork":false,"pushed_at":"2024-04-02T03:49:22.000Z","size":3887,"stargazers_count":39,"open_issues_count":5,"forks_count":15,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-11-05T02:36:10.973Z","etag":null,"topics":["hadoop","hive","java","kerberos","kubernetes","spark","yarn"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/melin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-08-31T13:12:08.000Z","updated_at":"2024-10-23T01:59:09.000Z","dependencies_parsed_at":"2024-01-14T03:50:40.764Z","dependency_job_id":"407eb197-d14e-40b9-8ba2-93f94a48a722","html_url":"https://github.com/melin/spark-jobserver","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/melin%2Fspark-jobserver","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/melin%2Fspark-jobserver/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/melin%2Fspark-jobserver/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/melin%2Fspark-jobserver/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/melin","download_url":"https://codeload.github.com/melin/spark-jobserver/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247134471,"owners_count":20889399,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hadoop","hive","java","kerberos","kubernetes","spark","yarn"],"created_at":"2024-08-01T14:00:47.975Z","updated_at":"2025-04-04T06:31:40.582Z","avatar_url":"https://github.com/melin.png","language":"JavaScript","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"readme":"## REST job server for Apache Spark (Spark as a Service)\n\njob server 来源以前做[大数据平台](https://github.com/melin/superior-sql-parser/tree/master/imgs)的设计，同时支持sql、jar、python 任务提交运行，对比其他常见解决方案：\n\n|      | Spark JobServer                                                                                                                                                                                                                                                               | Kyuubi                                                                                                                       | Spark ThriftServer(Spark on Hive) | HiveServer2(Hive on Spark) |\n|:---------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------|:----------------------------------|:---------------------------|\n| 任务类型 | SQL/JAR/Python                                                                                                                                                                                                                                                                | SQL                                                                                                                          | SQL                               | SQL                        |\n| 提交方式 | Rest API                                                                                                                                                                                                                                                                      | JDBC                                                                                                                         | JDBC                              | JDBC                       |\n| 编译   | Spark Catalyst/引擎端                                                                                                                                                                                                                                                            | Spark Catalyst/引擎端                                                                                                           | Spark Catalyst/服务端                | HiveOptimizer/服务端          |\n| 执行   | Spark App共享策略：可以根据Yarn 资源评估能够启动多少spark app, Jobserver 管理Spark app数量和生命周期。spark app 支持共享和非共享两种模式\u003cbr/\u003e\u003cb\u003e共享\u003c/b\u003e：同时只能运行一个任务，任务运行结束，spark app 状态为空闲，可以接收新的任务运行，比较适合对资源没有特殊要求的任务；\u003cbr/\u003e\u003cb\u003e非共享\u003c/b\u003e：启动一个新的spark app运行任务，任务运行结束，spark app会关闭，比适合任务运行需要很多资源，需要单独设置内存、cpu、executor数量的场景。 | 多种Spark APP共享测录：Server、User、Group、Connection。\u003cbr/\u003e这些策略都不能做到真正的资源隔离，例如User策略，同用户多个任务运行，存在一个spark app 运行多个任务，相互争抢资源，大任务可能饿死小任务 | 所有任务共享一个spark app                 | 提交查询对应一个app，相比原生spark执行慢   |\n\nSpark JobServer特点:\n1. 支持sql、jar、python 任务执行。\n2. 细粒度到任务资源完全隔离，避免大任务饿死小任务，kyuubi 隔离级别还是比较粗，最小是User级别隔离，存在同个用户多个任务相互争抢资源，不能做到任务级别隔离。\n3. Spark APP 资源池管理，根据计算集群资源数量，管理spark app 数量。避免过多任务提交到yarn，在yarn等待队列中，kyuubi不管理spark app 数量\n4. 大数据平台交互查询场景，不需要等待spark app 启动，快速执行sql 等任务。\n\n主要功能: \n1. 通过Rest API 提交spark 作业运行，支持sql，java/scala，python类型作业，解耦业务系统与spark 集群。\n2. Spark Job 运行资源相互隔离以及高可用性，每一个job 独立运行在一个Spark driver中。\n3. 预启动 Spark Driver，提高Job 启动速度，Driver 共享运行多个Job(同时只有一个job运行)\n4. 支持多Yarn集群部署，Client 提交Job 到指定集群运行。\n5. Driver 定制化，可以实现比较多能力，例如：表权限，碎片文件压缩，DQC等功能。\n6. 支持作业实例任务依赖(DAG),\n7. 支持spark streaming sql，方便对接kafka、debezium、canal、hudi cdc等流数据源，具体请参考 [spark-streaming-sql.md](docs%2Fspark-streaming-sql.md)\n8. 更加灵活的对调度任务并发控制和优先级，例如单个用户最大并发数量，不完全依赖底层yarn、k8s等资源管理能力。例如一些调度框架是把作业直接传给yarn 资源管理器，如果yarn资源不够，提交上去的任务全部在yarn 等待队列中。CDH 默认是公平调度，会导致任务无法按照优先级运行。\n9. 动态注册jdbc catalog，具体参考 [dynamic-jdbc-catalog.md](docs%2Fdynamic-jdbc-catalog.md)\n10. 支持kerberos 认证\n\n@TODO\n1. 集成k8s\n2. 支持作业定时调度, 可以减少其它调度系统依赖。(一个简易的方案，每天凌晨生成当天需要运行的作业实例，再轮训作业实例表，阿里云dataworks 每天千万级别的任务实例，也是轮训数据库方案)\n3. 集成数据权限、数据血缘(借鉴 kyuubi，但不依赖ranger)\n4. 完善调度任务优先级。\n5. spark jar 上传管理\n\n![架构](imgs/jobserver.png)\n![Hadoop 集群管理](imgs/cluster.png)\n![实例运行日志](imgs/instance.png)\n![与大数据开发平台集成，作业执行流程](imgs/jobflow.png)\n\n## 一、Build\n\n```\nmvn clean package -Prelease,hadoop-2 -DlibScope=provided\nmvn clean package -Prelease,hadoop-3 -DlibScope=provided\nmvn clean package -Prelease,cdh6 -DlibScope=provided\n```\n\n## 二、作业实例接口\n包含作业实例提交、作业实例状态查询、作业实例运行日志查询、终止作业实例接口，具体接口：[Rest API](https://github.com/melin/spark-jobserver/blob/master/jobserver-admin/src/main/java/io/github/melin/spark/jobserver/rest/JobServerRestApi.java)\n\n## 三、作业开发\nspark jar和python 作业类型，作业代码中不需要初始化sparkSession，SparkJobserver 初始化sparkSession，传递给作业，目的是为了统一规范作业配置，以及共享sparkSession。\n\n### 1、Spark jar 作业\n请参考: [Spark jar](https://github.com/melin/spark-jobserver/tree/master/jobserver-api)\n\n### 2、Spark pyhton 作业\npython 作业入口方法为：main，\n```python\ndef main(sparkSession):\n  print(\"hello world\")\n```\n\n## 四、Yarn Cluster 模式部署\n### 1、准备环境\nSpark 任务运行环境：Hadoop 2.7.7，Spark 3.3.2。为避免每次任务运行上传jar，提前把相关jar 上传到 hdfs 路径。根路径：/user/superior/spark-jobserver (可以在集群管理中通过，修改jobserver.driver.home 参数设置)，根路径下有不同jar 和 目录\n1. spark-jobserver-driver-0.1.0.jar  -- spark jobserver driver jar，jar 参考编译spark jobserver 部分。\n2. aspectjweaver-1.9.19.jar  -- aspectj 拦截spark 相关代码，实现功能增强，直接maven 仓库下载\n3. spark-3.3.1  --Spark 依赖所有jar，从spark 官网下载: spark-3.3.2-bin-hadoop2.tgz, 解压后把jars 目录下所有jar 上传到 spark-3.3.1 目录。\n4. 集成hudi(可选)，下载 https://repo1.maven.org/maven2/org/apache/hudi/hudi-spark3.3-bundle_2.12/0.12.1/hudi-spark3.3-bundle_2.12-0.12.1.jar 文件 上传到 spark-3.3.1 目录\n\n\n### 2、Yarn 集群配置要求\n1. yarn 开启日志聚合，方便出现问题查看日志\n2. yarn 配置spark_shuffle，spark 需要开启动态资源分配\n3. yarn 开启cgroup cpu 资源隔离，避免资源争抢\n4. 开启 history server\n5. hdfs block 设置为256M\n\n### 3、部署 jobserver admin 服务\n\n上传 spark-jobserver-0.1.0.tar.gz 文件到服务器，直接解压生成目录：spark-jobserver-0.1.0\n\u003e tar -zxf spark-jobserver-0.1.0.tar.gz\n\n创建数据jobserver，执行[jobserver.sql](build/script/jobserver.sql) 脚本，创建表。\n\n### 4、集群相关参数\n\n[参考代码](https://github.com/melin/spark-jobserver/blob/master/jobserver-admin/src/main/java/io/github/melin/spark/jobserver/SparkJobServerConf.java)\n\n### 5、启动服务\n\n```\n./bin/server.sh start dev\n-- 启动脚本有两个参数\n第一个参数可选值：start、stop、restart、status、log，启动status 和 log 不需要指定第二参数。\n第二个参数可选值：dev、test、production。对应spring boot profile，对应应用启动加载conf目录下application-[profile].properties 文件\n```\n\nWeb Console: http://ip:9001, 默认账号/密码: jobserver/jobserver2022\n\n## 五、相关项目\n1. https://gitee.com/melin/bee\n2. https://github.com/melin/superior-sql-parser\n3. https://github.com/melin/superior-sql-formatter\n4. https://github.com/melin/superior-sql-autocomplete\n5. https://github.com/melin/datatunnel\n6. https://github.com/melin/flink-jobserver\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmelin%2Fspark-jobserver","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmelin%2Fspark-jobserver","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmelin%2Fspark-jobserver/lists"}