{"id":13570090,"url":"https://github.com/melin/flink-jobserver","last_synced_at":"2025-04-04T06:31:48.026Z","repository":{"id":65102041,"uuid":"528302966","full_name":"melin/flink-jobserver","owner":"melin","description":"REST job server for Apache Flink","archived":false,"fork":false,"pushed_at":"2023-06-21T02:52:49.000Z","size":2132,"stargazers_count":16,"open_issues_count":1,"forks_count":11,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-11-05T02:36:12.263Z","etag":null,"topics":["flink","hadoop","hive","java","kerberos","kubernetes","yarn"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/melin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-08-24T06:56:57.000Z","updated_at":"2024-08-14T03:04:12.000Z","dependencies_parsed_at":"2024-01-14T03:50:33.063Z","dependency_job_id":"96b9f768-a1af-4f2c-bd06-f94d72812a11","html_url":"https://github.com/melin/flink-jobserver","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/melin%2Fflink-jobserver","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/melin%2Fflink-jobserver/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/melin%2Fflink-jobserver/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/melin%2Fflink-jobserver/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/melin","download_url":"https://codeload.github.com/melin/flink-jobserver/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247134487,"owners_count":20889400,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["flink","hadoop","hive","java","kerberos","kubernetes","yarn"],"created_at":"2024-08-01T14:00:48.118Z","updated_at":"2025-04-04T06:31:43.019Z","avatar_url":"https://github.com/melin.png","language":"JavaScript","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"readme":"## REST job server for Apache Flink (Flink as a Service)\n\n主要特点\n1. 通过Rest API 提交flink 作业运行，支持sql，java/scala 类型作业，解耦业务系统与Flink 集群。\n2. Flink Job 运行资源相互隔离，每一个job 独立运行在一个FLink driver中。\n3. 预启动 Flink Driver，特别是在WebIDE 交互场景，有效提高Job 启动速度，Driver 共享运行多个Job(同时只有一个job运行，有点类似pre job模式)\n4. 支持多集群部署，Client 提交Job 到指定集群运行，只支持 application 和 session 模式\n5. Driver 定制化，可以实现比较多能力，例如：表权限，碎片文件压缩，DQC等功能。\n6. 更加灵活的对调度任务并发和优先级控制，例如单个用户最大并发数量，不完全依赖底层yarn、k8s等资源管理能力。例如一些调度框架是把作业直接传给yarn 资源管理器，如果yarn资源不够，提交上去的任务全部在yarn 等待队列中。CDH 默认是公平调度，会导致任务无法按照优先级运行。\n\n![Flink 任务](imgs/jobserver.png)\n![集群管理](imgs/cluster.png)\n\n## 一、Build\n\n```\nmvn clean package -Prelease,hadoop-2 -DlibScope=provided\nmvn clean package -Prelease,hadoop-3 -DlibScope=provided\nmvn clean package -Prelease,cdh6 -DlibScope=provided\n```\n\n## 二、作业实例接口 \n包含作业实例提交、作业实例状态查询、作业实例运行日志查询、终止作业实例接口，具体接口：[Rest API](https://github.com/melin/flink-jobserver/blob/master/flink-admin/src/main/java/io/github/melin/flink/jobserver/web/rest/JobServerRestApi.java)\n\n### 1、Flink jar 作业\n请参考: [Flink jar](https://github.com/melin/flink-jobserver/tree/master/jobserver-api)\n\n## 三、Yarn Cluster 模式部署\n### 1、准备环境\nFlink 任务运行环境：Hadoop 2.7.7，Spark 3.3.0。为避免每次任务运行上传jar，提前把相关jar 上传到 hdfs 路径。根路径：/user/superior/flink-jobserver (可以在集群管理中通过，修改jobserver.driver.home 参数设置)，根路径下有不同jar 和 目录\n1. flink-jobserver-driver-0.1.0.jar  -- flink jobserver driver jar，jar 参考编译 flink jobserver 部分。\n2. aspectjweaver-1.9.9.1.jar  -- aspectj 拦截spark 相关代码，实现功能增强，直接maven 仓库下载 \n3. flink-1.16.0  --flink 依赖所有jar，从flink 官网下载: flink-1.16.0-bin-scala_2.12.tgz, 解压后把lib 目录下所有jar 上传到 flink-1.16.0 目录。\n4. 如果使用hudi，上传 hudi-flink[version]-bundle-[version].jar 到 flink-1.16.0 目录。\n\n### 2、Yarn 集群配置要求\n1. yarn 开启日志聚合，方便出现问题查看日志 \n2. yarn 开启cgroup cpu 资源隔离，避免资源争抢 \n3. 开启 history server \n4. hdfs block 设置为256M\n\n### 3、部署 jobserver admin 服务\n\n上传 flink-jobserver-0.1.0.tar.gz 文件到服务器，直接解压生成目录：flink-jobserver-0.1.0\n\u003e tar -zxf flink-jobserver-0.1.0.tar.gz\n\n创建数据jobserver，执行 script/jobserver.sql 脚本，创建表。\n\n### 4、集群相关参数\n\n[参考代码](https://github.com/melin/flink-jobserver/blob/master/jobserver-admin/src/main/java/io/github/melin/flink/jobserver/FlinkJobServerConf.java)\n\n### 5、启动服务\n```\n./bin/server.sh start dev\n-- 启动脚本有两个参数\n第一个参数可选值：start、stop、restart、status、log，启动status 和 log 不需要指定第二参数。\n第二个参数可选值：dev、test、production。对应spring boot profile，对应应用启动加载conf目录下application-[profile].properties 文件\n```\n\n## 四、相关项目\n1. https://gitee.com/melin/bee\n2. https://github.com/melin/superior-sql-parser\n3. https://github.com/melin/superior-sql-formatter\n4. https://github.com/melin/superior-sql-autocomplete\n5. https://github.com/melin/datatunnel\n6. https://github.com/melin/spark-jobserver\n\n## 五、参考资料\n1. [Flink SQL通过Hudi HMS Catalog读写Hudi并同步Hive表](https://mp.weixin.qq.com/s/WpvOvFv-iAzdCwmOO5oQ4Q)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmelin%2Fflink-jobserver","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmelin%2Fflink-jobserver","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmelin%2Fflink-jobserver/lists"}