{"id":13487258,"url":"https://github.com/zhp8341/flink-streaming-platform-web","last_synced_at":"2025-05-15T04:00:25.451Z","repository":{"id":37050000,"uuid":"304351681","full_name":"zhp8341/flink-streaming-platform-web","owner":"zhp8341","description":"基于flink的实时流计算web平台","archived":false,"fork":false,"pushed_at":"2024-08-31T12:22:43.000Z","size":12221,"stargazers_count":1829,"open_issues_count":33,"forks_count":688,"subscribers_count":50,"default_branch":"master","last_synced_at":"2025-04-11T14:19:22.435Z","etag":null,"topics":["flink","flink-sql","sql","streaming"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zhp8341.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-10-15T14:26:20.000Z","updated_at":"2025-04-09T14:52:39.000Z","dependencies_parsed_at":"2024-06-29T04:20:09.946Z","dependency_job_id":"4e91b410-ef86-4197-bd75-9a1db783adcd","html_url":"https://github.com/zhp8341/flink-streaming-platform-web","commit_stats":null,"previous_names":[],"tags_count":26,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhp8341%2Fflink-streaming-platform-web","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhp8341%2Fflink-streaming-platform-web/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhp8341%2Fflink-streaming-platform-web/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhp8341%2Fflink-streaming-platform-web/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zhp8341","download_url":"https://codeload.github.com/zhp8341/flink-streaming-platform-web/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254270640,"owners_count":22042858,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["flink","flink-sql","sql","streaming"],"created_at":"2024-07-31T18:00:57.121Z","updated_at":"2025-05-15T04:00:25.388Z","avatar_url":"https://github.com/zhp8341.png","language":"Java","funding_links":[],"categories":["Java","大数据"],"sub_categories":[],"readme":"\n## 该项目获得 [Flink Forward Asia Hackathon (2021) 季军 ](https://mp.weixin.qq.com/s/J2OkX_D34ktfwSfhwdmXYQ)    \n\n**国内文章备用地址（因为github上面 图片效果可能看不到，原因你懂的 哈哈）**\n\nhttps://xie.infoq.cn/article/1af0cb75be056fea788e6c86b \n\n\n\n**github地址**  https://github.com/zhp8341/flink-streaming-platform-web\n\n\n**国内gitee 地址** https://gitee.com/zhuhuipei/flink-streaming-platform-web\n\n\n### 如需[技术付费服务点击查看](/docs/service.md)\n\n\n\n## 一、简介 \n  flink-streaming-platform-web系统是基于[Apache Flink](https://flink.apache.org) 封装的一个可视化的、轻量级的flink web客户端系统，用户只需在web\n  界面进行sql配置就能完成流计算任务。\n  \n  **主要功能**:包含任务配置、启/停任务、告警、日志等功能，支持sql语法提示，格式化、sql语句校验。 \n   \n  **目的**:减少开发、降低成本 完全实现sql化 流计算任务。 😂\n\n\n   该项目获得 [Flink Forward Asia Hackathon (2021) 季军 ](https://mp.weixin.qq.com/s/J2OkX_D34ktfwSfhwdmXYQ)\n\n   [点击查看WEB页面功能显示效果](/docs/img.md)\n\n   **如需技术支持[详见](/docs/service.md)**\n\n\n### 1、主要功能\n\n   * **[1] 任务支持单流 、双流、 单流与维表等。**\n   * **[2] 支持本地模式、yarn-per模式、STANDALONE模式 Application模式**\n   * **[3] 支持catalog、hive。**\n   * **[4] 支持自定义udf、连接器等,完全兼容官方连接器。**\n   * **[5] 支持sql的在线开发，语法提示，格式化。**\n   * **[6] 支持钉钉告警、自定义回调告警、自动拉起任务。**\n   * **[7] 支持自定义Jar提交任务。**\n   * **[8] 支持多版本flink版本（需要用户编译对应flink版本）。**\n   * **[9] 支持自动、手动savepoint备份，并且从savepoint恢复任务。**\n   * **[10] 支持批任务如：hive。**\n   * **[11] 连接器、udf等三jar管理**\n\n  **目前flink版本已经升级到1.16.7**\n\n\n  **如果您觉得还不错请在右上角点一下star 谢谢 🙏 大家的支持是开源最大动力**\n  \n    \n### 2、效果及源码文档\n\n1、 [点击查看WEB页面功能显示效果](/docs/img.md)\n\n2、 [源码讲解文档](/docs/source.md)\n\n**如需技术支持[详见](/docs/service.md)**\n\n## 二、环境搭建及安装\n\n1、 [Flink 和 flink-streaming-platform-web 安装部署](/docs/deploy.md)\n\n2、 [在IDEA中执行运行调试](/docs/idea-run.md)\n\n3、 [docker容器化制作镜像(参考)](/docs/tristan-deploy-use)\n\n4、[flink-streaming-platform-web 体验（基于docker）适合体验者](/docs/docker-demo.md)\n\n5、关于k8s理论上是可行的，需要结合flink版本（或者hadoop版本）定制镜像\n\n**如需技术支持[详见](/docs/service.md)**\n\n## 三、功能介绍\n\n### 3.1 配置操作\n\n1、[sql任务配置使用手册](/docs/manual-sql.md)\n\n2、[jar任务配置使用手册](/docs/manual-jar.md)\n\n\n### 3.2 sql配置demo\n\n\n1、[demo1 单流kafka写入mysqld 参考 ](/docs/sql_demo/demo_1.md)\n\n2、[demo2 双流kafka写入mysql 参考](/docs/sql_demo/demo_2.md)\n\n3、[demo3 kafka和mysql维表实时关联写入mysql 参考](/docs/sql_demo/demo_3.md)\n\n4、[demo4 滚动窗口 ](/docs/sql_demo/demo_4.md)\n\n5、[demo5 滑动窗口](/docs/sql_demo/demo_5.md)\n\n6、[demo6 JDBC CDC的使用示例](/docs/sql_demo/demo_6.md)\n\n7、[demo7 datagen简介](/docs/sql_demo/demo_datagen.md)\n\n8、[catalog 使用示例](/docs/catalog.md)\n\n9、[hive批任务 使用示例](/docs/sql_demo/demo_batch.md)\n\n\n###  3.2 hello-word demo\n\n\n**请使用下面的sql进行环境测试 用于新用户跑一个hello word 对平台有个感知的认识**\n\n```sql\n\n  CREATE TABLE source_table (\n  f0 INT,\n  f1 INT,\n  f2 STRING\n ) WITH (\n  'connector' = 'datagen',\n  'rows-per-second'='5'\n );\n  \n  \n CREATE TABLE print_table (\n  f0 INT,\n  f1 INT,\n  f2 STRING\n ) WITH (\n  'connector' = 'print'\n );\n  \n  \n  insert into print_table select f0,f1,f2 from source_table;\n \n```\n\n\n**官方相关连接器下载** \n\n请移步 https://nightlies.apache.org/flink/flink-docs-release-1.14/zh/docs/connectors/table/overview/\n\n\n\n\n\n\n##  四、支持flink sql官方语法\n\n\n| 支持的sql语法        | \n| --------   | \n| INSERT   INTO         |     \n| INSERT   OVERWRITE          |   \n|  CREATE   TABLE   |      \n|  CREATE    FUNCTION |      \n| CREATE    VIEW  |      \n| USE   CATALOG   |      \n|  DROP     |      \n|  ALTER     |      \n|  SHOW CATALOGS     |      \n|  SHOW DATABASES     |      \n|  SHOW TABLES     |      \n|  SHOW FUNCTIONS     |      \n| CREATE CATALOG      |      \n|   SET   |      \n|   SELECT (不支持)    | \n\n\n\n\n\n\n##  五、其他\n1、由于hadoop集群环境不一样可能导致部署出现困难,整个搭建比较耗时.\n\n2、由于es 、hbase等版本不一样可能需要下载源码重新选择对应版本 源码地址 [https://github.com/zhp8341/flink-streaming-platform-web](https://github.com/zhp8341/flink-streaming-platform-web)\n\n交流和解答\n\n\n\n\n\n\n 完全按照1.14.3的连接器相关的配置详见\n\nhttps://nightlies.apache.org/flink/flink-docs-release-1.13/zh/docs/connectors/table/kafka/\n\n\n\n如果需要使用到连接器请去官方下载\n如：kafka 连接器 https://nightlies.apache.org/flink/flink-docs-release-1.13/zh/docs/connectors/table/kafka/\n\n**第一种下载连接器后直接放到 flink/lib/目录下就可以使用了**\n\n    1、该方案存在jar冲突可能，特别是连接器多了以后\n    2、在非yarn模式下每次新增jar需要重启flink集群服务器\n\n\n**第二种放到http的服务下填写到三方地址**\n\n    公司内部建议放到内网的某个http服务\n    http://ccblog.cn/jars/flink-connector-jdbc_2.11-1.12.0.jar\n    http://ccblog.cn/jars/flink-sql-connector-kafka_2.11-1.12.0.jar\n    http://ccblog.cn/jars/flink-streaming-udf.jar\n    http://ccblog.cn/jars/mysql-connector-java-5.1.25.jar\n\n    如果使用jar管理功能 可直接填写jar包名字\n    flink-connector-jdbc_2.11-1.12.0.jar\n    flink-streaming-udf.jar\n    mysql-connector-java-5.1.25.jar\n    \n\n\n ![图片](http://img.ccblog.cn/flink/9.png)\n\n 多个url使用换行\n\n**自定义连接器打包的时候需要打成shade 并且解决jar的冲突**\n\n\n**个人建议使用第二种方式，每个任务之间jar独立，如果把所有连接器放到lib 可能会和其他任务的jar冲突**\n**公用的可以放到flink/lib目录里面  如：mysql驱动 kafka连接器等**\n\n\n##  六、问题\n\n1、 [可能遇到的问题和解决方案](/docs/question.md)\n\n\n\n##  七、RoadMap\n\n1、批量任务将加入调度系统\n\n\n\n\n\n##  八、源码编译\n\n[根据自己的flink版本重新编译web客户端](/docs/compile.md)\n\n\n ##  九、联系方式 \n \n 请加钉钉群： 34315096 \n\n钉钉 \n\n![钉钉群](./docs/img/dd.png)\n\n \n ##  十、使用情况 \n\n \n[2021-03-18 和 2021-03-19 两天 做的调研 ](/docs/img2.md)\n\n\n## 十一、 鸣谢\n感谢文香炯（湖南草花互动科技股份公司）贡献前端代码 ：https://gitee.com/wenxiangjiong/flink-streaming-platform-web.git\n\n\n## 十二、 技术支持\n### 如需[技术付费服务点击查看](/docs/service.md)\n     \n## 十三、 捐赠\n\n\n\n![支付宝](./docs/img/zfb.png)\n![微信](./docs/img/wx.png)\n \n \n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzhp8341%2Fflink-streaming-platform-web","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzhp8341%2Fflink-streaming-platform-web","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzhp8341%2Fflink-streaming-platform-web/lists"}