{"id":19990368,"url":"https://github.com/JoeyBling/webkettle","last_synced_at":"2025-05-04T09:35:23.138Z","repository":{"id":37335682,"uuid":"134533177","full_name":"JoeyBling/webkettle","owner":"JoeyBling","description":"基于web版kettle开发的一套分布式综合调度,管理,ETL开发的用户专业版B/S架构工具","archived":false,"fork":false,"pushed_at":"2024-06-25T02:57:56.000Z","size":9757,"stargazers_count":559,"open_issues_count":21,"forks_count":261,"subscribers_count":39,"default_branch":"master","last_synced_at":"2024-11-08T16:48:40.018Z","etag":null,"topics":["etl","extjs","kettle","kettle-web","mysql","quartz","spring-mvc"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JoeyBling.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":[],"patreon":null,"open_collective":"zhousiwei","ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":["https://zhousiwei.gitee.io/ibooks/about/#%E6%89%93%E8%B5%8F","https://joeybling.github.io/ibooks/about/#%E6%89%93%E8%B5%8F"]}},"created_at":"2018-05-23T07:47:56.000Z","updated_at":"2024-11-06T06:47:43.000Z","dependencies_parsed_at":"2024-11-08T16:37:28.548Z","dependency_job_id":"da18d6b2-478f-4792-b35f-031cb49a5a86","html_url":"https://github.com/JoeyBling/webkettle","commit_stats":{"total_commits":21,"total_committers":1,"mean_commits":21.0,"dds":0.0,"last_synced_commit":"f59006d639dc76f1aeb688fea88f2a29c8dd3904"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JoeyBling%2Fwebkettle","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JoeyBling%2Fwebkettle/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JoeyBling%2Fwebkettle/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JoeyBling%2Fwebkettle/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JoeyBling","download_url":"https://codeload.github.com/JoeyBling/webkettle/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224214059,"owners_count":17274524,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["etl","extjs","kettle","kettle-web","mysql","quartz","spring-mvc"],"created_at":"2024-11-13T04:51:11.282Z","updated_at":"2024-11-13T04:51:35.052Z","avatar_url":"https://github.com/JoeyBling.png","language":"JavaScript","funding_links":["https://opencollective.com/zhousiwei","https://zhousiwei.gitee.io/ibooks/about/#%E6%89%93%E8%B5%8F","https://joeybling.github.io/ibooks/about/#%E6%89%93%E8%B5%8F"],"categories":["JavaScript"],"sub_categories":[],"readme":"\n![star](https://img.shields.io/github/stars/JoeyBling/webkettle \"star\")\n![fork](https://img.shields.io/github/forks/JoeyBling/webkettle \"fork\")\n[![webkettle交流群](https://img.shields.io/badge/QQ群-487063343-red.svg \"webkettle交流群\")](https://jq.qq.com/?_wv=1027\u0026k=55kiWBY \"webkettle交流群\")\n[![LICENSE](https://img.shields.io/github/license/JoeyBling/webkettle \"LICENSE\")](./LICENSE \"LICENSE\")\n\n\u003e **欢迎相同爱好者加入一起维护此项目**\n\n#  webKettleETL产品介绍\n-------------------------\n\n-  webkettle平台，创造性的将平台构建为B/S架构的ETL模型设计以及集成用户专业调度管理的分布式ETL建模运维系统。\n-  系统分为七大模块:模型、平台、任务、定时调度、日志、节点、用户.\n-  模型模块进行ETL模型开发，在B/S系统中用拖拽的方式设计数据流逻辑。\n-  其他六个模块为用户专业调度管理系统,更多是面向业务运维人员，关注于整个流程的执行情况和数据的导入导出细节信息,以及对任务的综合调度。完全支持集群和单机两种运行模式。并通过用户权限的管控让平台的资源分配变得更加专业。\n\n# 模块展示\n--------------------------\n- **平台**（展示平台概况）\n\n![](./examples/images/平台概况_mini.png \"平台概况\")\n\n- **模型设计**（进行ETL模型设计调试）\n\n![](./examples/images/ETL模型设计调试_mini.png \"ETL模型设计调试\")\n\n- **任务管理**（对ETL任务进行综合的调度管理以及监控）\n\n![](./examples/images/作业管理_mini.png \"作业管理\")\n\n![](./examples/images/任务监控_mini.png \"任务监控\")\n\n- **日志**（ETL任务的执行日志模块）\n\n![](./examples/images/ETL任务的执行日志模块_mini.png \"ETL任务的执行日志模块\")\n\n-  **节点管理**（ETL任务的执行引擎节点管理）\n\n![](./examples/images/节点管理_mini.png \"节点管理\")\n\n-  **定时调度**（定时ETL任务管理）\n\n![](./examples/images/定时调度_mini.png \"定时调度\")\n\n-  **用户管理**（平台的用户权限管理系统）\n\n![](./examples/images/用户管理_mini.png \"用户管理\")\n\n#  webKettleETL技术实现\n-------------------------\n- ETL是基于流行的ETL工具kettle开发的B/S架构版本的ETL产品,所使用的底层引擎是调用kettle的api\n- ETL任务以元数据的方式储存在资源库里面,通过读取资源库元数据,对元数据进行解析后可将ETL任务通过MXGraph展现至B/S架构的系统中,实现了在B/S架构上面开发ETL任务的功能,属于系统的模型开发模块功能.\n- 通过对元数据的操作可以对ETL任务进行综合性的管理以及监控,通过Quartz定时框架可以实现对任务的定时调度\n- 根据kettle执行引擎可以部署多台服务器节点的特性,再通过系统跟远程的节点发送HTTP请求,可以实现在多节点上分布式的执行ETL任务,并可以实现实时监控ETL的任务运行情况\n- 基于高度自由化的B/S框架,可以嵌入多元化的需求,于是我们在中添加了用户管理模块,通过用户权限的概念对所有的ETL任务进行人性化的运维管理\n- 整个项目使用Maven进行构建管理,多模块之间低耦合,高拓展性\n\n![](./examples/images/技术实现_mini.png \"webKettleETL技术实现\")\n\n\n# webKettleETL整体功能流程\n--------------------------\n![](./examples/images/功能流程_mini.png \"功能流程\")\n![](./examples/images/开发里程碑_mini.png \"开发里程碑\")\n\n# 技术选型\n--------------------------\n\n- 核心框架：```Spring Framework```\n- 任务调度：```Spring + Quartz```\n- 持久层框架：```MyBatis```\n- 会话管理：```Spring-Session```\n- 日志管理：```Log4j```\n- 前端框架：```EXTJS3.4+MXGraph2.3```\n- 项目管理: ```Maven3.2.3```\n- ETL底层引擎:```kettle7.0```\n\n# 项目结构\n\u003e 项目结构图\n\n![](./examples/images/项目结构_mini.png \"项目结构\")\n\n# 启动说明\n--------------------------\n\n\u003e **附加说明：启动Tomcat或其他服务器时，请设置路径为 /**\n\n1. 本系统采用单资源库模式，数据源连接在```dispatch-servlet.xml```中配置，系统启动后就会读取该资源库\n2. **数据库脚本在项目的sql目录下**\n3. 数据库暂时只支持MySQL，本系统在MySQL5.5.20版本上测试，其他版本尚未测试\n4. 本例使用Maven3.2.3构建，启动服务器后访问http://localhost:8080/\n5. 支持`IE9`及以上、`FireFox`等浏览器，IE6-IE8需要做特殊化处理，其他浏览器未测试\n6. jdk版本需要```1.8```以上\n7. 管理员账号密码：```admin/admin```\n\n### 本地预览\n```bash\nmvn install\ncd ./kettle-webapp\nmvn clean tomcat7:run\n```\n\n### 编译部署\n```bash\ncd ./kettle-webapp\nmvn clean tomcat7:deploy\n```\n\n# 许可证\n[![LICENSE](https://img.shields.io/github/license/JoeyBling/webkettle \"LICENSE\")](./LICENSE \"LICENSE\")","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FJoeyBling%2Fwebkettle","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FJoeyBling%2Fwebkettle","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FJoeyBling%2Fwebkettle/lists"}