{"id":20917851,"url":"https://github.com/aceld/kis-flow","last_synced_at":"2025-04-05T06:10:20.993Z","repository":{"id":212427673,"uuid":"730714815","full_name":"aceld/kis-flow","owner":"aceld","description":" Stream processing framework based on Golang.","archived":false,"fork":false,"pushed_at":"2024-09-10T02:49:44.000Z","size":250,"stargazers_count":372,"open_issues_count":3,"forks_count":33,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-11-24T17:16:01.362Z","etag":null,"topics":["framework","golang","stream"],"latest_commit_sha":null,"homepage":"https://github.com/aceld/kis-flow/wiki","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aceld.png","metadata":{"files":{"readme":"README-CN.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["aceld"],"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"lfx_crowdfunding":null,"polar":null,"custom":null}},"created_at":"2023-12-12T14:14:27.000Z","updated_at":"2024-11-22T16:43:44.000Z","dependencies_parsed_at":"2024-01-09T10:51:06.153Z","dependency_job_id":"47773516-1639-46b9-bbdd-a1009050dc10","html_url":"https://github.com/aceld/kis-flow","commit_stats":null,"previous_names":["aceld/kis-flow"],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aceld%2Fkis-flow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aceld%2Fkis-flow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aceld%2Fkis-flow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aceld%2Fkis-flow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aceld","download_url":"https://codeload.github.com/aceld/kis-flow/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247294541,"owners_count":20915340,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["framework","golang","stream"],"created_at":"2024-11-18T16:36:10.389Z","updated_at":"2025-04-05T06:10:20.968Z","avatar_url":"https://github.com/aceld.png","language":"Go","readme":"# \u003cimg width=\"250px\" src=\"https://github.com/aceld/kis-flow/assets/7778936/8729d750-897c-4ba3-98b4-c346188d034e\" /\u003e\n[English](README.md) | 简体中文\n\n[![License](https://img.shields.io/badge/License-MIT-black.svg)](LICENSE)\n[![Discord](https://img.shields.io/badge/KisFlow-Discord-blue.svg)](https://discord.gg/xQ8Xxfyfcz)\n[![KisFlow-tutorial](https://img.shields.io/badge/KisFlowTutorial-YuQue-red.svg)](https://www.yuque.com/aceld/kis-flow) \n[![KisFlow-Doc](https://img.shields.io/badge/KisFlow-Doc-green.svg)](https://www.yuque.com/aceld/kis-flow-doc)\n\n\n#### KisFlow(Keep It Simple Flowing)\n\n基于Golang的流式计算框架. 为保持简单的流动，强调在进行各种活动或工作时保持简洁、清晰、流畅的过程。\n\n\n\n## KisFlow源代码\n\nGithub\nGit: https://github.com/aceld/kis-flow\n\nGitCode\nGit: https://gitcode.com/aceld/kis-flow\n\nGitee\nGit: https://gitee.com/Aceld/kis-flow\n\n## 《KisFlow开发者文档》\n\n[ \u003c KisFlow Wiki : English \u003e ](https://github.com/aceld/kis-flow/wiki)\n\n[ \u003c KisFlow 文档 : 简体中文\u003e ](https://www.yuque.com/aceld/kis-flow-doc)\n\n\n## 在线开发教程\n\n\n| platform | Entry                                                                                                                                              | \n| ---- |----------------------------------------------------------------------------------------------------------------------------------------------------| \n| \u003cimg src=\"https://user-images.githubusercontent.com/7778936/236784004-b6d99e26-b1ab-4bc3-988e-7a46108b85fe.png\" width = \"100\" height = \"100\" alt=\"\" align=center /\u003e| [Practical Tutorial for a Streaming Computation Framework Based on Golang](https://dev.to/aceld/part-1-golang-framework-hands-on-kisflow-streaming-computing-framework-overview-8fh) | \n|\u003cimg src=\"https://user-images.githubusercontent.com/7778936/236784168-6528a9b8-d37b-4b02-a37c-b9988d7508d8.jpeg\" width = \"100\" height = \"100\" alt=\"\" align=center /\u003e| [《基于Golang的流式计算框架实战教程》](https://www.yuque.com/aceld/hsa94o)                                                                                              |\n\n\n## KisFlow系统定位\n\nKisFlow为业务上游计算层，上层接数仓/其他业务方ODS层、下游接本业务存储数据中心。\u003cbr /\u003e\n\n\u003cimg width=\"700px\" src=\"https://github.com/aceld/kis-flow/assets/7778936/b9e1957a-2d11-45d9-84c1-e92c9ac833cc\" /\u003e\n\n\n## KisFlow整体架构图\n\n| 层级    | 层级说明                                                                               | 包括子模块                                                                                                                                                                                                                                                                                                           |\n|-------|------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| 流式计算层 | 为KisFlow上游计算层，直接对接业务存储及数仓ODS层，如上游可以为Mysql Binlog、日志、接口数据等，为被动消费模式，提供KisFlow实时计算能力。 | **KisFlow**：分布式批量消费者,一个KisFlow是由多个KisFunction组合。\u003cbr /\u003e\u003cbr /\u003e**KisConnectors**：计算数据流流中间状态持久存储及连接器。\u003cbr /\u003e\u003cbr /\u003e**KisFunctions**：支持算子表达式拼接，Connectors集成、策略配置、Stateful Function模式、Slink流式拼接等。\u003cbr /\u003e\u003cbr /\u003e**KisConfig：** KisFunction的绑定的流处理策略，可以绑定ReSource让Function具有固定的独立流处理能力。\u003cbr /\u003e\u003cbr /\u003e**KisSource：** 对接ODS的数据源 |\n| 任务调度层 | 定时任务调度及执行器业务逻辑，包括任务调度平台、执行器管理、调度日志及用户管理等。提供KisFlow的定时任务、统计、聚合运算等调度计算能力。            | **任务调度平台可视化**：包括任务的运行报表、调度报表、成功比例、任务管理、配置管理、GLUE IDE等可视化管理平台。\u003cbr /\u003e\u003cbr /\u003e执行器管理**KisJobs**：Golang SDK及计算自定义业务逻辑、执行器的自动注册、任务触发、终止及摘除等。\u003cbr /\u003e\u003cbr /\u003e**执行器场景KisScenes：** 根据业务划分的逻辑任务集合。\u003cbr /\u003e\u003cbr /\u003e**调度日志及用户管理**：任务调度日志收集、调度详细、调度流程痕迹等。                                                                              |\n\n![KisFlow架构图drawio](https://github.com/aceld/kis-flow/assets/7778936/3b829bdb-600d-4ab9-9e62-e14f90737cc3)\n\n![KisFlow架构设计-KisFlow整体结构 drawio](https://github.com/aceld/kis-flow/assets/7778936/efc1b29d-9dd4-4945-a35a-fb9a618002d7)\n\nKisFlow是一种流式概念形态，具体表现的特征如下：\u003cbr /\u003e\n\n1、一个KisFlow可以由任意KisFunction组成，且KisFlow可以动态的调整长度。\u003cbr /\u003e\n\n2、一个KisFunction可以随时动态的加入到某个KisFlow中，且KisFlow和KisFlow之间的关系可以通过KisFunction的Load和Save节点的加入，进行动态的并流和分流动作。\u003cbr /\u003e\n\n3、KisFlow在编程行为上，从面向流进行数据业务编程，变成了面向KisFunction的函数单计算逻辑的开发，接近FaaS(Function as a\nservice)体系。\n\n## Example\n\n下面是简单的应用场景案例，具体应用单元用例请 参考\n\nhttps://github.com/aceld/kis-flow-usage\n\n#### 《KisFlow开发者文档》\n\nhttps://www.yuque.com/aceld/kis-flow-doc\n\n#### 安装KisFlow\n\n```bash\n$go get github.com/aceld/kis-flow\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e1. Quick Start（快速开始）\u003c/summary\u003e\n\n### 案例源代码\n\nhttps://github.com/aceld/kis-flow-usage/tree/main/1-quick_start\n\n### 项目目录\n\n```bash\n├── faas_stu_score_avg.go\n├── faas_stu_score_avg_print.go\n└── main.go\n```\n\n### Flow\n\n\u003cimg width=\"770\" alt=\"image\" src=\"https://github.com/aceld/kis-flow/assets/7778936/3747ed10-aba1-417e-a3c1-c6205a02444b\"\u003e\n\n### Main\n\n\u003e main.go\n\n```go\npackage main\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"github.com/aceld/kis-flow/common\"\n\t\"github.com/aceld/kis-flow/config\"\n\t\"github.com/aceld/kis-flow/flow\"\n\t\"github.com/aceld/kis-flow/kis\"\n)\n\nfunc main() {\n\tctx := context.Background()\n\n\t// Create a new flow configuration\n\tmyFlowConfig1 := config.NewFlowConfig(\"CalStuAvgScore\", common.FlowEnable)\n\n\t// Create new function configuration\n\tavgStuScoreConfig := config.NewFuncConfig(\"AvgStuScore\", common.C, nil, nil)\n\tprintStuScoreConfig := config.NewFuncConfig(\"PrintStuAvgScore\", common.E, nil, nil)\n\n\t// Create a new flow\n\tflow1 := flow.NewKisFlow(myFlowConfig1)\n\n\t// Link functions to the flow\n\t_ = flow1.Link(avgStuScoreConfig, nil)\n\t_ = flow1.Link(printStuScoreConfig, nil)\n\n\t// Submit a string\n\t_ = flow1.CommitRow(`{\"stu_id\":101, \"score_1\":100, \"score_2\":90, \"score_3\":80}`)\n\t// Submit a string\n\t_ = flow1.CommitRow(`{\"stu_id\":102, \"score_1\":100, \"score_2\":70, \"score_3\":60}`)\n\n\t// Run the flow\n\tif err := flow1.Run(ctx); err != nil {\n\t\tfmt.Println(\"err: \", err)\n\t}\n\n\treturn\n}\n\nfunc init() {\n\t// Register functions\n\tkis.Pool().FaaS(\"AvgStuScore\", AvgStuScore)\n\tkis.Pool().FaaS(\"PrintStuAvgScore\", PrintStuAvgScore)\n}\n```\n\n### Function1\n\n\u003e faas_stu_score_avg.go\n\n```go\npackage main\n\nimport (\n\t\"context\"\n\t\"github.com/aceld/kis-flow/kis\"\n\t\"github.com/aceld/kis-flow/serialize\"\n)\n\ntype AvgStuScoreIn struct {\n\tserialize.DefaultSerialize\n\tStuId  int `json:\"stu_id\"`\n\tScore1 int `json:\"score_1\"`\n\tScore2 int `json:\"score_2\"`\n\tScore3 int `json:\"score_3\"`\n}\n\ntype AvgStuScoreOut struct {\n\tserialize.DefaultSerialize\n\tStuId    int     `json:\"stu_id\"`\n\tAvgScore float64 `json:\"avg_score\"`\n}\n\n// AvgStuScore(FaaS) 计算学生平均分\nfunc AvgStuScore(ctx context.Context, flow kis.Flow, rows []*AvgStuScoreIn) error {\n\tfor _, row := range rows {\n\n\t\tout := AvgStuScoreOut{\n\t\t\tStuId:    row.StuId,\n\t\t\tAvgScore: float64(row.Score1+row.Score2+row.Score3) / 3,\n\t\t}\n\n\t\t// 提交结果数据\n\t\t_ = flow.CommitRow(out)\n\t}\n\n\treturn nil\n}\n```\n\n### Function2\n\n\u003e faas_stu_score_avg_print.go\n\n```go\npackage main\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"github.com/aceld/kis-flow/kis\"\n\t\"github.com/aceld/kis-flow/serialize\"\n)\n\ntype PrintStuAvgScoreIn struct {\n\tserialize.DefaultSerialize\n\tStuId    int     `json:\"stu_id\"`\n\tAvgScore float64 `json:\"avg_score\"`\n}\n\ntype PrintStuAvgScoreOut struct {\n\tserialize.DefaultSerialize\n}\n\nfunc PrintStuAvgScore(ctx context.Context, flow kis.Flow, rows []*PrintStuAvgScoreIn) error {\n\n\tfor _, row := range rows {\n\t\tfmt.Printf(\"stuid: [%+v], avg score: [%+v]\\n\", row.StuId, row.AvgScore)\n\t}\n\n\treturn nil\n}\n```\n\n### OutPut\n\n```bash\nAdd KisPool FuncName=AvgStuScore\nAdd KisPool FuncName=PrintStuAvgScore\nfuncName NewConfig source is nil, funcName = AvgStuScore, use default unNamed Source.\nfuncName NewConfig source is nil, funcName = PrintStuAvgScore, use default unNamed Source.\nstuid: [101], avg score: [90]\nstuid: [102], avg score: [76.66666666666667]\n```\n\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003e2. Quick Start With Config（快速开始）\u003c/summary\u003e\n\n### 案例源代码\n\nhttps://github.com/aceld/kis-flow-usage/tree/main/2-quick_start_with_config\n\n项目目录\n\n```bash\n├── Makefile\n├── conf\n│   ├── flow-CalStuAvgScore.yml\n│   ├── func-AvgStuScore.yml\n│   └── func-PrintStuAvgScore.yml\n├── faas_stu_score_avg.go\n├── faas_stu_score_avg_print.go\n└── main.go\n```\n\n### Flow\n\n\u003cimg width=\"770\" alt=\"image\" src=\"https://github.com/aceld/kis-flow/assets/7778936/3747ed10-aba1-417e-a3c1-c6205a02444b\"\u003e\n\n### Config\n\n#### (1) Flow Config\n\n\u003e conf/flow-CalStuAvgScore.yml\n\n```yaml\nkistype: flow\nstatus: 1\nflow_name: CalStuAvgScore\nflows:\n  - fname: AvgStuScore\n  - fname: PrintStuAvgScore\n```\n\n#### (2) Function1 Config\n\n\u003e conf/func-AvgStuScore.yml\n\n```yaml\nkistype: func\nfname: AvgStuScore\nfmode: Calculate\nsource:\n  name: 学生学分\n  must:\n    - stu_id\n```\n\n#### (3) Function2(Slink) Config\n\n\u003e conf/func-PrintStuAvgScore.yml\n\n```yaml\nkistype: func\nfname: PrintStuAvgScore\nfmode: Expand\nsource:\n  name: 学生学分\n  must:\n    - stu_id\n```\n\n### Main\n\n\u003e main.go\n\n```go\npackage main\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"github.com/aceld/kis-flow/file\"\n\t\"github.com/aceld/kis-flow/kis\"\n)\n\nfunc main() {\n\tctx := context.Background()\n\n\t// Load Configuration from file\n\tif err := file.ConfigImportYaml(\"conf/\"); err != nil {\n\t\tpanic(err)\n\t}\n\n\t// Get the flow\n\tflow1 := kis.Pool().GetFlow(\"CalStuAvgScore\")\n\tif flow1 == nil {\n\t\tpanic(\"flow1 is nil\")\n\t}\n\n\t// Submit a string\n\t_ = flow1.CommitRow(`{\"stu_id\":101, \"score_1\":100, \"score_2\":90, \"score_3\":80}`)\n\t// Submit a string\n\t_ = flow1.CommitRow(`{\"stu_id\":102, \"score_1\":100, \"score_2\":70, \"score_3\":60}`)\n\n\t// Run the flow\n\tif err := flow1.Run(ctx); err != nil {\n\t\tfmt.Println(\"err: \", err)\n\t}\n\n\treturn\n}\n\nfunc init() {\n\t// Register functions\n\tkis.Pool().FaaS(\"AvgStuScore\", AvgStuScore)\n\tkis.Pool().FaaS(\"PrintStuAvgScore\", PrintStuAvgScore)\n}\n```\n\n### Function1\n\n\u003e faas_stu_score_avg.go\n\n```go\npackage main\n\nimport (\n\t\"context\"\n\t\"github.com/aceld/kis-flow/kis\"\n\t\"github.com/aceld/kis-flow/serialize\"\n)\n\ntype AvgStuScoreIn struct {\n\tserialize.DefaultSerialize\n\tStuId  int `json:\"stu_id\"`\n\tScore1 int `json:\"score_1\"`\n\tScore2 int `json:\"score_2\"`\n\tScore3 int `json:\"score_3\"`\n}\n\ntype AvgStuScoreOut struct {\n\tserialize.DefaultSerialize\n\tStuId    int     `json:\"stu_id\"`\n\tAvgScore float64 `json:\"avg_score\"`\n}\n\n// AvgStuScore(FaaS) 计算学生平均分\nfunc AvgStuScore(ctx context.Context, flow kis.Flow, rows []*AvgStuScoreIn) error {\n\tfor _, row := range rows {\n\n\t\tout := AvgStuScoreOut{\n\t\t\tStuId:    row.StuId,\n\t\t\tAvgScore: float64(row.Score1+row.Score2+row.Score3) / 3,\n\t\t}\n\n\t\t// 提交结果数据\n\t\t_ = flow.CommitRow(out)\n\t}\n\n\treturn nil\n}\n```\n\n### Function2\n\n\u003e faas_stu_score_avg_print.go\n\n```go\npackage main\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"github.com/aceld/kis-flow/kis\"\n\t\"github.com/aceld/kis-flow/serialize\"\n)\n\ntype PrintStuAvgScoreIn struct {\n\tserialize.DefaultSerialize\n\tStuId    int     `json:\"stu_id\"`\n\tAvgScore float64 `json:\"avg_score\"`\n}\n\ntype PrintStuAvgScoreOut struct {\n\tserialize.DefaultSerialize\n}\n\nfunc PrintStuAvgScore(ctx context.Context, flow kis.Flow, rows []*PrintStuAvgScoreIn) error {\n\n\tfor _, row := range rows {\n\t\tfmt.Printf(\"stuid: [%+v], avg score: [%+v]\\n\", row.StuId, row.AvgScore)\n\t}\n\n\treturn nil\n}\n```\n\n### OutPut\n\n```bash\nAdd KisPool FuncName=AvgStuScore\nAdd KisPool FuncName=PrintStuAvgScore\nAdd FlowRouter FlowName=CalStuAvgScore\nstuid: [101], avg score: [90]\nstuid: [102], avg score: [76.66666666666667]\n```\n\n\u003c/details\u003e\n\n\n---\n\n### 开发者\n\n* 刘丹冰([@aceld](https://github.com/aceld))\n* 胡辰豪([@ChenHaoHu](https://github.com/ChenHaoHu))\n\nThanks to all the developers who contributed to KisFlow!\n\n\u003ca href=\"https://github.com/aceld/kis-flow/graphs/contributors\"\u003e\n  \u003cimg src=\"https://contrib.rocks/image?repo=aceld/kis-flow\" /\u003e\n\u003c/a\u003e    \n\n### 加入KisFlow 社区\n\n| platform                                                                                                                                                             | Entry                                                                                                                                                                                                    | \n|----------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| \n| \u003cimg src=\"https://user-images.githubusercontent.com/7778936/236775008-6bd488e3-249a-4d43-8885-7e3889e11e2d.png\" width = \"100\" height = \"100\" alt=\"\" align=center /\u003e  | https://discord.gg/xQ8Xxfyfcz                                                                                                                                                                            | \n| \u003cimg src=\"https://user-images.githubusercontent.com/7778936/236775137-5381f8a6-f534-49c4-8628-e52bf245c3bc.jpeg\" width = \"100\" height = \"100\" alt=\"\" align=center /\u003e | 加微信: `ace_ld`  或扫二维码，备注`flow`即可。\u003c/br\u003e\u003cimg src=\"https://user-images.githubusercontent.com/7778936/236781258-2f0371bd-5797-49e8-a74c-680e9f15843d.png\" width = \"150\" height = \"150\" alt=\"\" align=center /\u003e |\n| \u003cimg src=\"https://user-images.githubusercontent.com/7778936/236778547-9cdadfb6-0f62-48ac-851a-b940389038d0.jpeg\" width = \"100\" height = \"100\" alt=\"\" align=center /\u003e | \u003cimg src=\"https://s1.ax1x.com/2020/07/07/UFyUdx.th.jpg\" height = \"150\"  alt=\"\" align=center /\u003e **WeChat Public Account**                                                                                 |\n| \u003cimg src=\"https://user-images.githubusercontent.com/7778936/236779000-70f16c8f-0eec-4b5f-9faa-e1d5229a43e0.png\" width = \"100\" height = \"100\" alt=\"\" align=center /\u003e  | \u003cimg src=\"https://github.com/aceld/zinx/assets/7778936/461b409f-6337-48a8-826b-a7a746aaee31\" width = \"150\" height = \"150\" alt=\"\" align=center /\u003e **QQ Group**                                                                                 |\n\n","funding_links":["https://github.com/sponsors/aceld"],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faceld%2Fkis-flow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faceld%2Fkis-flow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faceld%2Fkis-flow/lists"}