{"id":19283834,"url":"https://github.com/tencentblueking/bamboo-engine","last_synced_at":"2025-04-04T19:07:11.768Z","repository":{"id":36964528,"uuid":"403897409","full_name":"TencentBlueKing/bamboo-engine","owner":"TencentBlueKing","description":"A event-driven workflow engine for Python","archived":false,"fork":false,"pushed_at":"2025-02-08T06:14:18.000Z","size":5241,"stargazers_count":156,"open_issues_count":28,"forks_count":57,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-03-28T18:08:11.018Z","etag":null,"topics":["blueking","bpmnengine","flowengine","flowqueue","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TencentBlueKing.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-09-07T08:16:47.000Z","updated_at":"2025-03-26T14:30:11.000Z","dependencies_parsed_at":"2023-10-20T07:00:30.454Z","dependency_job_id":"bb8c627e-5bef-474d-8b2c-302468e125e2","html_url":"https://github.com/TencentBlueKing/bamboo-engine","commit_stats":null,"previous_names":[],"tags_count":103,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TencentBlueKing%2Fbamboo-engine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TencentBlueKing%2Fbamboo-engine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TencentBlueKing%2Fbamboo-engine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TencentBlueKing%2Fbamboo-engine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TencentBlueKing","download_url":"https://codeload.github.com/TencentBlueKing/bamboo-engine/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247234921,"owners_count":20905854,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["blueking","bpmnengine","flowengine","flowqueue","python"],"created_at":"2024-11-09T21:35:26.818Z","updated_at":"2025-04-04T19:07:11.744Z","avatar_url":"https://github.com/TencentBlueKing.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# bamboo-engine: A event-driven workflow engine for Python\n\n[![license](https://img.shields.io/badge/license-MIT-brightgreen.svg?style=flat)](https://github.com/TencentBlueKing/bamboo-engine/blob/master/LICENSE.txt)\n[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](https://github.com/TencentBlueKing/bamboo-engine/pulls)\n[![Python 3.6](https://img.shields.io/badge/python-v3.6-blue)](https://github.com/TencentBlueKing/bamboo-engine/)\n[![Python 3.7](https://img.shields.io/badge/python-v3.7-blue)](https://github.com/TencentBlueKing/bamboo-engine/)\n[![codecov](https://codecov.io/gh/TencentBlueKing/bamboo-engine/branch/master/graph/badge.svg?token=ROH54UE7B8)](https://codecov.io/gh/TencentBlueKing/bamboo-engine)\n[![BK Pipelines Status](https://api.bkdevops.qq.com/process/api/external/pipelines/projects/blueapps/p-d620b6131c994e76ba292ae359c162f1/badge?X-DEVOPS-PROJECT-ID=blueapps)](https://api.bkdevops.qq.com/process/api/external/pipelines/projects/blueapps/p-d620b6131c994e76ba292ae359c162f1/badge?X-DEVOPS-PROJECT-ID=blueapps)\n\n[(English Documents Available)](README_en.md)\n\nbamboo-engine 是一个通用的流程引擎，他可以解析，执行，调度由用户创建的流程任务，并提供了如暂停，撤销，跳过，强制失败，重试和重入等等灵活的控制能力和并行、子流程等进阶特性，并可通过水平扩展来进一步提升任务的并发处理能力。\n\n\u003c!-- TOC --\u003e\n- [整体设计](#整体设计)\n- [Quick start](#quick-start)\n  - [1. 安装依赖](#1-安装依赖)\n  - [2. 项目初始化](#2-项目初始化)\n  - [3. 执行流程](#3-执行流程)\n- [benchmark](#benchmark)\n\n\u003c!-- /TOC --\u003e\n- 使用文档\n  - [核心概念](./docs/user_guide/basic_concept.md)\n  - [流程编排](./docs/user_guide/flow_orchestration.md)\n  - [流程构造器](./docs/user_guide/flow_builder.md)\n  - [流程构造生成树字段摘要](./docs/user_guide/builded_pipeline_tree_schema.md)\n  - [SPLICE 变量](./docs/user_guide/splice_var.md)\n  - [Engine API](./docs/user_guide/engine_api.md)\n  - [Metrics](./docs/user_guide/monitor.md)\n  - 升级指引\n    - [如何升级到 2.x 版本](./docs/upgrade/bamboo_engine_2.md)\n- 运行时文档\n  - bamboo-pipeline\n    - [自定义组件](./docs/user_guide/custom_component.md)\n    - [运行自定义组件](./docs/user_guide/run_your_component.md)\n    - [自定义结束事件](./docs/user_guide/custom_end_event.md)\n    - [组件单元测试](./docs/user_guide/component_unit_test.md)\n    - [Worker 配置](./docs/user_guide/workers.md)\n    - [引擎管理端配置](./docs/user_guide/engine_admin_config.md)\n    - [运行时信号](./docs/user_guide/signals.md)\n    - [运行时钩子](./docs/user_guide/engine_hook.md)\n    - [流程回退](./docs/user_guide/rollback.md)\n    - [重置某个异常节点的输出](./docs/user_guide/update_node_output.md)\n    - [设置](./docs/user_guide/settings.md)\n    - [增强包 - 节点超时功能](./docs/user_guide/node_timeout_introduction.md)\n    - [流程从指定位置开始](./docs/user_guide/start_the_pipeline_at_the_specified_location.md)\n\n\n## 整体设计\n\n![](./docs/assets/img/engine_arch.png)\n\nbamboo-engine 是流程引擎核心模块、调度逻辑的定义和实现，并没有内置特定的引擎运行时，需要搭配实现了 `bamboo_engine.eri.interfaces.EngineRuntimeInterface` 接口的引擎运行时使用，目前提供了以下运行时可供使用：\n\n- 基于 Django，Celery 的运行时：[bamboo-pipeline](./runtime/bamboo-pipeline)\n\n\nengine 模块结构：\n\n![](./docs/assets/img/code_arch.png)\n\n## Quick start\n\n### 1. 安装依赖\n\n```\n$ pip install bamboo-pipeline\n```\n### 2. 项目初始化\n\n由于 `bamboo-pipeline` 运行时基于 Django 实现，所以需要新建一个 Django 项目：\n\n```\n$ django-admin startproject bamboo_engine_playground\n$ cd bamboo_engine_playground\n```\n\n在 `bamboo_engine_playground.settings.py` 下添加如下配置：\n\n```python\nfrom pipeline.eri.celery.queues import *\nfrom celery import Celery\n\napp = Celery(\"proj\")\n\napp.config_from_object(\"django.conf:settings\")\n\nINSTALLED_APPS = [\n    ...\n    \"pipeline.component_framework\",\n    \"pipeline.eri\",\n    ...\n]\n```\n\n在 `bamboo_engine_playground` 目录下初始化数据库：\n\n```\n$ python manage.py migrate\n```\n\n### 3. 执行流程\n\n首先在 `bamboo_engine_playground` 目录下启动 celery worker：\n\n```\n$ DJANGO_SETTINGS_MODULE=bamboo_engine_playground.settings celery worker -A bamboo_engine_playground.settings -Q er_execute,er_schedule -l info\n```\n\n创建并执行一个简单的流程：\n\n![](./docs/assets/img/simple_example.png)\n\n```python\nimport time\n\nfrom bamboo_engine import api\nfrom bamboo_engine.builder import *\nfrom pipeline.eri.runtime import BambooDjangoRuntime\n\n# 使用 builder 构造出流程描述结构\nstart = EmptyStartEvent()\n# 这里先使用 bamboo-pipeline 自带的示例组件，我们会在后续的章节中学习如何自定义组件\nact = ServiceActivity(component_code=\"example_component\")\nend = EmptyEndEvent()\n\nstart.extend(act).extend(end)\n\npipeline = builder.build_tree(start)\n\n# 执行流程对象\nruntime = BambooDjangoRuntime()\n\n# 如果需要忽略流程中的环检查，可以指定为True cycle_tolerate\n# api.run_pipeline(runtime=runtime, pipeline=pipeline, cycle_tolerate=True)\napi.run_pipeline(runtime=runtime, pipeline=pipeline)\n\n# 等待 1s 后获取流程执行结果\ntime.sleep(1)\n\nresult = api.get_pipeline_states(runtime=runtime, root_id=pipeline[\"id\"])\n\nprint(result.data)\n```\n\n随后我们就能够看到流程的状态信息，如下所示，流程中的所有节点已经执行成功：\n\n```python\n{'pc31c89e6b85a4e2c8c5db477978c1a57': {'id': 'pc31c89e6b85a4e2c8c5db477978c1a57',\n  'state': 'FINISHED',\n  'root_id:': 'pc31c89e6b85a4e2c8c5db477978c1a57',\n  'parent_id': 'pc31c89e6b85a4e2c8c5db477978c1a57',\n  'version': 'vaf47e56f2f31401e979c3c47b2a0c285',\n  'loop': 1,\n  'retry': 0,\n  'skip': False,\n  'created_time': datetime.datetime(2021, 3, 10, 3, 45, 54, 688664, tzinfo=\u003cUTC\u003e),\n  'started_time': datetime.datetime(2021, 3, 10, 3, 45, 54, 688423, tzinfo=\u003cUTC\u003e),\n  'archived_time': datetime.datetime(2021, 3, 10, 3, 45, 54, 775165, tzinfo=\u003cUTC\u003e),\n  'children': {'e42035b3f98374062921a191115fc602e': {'id': 'e42035b3f98374062921a191115fc602e',\n    'state': 'FINISHED',\n    'root_id:': 'pc31c89e6b85a4e2c8c5db477978c1a57',\n    'parent_id': 'pc31c89e6b85a4e2c8c5db477978c1a57',\n    'version': 've2d0fa10d7d842a1bcac25984620232a',\n    'loop': 1,\n    'retry': 0,\n    'skip': False,\n    'created_time': datetime.datetime(2021, 3, 10, 3, 45, 54, 744490, tzinfo=\u003cUTC\u003e),\n    'started_time': datetime.datetime(2021, 3, 10, 3, 45, 54, 744308, tzinfo=\u003cUTC\u003e),\n    'archived_time': datetime.datetime(2021, 3, 10, 3, 45, 54, 746690, tzinfo=\u003cUTC\u003e)},\n   'e327f83de42df4ebfab375c271bf63d29': {'id': 'e327f83de42df4ebfab375c271bf63d29',\n    'state': 'FINISHED',\n    'root_id:': 'pc31c89e6b85a4e2c8c5db477978c1a57',\n    'parent_id': 'pc31c89e6b85a4e2c8c5db477978c1a57',\n    'version': 'v893cdc14150d4df5b20f2db32ba142b3',\n    'loop': 1,\n    'retry': 0,\n    'skip': False,\n    'created_time': datetime.datetime(2021, 3, 10, 3, 45, 54, 753321, tzinfo=\u003cUTC\u003e),\n    'started_time': datetime.datetime(2021, 3, 10, 3, 45, 54, 753122, tzinfo=\u003cUTC\u003e),\n    'archived_time': datetime.datetime(2021, 3, 10, 3, 45, 54, 758697, tzinfo=\u003cUTC\u003e)},\n   'e6c7d7a3721ca4b19a5a7f3b34d8387bf': {'id': 'e6c7d7a3721ca4b19a5a7f3b34d8387bf',\n    'state': 'FINISHED',\n    'root_id:': 'pc31c89e6b85a4e2c8c5db477978c1a57',\n    'parent_id': 'pc31c89e6b85a4e2c8c5db477978c1a57',\n    'version': 'v0c661ee6994d4eb4bdbfe5260f9a9f22',\n    'loop': 1,\n    'retry': 0,\n    'skip': False,\n    'created_time': datetime.datetime(2021, 3, 10, 3, 45, 54, 767563, tzinfo=\u003cUTC\u003e),\n    'started_time': datetime.datetime(2021, 3, 10, 3, 45, 54, 767384, tzinfo=\u003cUTC\u003e),\n    'archived_time': datetime.datetime(2021, 3, 10, 3, 45, 54, 773341, tzinfo=\u003cUTC\u003e)}}}}\n```\n\n恭喜你，你已经成功的创建了一个流程并把它运行起来了！\n\n## benchmark\n\n\n测试环境：\n\n- MacBook Pro（16 英寸，2019）\n- 处理器：2.6 GHz 六核Intel Core i7\n- 内存：32 GB 2667 MHz DDR4\n- OS：macOS Big Sur 11.2.1\n- Broker：RabbitMQ 3.8.2\n- MySQL：5.7.22\n- worker 启动命令（单个 worker 进程 -c 参数不变，通过增加进程来提高并发处理能力）\n  - python manage.py celery worker -c 100 -P gevent -l info -Q er_execute -n execute_%(process_num)02d\n  - python manage.py celery worker -c 100 -P gevent -l info -Q er_schedule -n schedule_%(process_num)02d\n\n| 测试场景                          | worker concurrency | 流程执行耗时(s) |\n| --------------------------------- | ------------------ | --------------- |\n| 100个流程(单流程17个节点)并发执行 | 100                | 25.98           |\n| 100个流程(单流程17个节点)并发执行 | 200                | 14.75           |\n| 100个流程(单流程17个节点)并发执行 | 500                | 8.29            |\n| 100个流程(单流程17个节点)并发执行 | 1000               | 6.78            |\n| 1000节点大流程                    | 100                | 19.33           |\n| 1000节点大流程                    | 200                | 12.5            |\n| 1000节点大流程                    | 500                | 11              |\n| 1000节点大流程                    | 1000               | 7.5             |\n\n![](./benchmark/EXECUTION%20水平扩展/Line-20210309.png)\n\n## Roadmap\n\n- [版本日志](release.md)\n\n## Support\n\n- [蓝鲸论坛](https://bk.tencent.com/s-mart/community)\n- [蓝鲸 DevOps 在线视频教程](https://bk.tencent.com/s-mart/video/)\n- 联系我们，技术交流QQ群：\n\n\u003cimg src=\"https://github.com/Tencent/bk-PaaS/raw/master/docs/resource/img/bk_qq_group.png\" width=\"250\" hegiht=\"250\" align=center /\u003e\n\n\n## BlueKing Community\n\n- [BK-CI](https://github.com/Tencent/bk-ci)：蓝鲸持续集成平台是一个开源的持续集成和持续交付系统，可以轻松将你的研发流程呈现到你面前。\n- [BK-BCS](https://github.com/Tencent/bk-bcs)：蓝鲸容器管理平台是以容器技术为基础，为微服务业务提供编排管理的基础服务平台。\n- [BK-PaaS](https://github.com/Tencent/bk-PaaS)：蓝鲸PaaS平台是一个开放式的开发平台，让开发者可以方便快捷地创建、开发、部署和管理SaaS应用。\n- [BK-SOPS](https://github.com/Tencent/bk-sops)：标准运维（SOPS）是通过可视化的图形界面进行任务流程编排和执行的系统，是蓝鲸体系中一款轻量级的调度编排类SaaS产品。\n- [BK-CMDB](https://github.com/Tencent/bk-cmdb)：蓝鲸配置平台是一个面向资产及应用的企业级配置管理平台。\n\n## Contributing\n\n如果你有好的意见或建议，欢迎给我们提 Issues 或 Pull Requests，为蓝鲸开源社区贡献力量。\n\n1. 本项目使用 [Poetry](https://python-poetry.org/) 进行开发、构建及发布，本地开发环境搭建请参考 Poetry 官方文档\n2. PR 需要通过 CI 中的所有代码风格检查，单元测试及集成测试才可被接受合并\n3. 新增加的模块请确保完备的单元测试覆盖\n\n## License\n\n基于 MIT 协议， 详细请参考[LICENSE](LICENSE.txt)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftencentblueking%2Fbamboo-engine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftencentblueking%2Fbamboo-engine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftencentblueking%2Fbamboo-engine/lists"}