{"id":21421306,"url":"https://github.com/owenliang/bigpipe","last_synced_at":"2025-07-14T07:32:39.914Z","repository":{"id":81719369,"uuid":"96972091","full_name":"owenliang/bigpipe","owner":"owenliang","description":"以Kafka为存储介质，提供异步Http RPC的中间件","archived":false,"fork":false,"pushed_at":"2018-05-08T05:55:26.000Z","size":2666,"stargazers_count":89,"open_issues_count":0,"forks_count":25,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-04-08T04:04:07.711Z","etag":null,"topics":["agent","golang","http","kafka"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/owenliang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-07-12T06:17:22.000Z","updated_at":"2025-01-03T21:46:17.000Z","dependencies_parsed_at":null,"dependency_job_id":"1f3be7e4-c97d-4a8c-941d-750b6e6cbf3c","html_url":"https://github.com/owenliang/bigpipe","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/owenliang/bigpipe","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/owenliang%2Fbigpipe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/owenliang%2Fbigpipe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/owenliang%2Fbigpipe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/owenliang%2Fbigpipe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/owenliang","download_url":"https://codeload.github.com/owenliang/bigpipe/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/owenliang%2Fbigpipe/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265255275,"owners_count":23735223,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent","golang","http","kafka"],"created_at":"2024-11-22T20:33:19.403Z","updated_at":"2025-07-14T07:32:39.905Z","avatar_url":"https://github.com/owenliang.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# bigpipe\n一个基于Kafka的中间件，旨在简化服务间异步Http调用的复杂度\n\n[![Build Status](https://travis-ci.org/owenliang/bigpipe.svg?branch=master)](https://travis-ci.org/owenliang/bigpipe)\n\n# 功能更新\n\n* 2018-04-04：增加了consumer粒度的熔断器，防止下游不可用导致大量流量持续损失，这是一个可选功能\n\n# 环境要求\n* Kafka \u003e= 0.9\n* Go \u003e= 1.8\n\n# 特性\n* 横向扩展：完全支持rebalanced-consumer-group，多进程部署即可实现partition自动负载均衡\n* 配置热加载：在线服务无需重启即可加载配置，流量0损失\n* 优雅退出：处理完剩余任务后退出，流量0损失\n* 超强性能：无锁，协程并发，单进程即可充分利用多核，满足一般流量需求\n* 容错机制：对下游实施限速、限并发、熔断三种保护机制\n\n# 安装依赖\n* [librdkafka（必须为0.9.5版本，编译时指定--prefix=/usr）](https://github.com/edenhill/librdkafka/releases/tag/v0.9.5)\n\n# GO包依赖\n* [confluentinc-kafka-go（无需自行下载，当前依赖为0.9.4版本）](https://github.com/confluentinc/confluent-kafka-go)\n\n# 编译方法\n* 设置GOPATH为项目根目录\n* 进入GOPATH目录，安装glide包管理：mkdir -p bin \u0026\u0026 curl https://glide.sh/get | sh\n* 执行sh build.sh\n\n# 潜在的安装问题\n* 编译bigpipe时候可能遇到如下报错，需要export PKG_CONFIG_PATH=/usr/lib/pkgconfig\n    pkg-config --cflags rdkafka\n    Package rdkafka was not found in the pkg-config search path.\n    Perhaps you should add the directory containing `rdkafka.pc'\n    to the PKG_CONFIG_PATH environment variable\n    No package 'rdkafka' found\n* 运行bigpipe时候可能遇到找不到符号librdkafka符号的问题，那是因为编译librdkafka没有指定--prefix=/usr，可以重新编译安装或者export LD_LIBRARY_PATH=/usr/local/lib\n\n# 使用方法\n* 运行：./bigpipe -config /path/to/bigpipe.json\n* 退出：killall bigpipe\n* 热加载：killall -USR1 bigpipe\n\n# 调用示例\n异步调用\n    \n    curl localhost:10086/rpc/call -d '{\"acl\": {\"name\":\"system-1\",\"secret\":\"i am good\"},\"url\":\"http://localhost:10086/rpc/server/mock\",\"data\":\"hello123123123\",\"partition_key\": \"暂时用不到\"}'\n    \n    {\n        \"data\": \"\",\n        \"errno\": 0,\n        \"msg\": \"发送成功\"\n    }\n\n接收端(PHP为例)\n\n    $data = file_get_contents(\"php://input\"); // $data的值为hello123123123\n\n统计信息\n    \n    curl localhost:10086/stats\n    \n    {\n        \"data\": {\n            \"clientStats\": {\n                \"test\": {\n                    \"rpcFail\": 0,\n                    \"rpcRetries\": 0,\n                    \"rpcSuccess\": 22,\n                    \"rpcTotal\": 22\n                }\n            },\n            \"consumerStats\": [\n                {\n                    \"groupId\": \"G1\",\n                    \"handleMessage\": 11,\n                    \"invalidMessage\": 0,\n                    \"topic\": \"test\"\n                },\n                {\n                    \"groupId\": \"G2\",\n                    \"handleMessage\": 11,\n                    \"invalidMessage\": 0,\n                    \"topic\": \"test\"\n                }\n            ],\n            \"producerStats\": {\n                \"test\": {\n                    \"deliveryFail\": 0,\n                    \"deliverySuccess\": 11\n                }\n            },\n            \"serverStats\": {\n                \"acceptedCall\": 11,\n                \"overloadCall\": 0,\n                \"receivedCall\": 11\n            }\n        },\n        \"errno\": 0,\n        \"msg\": \"success\"\n    }\n\n# 配置说明\n    {\n      \"log.level\": 5,\n      \"log.directory\": \"./logs\",\n    \n      \"kafka.bootstrap.servers\": \"localhost:9092\",\n      \"kafka.topics\": [\n        {\"name\": \"test\", \"partitions\": 3}\n      ],\n    \n      \"kafka.producer.retries\": 2,\n      \"kafka.producer.acl\": [\n        {\"name\": \"system-1\", \"secret\": \"you are good\", \"topic\": \"test\"},\n        {\"name\": \"system-2\", \"secret\": \"you are bad\", \"topic\": \"test\"}\n      ],\n    \n      \"kafka.consumer.list\": [\n        {\"topic\": \"test\", \"groupId\": \"G1\", \"rateLimit\": 100, \"timeout\": 3000, \"retries\": 2, \"concurrency\": 5, \"circuitBreaker\": {\"breakPeriod\": 10, \"recoverPeriod\": 30, \"winSize\": 60, \"minStats\": 100, \"healthRate\": 0.85}},\n        {\"topic\": \"test\", \"groupId\": \"G2\", \"rateLimit\": 100, \"timeout\": 3000, \"retries\": 2, \"concurrency\": 5}\n      ],\n    \n      \"http.server.port\": 10086,\n      \"http.server.handler.channel.size\": 500000,\n      \"http.server.read.timeout\": 5000,\n      \"http.server.write.timeout\": 5000\n    }\n\n下面加粗的配置项，支持热加载。\n* **log.level**：日志级别，FATAL=1|ERROR=2|WARNING=3|INFO=4|DEBUG=5\n* **log.directory**：日志存储路径，目录必须存在\n* **kafka.bootstrap.servers**： kafka服务器列表，逗号分隔\n* **kafka.topics**：允许读写的topic列表\n* **kafka.producer.retries**：投递消息到kafka的重试次数\n* **kafka.producer.acl**：投递消息的权限账号/密码，调用者必须符合其中的某条规则\n* **kafka.producer.acl.name**：权限账号\n* **kafka.producer.acl.secret**：权限密码\n* **kafka.producer.acl.name.topic**：授权的topic名称\n* **kafka.consumer.list**：kafka消费组列表\n* **kafka.consumer.list.topic**：消费的topic\n* **kafka.consumer.list.groupId**：消费组id，相同topic下相同groupId将彼此进行rebalanced负载均衡\n* **kafka.consumer.list.rateLimit**：流速限制，即每秒的最大调用次数\n* **kafka.consumer.list.timeout**：超时时间，即每个调用最大等待应答的时间（毫秒）\n* **kafka.consumer.list.retries**：重试次数，即每个调用连续失败的最大次数\n* **kafka.consumer.list.concurrency**：并发限制，即同一时刻最多并发的请求个数\n* **kafka.consumer.list.circuitBreaker**: 熔断器，可选\n* **kafka.consumer.list.circuitBreaker.breakPeriod**: 熔断冻结时间（此期间不透过任何请求），单位秒\n* **kafka.consumer.list.circuitBreaker.recoverPeriod**: 熔断慢恢复时间（此期间逐渐增大放量请求），单位秒\n* **kafka.consumer.list.circuitBreaker.winSize**: 统计滑动窗口（检查这段窗口时间内的成功率），单位秒\n* **kafka.consumer.list.circuitBreaker.minStats**: 滑动窗口内样本少于此数值则直接认为服务健康\n* **kafka.consumer.list.circuitBreaker.healthRate**: 统计的健康阀值，区间[0,1]，成功率低于此值认为服务不健康\n\n* http.server.port：服务端监听地址，http协议\n* http.server.handler.channel.size：收到的异步调用缓冲队列大小，堆积超过队列大小将返回请求失败\n* http.server.read.timeout：服务端读取请求的超时（毫秒）\n* http.server.write.timeout：服务端发送应答的超时（毫秒）\n\n\n# 工作原理\n* server模块：接收异步Http调用\n* handler模块：处理Http请求并传递给producer\n* producer模块：将异步调用序列化，投递到kafka\n* log模块：线程安全的异步日志\n* config模块：基于json的配置\n* client模块：异步http客户端，支持超时、重试、并发控制、流速控制\n* consumer模块：读取kafka中的消息，发送给下游\n* stats模块：基于原子变量的程序统计\n\n# 运维建议\n* 关于扩展性：kafka topic预分配足够的partition，保证bigpipe可横向扩展\n* 关于可用性：bigpipe至少部署2个等价节点，利用lvs/haproxy负载均衡，或者客户端负载均衡\n\n# 参考文档\n[bigpipe设计PPT](https://gitlab-team.smzdm.com/smzdm/bigpipe/raw/master/doc/bigpipe%E5%88%86%E4%BA%AB.pptx)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fowenliang%2Fbigpipe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fowenliang%2Fbigpipe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fowenliang%2Fbigpipe/lists"}