{"id":13469981,"url":"https://github.com/blackbeans/kiteq","last_synced_at":"2025-03-26T09:32:00.402Z","repository":{"id":24610122,"uuid":"28018842","full_name":"blackbeans/kiteq","owner":"blackbeans","description":"kiteq is a distributed mq framework ","archived":false,"fork":false,"pushed_at":"2023-06-20T06:54:20.000Z","size":7321,"stargazers_count":785,"open_issues_count":0,"forks_count":178,"subscribers_count":108,"default_branch":"master","last_synced_at":"2024-10-30T00:52:40.558Z","etag":null,"topics":["2pc","golang","mq","notify"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/blackbeans.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2014-12-15T03:29:58.000Z","updated_at":"2024-10-29T16:31:41.000Z","dependencies_parsed_at":"2024-01-14T08:01:54.516Z","dependency_job_id":"9dc54890-c0ba-4b72-a124-eb8affe27104","html_url":"https://github.com/blackbeans/kiteq","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blackbeans%2Fkiteq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blackbeans%2Fkiteq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blackbeans%2Fkiteq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blackbeans%2Fkiteq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/blackbeans","download_url":"https://codeload.github.com/blackbeans/kiteq/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245626053,"owners_count":20646284,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["2pc","golang","mq","notify"],"created_at":"2024-07-31T16:00:20.489Z","updated_at":"2025-03-26T09:32:00.020Z","avatar_url":"https://github.com/blackbeans.png","language":"Go","funding_links":[],"categories":["Go","Misc","Message Queue Middlewares"],"sub_categories":[],"readme":"KiteQ ![image](./doc/logo.jpg)\n=======\n\n基于go+protobuff实现的多种持久化方案的mq框架\n\n#### WIKI(https://kiteq/wiki)\n\n#### 简介\n    * 基于zk/etcd维护发送方、订阅方、broker订阅发送关系、支持水平、垂直方面的扩展\n    * 基于与topic以及第二级messageType订阅消息\n    * 基于mysql、文件存储方式(file、rocksdb)多重持久层消息存储\n    * 保证可靠异步投递\n    * 支持两阶段提交分布式事务\n    * 自定义group内的Topic级别的流控措施，保护订阅方安全\n    * kiteserver的流量保护\n    * 客户端连接上报系统预热时间，根据预热时间逐步放量推送消息\n    * Client Supported：Java、Go、CPP、PHP\n\n#### 工程结构\n    kiteq/\n    ├── README.md\n    ├── conf              配置信息\n    ├── log               log4go的配置\n    ├── build.sh          安装脚本\n    ├── doc               文档\n    ├── handler           KiteQ所需要的处理Handler\n    ├── kiteq.go          KiteQ对外启动入口        \n    └── server             KiteQ的Server端组装需要的组件\n\n##### 概念：\n    \n    * Binding:订阅关系，描述订阅某种消息类型的数据结构\n    * Consumer : 消息的消费方\n    * Producer : 消息的发送方\n    * Topic: 消息的主题比如 Trade则为消息主题，一般可以定义为某种业务类型\n    * MessageType: 第二级别的消息类型，比如Trade下存在支付成功的pay-succ-200的消息类型\n    \n#### Zookeeper数据结构\n\n        KiteServer : /kiteq/server/${topic}/ip:port\n        Producer   : /kiteq/pub/${topic}/${groupId}/ip:port\n        Consumer   : /kiteq/sub/${topic}/${groupId}-bind/#$data(bind)\n\n##### 订阅方式: \n\n        Direct (直接订阅)： 明确的Topic+MessageType订阅消息\n        Regx(正则式订阅):  Topic级别下，对MessageType进行正则匹配方式订阅消息\n        Fanout(广播式订阅): Topic级别下，订阅所有的MessageType的消息\n\n##### 持久订阅和非持久订阅\n\n        持久订阅 ： 分组集群内的机器哪怕是全部离线，消息也不会丢弃，上线后KiteQ推送\n\n        非持久订阅： 分组集群内的机器全部离线，消息不会保留，直接丢弃\n\n#### 模块图\n  ![image](./doc/kiteq_dep.png)\n\n  - 描述：\n\n    [KiteQ-Common](https://github.com/blackbeans/kiteq-common)  提供了对存储、网络传输协议、zk管理\n\n    [KiteQ-OPS](https://github.com/blackbeans/kiteq-ops) 提供了KiteQ集群图形化监控\n\n    [Turbo](https://github.com/blackbeans/turbo) 提供通用网络层的封装\n\n#### 架构图\n  ![image](./doc/kiteq_arch.png)\n  \n  - 流程描述：\n    \n  KiteQ：\n  \n        1.1 KiteQ启动会推送本机可以支持的接收和投递的Topic到注册中心(zk/etcd) /kiteq/server节点，同时监听是否订阅本机支持Topic的订阅分组订阅关系变化\n    \n        1.2 在订阅关系发生变化时，注册中心通过watch机制通知KiteQ更新本地的订阅关系，并可以提供对新增订阅分组的消息投递工作\n    \n Producer：\n \n        2.1 Producer启动后将本应用需要发送的消息的Topic列表推送到注册中心 /kiteq/pub节点(这部分只是用来集中化管理publish的分组)\n    \n        2.2 通过配置的消息Topic列表获取注册中心对应KiteQ节点的IP:Port，并增加watch观测KiteQ节点的变化，做本地KiteQ地址列表的变化\n    \n        3.1 通过获取的IP:Port在本地发起TCP长连接，并保持固定周期的心跳\n    \n        3.2 KiteQ对长连接验证授权完成后，返回连接建立成功，此后即可发布对应的消息了。\n    \n Consumer:\n \n        2.1 Consumer启动后会将本分组所订阅的订阅关系Binding推送到注册中心/kiteq/sub节点。\n    \n        2.2 通过本地订阅关系订阅的Topic列表获取注册中心对应KiteQ节点的IP:Port，并增加watch观测KiteQ节点的变化，做本地KiteQ地址列表的变化\n    \n        3.1、3.2 与Producer一致，在3.2完成后，KiteQ会在有对应消息来到时，推送给本分组内的随机一台Consumer\n   \n KiteQ-OPS:\n    \n        基于注册中心以及KiteQ暴露的Http接口提供系统的数据，并以图表方式呈现KiteQ当前状态\n   \n    - 注：\n        发布订阅者没有绝对的角色区分。同一个分组既可以作为发布者也可以作为消息的订阅方。\n\n#####  两阶段提交：\n\n    因为引入了异步投递方案，所以在有些场景下需要本地执行某个事务成功的时候，本条消息才可以被订阅方消费。\n    例如：\n        用户购买会员支付成功成功需要修改本地用户账户Mysql的余额、并且告知会员系统为用户的会员期限延长。\n        这个时候就会碰到、必须在保证mysql操作成功的情况下，会员系统才可以接收到会员延期的消息。\n    \n    对于以上的问题，KiteQ的处理如下：\n        1. 发送一个UnCommit的消息到KiteQ ,KiteQ 不会对Uncommite的消息做投递操作\n        2. KiteQ定期对UnCommit的消息向Producer发送TxAck的询问\n        3. 直到Producer明确告诉Commit或者Rollback该消息\n        4. Commit会走正常投递流程、Rollback会对当前消息回滚即删除操作。\n\n#####  Quickstart \n\nDockerHub\n\n```shell\n \n  #install zookeeper\n  docker pull zookeeper \n  #start zookeeper\n  docker run --name zk001 --network=host  -t zookeeper\n  #start kiteq server\n  docker run --name kiteq001 --network=host  -d hubean/kiteq:v1.0.0  -clusterName=rocksdb_dev -configPath=./conf/cluster.toml -pport=13801 -bind=:13800\n  \n  #start your client \n    \n  code with kiteq-client-go(https://github.com/blackbeans/kiteq-client-go)\n```\n\n\n* Note :\n    \n        KiteQ's Config is conf/cluster.toml\n            \n        You can use command 'sh kiteq -clusterName=${cluster_name} -configPath=${toml_path}'\n        \n        Recommend Using supervisor to manager kiteq .\n         \n* Registry Of KiteQ :\n\n        zk:    zk://localhost:2181,localhost:2181 (stable)\n\n        etcd:  etcd://http://localhost:2379,http://localhost:2379 (unstable)\n\n#### KiteQ Start Mode\n\n- How to use Rocksdb for kiteq's message storage\n\n```shell\n\n go run kiteq.go -clusterName=rocksdb_dev -configPath=./conf/cluster.toml -pport=13801 -bind=:13800\n \n```\n\n- How to use memory for kiteq's message storage\n\n```shell\n\n go run kiteq.go -clusterName=memory_dev -configPath=./conf/cluster.toml -pport=13801 -bind=:13800\n \n```\n\n- How to use RDS/Mysql for kiteq's message storage\n\n```shell\n  \n  install mysql \u0026  initial message table by running  store/mysql/table.sh \n  \n  go run kiteq.go -clusterName=mysql_dev -configPath=./conf/cluster.toml -pport=13801 -bind=:13800\n \n```\n\n##### Client Supported：\n\n[Go-Client](https://github.com/blackbeans/kiteq-client-go)\n\n[Java-Client](https://github.com/blackbeans/kiteq-client-java) \n\n[PHP-Client](https://github.com/blackbeans/kiteq-client-php) (Deprecated)\n\n[CPP-Client](https://github.com/quguangjie/kiteq-client-cpp) (Deprecated)\n\n#### Contact us\n\nQQ Group: 139685004(Kiteq实战群)\n\n\n\n\n\n\n\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblackbeans%2Fkiteq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fblackbeans%2Fkiteq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblackbeans%2Fkiteq/lists"}