{"id":13635422,"url":"https://github.com/houbb/mq","last_synced_at":"2025-08-21T07:33:23.344Z","repository":{"id":38449438,"uuid":"474621875","full_name":"houbb/mq","owner":"houbb","description":"The message queue in java.(java 简易版本 mq 实现)","archived":false,"fork":false,"pushed_at":"2022-11-30T04:44:11.000Z","size":281,"stargazers_count":122,"open_issues_count":2,"forks_count":28,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-11-24T20:51:51.527Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/houbb.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-03-27T11:49:21.000Z","updated_at":"2024-11-13T16:43:01.000Z","dependencies_parsed_at":"2022-08-19T15:51:45.373Z","dependency_job_id":null,"html_url":"https://github.com/houbb/mq","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/houbb%2Fmq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/houbb%2Fmq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/houbb%2Fmq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/houbb%2Fmq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/houbb","download_url":"https://codeload.github.com/houbb/mq/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230501172,"owners_count":18236061,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T00:00:45.390Z","updated_at":"2024-12-19T21:09:38.497Z","avatar_url":"https://github.com/houbb.png","language":"Java","funding_links":[],"categories":["测试","进程间通信"],"sub_categories":["客户端","Spring Cloud框架"],"readme":"# mq\n\n[mq](https://github.com/houbb/mq) 是基于 netty 实现的 java mq 框架，类似于 rocket mq。\n\n[![Build Status](https://travis-ci.com/houbb/mq.svg?branch=master)](https://travis-ci.com/houbb/mq)\n[![Maven Central](https://maven-badges.herokuapp.com/maven-central/com.github.houbb/mq/badge.svg)](http://mvnrepository.com/artifact/com.github.houbb/mq)\n[![](https://img.shields.io/badge/license-Apache2-FF0080.svg)](https://github.com/houbb/mq/blob/master/LICENSE.txt)\n[![Open Source Love](https://badges.frapsoft.com/os/v2/open-source.svg?v=103)](https://github.com/houbb/nlp-common)\n\n\u003e [变更日志](https://github.com/houbb/mq/blob/master/CHANGELOG.md)\n\n主要用于个人学习，由渐入深，理解 mq 的底层实现原理。\n\n## 特性\n\n- 基于 netty4 的客户端调用服务端\n\n- timeout 超时处理\n\n- broker 启动的 check 检测服务可用性\n\n- load balance 负载均衡\n\n- 基于 TAG 的消息过滤，broker 端实现\n\n- 生产者的消息同步发送，ONE WAY 发送\n\n- 生产消息的批量发送\n\n- 消息状态的批量确认\n\n- fail 支持 failOver failFast 等失败处理策略\n\n- heartbeat 服务端心跳\n\n- AT LEAST ONCE 最少一次原则\n\n# 快速入门\n\n## 测试\n\n### 注册中心\n\n依赖 maven 包：\n\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.github.houbb\u003c/groupId\u003e\n    \u003cartifactId\u003emq-broker\u003c/artifactId\u003e\n    \u003cversion\u003e0.1.3\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n代码实现：\n\n```java\nMqBroker broker = new MqBroker();\nbroker.start();\n```\n\n### 消费者\n\n依赖 maven 包：\n\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.github.houbb\u003c/groupId\u003e\n    \u003cartifactId\u003emq-consumer\u003c/artifactId\u003e\n    \u003cversion\u003e0.1.3\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n代码实现：\n\n\n```java\nfinal MqConsumerPush mqConsumerPush = new MqConsumerPush();\nmqConsumerPush.start();\n\nmqConsumerPush.subscribe(\"TOPIC\", \"TAGA\");\nmqConsumerPush.registerListener(new IMqConsumerListener() {\n    @Override\n    public ConsumerStatus consumer(MqMessage mqMessage, IMqConsumerListenerContext context) {\n        System.out.println(\"---------- 自定义 \" + JSON.toJSONString(mqMessage));\n        return ConsumerStatus.SUCCESS;\n    }\n});\n```\n\n### 生产者\n\n依赖 maven 包：\n\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.github.houbb\u003c/groupId\u003e\n    \u003cartifactId\u003emq-producer\u003c/artifactId\u003e\n    \u003cversion\u003e0.1.3\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n代码实现：\n\n```java\nMqProducer mqProducer = new MqProducer();\nmqProducer.start();\n\nString message = \"HELLO MQ!\";\nMqMessage mqMessage = new MqMessage();\nmqMessage.setTopic(\"TOPIC\");\nmqMessage.setTags(Arrays.asList(\"TAGA\", \"TAGB\"));\nmqMessage.setPayload(message);\n\nSendResult sendResult = mqProducer.send(mqMessage);\nSystem.out.println(JSON.toJSON(sendResult));\n```\n\n# 前言\n\n工作至今，接触 mq 框架已经有很长时间。\n\n但是对于其原理一直只是知道个大概，从来没有深入学习过。\n\n以前一直想写，但由于各种原因被耽搁。\n\n## 技术准备\n\n[Java 并发实战学习](https://houbb.github.io/2019/01/18/jcip-00-overview)\n\n[TCP/IP 协议学习笔记](https://houbb.github.io/2019/04/05/protocol-tcp-ip-01-overview-01)\n\n[Netty 权威指南学习](https://houbb.github.io/2019/05/10/netty-definitive-gudie-00-overview)\n\n这些技术的准备阶段，花费了比较长的时间。\n\n也建议想写 mq 框架的有相关的知识储备。\n\n其他 mq 框架使用的经验此处不再赘述。\n\n## 快速迭代\n\n原来一直想写 mq，却不行动的原因就是想的太多，做的太少。\n\n想一下把全部写完，结果就是啥都没写。\n\n所以本次的开发，每个代码分支做的事情实际很少，只做一个功能点。\n\n陆陆续续经过近一个月的完善，对 mq 框架有了自己的体会和进一步的认知。\n\n代码实现功能，主要参考 [Apache Dubbo](https://dubbo.apache.org/zh/docs/introduction/)\n\n# 文档\n\n## 文档\n\n文档将使用 markdown 文本的形式，补充 code 层面没有的东西。\n\n[【mq】从零开始实现 mq-01-生产者、消费者启动 ](https://mp.weixin.qq.com/s/moF528JiVG9dqCi5oFMbVg)\n\n[【mq】从零开始实现 mq-02-如何实现生产者调用消费者？](https://mp.weixin.qq.com/s/_OF4hbh9llaxN27Cv_cToQ)\n\n[【mq】从零开始实现 mq-03-引入 broker 中间人](https://mp.weixin.qq.com/s/BvEWsLp3_35yFVRqBOxS2w)\n\n[【mq】从零开始实现 mq-04-启动检测与实现优化](https://mp.weixin.qq.com/s/BvEWsLp3_35yFVRqBOxS2w)\n\n[【mq】从零开始实现 mq-05-实现优雅停机](https://mp.weixin.qq.com/s/BvEWsLp3_35yFVRqBOxS2w)\n\n[【mq】从零开始实现 mq-06-消费者心跳检测 heartbeat](https://mp.weixin.qq.com/s/lsvm9UoQWK98Jy3kuS2aNg)\n\n[【mq】从零开始实现 mq-07-负载均衡 load balance](https://mp.weixin.qq.com/s/ZNuecNeVJzIPCp252Hn4GQ)\n\n[【mq】从零开始实现 mq-08-配置优化 fluent](https://mp.weixin.qq.com/s/_O20KKdGwxMcHc87rcuWug)\n\n[【mq】从零开始实现 mq-09-消费者拉取消息 pull message](https://mp.weixin.qq.com/s/bAqOJ4fKWTAVet0Oqv8S0g)\n\n[【mq】从零开始实现 mq-10-消费者拉取消息回执 pull message ack](https://mp.weixin.qq.com/s/OgcQI-Go1ZS9-pdLtYwkcg)\n\n[【mq】从零开始实现 mq-11-消费者消息回执添加分组信息 pull message ack groupName](https://mp.weixin.qq.com/s/3RnB7KhZB3n8yGI6Z02-bw)\n\n[【mq】从零开始实现 mq-12-消息的批量发送与回执](https://mp.weixin.qq.com/s/tg0gxwbGWd7cn_RGMiEWew)\n\n[【mq】从零开始实现 mq-13-注册鉴权 auth](https://mp.weixin.qq.com/s/SzWAqyHpeTrDQyUTknsJGQ)\n\n\n## 代码注释\n\n代码有详细的注释，便于阅读和后期维护。\n\n## 测试\n\n目前测试代码算不上完善。后续将陆续补全。\n\n# mq 模块\n\n| 模块 | 说明 |\n|:---|:---|\n| mq-common | 公共代码 |\n| mq-broker | 注册中心 |\n| mq-producer | 服务端 |\n| mq-consumer | 客户端 |\n| mq-test | 测试模块 |\n\n# 测试代码\n\n这部分测试代码可以关注公众号【老马啸西风】，后台回复【mq】领取。\n\n![qrcode](qrcode.jpg)\n\n# 后期 ROAD-MAP\n\n- [ ] all 模块\n\n- [x] check broker 启动检测\n  \n- [x] 关闭时通知 register center\n\n- [x] 优雅关闭添加超时设置\n  \n- [x] heartbeat 心跳检测机制\n\n- [x] 完善 load-balance 实现 + shardingkey 粘性消费、请求\n\n- [x] 失败重试的拓展\n\n- [x] 消费者 pull 策略实现\n\n- [x] pull 消息消费的 ACK 处理\n\n- [x] broker springboot 实现\n\n- [x] 消息的 ack 处理，要基于 groupName 进行处理\n\n- [x] 消息的回溯消费 offset \n\n- [x] 消息的批量发送，批量 ACK\n\n- [x] 添加注册鉴权，保证安全性\n\n- [ ] 顺序消息 \n\n- [ ] 事务消息\n\n- [ ] 定时消息\n\n- [ ] 流量控制 back-press 反压\n\n- [ ] 消息可靠性\n\n- [ ] offline message 离线消息\n\n- [ ] dead message 死信队列\n\n- [ ] 断线重连\n\n\n# 中间件等工具开源矩阵\n\n[heaven: 收集开发中常用的工具类](https://github.com/houbb/heaven)\n\n[rpc: 基于 netty4 实现的远程调用工具](https://github.com/houbb/rpc)\n\n[mq: 简易版 mq 实现](https://github.com/houbb/mq)\n\n[ioc: 模拟简易版 spring ioc](https://github.com/houbb/ioc)\n\n[mybatis: 简易版 mybatis](https://github.com/houbb/mybatis)\n\n[cache: 渐进式 redis 缓存](https://github.com/houbb/cache)\n\n[jdbc-pool: 数据库连接池实现](https://github.com/houbb/jdbc-pool)\n\n[sandglass: 任务调度时间工具框架](https://github.com/houbb/sandglass)\n\n[sisyphus: 支持注解的重试框架](https://github.com/houbb/sisyphus)\n\n[resubmit: 防止重复提交框架，支持注解](https://github.com/houbb/resubmit)\n\n[auto-log: 日志自动输出](https://github.com/houbb/auto-log)\n\n[async: 多线程异步并行框架](https://github.com/houbb/async)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhoubb%2Fmq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhoubb%2Fmq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhoubb%2Fmq/lists"}