{"id":15014251,"url":"https://github.com/xjtuwsn/cranemq","last_synced_at":"2025-04-12T07:42:13.725Z","repository":{"id":203761112,"uuid":"696766451","full_name":"xjtuwsn/cranemq","owner":"xjtuwsn","description":"一个简单的Java消息队列，具有高效、可靠、分布式等特点 (A simple implemation of message queue in Java)","archived":false,"fork":false,"pushed_at":"2023-10-26T14:23:58.000Z","size":1343,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-26T02:51:12.986Z","etag":null,"topics":["distribution","java","message-queue","mq","netty"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xjtuwsn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-26T11:54:36.000Z","updated_at":"2024-08-21T12:48:10.000Z","dependencies_parsed_at":null,"dependency_job_id":"bd947751-e9ff-4a38-b596-d2986666be57","html_url":"https://github.com/xjtuwsn/cranemq","commit_stats":{"total_commits":39,"total_committers":1,"mean_commits":39.0,"dds":0.0,"last_synced_commit":"5a593354485a957868e54835aa836b862fd1d314"},"previous_names":["xjtuwsn/cranemq"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xjtuwsn%2Fcranemq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xjtuwsn%2Fcranemq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xjtuwsn%2Fcranemq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xjtuwsn%2Fcranemq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xjtuwsn","download_url":"https://codeload.github.com/xjtuwsn/cranemq/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248536981,"owners_count":21120683,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distribution","java","message-queue","mq","netty"],"created_at":"2024-09-24T19:45:22.414Z","updated_at":"2025-04-12T07:42:13.706Z","avatar_url":"https://github.com/xjtuwsn.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"## 介绍\n\n出于对消息队列的兴趣，以及为了更好的学习[rocketmq](https://github.com/apache/rocketmq)和[kafka](https://github.com/apache/kafka)，我选择对mq的核心功能进行了实现。基于这些开源mq的架构和设计思想，我尝试用自己的方式进行了实现，该项目也主要是出于**个人学习目的**\n\n目前已实现如下功能：\n\n+ 生产者以同步、异步、单向方式发送消息\n+ 单个消息和批量消息\n+ 发送普通消息、顺序消息、延时消息\n+ 超时检测和失败重试\n+ 钩子函数\n+ 发送队列负载均衡\n+ 消费者订阅主题\n+ pull和push方式获取消息\n+ 历史消息消费\n+ 批量拉取\n+ 消费者组管理与监听\n+ 消费失败重试\n+ 广播消费模式和集群消费模式\n+ 消息过滤\n+ 本地位移管理\n+ broker端持久化存储消息\n+ 管理消费者位移\n+ 管理生产者和消费者，心跳机制\n+ 向注册中心注册\n+ 提供重试队列、延时队列和死信队列\n+ 多种注册中心\n+ 支持灰度发布\n+ 与spring框架集成\n\n## 特性\n\n1. 基于发布-订阅模型实现消息队列，通过生产者组和消费者组统一管理\n\n2. 底层基于Netty构建高效通信框架，使用多种自定义协议完成通讯\n\n3. 支持普通消息的发送和接受，同时基于shardingkey和分布式锁和本地锁实现顺序消息，消费者支持pull与push两种消费模式，并提供多种生产者队列选择的负载均衡策略\n\n4. 实现多种通信调用方式，并提供失败重试和超时控制机制\n\n5. 所有消息全部持久化在磁盘，保证可靠性，同时基于mmap和pagecahe机制实现高效存储和读写分离，建立消息索引文件加速读取，不依赖其它数据库\n\n6. 通过不同模式的消费者位移持久化管理和确认机制，保证消息至少被消费一次，并且可以消费历史消息\n\n7. 通过心跳机制定期摘除不活跃连接\n\n8. 实现消费者组和服务端变化时的rebalance机制，实现消息重分配，内置多种分配策略\n9. 内部实现时间轮来进行高效的延时任务调度，支持生产者生产任意时间延时消息，并持久化延迟任务，引入提交机制进行崩溃恢复\n10. 借助延时队列实现消息失败定时重试，并将多次失败消息加入死信队列，支持死信队列的查看\n11. 使用线程池提升效率，通过事件监听和请求队列，生产者和消费者模式进行解耦，并通过各种机制保证并发安全\n12. 支持灰度消息分流，保证灰度发布前与发布后订阅关系与消费进度一致性\n13. 实现注册中心，提供队列发现能力，方便进行集群扩展，支持更换默认注册中心为zookeeper和nacos\n14. 支持水平扩展，目前支持多主集群\n15. 与Spring和SpringBoot框架进行集成，使用注解和starter简化配置，方便调用\n\n\n\n## 快速开始\n\n### 项目结构\n\n``` shell\n- broker                               : MQ服务端实现，负责消息存储、通信、客户端管理等 \n- client                               : MQ客户端实现，包括消费者和生产者\n- common                               : 通用属性\n- example                              : 使用示例\n     - example-frameless               : 无框架使用\n     - example-spring                  : 结合spring使用\n- extension                            : 扩展\n     - cranemq-spring-boot-starter     : springboot-starter\n     - nacos-extension                 : nacos注册中心集成\n     - zookeeper-extension             : zookeeper注册中心集成\n- registry                             : 轻量注册中心\n- test\n```\n\n### Broker\n\nbroker端采用SpringBoot编写，下载源码后，使用maven对项目进行编译，产生jar包运行\n\n#### 配置文件\n\n无其它依赖，需要指定配置文件位置，在`broker/resources`下有示例配置文件，配置优先级：\n\n1. 启动时以命令行方式给处配置文件全路径\n2. 在`application.yaml`中配置路径\n\n#### 端口\n\n默认情况下broker需要使用两个端口：\n\n+ 与客户端通信端口`6086`\n+ 队列查看页面端口`7654`\n  + 启动后访问`http://localhost:7654/`查看队列信息\n\n\n\n### Registry\n\n自带配置中心可以直接启动，默认端口`11111`，可以使用`zookeeper`或`nacos`\n\n\n\n### 消费者和生产者\n\n无框架使用引入\n\n``` java\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.github.xjtuwsn\u003c/groupId\u003e\n    \u003cartifactId\u003eclient\u003c/artifactId\u003e\n    \u003cversion\u003e0.0.1\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n`SpringBoot`引入\n\n``` java\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.github.xjtuwsn\u003c/groupId\u003e\n    \u003cartifactId\u003ecranemq-spring-boot-starter\u003c/artifactId\u003e\n    \u003cversion\u003e0.0.1\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n具体使用示例参考`example`模块\n\n\n\n## 完善\n\n因为个人时间和水平(比较关键)问题，所以项目还有着不少的问题，和许多可以完善的点，包括但不仅限于以下几点：\n\n+ 完整测试，目前仅做了功能性测试，各种情况还没有考虑到\n+ 性能优化，通过压测和监控解决性能瓶颈\n+ 代码规范，代码写的比较仓促，耦合度高\n\n功能方面：\n\n+ 集群扩展，目前仅支持多主集群，实现方便而可靠性差，期望实现主从集群\n+ 消费者流量控制\n+ ACL\n+ 事务\n+ 消息压缩，存储优化\n+ 等等\n\n## 联系\n\n+ e-mail：`wwshining@qq.com`","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxjtuwsn%2Fcranemq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxjtuwsn%2Fcranemq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxjtuwsn%2Fcranemq/lists"}