{"id":15019341,"url":"https://github.com/weihubeats/mq-idempotent","last_synced_at":"2026-02-13T14:22:15.134Z","repository":{"id":38204352,"uuid":"426048315","full_name":"weihubeats/mq-idempotent","owner":"weihubeats","description":"mq幂等框架开箱即用(支持RocketMQ、RabbitMQ、Kafka)mq idempotent framework (support RocketMQ, RabbitMQ, Kafka)","archived":false,"fork":false,"pushed_at":"2023-02-08T03:02:50.000Z","size":616,"stargazers_count":35,"open_issues_count":5,"forks_count":7,"subscribers_count":2,"default_branch":"develop","last_synced_at":"2025-01-31T00:34:38.588Z","etag":null,"topics":["idempotent","idempotent-kafka","idempotent-rabbitmq","idempotent-rocketmq","java","rabbitmq","rocketmq","springboot"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/weihubeats.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-11-09T01:17:20.000Z","updated_at":"2024-12-21T08:02:27.000Z","dependencies_parsed_at":"2024-10-10T05:31:06.028Z","dependency_job_id":"20813596-4c7a-4909-972d-3527901345df","html_url":"https://github.com/weihubeats/mq-idempotent","commit_stats":{"total_commits":88,"total_committers":5,"mean_commits":17.6,"dds":"0.44318181818181823","last_synced_commit":"87c9edb33d62cdabb8c0e67a719ed014ebb3e88d"},"previous_names":["weihubeats/wh-mq-idempotent"],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/weihubeats%2Fmq-idempotent","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/weihubeats%2Fmq-idempotent/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/weihubeats%2Fmq-idempotent/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/weihubeats%2Fmq-idempotent/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/weihubeats","download_url":"https://codeload.github.com/weihubeats/mq-idempotent/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":237944134,"owners_count":19391588,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["idempotent","idempotent-kafka","idempotent-rabbitmq","idempotent-rocketmq","java","rabbitmq","rocketmq","springboot"],"created_at":"2024-09-24T19:53:21.645Z","updated_at":"2025-10-24T09:30:44.248Z","avatar_url":"https://github.com/weihubeats.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"![img.png](static/img/logo.png)\n\n## mq-Idempotent\n\n通用的mq消息幂等去重框架，开箱即用,支持主流所有mq\u003c/br\u003e\nGeneral mq message idempotent deduplication framework, out of the box, supports all mainstream mq\n\n\u003ca href=\"README_en.md\" target=\"_blank\"\u003e\u003cb\u003eEnglish\u003c/b\u003e\u003c/a\u003e\n\n1. 需要项目是Springboot项目\n2. 原理很简单基于Spring AOP + Redis做的\n3. 现在暂时只支持aliyun ons-client、RocketMQ Client、Spring Boot RabbitMQ.不支持的可以自己实现MessageConverter去定义,非常轻量方便\n4. 由于目前源代码非常轻量，所以不引用jar直接copy源代码到项目中使用也是可以的\n\n\u003e 由于本地没有rabbitmq、Kafka等,但是需要支持也非常方便,只需要实现MessageConverter 接口,后面有详细说明\n\n## 分支说明\n- master: 最新稳定代码\n- develop: 正在开发中的代码\n\n## 使用\n\n####\n必要条件\n1. redis redis使用的是redission\n2. springboot\n3. jdk8+\n\n#### 1. 添加依赖\n##### rocketmq-client\n- maven\n```xml\n\u003cdependency\u003e\n  \u003cgroupId\u003eio.github.weihubeats\u003c/groupId\u003e\n  \u003cartifactId\u003emq-idempotent-spring-boot-starter\u003c/artifactId\u003e\n  \u003cversion\u003e1.1.6\u003c/version\u003e\n\u003c/dependency\u003e\n```\n- gradle\n```xml\nimplementation 'io.github.weihubeats:mq-idempotent-spring-boot-starter:1.1.6'\n```\n\n##### aliyun ons-client\n```xml\n\u003cdependency\u003e\n\u003cgroupId\u003eio.github.weihubeats\u003c/groupId\u003e\n\u003cartifactId\u003emq-idempotent-spring-boot-starter-aliyun-rocketmq\u003c/artifactId\u003e\n\u003cversion\u003e1.1.6\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n- gradle\n```xml\nimplementation 'io.github.weihubeats:mq-idempotent-spring-boot-starter-aliyun-rocketmq:1.1.6'\n```\n#### 2. 在需要幂等的方法上添加注解\n```java\n@Idempotent\n```\n\n```java\n    @Idempotent\n    public void testConsumer(Message message) {\n        String msg = new String(message.getBody());\n        System.out.println(\"消息id \" + message.getMsgID());\n        System.out.println(\"消息key \" + message.getKey());\n        System.out.println(\"消费成功, msg \" + msg);\n    }\n```\n\n\u003e 注意事项,由于是基于AOP实现的,所以需要注意AOP失效场景导致的问题\n\u003e 添加住的方法的参数必须是 Message,方法返回值必须是void 或者 boolean 因为aop要处理重复消费直接返回 true 或者 null\n\u003e 更详细实用方式请参考模块 wh-mq-Idempotent-samples\n\n## 例子参考\n使用例子请参考 wh-mq-Idempotent-samples 模块\n\n\n## 自定义配置\n\n- 基于配置文件自动配置\n```yaml\nidempotent:\n  redisKey: \"mq:unique:\"\n  redisValue : \"ss\"\n  tryLockTime: 2\n  redisTimeOut: 3\n```\n- 基于JavaBean\n```java\n    @Bean\n    public IdempotentConfig idempotentConfig() {\n        IdempotentConfig idempotentConfig = new IdempotentConfig();\n        // 去重 redis key名 默认 mq::unique::\n        idempotentConfig.setRedisKey(redisKey);\n        // 去重redis value 默认 s\n        idempotentConfig.setRedisValue(redisValue);\n        // 去重redis尝试获取锁等待时间 默认1s 单位秒\n        idempotentConfig.setTryLockTime(tryLockTime);\n        // 设置处理成功消息存放redis时间 默认 3天\n        idempotentConfig.setRedisTimeOut(redisTimeOut);\n        return idempotentConfig;\n\n    }\n```\n\n## 自定义mq去重\n引入依赖\n```java\n\u003cdependency\u003e\n\u003cgroupId\u003eio.github.weihubeats\u003c/groupId\u003e\n\u003cartifactId\u003emq-idempotent-core\u003c/artifactId\u003e\n\u003cversion\u003e1.1.6\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n实现接口 MessageConverter.java\n例如 支持rocketMQ\n```java\n@Component\npublic class RocketMQMessageConverter implements MessageConverter\u003cMessageExt\u003e {\n\n\n    @Override\n    public String getUniqueKey(MessageExt messageExt, String field, Method method, Object[] args) {\n        return StringUtils.isNoneBlank(messageExt.getKeys()) ? messageExt.getKeys() :messageExt.getMsgId();\n    }\n}\n\n```\n\n## 模块说明\n- wh-core 核心实现\n- wh-mq-rocketmq rocketmq幂等核心实现\n- wh-mq-aliyun-rocketmq 阿里云client幂等核心实现\n- wh-mq-Idempotent-samples 使用例子\n\n## 设计思路\n\n![img.png](static/img/img.png)\n\n目前redis实现的大致思路如下\n1. 消费者获取到MQ消费信息\n2. 基于配置的MQ消息中的业务唯一键去reids(Mysql) 判断是否已消费\n3. 如果没有消费则加锁防止并发问题,加锁成功则消费,失败则返回消费失败让MQ重新投递,因为这里防止第一个抢到锁的线程执行失败,所以不能直接返回成功,需要后面的任务重新投递到MQ重新消费\n4. 执行业务代码成功后写入消费成功(写入redis或更新Mysql)\n5. 释放锁\n\n## 发布版本\n\n- 1.0.4:支持阿里云RocketMQ Client\n- 1.0.5:新增支持开源RocketMQ Client，新增自动化配置 `IdempotentConfig.java`\n- 1.0.6:优化重复消费重复投递问题，优化代码结构\n- 1.0.7:优化redis key 过期时间\n- 1.0.8:优化AOP实现方式\n- 1.1.9:添加对Spring Boot RabbitMQ支持\n- 1.1.0-Release:优化消息存储细节，自定义消息转换策略添加methdo、args \n- 1.1.1-Release: 修复分布式锁异常问题\n- 1.1.2-Release: 修复分布式锁异常问题\n- 1.1.4-Release: 添加事务幂等处理，添加飞书报警，添加自定义spi异常报警\n\n## 未来版本\n1. 支持kafka\n2. 支持Mysql去重\n3. 网络两将军问题处理\n## 正在开发中。。。。。。\n\n## 期待你的加入\n\n作者微信:\n![作者微信](static/img/wx.jpg)\n\n## 鸣谢\n\n感谢 JetBrains 提供的免费开源 License：\n\n\u003cp\u003e\n    \u003cimg src=\"https://images.gitee.com/uploads/images/2020/0406/220236_f5275c90_5531506.png\" alt=\"图片引用自lets-mica\" style=\"float:left;\"\u003e\n\u003c/p\u003e","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fweihubeats%2Fmq-idempotent","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fweihubeats%2Fmq-idempotent","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fweihubeats%2Fmq-idempotent/lists"}