{"id":13694553,"url":"https://github.com/Jaskey/RocketMQDedupListener","last_synced_at":"2025-05-03T04:30:32.297Z","repository":{"id":38354422,"uuid":"264127183","full_name":"Jaskey/RocketMQDedupListener","owner":"Jaskey","description":"RocketMQ消息幂等去重消费者，支持使用MySQL或者Redis做幂等表，开箱即用","archived":false,"fork":false,"pushed_at":"2021-08-18T06:44:49.000Z","size":103,"stargazers_count":235,"open_issues_count":0,"forks_count":87,"subscribers_count":10,"default_branch":"master","last_synced_at":"2024-11-12T21:38:52.403Z","etag":null,"topics":["deduplication","rocketmq","rocketmq-client"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Jaskey.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-05-15T07:31:30.000Z","updated_at":"2024-11-07T13:04:15.000Z","dependencies_parsed_at":"2022-08-25T05:00:43.044Z","dependency_job_id":null,"html_url":"https://github.com/Jaskey/RocketMQDedupListener","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jaskey%2FRocketMQDedupListener","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jaskey%2FRocketMQDedupListener/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jaskey%2FRocketMQDedupListener/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jaskey%2FRocketMQDedupListener/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Jaskey","download_url":"https://codeload.github.com/Jaskey/RocketMQDedupListener/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252144275,"owners_count":21701379,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deduplication","rocketmq","rocketmq-client"],"created_at":"2024-08-02T17:01:34.665Z","updated_at":"2025-05-03T04:30:31.928Z","avatar_url":"https://github.com/Jaskey.png","language":"Java","funding_links":[],"categories":["Java"],"sub_categories":[],"readme":"# RocketMQDedupListener\n通用的RocketMQ消息幂等去重消费者工具类，开箱即用\n\n1. 支持利用Redis或者MySQL做幂等表。\n2. 支持业务主键去重或消息ID去重（默认）\n3. 支持消息重复并发控制（重复的消息消费成功/失败前，不会同时消费第二条）\n4. 接近于EXACTLY-ONCE语义（消息只会且仅会被成功消费一次），极端场景下则为ATLEAST-ONCE语义（消息至少被成功消费一次，不会因为去重的增强而丢失消息）\n\n\n# 内置去重原理\n\n见以下流程图\n\n![image](https://raw.githubusercontent.com/Jaskey/RocketMQDedupListener/master/dedup-flow.png)\n\n\n# 去重实现的来龙去脉\n\n可以参考本人在官方微信发表的博文： [RocketMQ消息幂等的通用解决方案](https://mp.weixin.qq.com/s/X25Jw-sz3XItVrXRS6IQdg) \n\n# Quick Start\n\n### 1、继承`DedupConcurrentListener`类，实现消费回调和去重键的设置回调\n\n```\npublic class SampleListener extends DedupConcurrentListener {\n\n    public SampleListener(DedupConfig dedupConfig) {\n        super(dedupConfig);\n    }\n\n    //基于什么做消息去重，每一类不同的消息都可以不一样，做去重之前会尊重此方法返回的值\n    @Override\n    protected String dedupMessageKey(MessageExt messageExt) {\n        //为了简单示意，这里直接使用消息体作为去重键，正式使用时候不建议这样使用\n        if (\"TEST-TOPIC\".equals(messageExt.getTopic())) {\n            return new String(messageExt.getBody());\n        } else {//其他使用默认的配置（消息id）\n            return super.dedupMessageKey(messageExt);\n        }\n    }\n\n    @Override\n    protected boolean doHandleMsg(MessageExt messageExt) {\n        switch (messageExt.getTopic()) {\n            case \"TEST-TOPIC\":\n                log.info(\"假装消费很久....{} {}\", new String(messageExt.getBody()), messageExt);\n                try {\n                    Thread.sleep(3000);\n                } catch (InterruptedException e) {}\n                break;\n        }\n        return true;\n    }\n}\n```\n\n### 2、使用此实例启动RocketMQ 消费者\n\n```\n\n            DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(\"TEST-APP1\");\n            consumer.subscribe(\"TEST-TOPIC\", \"*\");\n\n            //START:区别于普通RocketMQ使用的代码\n            String appName = consumer.getConsumerGroup();//针对什么应用做去重，相同的消息在不同应用的去重是隔离处理的\n            StringRedisTemplate stringRedisTemplate = null;// 这里省略获取StringRedisTemplate的过程，具体的消息幂等表会保存到Redis中\n            DedupConfig dedupConfig = DedupConfig.enableDedupConsumeConfig(appName, stringRedisTemplate);\n            DedupConcurrentListener messageListener = new SampleListener(dedupConfig);\n            //END:区别于普通RocketMQ使用的代码\n\n\n            consumer.registerMessageListener(messageListener);\n            consumer.start();\n        \n```\n\n注：\n\n1. 以上省略了RocketMQ消费者的配置及`StringRedisTemplate`的获取过程，需要使用者自己准备。\n2. `RocketMQDedupListener`支持使用Redis或MySQL进行去重，更多使用详情请见`SampleListener.java`\n\n\n\n\n# 使用场景测试\n\n以下是部分实验的输出日志及现象，读者可以参考观察实验（基于`SampleListener.java`的实现）\n\n\n## 一、测试普通的消息重复：\n\n1. 模拟正常消息：发送消息到TEST-TOPIC， 报文为，test-ljj-msg1234\n2. 模拟重复消息：隔几秒后（这个例子需要大于3秒），再发送消息到TEST-TOPIC，报文一样是test-ljj-msg1234\n\n\n\n### 日志输出如下：\n\n\n```\n[INFO] 2020-05-15 11:06:17,697 []  \u003e\u003e\u003e 假装消费很久....test-ljj-msg1234 MessageExt [queueId=1, storeSize=169, queueOffset=0, sysFlag=0, bornTimestamp=1589511454575, bornHost=/10.13.32.179:52637, storeTimestamp=1589511454576, storeHost=/10.13.32.179:10911, msgId=0A0D20B300002A9F000000003EEA31B0, commitLogOffset=1055535536, bodyCRC=1038040938, reconsumeTimes=0, preparedTransactionOffset=0, toString()=Message [topic=TEST-TOPIC, flag=0, properties={MIN_OFFSET=0, MAX_OFFSET=1, CONSUME_START_TIME=1589511977328, UNIQ_KEY=0A0D20B3632A5B2133B14A730F6F014A, WAIT=true}, body=16]]\n[INFO] 2020-05-15 11:06:20,748 []  \u003e\u003e\u003e consume [1] msg(s) all successfully\n[WARN] 2020-05-15 11:06:26,504 []  \u003e\u003e\u003e message has been consumed before! dedupKey = DedupElement={\"application\":\"repay-platform-core\",\"msgUniqKey\":\"test-ljj-msg1234\",\"tag\":\"\",\"topic\":\"TEST-TOPIC\"}, msgId : 0A0D20B3632A5B2133B14A7332DB014B , so just ack. RedisPersist\n[INFO] 2020-05-15 11:06:26,504 []  \u003e\u003e\u003e consume [1] msg(s) all successfully\n```\n\n\n说明：\n\n可以看到第二条消息被直接幂等掉了，没有进入业务的测试代码\n\n\n\n## 测试并发重复消费：\n\n\n1. 模拟正常消息：发送消息到TEST-TOPIC， 报文为 test-ljj-msg123\n2. 模拟重复消息：马上（这个例子需要小于3秒）再发送消息到TEST-TOPIC，报文一样是 test-ljj-msg123\n\n\n由于这里需要一些特殊说明，以下日志增加了注释\n\n\n```\n33秒第一条消息到达，这里消息会消费3秒\n[INFO] 2020-05-15 11:07:33,756 []  \u003e\u003e\u003e 假装消费很久....test-ljj-msg123 MessageExt [queueId=1, storeSize=168, queueOffset=2, sysFlag=0, bornTimestamp=1589511530879, bornHost=/10.13.32.179:52651, storeTimestamp=1589511530881, storeHost=/10.13.32.179:10911, msgId=0A0D20B300002A9F000000003EEA3302, commitLogOffset=1055535874, bodyCRC=146853239, reconsumeTimes=0, preparedTransactionOffset=0, toString()=Message [topic=TEST-TOPIC, flag=0, properties={MIN_OFFSET=0, MAX_OFFSET=3, CONSUME_START_TIME=1589512053623, UNIQ_KEY=0A0D20B3632A5B2133B14A74397F014C, WAIT=true}, body=15]]\n\n35秒重复消息到达，发现前面的消息还在消费\n[WARN] 2020-05-15 11:07:35,884 []  \u003e\u003e\u003e the same message is considered consuming, try consume later dedupKey : DedupElement={\"application\":\"repay-platform-core\",\"msgUniqKey\":\"test-ljj-msg123\",\"tag\":\"\",\"topic\":\"TEST-TOPIC\"}, 0A0D20B3632A5B2133B14A7441FB014D, RedisPersit\n消费按消费失败处理，触发延迟消费\n[WARN] 2020-05-15 11:07:35,884 []  \u003e\u003e\u003e consume [1] msg(s) fails, ackIndex = [-1]\n\n36秒第一条消息消费成功\n[INFO] 2020-05-15 11:07:36,801 []  \u003e\u003e\u003e consume [1] msg(s) all successfully\n\n46秒第二条消息延迟消费开始，发现这条消息已经被成功消费\n[WARN] 2020-05-15 11:07:46,024 []  \u003e\u003e\u003e message has been consumed before! dedupKey = DedupElement={\"application\":\"repay-platform-core\",\"msgUniqKey\":\"test-ljj-msg123\",\"tag\":\"\",\"topic\":\"TEST-TOPIC\"}, msgId : 0A0D20B3632A5B2133B14A7441FB014D , so just ack. RedisPersit\n\n直接按照消费成功处理\n[INFO] 2020-05-15 11:07:46,024 []  \u003e\u003e\u003e consume [1] msg(s) all successfully\n```\n\n\n说明：\n\n可以看到第二条消息在第一条消息消费的过程中就投递到消费者了，这时候去重逻辑做了并发控制，保证了业务代码的安全。\n\n------------------------\n\n## MYSQL去重支持\n若希望使用MYSQL存储消息消费记录，使用上仅需把StringRedisTemplate改成JdbcTemplate：\n\n            JdbcTemplate jdbcTemplate = null;// 这里省略获取JDBCTemplate的过程，幂等表将使用MySQL的t_rocketmq_dedup存储\n            DedupConfig dedupConfig = DedupConfig.enableDedupConsumeConfig(appName, jdbcTemplate);\n\n\n\n同时需要预先建立一张消息去重表，结构如下:\n\n```\n-- ----------------------------\n-- Table structure for t_rocketmq_dedup\n-- ----------------------------\nDROP TABLE IF EXISTS `t_rocketmq_dedup`;\nCREATE TABLE `t_rocketmq_dedup` (\n`application_name` varchar(255) NOT NULL COMMENT '消费的应用名（可以用消费者组名称）',\n`topic` varchar(255) NOT NULL COMMENT '消息来源的topic（不同topic消息不会认为重复）',\n`tag` varchar(16) NOT NULL COMMENT '消息的tag（同一个topic不同的tag，就算去重键一样也不会认为重复），没有tag则存\"\"字符串',\n`msg_uniq_key` varchar(255) NOT NULL COMMENT '消息的唯一键（建议使用业务主键）',\n`status` varchar(16) NOT NULL COMMENT '这条消息的消费状态',\n`expire_time` bigint(20) NOT NULL COMMENT '这个去重记录的过期时间（时间戳）',\nUNIQUE KEY `uniq_key` (`application_name`,`topic`,`tag`,`msg_uniq_key`) USING BTREE\n) ENGINE=InnoDB DEFAULT CHARSET=utf8 ROW_FORMAT=COMPACT;\n\n\n```\n\n说明:因为需要支持不同的应用，所以需要存储application_name，因为同一个业务主键可能来自不同的topic/tag，所以也需要存储起来。\n\n\n### 一直消费失败会否引起死循环\n\n不会。失败/消费中触发的延迟消费依赖与RocketMQ原生的重试机制，默认是16次。如果有希望调整延迟的时间和重试次数，需要自行调整Consumer配置。\n\n# 关于作者\n\nApache RocketMQ Committer，知乎专栏 [RocketMQ详解](https://zhuanlan.zhihu.com/rocketmq)作者，RoceketMQ官微投稿者\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FJaskey%2FRocketMQDedupListener","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FJaskey%2FRocketMQDedupListener","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FJaskey%2FRocketMQDedupListener/lists"}