{"id":16569239,"url":"https://github.com/kevinleeex/redeq","last_synced_at":"2025-10-29T00:31:53.956Z","repository":{"id":52077319,"uuid":"407191167","full_name":"kevinleeex/redeq","owner":"kevinleeex","description":"Yet Another Redis Delayed Queue，又一个基于Redis的延迟队列设计与实现","archived":false,"fork":false,"pushed_at":"2022-08-04T03:21:55.000Z","size":690,"stargazers_count":11,"open_issues_count":3,"forks_count":5,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-02-01T20:45:02.088Z","etag":null,"topics":["delayed-queue","redis"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kevinleeex.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-09-16T14:13:06.000Z","updated_at":"2024-02-26T14:59:26.000Z","dependencies_parsed_at":"2022-09-04T19:10:30.846Z","dependency_job_id":null,"html_url":"https://github.com/kevinleeex/redeq","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kevinleeex%2Fredeq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kevinleeex%2Fredeq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kevinleeex%2Fredeq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kevinleeex%2Fredeq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kevinleeex","download_url":"https://codeload.github.com/kevinleeex/redeq/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238751434,"owners_count":19524551,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["delayed-queue","redis"],"created_at":"2024-10-11T21:12:50.008Z","updated_at":"2025-10-29T00:31:53.580Z","avatar_url":"https://github.com/kevinleeex.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\u003cimg style=\"display:inline-block\" width='800' src=\"./assets/redeq-logo.png\"/\u003e\u003cp\u003e\n    \u003cspan style=\"font-size: 14px\"\u003eVersion: 1.1.1-ALPHA\u003c/span\u003e\u003cbr\u003e\n    \u003cspan\u003e\"ReDeQ - 另一个基于Redis的延迟队列\"\u003c/span\u003e\u003cbr\u003e\n    \u003cspan style=\"font-size: 12px;color= #95dafc\"\u003e-- Created by \u003ca\u003eKevin T. Lee\u003c/a\u003e --\u003c/span\u003e\n    \u003c/p\u003e\n   \u003ca href=\"./LICENSE\"\u003e\u003cimg alt=\"MIT\" src=\"https://img.shields.io/badge/LICENSE%20-MIT-green.svg?longCache=true\u0026style=for-the-badge\"\u003e\u003c/a\u003e\n   \u003ca href=\"http://lidengju.com\"\u003e\u003cimg alt=\"Code\" src=\"https://img.shields.io/badge/Code%20with-Heart-red.svg?longCache=true\u0026style=for-the-badge\"\u003e\u003c/a\u003e\n   \u003ca href=\"https://github.com/kevinleeex/redeq\"\u003e\u003cimg alt=\"Version\" src=\"https://img.shields.io/badge/Version-1.1.1_ALPHA-blue.svg?longCache=true\u0026style=for-the-badge\"\u003e\u003c/a\u003e\n   \n\u003c/div\u003e\n\n# ReDeQ - Yet Another Redis Delayed Queue\n\n[![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=kevinleeex_redeq\u0026metric=alert_status)](https://sonarcloud.io/dashboard?id=kevinleeex_redeq)\n\n另一个基于Redis的延迟队列\n\n\u003e 作者：Li, Dengju，字数：2606，预计花费时间：15分钟\n\n- [ReDeQ - Yet Another Redis Delayed Queue](#redeq---yet-another-redis-delayed-queue)\n   * [Overview](#overview)\n   * [Implementation](#implementation)\n   * [Features](#features)\n   * [Installation](#installation)\n   * [Usage](#usage)\n      + [with spring-boot](#with-spring-boot)\n         - [Dependency](#dependency)\n         - [Config](#config)\n         - [Example](#example)\n      + [without spring-boot](#without-spring-boot)\n   * [Design](#design)\n      + [Service Diagram](#service-diagram)\n      + [ReDeQ-Design](#redeq-design)\n      + [Components](#components)\n      + [State Diagram](#state-diagram)\n   * [FAQ](#faq)\n   * [References](#references)\n   * [License](#license)\n\n## Overview\n\n我们可能经常面临如需要多少分钟后延迟重试任务，多少分钟后消息或邮件通知用户，或者一段时间后取消任务、订单或者变更状态等类似需求。显然，我们可以通过JDK中的`TimerTask\n`来直接设定定时任务，但是这样并不方便我们对已执行或准备执行的任务进行很好的管理，也不适合在分布式场景下对大量的对象设置定时任务。在这样的需求场景下，延迟队列可以派上用场，目前有这样的几种方案来实现延迟队列：\n\n1. 基于JUC包中的延迟队列(`DelayQueue`)，它是一个无界阻塞队列(`BlockingQueue`)，本质封装了一个基于完全二叉堆优先队列(`PriorityQueue`)\n   ，实现较小元素（较近的时间）排在队首，并且限制了可以出队的时间，以实现延迟队列。\n2. 基于Quartz定时任务实现，在Redis和RabbitMQ没有广泛实现的时候，常使用这种方式实现。\n3. 基于Redis的Zset数据结构实现延迟队列的效果，可以将待执行时间作为score来进行排序，通过获取特定范围score的数据进行消费以实现延迟队列。\n4. 基于RabbitMQ来实现延迟队列，通过设置消息的存活时间(Time To Live, TTL)，以及死信交换机制，来实现当消息过期时，将其转发到指定队列重新消费。\n5. 同样利用过期机制，Redis可以为key设置过期回调，来实现监听Key的过期事件，从而实现延迟重新消费。\n6. 基于哈希时间轮盘算法(HashedWheelTimer)，如在Netty的心跳监测实现中使用了这种方案的延迟队列，基于JUC中的DelayQueue，多了一层轮盘算法封装。\n\n本中间件基于Redis实现延迟队列，期望设计分布式环境下安全、易用、高效的延迟队列实现，并提供给用户便捷的接口来对延迟队列进行操作。我将此延迟队列中间件命名为ReDeQ`['redik(ju:)]`。\n\n## Implementation\n\n- Java 8\n- redisson 3.16.1\n- spring-boot 2.3.4\n- redis server 5.0.3\n- gson 2.8.5\n- slf4j 1.7.25\n\n## Features\n\n- [x] 支持参数自动配置开箱即用\n- [x] 准实时性，基于定时扫描频率\n- [x] 高性能，支持多topic延迟队列消费\n- [x] 可靠性，延迟队列重试机制，保证至少一次消费\n- [x] 支持分布式集群、单机，使用Redisson分布式锁进行资源控制\n- [x] 重试机制\n- [x] 容量限制\n- [x] 增删作业时的命令批量执行\n- [x] 优雅结束服务\n- [ ] 消费能力心跳监测（当实例不具备消费能力时，应考虑降级处理，避免造成消息积压）\n- [ ] 完善的客户端（支持取消订阅主题、查看延迟队列元素等功能）\n\n## Installation\n\n```shell\nmvn clean compile install\n```\n\n## Usage\n\n### with spring-boot\n\nSee `redeq-spring-boot-example` example project. \n\n#### Dependency\n\n添加如下依赖\n\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.lidengju\u003c/groupId\u003e\n    \u003cartifactId\u003eredeq-spring-boot-starter\u003c/artifactId\u003e\n    \u003cversion\u003e1.1.1-ALPHA\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n#### Config\n\n```yaml\nredeq:\n  # 配置redis连接信息，配置后会注入RedissonClient，默认集群模式\n  redis:\n    hosts: 127.0.0.1:7001,127.0.0.1:7002,127.0.0.1:7003,127.0.0.1:7004,127.0.0.1:7005,127.0.0.1:7006 # redis hosts, 逗号分隔, 用于配置redisson\n    password: p@33word # redis password\n  # 下面非必须配置\n  app:\n    prefix: \"\" # 应用前缀，用于redis键前缀\n    verbose: false # 是否打印冗余日志\n    delay: 60 # 默认任务延迟时间，单位秒，默认60秒\n    retry: 3 # 默认重试次数，默认3次\n    schedule: 5 # 扫描延迟队列时间，默认5秒，减少提高实时性\n    poll-queue-timeout: 5 # 从就绪队列获取任务的阻塞等待时间，默认5秒\n    max-pool: 500000 # 作业池的最大容量，超过将抛出JOB_POOL_EXCEEDED异常，默认50万\n    max-subscribers: 10 # 最多允许订阅的数量\n    concurrency: 1 # 迁移作业的并行量，必须是2的幂\n  lock:\n    acquire-lock-timeout: 3 # 加锁阻塞时间，默认3秒\n    expire-lock-timeout: 20 # 加锁过期时间，默认20秒，过期时间比poll时间长，避免锁提前过期释放\n```\n\n#### Example\n\n**添加Redisson配置**(配置redeq.redis可自动注入)\n\n```java\n@Configuration\n@Slf4j\npublic class RedissonConfig {\n    @Value(\"${redeq.redis.hosts}\")\n    private String hosts;\n    @Value(\"${redeq.redis.password:}\")\n    private String password;\n\n    @Bean\n    public RedissonClient redissonClient() {\n        Config config = new Config();\n        String[] hostArray = Arrays.stream(hosts.split(\",\"))\n                .map(x -\u003e \"redis://\" + x.trim())\n                .toArray(String[]::new);\n        // redis集群模式 \n        config.useClusterServers().addNodeAddress(hostArray);\n        log.info(\"Redis cluster nodes added: {}\", Arrays.toString(hostArray));\n        return Redisson.create(config);\n    }\n}\n```\n\n**在需要重试的地方添加作业**\n\n```java\npublic class Service {\n    @Autowired\n    private RedeqClient redeqClient;\n\n    public void proc(Model model) {\n        try {\n            firstProc(model);\n        } catch (RedeqException e) {\n            DelayedJob.Builder delayedJobBuilder = new DelayedJob.Builder();\n            DelayedJob job = delayedJobBuilder\n                    .withBase(model.getTopic(), model.getId())\n                    .withBody(model)\n                    .build();\n            if (redeqClient.add(job) \u003e 0) {\n                continue;\n            }\n        }\n        sencondProc(model);\n    }\n}\n```\n\n\n**延迟任务消费**\n\n```java\n/**\n* 消费线程类，对于延迟任务消费情况进行处理\n*/\n@Component\n@Slf4j\npublic class TestConsumer {\n    @Autowired\n    private RedeqClient redeqClient;\n\n    @PostConstruct\n    private void init() {\n        // 当前消费线程要消费的主题\n        List\u003cString\u003e topics = Lists.newArrayList(\"testTopic\");\n        try {\n            redeqClient.subscribe(topics, new AbstractConsumeService() {\n                // 必须复写\n                @Override\n                public boolean onConsume(DelayedJob job) {\n                    log.info(\"正在消费[{}]\", job.getTopicId());\n                    Model model = (JsonToModel) job.getBody();\n                    try{\n        \t\tfirstProc(model);\n                  \t} catch(RedeqException e){\n                      log.info(\"first proc failed!\");\n                      // 返回false重试\n                      return false;\n                  \t}\n    \t\t\tsencondProc(model);\n                    return true;\n                }\n                \n                @Override\n                public void onSucceed(DelayedJob job) {\n                    log.info(\"成功后执行\");\n                }\n\n                @Override\n                public void onFailed(DelayedJob job) {\n                    log.info(\"超出重试次数后执行\");\n                    Model model = (JsonToModel) job.getBody();\n                    secondProc(model);\n                }\n\n                @Override\n                public void onRetry(DelayedJob job) {\n                    log.info(\"添加重试后执行\");\n                }\n            });\n        } catch (Exception e) {\n            log.error(\"启动Redeq消费线程失败\", e);\n        }\n    }\n}\n```\n\n### without spring-boot\n\nSee `redeq-example` example project.\n\nTODO \n\n## Design\n\n### Service Diagram\n\n```mermaid\ngraph TD;\n\tA[(Broker)] --\u003e |poll| B[Comsumer Service]\n\tB --\u003e C(comsuming)\n\tC --\u003e |process| D{Process succeed? }\n\tD --\u003e |Yes| E(Post Processing)\n\tD --\u003e |No| F[fa:fa-star ReDeQ]\n\tF -.-\u003e |Retry at next execution time| C\n```\n### ReDeQ-Design\n\n![image-20210909233309405](assets/redeq-design.png)\n\n\n\n\u003ccenter\u003eREDEQ设计架构图。\u003csmall\u003eCredit: Li, Dengju\u003c/small\u003e\u003c/center\u003e\n\n### Components\n\n- **Job Pool：** 作业池，基于redis中的hash数据结构实现，用于存放一个作业/消息的完整内容，用于消费时根据唯一id获取相应内容；\n- **Bucket Queue：** 延迟桶队列，本质为一个优先队列，基于redis中的sorted set数据结构实现，用于按待执行时间对待执行作业进行排序，定时将到期的作业迁移至就绪队列作为任务等待消费；通过对Job的id进行哈希并取余，将其放入特定编码的桶队列中，以实现不同实例并行迁移作业；\n- **Ready Queue：** 就绪队列，本质为一个无界单向队列，基于redis中的list数据结构实现，用于存放待消费的任务，供用户的消费线程进行消费；根据Job的topic不同，其会被放入不同的主题就绪队列，以实现多实例并行消费。\n- **Timer：** 用于控制从**Bucket Queue** 向**Ready Queue**迁移任务的周期，默认为每5秒，周期越短实时性越高，但是可能会产生性能影响；\n- **Client：** 客户端，提供了常用的API来增加作业、删除作业、获取待消费的任务、启动消费线程等；\n- **Consumer：** 消费者，提供了模板方法，便于用户编写消费成功、消费重试、消费失败的监听事件的相应处理\n\n### State Diagram\n\n```mermaid\nstateDiagram\nstate add_fork \u003c\u003cfork\u003e\u003e\nA : Job Pool\nB : Bucket Queue\n\t[*] --\u003e add_fork:ADD a new delayed job to\n\tadd_fork --\u003e A\n\tadd_fork --\u003e B\n```\n\n\u003ccenter\u003e向任务池和延迟队列添加一个延迟任务\u003c/center\u003e\n\n```mermaid\nstateDiagram\nstate del_fork \u003c\u003cfork\u003e\u003e\nA : Job Pool\nB : Bucket Queue\n\t[*] --\u003e del_fork:Remove a delayed job from\n\tdel_fork --\u003e A\n\tdel_fork --\u003e B\n```\n\n\u003ccenter\u003e从任务池和延迟队列移除一个延迟任务\u003c/center\u003e\n\n```mermaid\nstateDiagram\n[*] --\u003e A\nA : Bucket Queue\nB : Ready Queue\n\tA --\u003e B:Schedule - transfering the expired job to ready queue\n```\n\n\u003ccenter\u003e固定时间周期将到期的延迟任务放入就绪队列\u003c/center\u003e\n\n```mermaid\nstateDiagram\n[*] --\u003e A\nA : Ready Queue\nB : Consumer(with topic)\n\tA --\u003e B:Consume - poll a task from the specified topical ready queue\nB --\u003e[*]\n```\n\n\u003ccenter\u003e从特定主题就绪队列获取一个任务，没有则阻塞等待\u003c/center\u003e\n\n## FAQ\n\n1. 如何多实例部署？\n\n   \u003e ReDeQ依赖Redisson实现的分布式锁进行资源控制，有关redis的运行模式由Redisson进行管理。\n\n2. 作业如何并行消费？\n\n   \u003e 为作业设置不同的Topic，根据实例数增加ReDeQ的`concurrency`数值，作业会根据其jobId，通过hash并对`concurrency`取模后，得到`[0, concurrency)`的下标`n`，其会被路由到`prefix`+`REDEQ:BUCKET`+`n`对应的队列中；注意，此数值减小后，若原本的Bucket Queue还有作业未被消费，则其可能不会被消费，应考虑在重新部署应用手动将原Bucket Queue中的作业迁移到现在的首个队列，如`REDEQ:BUCKET0`中。\n\n3. 作业如何被路由？\n\n   \u003e `DelayedJob`的`srcId`属性（其值等于初始的`jobId`），通过如下哈希取模得到路由下标：\n   \u003e\n   \u003e ```java\n   \u003e // 1. 取绝对值\n   \u003e // 2. 通过\u0026与位运算替代取模%，前提是modBy满足(2^n)\n   \u003e routeId = (srcId.hashCode() \u0026 Integer.MAX_VALUE) \u0026 (modBy-1);\n   \u003e // 3. 为使modBy满足条件，因此在设置modBy即concurrency的值时，获取其最接近的2的幂\n   \u003e // get the closest power of 2 according given number\n   \u003e concurrency = Math.max(1, concurrency);\n   \u003e concurrency = (int) Math.pow(2, Math.floor(Math.log(concurrency)/Math.log(2)));\n   \u003e ```\n\n4. 为什么重试作业需要重命名作业ID？\n\n   \u003e 在**添加**和**删除**作业时，会使用单个作业的`jobId`添加可重入锁，不同作业添加的过程不会产生冲突；但当执行**迁移**操作时，其是对整个迁移过程加锁，本质是执行往**就绪队列**增加和从**桶队列**删除的操作，这个过程不是原子事务执行的，其中可能会有其他指令插入，若不重命名，在根据key删除时，可能会将此key误删除；同理，若插入相同jobId作业，也有可能造成此问题。\n   \u003e\n   \u003e 若根据score(即时间戳)对**桶队列**进行删除，则要注意由于不同实例的时钟偏移或时间延迟造成误删除。\n   \u003e\n   \u003e 注意，redis的`zset`数据结构根据**score**删除的时间复杂度为`O(log(N) + M)`，而根据**value**删除的时间复杂度是`O(M * log(N))` M是删除的个数。\n\n## References\n\n- [Redisson参考手册](http://bookstack.cn/read.redisson-wiki-zh)\n- [有赞延迟队列](http://tech.youzan.com/queuing_delay)\n\n## License\n\n[MIT](./LICENSE)开源协议\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkevinleeex%2Fredeq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkevinleeex%2Fredeq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkevinleeex%2Fredeq/lists"}