{"id":26546874,"url":"https://github.com/changsongl/delay-queue","last_synced_at":"2025-03-22T05:28:12.877Z","repository":{"id":42178710,"uuid":"325749588","full_name":"changsongl/delay-queue","owner":"changsongl","description":"延迟队列，高可用。Delay Queue","archived":false,"fork":false,"pushed_at":"2022-03-19T16:35:01.000Z","size":869,"stargazers_count":38,"open_issues_count":6,"forks_count":11,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-12-18T08:39:30.463Z","etag":null,"topics":["delay-queue","delayed-job","delayqueue","golang","redis"],"latest_commit_sha":null,"homepage":"https://tech.youzan.com/queuing_delay/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/changsongl.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-12-31T08:07:51.000Z","updated_at":"2024-07-04T09:07:35.000Z","dependencies_parsed_at":"2022-08-31T13:42:46.763Z","dependency_job_id":null,"html_url":"https://github.com/changsongl/delay-queue","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/changsongl%2Fdelay-queue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/changsongl%2Fdelay-queue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/changsongl%2Fdelay-queue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/changsongl%2Fdelay-queue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/changsongl","download_url":"https://codeload.github.com/changsongl/delay-queue/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244912802,"owners_count":20530764,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["delay-queue","delayed-job","delayqueue","golang","redis"],"created_at":"2025-03-22T05:28:12.374Z","updated_at":"2025-03-22T05:28:12.820Z","avatar_url":"https://github.com/changsongl.png","language":"Go","readme":"# delay-queue\n\n[![Go Report Card](https://goreportcard.com/badge/github.com/changsongl/delay-queue)](https://goreportcard.com/report/github.com/changsongl/delay-queue)\n[![Build Status](https://travis-ci.com/changsongl/delay-queue.svg?branch=main)](https://travis-ci.com/changsongl/delay-queue)\n[![Hits](https://hits.seeyoufarm.com/api/count/incr/badge.svg?url=https%3A%2F%2Fgithub.com%2Fchangsongl%2Fdelay-queue\u0026count_bg=%232BBC8A\u0026title_bg=%23555555\u0026icon=artstation.svg\u0026icon_color=%23C7C7C7\u0026title=Visitor\u0026edge_flat=false)](https://hits.seeyoufarm.com)\n\nTranslations:\n\n- [中文文档](./README_ZH.md)\n\n````\n  ____       _                ___                        \n |  _ \\  ___| | __ _ _   _   / _ \\ _   _  ___ _   _  ___ \n | | | |/ _ \\ |/ _` | | | | | | | | | | |/ _ \\ | | |/ _ \\\n | |_| |  __/ | (_| | |_| | | |_| | |_| |  __/ |_| |  __/\n |____/ \\___|_|\\__,_|\\__, |  \\__\\_\\\\__,_|\\___|\\__,_|\\___|\n                     |___/   \n````\n\n### Introduction\nThis project is a delay queue. It is based on Youzan 有赞 delay queue. Currently,\nit is based on Redis for storage.\n\n### Highly Available\n![arch](./doc/arch/Delay-Queue-HA.jpg)\n\n### How to run the delay queue?\n````shell\n# clone project\ngit clone https://github.com/changsongl/delay-queue.git\n\n# build the project\nmake\n\n# run the project\nbin/delayqueue\n````\n\n````shell\n# flags\nbin/delayqueue -help\n  -config.file string\n        config file (default \"../../config/config.yaml\")\n  -config.type string\n        config type: yaml, json\n  -env string\n        delay queue env: debug, release (default \"release\")\n  -version\n        display build info\n````\n\nThe default configuration file is `config/config.yaml.example`.\n\n### Usage\n- ##### SDK [Link](https://github.com/changsongl/delay-queue-client)\n\n- ##### Http\n\n````\n// Push job\nPOST 127.0.0.1:8000/topic/mytopic/job\nbody: {\"id\": \"myid1\",\"delay\":10, \"ttr\":4, \"body\":\"body\"}\n\n// response \n{\n    \"message\": \"ok\",\n    \"success\": true\n}\n````\n\n````\n// Pop job (timeout: seconds)\nGET 127.0.0.1:8000/topic/mytopic/job?timeout=5\n\n// response\n{\n    \"message\": \"ok\",\n    \"success\": true,\n    \"data\": {\n        \"body\": \"body\",\n        \"delay\": 10,\n        \"id\": \"myid1\",\n        \"topic\": \"mytopic\",\n        \"ttr\": 4\n    }\n}\n````\n\n````\n// Delete job\nDELETE 127.0.0.1:8000/topic/mytopic/job/myid1\n\n// response\n{\n    \"message\": \"ok\",\n    \"success\": true\n}\n````\n\n````\n// Delete job\nPUT 127.0.0.1:8000/topic/mytopic/job/myid1\n\n// response\n{\n    \"message\": \"ok\",\n    \"success\": true\n}\n````\n\n### Designs\n\n#### Terms\n1. Job: It is a task to be processed, and it is related to only one topic.\n2. Topic: It is a set of jobs, it is implemented by a time-sorted queue.\n All consumers need to choose at least one topic to consume jobs.\n\n#### Job\nJobs contain many properties like:\n1. Topic: It could be a service name, users can define it depending on their\n business.\n2. ID: it is unique key for inside of a topic. It's used to search job information\n in a topic. The combination of a topic and an ID should be unique in your\n business.\n3. Delay: It defines how many second to be delay for the job. Unit: Second\n4. TTR(time to run): It is job processing timeout. If consumer process this\n job more than TTR seconds, it might be sent to other consumer, if a consumer\n pop the topic.\n5. Body: It is content of job. It is a string. You can put your json data to it.\n When you consume the job, you can decode it and run your logic.\n\n\n#### Component\n\n\u003eThere are 4 components in the delay queue.\n\u003e1. Job Pool: It saves all metadata of jobs.\n\u003e2. Delay Bucket: It is a time-sorted queue. It saves jobs that is waiting\n for being ready. There are more than one Bucket in the delay queue for\n higher throughput.\n\u003e3. Timer: It is a core component to scan the Delay Bucket. It pops out \n ready jobs from Buckets and put then inside ready queue.\n\u003e4. Ready Queue: It is a queue for storing all ready jobs, which can be\n popped now. It is also only store the job id for the consumers.\n\n\u003cimg alt=\"delay-queue\" src=\"https://tech.youzan.com/content/images/2016/03/delay-queue.png\" width=\"80%\"\u003e\n\n#### States\n\u003eThere are four states for jobs in the delay queue. The job can be only\n\u003e in one state at the time.\n\u003e1. Ready: It is ready to be consumed.\n\u003e2. Delay: It is waiting for the delay time, and it can't be consumed.\n\u003e3. reserved: It means the job has consumed by a consumer, but consumer\n\u003e hasn't ack the job. (Call delete、finish).\n\u003e4. Deleted: The job has finished or deleted.\n\n\u003cimg alt=\"job-state\" src=\"https://tech.youzan.com/content/images/2016/03/job-state.png\" width=\"80%\"\u003e\n\n### Monitor\nThis project is using Prometheus as the monitor tool. It exposes the metrics apis to Prometheus.\nYou can use Prometheus and Grafana as the monitor tools.\n\n````\n# HELP delay_queue_in_flight_jobs_numbers_in_bucket Gauge of the number of inflight jobs in each bucket\n# TYPE delay_queue_in_flight_jobs_numbers_in_bucket gauge\ndelay_queue_in_flight_jobs_numbers_in_bucket{bucket=\"dq_bucket_0\"} 0\ndelay_queue_in_flight_jobs_numbers_in_bucket{bucket=\"dq_bucket_1\"} 3\ndelay_queue_in_flight_jobs_numbers_in_bucket{bucket=\"dq_bucket_2\"} 0\ndelay_queue_in_flight_jobs_numbers_in_bucket{bucket=\"dq_bucket_3\"} 0\ndelay_queue_in_flight_jobs_numbers_in_bucket{bucket=\"dq_bucket_4\"} 0\ndelay_queue_in_flight_jobs_numbers_in_bucket{bucket=\"dq_bucket_5\"} 0\ndelay_queue_in_flight_jobs_numbers_in_bucket{bucket=\"dq_bucket_6\"} 0\ndelay_queue_in_flight_jobs_numbers_in_bucket{bucket=\"dq_bucket_7\"} 0\n.\n.\n.\n# HELP delay_queue_in_flight_jobs_numbers_in_queue Gauge of the number of inflight jobs in each queue\n# TYPE delay_queue_in_flight_jobs_numbers_in_queue gauge\ndelay_queue_in_flight_jobs_numbers_in_queue{queue=\"dq_queue_mytopic\"} 1\n````\n\n### What's the plan of this project?\nI will work on this project all the time! I will add more features and \n fix bugs, and I will make this project ready to use in production. Star\n Or Fork it if you like it. I'm very welcome to you for contribution.\n \n### How to contribute?\n1. Level a message in the unsigned issue.\n2. We will discuss how to do it, and I will assign the issue to you.\n3. Fork the project, and checkout your branch from \"develop\" branch.\n4. Submit the PR to \"develop\" branch.\n5. It will be merged after code review.\n\n### Stargazers\n[![Stargazers over time](https://starchart.cc/changsongl/delay-queue.svg)](https://starchart.cc/changsongl/delay-queue)\n\n### Reference\n\nYouzan Design Concept [Youzan Link](https://tech.youzan.com/queuing_delay/)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchangsongl%2Fdelay-queue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchangsongl%2Fdelay-queue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchangsongl%2Fdelay-queue/lists"}