{"id":19429184,"url":"https://github.com/ehsaniara/delay-box","last_synced_at":"2026-03-01T12:31:44.814Z","repository":{"id":247042532,"uuid":"822930872","full_name":"ehsaniara/delay-box","owner":"ehsaniara","description":"This tool simplifies the workflow by removing the need to write code to handle Redis and Kafka development complexities. It manages these tasks for you through straightforward REST calls.","archived":false,"fork":false,"pushed_at":"2025-04-01T16:43:05.000Z","size":7800,"stargazers_count":2,"open_issues_count":4,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-14T16:57:39.802Z","etag":null,"topics":["delayed-job","devops","distributed-systems","kafka","redis","scheduled-tasks","sre"],"latest_commit_sha":null,"homepage":"https://ehsaniara.github.io/delay-box/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ehsaniara.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-07-02T05:41:21.000Z","updated_at":"2025-04-01T16:41:58.000Z","dependencies_parsed_at":"2024-08-23T05:20:11.047Z","dependency_job_id":"f3afa807-b1f0-4978-9e73-1651281e2e21","html_url":"https://github.com/ehsaniara/delay-box","commit_stats":null,"previous_names":["ehsaniara/scheduler","ehsaniara/scheduler-worker","ehsaniara/delay-box"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ehsaniara/delay-box","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ehsaniara%2Fdelay-box","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ehsaniara%2Fdelay-box/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ehsaniara%2Fdelay-box/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ehsaniara%2Fdelay-box/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ehsaniara","download_url":"https://codeload.github.com/ehsaniara/delay-box/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ehsaniara%2Fdelay-box/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29969243,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-01T11:43:06.159Z","status":"ssl_error","status_checked_at":"2026-03-01T11:43:03.887Z","response_time":124,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["delayed-job","devops","distributed-systems","kafka","redis","scheduled-tasks","sre"],"created_at":"2024-11-10T14:18:20.300Z","updated_at":"2026-03-01T12:31:44.788Z","avatar_url":"https://github.com/ehsaniara.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DelayBox\n\n\n[![Go](https://github.com/ehsaniara/delay-box/actions/workflows/go.yml/badge.svg?branch=main)](https://github.com/ehsaniara/delay-box/actions/workflows/go.yml) [![Go Report Card](https://goreportcard.com/badge/github.com/ehsaniara/delay-box)](https://goreportcard.com/report/github.com/ehsaniara/delay-box)\n\n\n**DelayBox** is a High Throughput Distributed **ONE-TIME** Task Scheduler ⚡. It is an advanced system designed to manage and execute a vast number of tasks across a distributed network of servers with **sub-second accuracy**. Built on top of the **Redis** database, it leverages Redis's high-speed in-memory data store 🔥 for quick access and efficient task management.\n\n# Why DelayBox\n\n**DelayBox** is a concept coming from the bigger domain of scheduling and handling delays in distributed systems, mainly towards the improvement of resilience, consistency, and fault tolerance of distributed applications. It is a middleware component that can be used to buffer messages or events with the intention of creating delays to manage distributed timing issues.\n\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/diagram2.svg\" alt=\"Flow Architecture\" /\u003e\n\u003c/p\u003e\n\n## Read more\n\nHere’s how DelayBox can help in designing a distributed system [click for more.](WhyDelayBox.md)\n1. Handling Eventual Consistency: [click for more.](https://github.com/ehsaniara/delay-box/blob/main/WhyDelayBox.md#1-handling-eventual-consistency)\n2. Mitigating Network Partitions: [click for more.](https://github.com/ehsaniara/delay-box/blob/main/WhyDelayBox.md#2-mitigating-network-partitions)\n3. Load Management and Throttling: [click for more.](https://github.com/ehsaniara/delay-box/blob/main/WhyDelayBox.md#3-load-management-and-throttling)\n4. Failure Recovery and Redundancy: [click for more.](https://github.com/ehsaniara/delay-box/blob/main/WhyDelayBox.md#4-failure-recovery-and-redundancy)\n5. Decoupling Components in Event-Driven Architecture: [click for more.](https://github.com/ehsaniara/delay-box/blob/main/WhyDelayBox.md#5-decoupling-components-in-event-driven-architecture)\n6. Improving Reliability in Asynchronous Workflows: [click for more.](https://github.com/ehsaniara/delay-box/blob/main/WhyDelayBox.md#6-improving-reliability-in-asynchronous-workflows)\n7. Dealing with Temporal Anomalies: [click for more.](https://github.com/ehsaniara/delay-box/blob/main/WhyDelayBox.md#7-dealing-with-temporal-anomalies)\n8. Application in Geo-Distributed Systems: [click for more.](https://github.com/ehsaniara/delay-box/blob/main/WhyDelayBox.md#8-application-in-geo-distributed-systems)\n\n\n\n## General Idea\n\nDelayBox consists of 2 topics (`Scheduler` and `Executor`)\n\n# Use-Case\n\n🚀 This tool simplifies the workflow for niche task execution by removing the need to write code to handle Redis and Kafka development complexities. It manages these tasks for you through straightforward REST calls.\n\n✅ It is ideal for applications requiring massive parallel processing capabilities, such as data processing pipelines, large-scale simulations, and real-time analytics.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/WorkFlow-1.png\" alt=\"General Architecture\"/\u003e\n\u003c/p\u003e\n\n# Examples\n## Docker Compose\n\n### 3 Worker Nodes with Kafka\n\nclone the project \n```shell\ncd docker-multi-worker\ndocker-compose -f docker-compose.yml up -d\n```\nWait until all worker nodes are up and running (you'll see in their console: \"🚀 scheduler is ready!\"). Then run the following command to create 1000 tasks which designed to execute in 10 second. (tasks just simply print the date in the worker nodes console)\n\n```shell\nsh ./create-task.sh\n```\n**Note:** your terminal console will only print  `{\"message\":\"task created\"}` 1000 times, but the worker consoles shows the date. (in this example its docker logs)\n\nYou can try the docker compose example with 3 worker nodes example [docker-compose example with kafka](./docker-multi-worker) \n\n### 3 Worker Nodes \n\nKeep in mind that in this setup Delay-Box gets advantage og redis pub sub, so you are limited to single Redis node.\n\nclone the project\n```shell\ncd docker-multi-worker-no-kafka\nexport COMPOSE_PROJECT_NAME=delay-box\ndocker-compose -p delay-box -f docker-compose.yml up -d\n```\nWait until all worker nodes are up and running (you'll see in their console: \"🚀 scheduler is ready!\"). Then run the following command to create 1000 tasks which designed to execute in 10 second. (tasks just simply print the date in the worker nodes console)\n\n```shell\nsh ./create-task.sh\n```\n**Note:** your terminal console will only print  `{\"message\":\"task created\"}` 1000 times, but the worker consoles shows the date. (in this example its docker logs)\n\nYou can try the docker compose example with 3 worker nodes example [Docker-compose example without kafka](./docker-multi-worker-no-kafka)\n\n\n## Local Example\n\nFollowing task is type of `SHELL_CMD` which mean it will execute at any defined worker nodes on the given timestamp, (Payload is just the OS Date command)\n```shell\ncurl -X POST http://localhost:8088/task  \\\n  -H \"Content-Type: application/json\" -d \\\n  '{\"parameter\":{\"executionTimestamp\":\"1720672590913\",\"taskType\":\"SHELL_CMD\"},\"pyload\":\"ZGF0ZQ==\"}'\n```\n\nTo Get list of pending tasks (First 100 tasks)\n```shell\ncurl \"http://localhost:8088/task\"\n```\n\n### pyload\nThis fild is stored as byte format and published in kafka topic `taskExecutionTopic`\n\n\n### parameter\n\n| parameter name     | type   | required | description                                                                                                  |\n|--------------------|--------|----------|--------------------------------------------------------------------------------------------------------------|\n| taskType           | string | YES      | Defines the task type [Type of task](#taskType)                                                              |\n| executionTimestamp | number | NO       | With this parameter, the task is expected to be executed at the specified Unix epoch time (in milliseconds). |\n| delay              | number | NO       | With this parameter, to delay task execution in millisecond.                                                 |\n\n**Note:** If neither `executionTimestamp` nor `delay` is provided, the task will be executed immediately.\n\n\n### taskType\n\n| name      | description                                                                                                                                                                                                                                                                                                                 |\n|-----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| PUB_SUB   | This type is the basic schedule task, which you just publish payload in kafka topic `schedulerTopic` and  the payload will be published kafka topic `taskExecutionTopic` when its scheduled to be executed. \u003cbr/\u003e**Note:** `schedulerTopic` and `taskExecutionTopic` are already configured in the application config file. |\n| SHELL_CMD | In this type, your payload, which is a Linux command, will be executed. \u003cbr/\u003e**Note:** If you expect to run any application, it must be pre-installed on the worker machine prior to task execution.                                                                                                                        |\n\n\n\n# General Architecture\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/diagram1.png\" alt=\"General Architecture\"/\u003e\n\u003c/p\u003e\n\n\n## Enable Kafka\n\nTo enable kafka support add the following config.\n```shell\nkafka:\n  enabled: true\n  brokers: localhost:9092\n  group: scheduler\n  schedulerTopic: Scheduler\n  taskExecutionTopic: TaskExecution\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fehsaniara%2Fdelay-box","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fehsaniara%2Fdelay-box","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fehsaniara%2Fdelay-box/lists"}