{"id":23507374,"url":"https://github.com/simplyhexagonal/recurring-task-queue","last_synced_at":"2025-05-12T15:51:05.453Z","repository":{"id":57160411,"uuid":"375234288","full_name":"simplyhexagonal/recurring-task-queue","owner":"simplyhexagonal","description":"Versatile type-safe task queueing library for recurring user-editable tasks.","archived":false,"fork":false,"pushed_at":"2021-10-15T04:53:59.000Z","size":344,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-06T20:33:52.553Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/simplyhexagonal.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-06-09T05:17:18.000Z","updated_at":"2024-01-10T08:24:07.000Z","dependencies_parsed_at":"2022-08-26T15:22:25.144Z","dependency_job_id":null,"html_url":"https://github.com/simplyhexagonal/recurring-task-queue","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simplyhexagonal%2Frecurring-task-queue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simplyhexagonal%2Frecurring-task-queue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simplyhexagonal%2Frecurring-task-queue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simplyhexagonal%2Frecurring-task-queue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/simplyhexagonal","download_url":"https://codeload.github.com/simplyhexagonal/recurring-task-queue/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253768478,"owners_count":21961335,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-25T10:18:26.525Z","updated_at":"2025-05-12T15:51:05.429Z","avatar_url":"https://github.com/simplyhexagonal.png","language":"TypeScript","funding_links":["https://www.buymeacoffee.com/jeanlescure","https://opencollective.com/simplyhexagonal"],"categories":[],"sub_categories":[],"readme":"# Recurring Task Queue (RTQ)\n![Tests](https://github.com/simplyhexagonal/recurring-task-queue/workflows/tests/badge.svg)\n![Compatible with Typescript versions 4+](https://img.shields.io/badge/Typescript-4%2B-brightgreen)\n![Compatible with Node versions 14+](https://img.shields.io/badge/Node-14%2B-brightgreen)\n\n![Compatible with Chrome versions 60+](https://img.shields.io/badge/Chrome-60%2B-brightgreen)\n![Compatible with Firefox versions 60+](https://img.shields.io/badge/Firefox-60%2B-brightgreen)\n![Compatible with Safari versions 12+](https://img.shields.io/badge/Safari-12%2B-brightgreen)\n![Compatible with Edge versions 18+](https://img.shields.io/badge/Edge-18%2B-brightgreen)\n\nVersatile type-safe queueing library for a finite set of **recurring** user-editable tasks.\n\n![](https://raw.githubusercontent.com/simplyhexagonal/recurring-task-queue/main/assets/task-status-cycle-1.0.png)\n\n## Open source notice\n\nThis project is open to updates by its users, [I](https://github.com/jeanlescure) ensure that PRs are relevant to the community.\nIn other words, if you find a bug or want a new feature, please help us by becoming one of the\n[contributors](#contributors-) ✌️ ! See the [contributing section](#contributing)\n\n## Like this module? ❤\n\nPlease consider:\n\n- [Buying me a coffee](https://www.buymeacoffee.com/jeanlescure) ☕\n- Supporting Simply Hexagonal on [Open Collective](https://opencollective.com/simplyhexagonal) 🏆\n- Starring this repo on [Github](https://github.com/simplyhexagonal/recurring-task-queue) 🌟\n\n## Abstract\n\nLet's say you have been put in charge of developing a recurring task which:\n\n- runs every five minutes,\n- loads all images uploaded in the past 5 minutes into memory and processes them\n- has the ability to edit the path to the location of the images\n- displays the status of each 5 minute run to measure performance and catch any failed attempts\n\nThe first two items on the list are pretty easy to solve for by setting up a CloudWatch Event that\nevery 5 minutes calls an end-point on your REST API which will perform the task.\n\nIn regards to the other two items your intuition dictates that to be able to allow the users to\nedit the path to the location of the images, you should store the task definitions on the app's\ndatabase, which would also make sense to store the status and maybe even a log for each task run.\n\nYou also want to make sure that, in the future, when multiple recurring tasks are defined, they can\nbe queued and processed without any additional development.\n\nWith the previous in mind you end up with a diagram similar to this:\n\n![](https://raw.githubusercontent.com/simplyhexagonal/recurring-task-queue/main/assets/mock-diagram.png)\n\nNow you can specify the following sub-requirements to complete the task:\n\n- define the structure of task definitions which will be stored\n- track and store the status of each task\n- make sure that no matter how many times the REST end-point is called, only one instance of the task will run at a time\n- there needs to be proper error handling all the way to avoid having a situation where the app dies every 5 minutes due to an unforeseen error\n\nSome nice-to-haves would be:\n\n- having the tasks retry if they fail\n- be able to set a max number of retries\n- when more than one task is defined, on each call to the end-point any task that has completed will run again, and any task still running will be left as is\n- if a task depends on a third-party API with strict rate limits, you can specify in the task definition a wait period between runs to avoid hitting said rate limit\n- be able to send notifications when a task reaches the maximum number of retries and is flagged as `FAILED`\n\nThe good news is, `recurring-task-queue` (`RTQ`) handles all of the above for you!\n\n## Setup\n\nInstall:\n\n```sh\npnpm i @simplyhexagonal/recurring-task-queue\n\n# or\nyarn add @simplyhexagonal/recurring-task-queue\n\n# or\nnpm install @simplyhexagonal/recurring-task-queue\n```\n\nDefine a task handler:\n\n```ts\nimport { RTQTaskHandler } from '@simplyhexagonal/recurring-task-queue';\n\ninterface imgProcTaskOptions {\n  imgLocation: string;\n}\n\nconst imgProcTaskHandler: RTQTaskHandler\u003cimgProcTaskOptions\u003e  = async (taskOptions) =\u003e {\n  const rawImages = await loadImages(taskOptions.imgLocation);\n\n  return await processImages(rawImages);\n}\n```\n\nStore a task in a data source:\n\n```ts\nimport {\n  RTQTask,\n  RTQStatus,\n} from '@simplyhexagonal/recurring-task-queue';\n\n// NOTE: in real-world scenarios this object would be\n// generated from user input\nconst imgProcTaskDefinition: RTQTask\u003cimgProcTaskOptions\u003e = {\n    id: uid(),\n    status: RTQStatus.NEW,\n    waitTimeBetweenRuns: 200,\n    taskName: 'Image Processing',\n    maxRetries: 1,\n    retryCount: 0,\n    // since the task has never run, simply set \n    // the lastRun date to 1970-01-01T00:00:00.000Z\n    lastRun: new Date(0),\n    taskOptions: {\n      imgLocation: 'some/image/location/path',\n    },\n};\n\n// This would be your custom function which handles\n// saving tasks in your data source\ncreateTask(imgProcTaskDefinition);\n```\n\nNow you will need to instantiate `RTQ`  with the appropriate options to access your stored\ntask/queue data, the task handler you defined, an event handler, and your\ncustom error handling:\n\n```ts\nimport RTQ, {\n  RTQOptions,\n  RTQTask,\n  RTQQueueEntry,\n  RTQTaskHandler,\n} from '@simplyhexagonal/recurring-task-queue';\n\nconst options = RTQOptions {\n  fetchTasks: async () =\u003e { /* return RTQTask\u003cimgProcTaskOptions\u003e[] */ },\n  updateTask: async (task: RTQTask\u003cimgProcTaskOptions\u003e) =\u003e { /* ... */},\n  createQueueEntry: async (queueEntry: RTQQueueEntry) =\u003e { /* ... */},\n  fetchQueueEntries: async () =\u003e { /* return RTQQueueEntry[] */ },\n  removeQueueEntry: async (queueEntry: RTQQueueEntry) =\u003e { /* return RTQQueueEntry[] */ },\n  taskHandlers: [\n    imgProcTaskHandler,\n  ],\n  eventHandler: async (event: RTQEvent) =\u003e { /* ... */ },\n  errorHandler: (error: any) =\u003e { /* ... */ },\n  maxConcurrentTasks: 10, // leave undefined to have no limit\n}\n\nconst recurring = new RTQ(options);\n```\n\n## Ticking\n\nBased on the setup described in the previous section we ended up with the following RTQ instance:\n\n```ts\nconst recurring = new RTQ(options);\n```\n\nIt is important to remember that RTQ handles a queue of tasks which it processes using the task\nhandlers you define, nothing more. As such, to begin queuing and processing the tasks your must\nrun RTQ's `tick()` method.\n\nWe do NOT recomend using loops or intervals within your app to `tick` your task queue, but rather\nset an end-point which can be periodically called by another process, for example:\n\n```ts\nconst server = fastify();\n\nserver.route({\n  method: 'GET',\n  url: '/api/process-images',\n  handler: async () =\u003e {\n    recurring.tick(); // \u003c= This is where the magic happens\n\n    return 'Processing images...';\n  },\n});\n```\n\n## Event handling\n\nThere are two types of events defined by the actions RTQ performs on tasks and the queue:\n\n```ts\nenum RTQAction {\n  MODIFY_TASK_STATUS = 'MODIFY_TASK_STATUS',\n  MODIFY_QUEUE = 'MODIFY_QUEUE',\n}\n```\n\nThe event itself carries a lot more information about the action performed:\n\n```ts\ninterface RTQEvent {\n  timestamp: Date;\n  action: RTQActionEnum;\n  message: string;\n  reason: string;\n  additionalData: {[k: string]: any};\n  triggeredBy: string;\n}\n```\n\nThe `additionalData` varies depending on the action:\n\n```ts\n// if (action === RTQAction.MODIFY_TASK_STATUS)\nadditionalData = {\n  taskId,\n  taskName,\n  prevStatus,\n  status,\n}\n\n// if (action === RTQAction.MODIFY_QUEUE)\nadditionalData = {\n  id,\n  taskId,\n  queuedAt,\n}\n```\n\nSo, for example let's say you want to send a notification if a task's status changes to `FAILED`,\nthen you would define your event handler like this:\n\n```ts\nconst eventHandler = async (event: RTQEvent) =\u003e {\n  const {\n    action,\n    additionalData,\n  } = event;\n\n  if (\n    action === RTQAction.MODIFY_TASK_STATUS\n    \u0026\u0026 additionalData.status === RTQStatus.FAILED\n  ) {\n    makeSureTowelIsAtHand();\n    dontPanic();\n    notifyEveryLastOneOfUs();\n  }\n}\n```\n\n## WIP\n\n- **Documentation**\n\nIn the mean-time you can see more detailed use cases in the [examples](https://github.com/simplyhexagonal/recurring-task-queue/tree/main/examples) and the [jest tests](https://github.com/simplyhexagonal/recurring-task-queue/blob/main/src/index.test.ts).\n\n## Development\n\n```\npnpm\npnpm dev\npnpm test\npnpm build\npnpm release\n```\n\n## Contributing\n\nYes, thank you! This plugin is community-driven, most of its features are from different authors.\nPlease update the tests and don't forget to add your name to the `package.json` file.\n\n## Contributors ✨\n\nThanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section --\u003e\n\u003c!-- prettier-ignore-start --\u003e\n\u003c!-- markdownlint-disable --\u003e\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"center\"\u003e\u003ca href=\"https://jeanlescure.cr\"\u003e\u003cimg src=\"https://avatars2.githubusercontent.com/u/3330339?v=4\" width=\"100px;\" alt=\"\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eJean Lescure\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"#maintenance-jeanlescure\" title=\"Maintenance\"\u003e🚧\u003c/a\u003e \u003ca href=\"https://github.com/simplyhexagonal/recurring-task-queue/commits?author=jeanlescure\" title=\"Code\"\u003e💻\u003c/a\u003e \u003ca href=\"#userTesting-jeanlescure\" title=\"User Testing\"\u003e📓\u003c/a\u003e \u003ca href=\"https://github.com/simplyhexagonal/recurring-task-queue/commits?author=jeanlescure\" title=\"Tests\"\u003e⚠️\u003c/a\u003e \u003ca href=\"#example-jeanlescure\" title=\"Examples\"\u003e💡\u003c/a\u003e \u003ca href=\"https://github.com/simplyhexagonal/recurring-task-queue/commits?author=jeanlescure\" title=\"Documentation\"\u003e📖\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003c!-- markdownlint-enable --\u003e\n\u003c!-- prettier-ignore-end --\u003e\n\u003c!-- ALL-CONTRIBUTORS-LIST:END --\u003e\n\n## License\n\nCopyright (c) 2021-Present [RTQ Contributors](https://github.com/simplyhexagonal/recurring-task-queue/#contributors-).\u003cbr/\u003e\nLicensed under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimplyhexagonal%2Frecurring-task-queue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsimplyhexagonal%2Frecurring-task-queue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimplyhexagonal%2Frecurring-task-queue/lists"}