{"id":22489862,"url":"https://github.com/conveyor-mq/conveyor-mq","last_synced_at":"2025-08-02T22:31:28.633Z","repository":{"id":37802587,"uuid":"266829922","full_name":"conveyor-mq/conveyor-mq","owner":"conveyor-mq","description":"A fast, robust and extensible distributed task/job queue for Node.js, powered by Redis.","archived":false,"fork":false,"pushed_at":"2023-11-01T13:14:06.000Z","size":2814,"stargazers_count":50,"open_issues_count":15,"forks_count":3,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-04-23T20:04:52.122Z","etag":null,"topics":["distributed","job","job-queue","job-scheduler","jobs","node","nodejs","queue","redis","task","task-queue","task-scheduler","worker","worker-queue"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/conveyor-mq.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-05-25T16:38:31.000Z","updated_at":"2024-02-13T02:37:40.000Z","dependencies_parsed_at":"2023-02-07T20:46:10.190Z","dependency_job_id":null,"html_url":"https://github.com/conveyor-mq/conveyor-mq","commit_stats":null,"previous_names":[],"tags_count":19,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/conveyor-mq%2Fconveyor-mq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/conveyor-mq%2Fconveyor-mq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/conveyor-mq%2Fconveyor-mq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/conveyor-mq%2Fconveyor-mq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/conveyor-mq","download_url":"https://codeload.github.com/conveyor-mq/conveyor-mq/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228500536,"owners_count":17930084,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed","job","job-queue","job-scheduler","jobs","node","nodejs","queue","redis","task","task-queue","task-scheduler","worker","worker-queue"],"created_at":"2024-12-06T17:20:58.572Z","updated_at":"2024-12-06T17:23:21.579Z","avatar_url":"https://github.com/conveyor-mq.png","language":"TypeScript","funding_links":[],"categories":["TypeScript"],"sub_categories":[],"readme":"# Conveyor MQ\n\nA fast, robust and extensible distributed task/job queue for Node.js, powered by Redis.\n\n![Tests](https://github.com/jasrusable/conveyor-mq/workflows/Tests/badge.svg)\n![npm](https://img.shields.io/npm/v/conveyor-mq)\n[![Coverage Status](https://coveralls.io/repos/github/jasrusable/conveyor-mq/badge.svg?branch=master)](https://coveralls.io/github/jasrusable/conveyor-mq?branch=master)\n\n## Introduction\n\nConveyor MQ is a general purpose, distributed task/job queue for Node.js, powered by Redis.\n\nConveyor MQ implements a [reliable queue](https://redis.io/commands/rpoplpush#pattern-reliable-queue) which provides strong guarantees around the reliability of tasks in the event of network or server errors, for example. Conveyor MQ offers [at-least-once](https://www.cloudcomputingpatterns.org/at_least_once_delivery/) and [exactly-once](https://www.cloudcomputingpatterns.org/exactly_once_delivery/) task delivery through the use of error or stall task retries. Conveyor MQ is implemented using a highly efficient and performant, polling-free design making use of [`brpoplpush`](https://redis.io/commands/brpoplpush) from Redis.\n\n```js\nimport { createManager, createWorker } from 'conveyor-mq';\n\nconst queueName = 'my-queue';\nconst redisConfig = { host: '127.0.0.1', port: 6379 };\n\nconst manager = createManager({ queue: queueName, redisConfig });\nmanager.enqueueTask({ data: { x: 1, y: 2 } });\n\nconst worker = createWorker({\n  queue: queueName,\n  redisConfig,\n  handler: ({ task }) =\u003e {\n    console.log(`Processing task: ${task.id}`);\n    return task.data.x + task.data.y;\n  },\n});\n```\n\n## Features\n\n- Task management\n  - Retry tasks on error or stall with customizable retry strategies\n  - Tasks which expire\n  - Task execution timeouts\n  - Delayed/Scheduled tasks\n  - Task progress\n- Events\n  - Task, Queue and Worker events\n- Concurrent worker processing\n- Fast \u0026 efficient, polling-free design\n- Highly extensible design with [plugins](#plugins)\n- Task rate limits\n- Async/await/Promise APIs\n- Robust\n  - Atomic operations with Redis [transactions](https://redis.io/commands/multi)\n  - [At-least-once](https://www.cloudcomputingpatterns.org/at_least_once_delivery/) task delivery\n  - High test [code coverage](https://coveralls.io/github/jasrusable/conveyor-mq?branch=master)\n- High performance\n  - Minimised network overhead using Redis [pipelining](https://redis.io/topics/pipelining) and [multi commands](https://redis.io/commands/multi)\n  - Uses Redis [Lua scripting](https://redis.io/commands/eval) for improved performance and atomicity\n\n## Table of Contents\n\n1. [Introduction](#introduction)\n2. [Features](#features)\n3. [Table of Contents](#table-of-contents)\n4. [Quick Start Guide](#quick-start-guide)\n5. [Overview](#overview)\n   - [Tasks](#tasks)\n   - [Manager](#manager)\n   - [Enqueuing tasks](#enqueuing-tasks)\n   - [Task retries](#task-retries)\n   - [Worker](#worker)\n   - [Processing tasks](#processing-tasks)\n   - [Orchestrator](#orchestrator)\n   - [Stalled tasks](#stalled-tasks)\n   - [Scheduled tasks](#scheduled-tasks)\n   - [Listener](#listener)\n   - [Plugins](#plugins)\n   - [Sharing Redis connections](#sharing-redis-connections)\n   - [Debugging](#debugging)\n6. [API Reference](#api-reference)\n7. [Examples](#examples)\n   - [Simple example](#simple-example)\n   - [Scheduled task example](#scheduled-task-example)\n   - [Express example](#express-example)\n   - [Child/sub tasks example](#childsub-tasks-example)\n   - [Task types example](#task-types-example)\n   - [Shared Redis client example](#shared-redis-client-example)\n   - [Plugins example](#plugins-example)\n8. [Roadmap](#roadmap)\n9. [Contributing](#contributing)\n10. [License](#license)\n\n## Installation\n\nnpm:\n\n```bash\nnpm install --save conveyor-mq\n```\n\nyarn:\n\n```bash\nyarn add conveyor-mq\n```\n\nYou will also need Redis \u003e=3.2\n\n## Quick Start Guide\n\n```js\nimport {\n  createManager,\n  createWorker,\n  createOrchestrator,\n  createListener,\n} from 'conveyor-mq';\n\nconst redisConfig = { host: '127.0.0.1', port: 6379 };\nconst queue = 'myQueue';\n\n// Create a manager which is used to add tasks to the queue, and query various properties of a queue:\nconst manager = createManager({ queue, redisConfig });\n\n// Add a task to the queue by calling manager.enqueueTask:\nconst task = { data: { x: 1, y: 2 } };\nmanager.enqueueTask(task);\n\n// Schedule a task to be added to the queue later by calling manager.scheduleTask:\nconst scheduledTask = {\n  data: { x: 1, y: 2 },\n  enqueueAfter: new Date('2020-05-03'),\n};\nmanager.enqueueTask(scheduledTask);\n\n// Create a listener and subscribe to the task_complete event:\nconst listener = createListener({ queue, redisConfig });\nlistener.on('task_complete', ({ event }) =\u003e\n  console.log('Task complete:', event.task.id),\n);\n\n// Create a worker which will process tasks on the queue:\nconst worker = createWorker({\n  queue,\n  redisConfig,\n  handler: ({ task }) =\u003e {\n    return task.data.x + task.data.y;\n  },\n});\n\n// Create an orchestrator to monitor the queue for stalled tasks, and enqueue scheduled tasks:\nconst orchestrator = createOrchestrator({ queue, redisConfig });\n```\n\n## Overview\n\n### Tasks\n\nThe most basic implementation of a task is an object with a `data` key:\n\n```js\nconst myTask = { data: { x: 1, y: 2 } };\n```\n\n#### Task life cycle\n\nA task will move through various statuses throughout its life cycle within the queue:\n\n`scheduled`: The task has been scheduled to be enqueued at a later time. (Delayed/Scheduled task)\n\n`queued`: The task has been enqueued on the queue and is pending processing by a worker.\n\n`processing`: The task has picked up by a worker and is being processed.\n\n`success`: The task has been successfully processed by a worker.\n\n`failed`: The task has been unsuccessfully processed by a worker and has exhausted all error \u0026 stall retires.\n\nTask status flow diagram:\n\n```js\n                                   -\u003e success\n                                 /\nscheduled -\u003e queued -\u003e processing\n^          ^                    \\\n|--- or ---|----------\u003c (Stall \u0026 error reties)\n                                  \\\n                                    -\u003e failed\n\n```\n\n\\*Note: `success` and `failed` statuses both represent the final outcome of a task, after all stall/error retrying has been attempted and exhausted.\n\n### Manager\n\nA [manager](https://jasrusable.github.io/conveyor-mq/index.html#createmanager) is responsible for enqueuing tasks, as well as querying various properties of a queue.\nCreate a manager by calling `createManager` and passing a `queue` and `redisConfig` parameter.\n\nAdd a task to the queue by calling `manager.enqueueTask` with an object `{ task: { data: x: 1, y: 2} }`.\n\nFor more information, see [createManager](https://jasrusable.github.io/conveyor-mq/index.html#createmanager), [Enqueuing tasks](#enqueuing-tasks)\n\n```js\nimport { createManager } from 'conveyor-mq';\n\n// Create a manager instance:\nconst manager = createManager({\n  queue: 'my-queue',\n  redisConfig: { host: 'localhost', port: 6379 },\n});\n\n// Add a task to the queue:\nawait manager.enqueueTask({ data: { x: 1, y: 2 } });\n\n// Get a task:\nconst task = await manager.getTaskById('my-task-id');\n/*\n  task = {\n    ...\n    status: 'queued',\n    data: {\n      x: 1,\n      y: 2,\n    },\n    ...\n  }\n*/\n```\n\n### Enqueuing tasks\n\nTasks are added to a queue (enqueued) by using a manager's `enqueueTask` function.\n\n```js\nimport { createManager } from 'conveyor-mq';\n\nconst myTask = {\n  // A custom task id. If omitted, an id will be auto generated by manager.enqueueTask.\n  id: 'my-custom-id',\n\n  // Custom task data for processing:\n  data: { x: 1, y: 2 },\n\n  // The maximum number of times a task can be retried after due to an error:\n  errorRetryLimit: 3,\n\n  // The maximum number of times a task can be retried being after having stalled:\n  stallRetryLimit: 3,\n\n  // The maximum number of times a task can be retired at all (error + stall):\n  retryLimit: 5,\n\n  // The maximum time a task is allowed to execute for after which it will fail with a timeout error:\n  executionTimeout: 5000,\n\n  // Custom retry strategy:\n  retryBackoff: { strategy: 'linear', factor: 10 },\n\n  // Schedules a task to only be enqueued after this time:\n  enqueueAfter: new Date('2020-05-01'),\n\n  // Time after which a task will expire and fail if only picked up by a worker after the time:\n  expiresAt: new Date('2020-05-06'),\n\n  // Time after an acknowledgement after which a task will be considered stalled and re-enqueued by an orchestrator:\n  stallTimeout: 5000,\n\n  // Frequency at which a task is acknowledged by a worker when being processed:\n  taskAcknowledgementInterval: 1000,\n};\n\nconst manager = createManager({\n  queue: 'my-queue',\n  redisConfig: { host: 'localhost', port: 6379 },\n});\n\nconst enqueuedTask = await manager.enqueueTask(myTask);\n```\n\n### Task retries\n\nConveyor MQ implements a number of different task retry mechanisms which can be controlled by various task properties.\n\n`errorRetryLimit` controls the maximum number of times a task is allowed to be retried after encountering an error whilst being processed.\n\n```js\n// Create a task which can be retried on error a maximum of 2 times:\nconst task = { data: { x: 1, y: 2 }, errorRetryLimit: 2 };\n```\n\n`errorRetries` is the number of times a task has been retried because of an error.\n\n```js\n// See how many times a task has been retried due to an error:\nconst task = await manager.getTaskById('my-task-id');\n/*\n  task = {\n    ...\n    id: 'my-task-id',\n    errorRetries: 2,\n    ...\n  }\n*/\n```\n\n`stallRetryLimit` controls the maximum number of times a task is allowed to be retried after encountering becoming stalled whilst being processed.\n\n```js\n// Create a task which can be retried on stall a maximum of 2 times:\nconst task = { data: { x: 1, y: 2 }, stallRetryLimit: 2 };\n```\n\n`stallRetries` is the number of times a task has been retried after having stalled:\n\n```js\n// See how many times a task has been retried due to an error:\nconst task = await manager.getTaskById('my-task-id');\n/*\n  task = {\n    ...\n    id: 'my-task-id',\n    stallRetries: 2,\n    ...\n  }\n*/\n```\n\n`retryLimit` controls the maximum number of times a task is allowed to be retried after either stalling or erroring whilst being processed.\n\n```js\n// Create a task which can be retried on stall or error a maximum of 2 times:\nconst task = { data: { x: 1, y: 2 }, retryLimit: 2 };\n```\n\n`retries` is the number of times a task has been retried in total (error + stall retries)\n\n```js\n// See how many times a task has been retried in total:\nconst task = await manager.getTaskById('my-task-id');\n/*\n  task = {\n    ...\n    id: 'my-task-id',\n    retries: 2,\n    ...\n  }\n*/\n```\n\n### Worker\n\nA [worker](https://jasrusable.github.io/conveyor-mq/index.html#createworker) is responsible for taking enqueued tasks off of the queue and processing them.\nCreate a worker by calling `createWorker` with a `queue`, `redisConfig` and `handler` parameter.\n\nThe `handler` parameter should be a function which receives a task and is responsible for processing the task.\nThe handler should return a promise which should resolve if the task was successful, or reject if failed.\n\nFor more information, see [createWorker](https://jasrusable.github.io/conveyor-mq/index.html#createworker) and [Processing tasks](#processing-tasks)\n\n```js\nimport { createWorker } from 'conveyor-mq';\n\n// Create a worker which will start monitoring the queue for tasks and process them:\nconst worker = createWorker({\n  queue: 'my-queue',\n  redisConfig: { host: 'localhost', port: 6379 },\n  // Pass a handler which receives tasks, processes them, and then returns the result of a task:\n  handler: ({ task }) =\u003e {\n    return task.data.x + task.data.y;\n  },\n});\n```\n\n### Processing tasks\n\nTasks are processed on the queue by workers which can be created using `createWorker`. Once created, a worker will begin monitoring a queue for tasks to process using an efficient, non-polling implementation making use of the `brpoplpush` Redis command.\n\nA worker can paused and resumed by calling `worker.pause` and `worker.start` respectively.\n\n```js\nimport { createWorker } from 'conveyor-mq';\n\nconst worker = createWorker({\n  // Queue name:\n  queue: 'my-queue',\n\n  // Redis configuration:\n  redisConfig: { host: 'localhost', port: 6379 },\n\n  // A handler function to process tasks:\n  handler: async ({ task, updateTaskProgress }) =\u003e {\n    await updateTaskProgress(100);\n    return 'some-task-result';\n  },\n\n  // The number of concurrent tasks the worker can processes:\n  concurrency: 10,\n\n  // The retry delay when retrying a task after it has errored:\n  getRetryDelay: ({ task }) =\u003e (task.retries + 1) * 100,\n\n  // Task success callback:\n  onTaskSuccess: ({ task }) =\u003e {\n    console.log('Task processed successfully', result);\n  },\n\n  // Task error callback:\n  onTaskError: ({ task, error }) =\u003e {\n    console.log('Task had an error', error);\n  },\n\n  // Task fail callback:\n  onTaskFailed: ({ task, error }) =\u003e {\n    console.log('Task failed with error', error);\n  },\n\n  // Worker idle callback. Called when the worker becomes idle:\n  onIdle: () =\u003e {\n    console.log('worker is now idle and not processing tasks');\n  },\n\n  // Amount of time since processing a task after which the worker is considered idle and the onIdle callback is called.\n  idleTimeout: 250,\n\n  // Worker ready callback, called once a worker is ready to start processing tasks:\n  onReady: () =\u003e {\n    console.log('Worker is now ready to start processing tasks');\n  },\n\n  // Control whether the worker should start automatically, else worker.start() must be called manually:\n  autoStart: true,\n\n  // Remove tasks once they are processed successfully\n  removeOnSuccess = false,\n\n  // Remove tasks once they are fail to be processed successfully\n  removeOnFailed = false,\n});\n```\n\n### Orchestrator\n\nAn [orchestrator](https://jasrusable.github.io/conveyor-mq/index.html#createorchestrator) is responsible for various queue maintenance operations including re-enqueueing stalled tasks, and enqueueing delayed/scheduled tasks.\nCreate an orchestrator by calling `createOrchestrator` with a `queue` and `redisConfig` parameter. The orchestrator will then begin monitoring the queue for stalled tasks and re-enqueueing them if needed, as well as enqueueing scheduled tasks.\n\nFor more information, see [createOrchestrator](https://jasrusable.github.io/conveyor-mq/index.html#createorchestrator) and [Stalling tasks](#stalled-tasks)\n\n```js\nimport { createOrchestrator } from 'conveyor-mq';\n\n// Create an orchestrator:\nconst orchestrator = createOrchestrator({\n  queue: 'my-queue',\n  redisConfig: { host: 'localhost', port: 6379 },\n});\n```\n\n### Stalled tasks\n\nAs part of the at-least-once task delivery strategy, Conveyor MQ implements stalled or stuck task checking and retrying.\n\nWhile a `worker` is busy processing a task, it will periodically acknowledge that it is still currently processing the task. The interval at which a processing task is acknowledged by a worker during processing at is controlled by `Task.taskAcknowledgementInterval` and otherwise falls back to `Worker.defaultTaskAcknowledgementInterval`.\n\nA task is considered stalled if while it is being processed by a worker, the worker fails to acknowledge that it is currently working on the task. This situation occurs mainly when either a worker goes offline or crashes unexpectedly whilst processing a task, or if the Node event loop on the worker becomes blocked while processing a task.\n\nThe time since a task was last acknowledged after which it is considered stalled is controlled by `Task.stallInterval` and otherwise falls back to `Worker.defaultStallInterval`.\n\n\u003e _Note_: An orchestrator is required to be running on the queue which will monitor and re-enqueue any stalled tasks. It is recommended to have only a single orchestrator run per queue to minimize Redis overhead, however multiple orchestrators can be run simultaneously.\n\n### Scheduled tasks\n\nTasks can be scheduled to be added to the queue at some future point in time. To schedule a task, include a `enqueueAfter` property on a task and call `manager.scheduleTask`:\n\n```js\nconst scheduledTask = {\n  data: { x: 1, y: 2 },\n  enqueueAfter: new Date('2020-05-15'),\n};\n\nconst { task: enqueuedTask } = await manager.scheduleTask(scheduledTask);\n/*\n  enqueuedTask = {\n    ...\n    data: { x: 1, y: 2 },\n    status: 'scheduled',\n    enqueueAfter: '2020-05-15',\n    ...\n  }\n*/\n```\n\n\u003e _Note_: An orchestrator is required to be running on the queue which will monitor and enqueue any scheduled tasks. It is recommended to have only a single orchestrator run per queue to minimize Redis overhead, however multiple orchestrators can be run simultaneously.\n\n### Listener\n\nA [listener](https://jasrusable.github.io/conveyor-mq/index.html#createlistener) is responsible for listening and subscribing to [events](https://jasrusable.github.io/conveyor-mq/enums/eventtypes.html). Use `listener.on` to subscribe to various task, queue and worker related events.\n\nFor more information, see [createListener](https://jasrusable.github.io/conveyor-mq/index.html#createlistener) and [Event](https://jasrusable.github.io/conveyor-mq/interfaces/event.html)\n\n```js\nimport { createListener } from 'conveyor-mq';\n\n// Create a listener:\nconst listener = createListener({\n  queue: 'my-queue',\n  redisConfig: { host: 'localhost', port: 6379 },\n});\n\n// Listen for the 'task_complete' event:\nlistener.on('task_complete', ({ event }) =\u003e {\n  console.log(`Task ${event.task.id} has completed!`),\n});\n```\n\n### Plugins\n\nConveyor MQ is highly extensible through its plugin \u0026 hooks architecture. The `createManager`, `createWorker` and `createOrchestrator` functions have an optional `hooks` parameter through which various hook functions can be passed to hook into the various queue lifecycle actions.\nPlugins can be created by implementing hook functions, and then calling `registerPlugins` to register plugins.\n\n#### Create a plugin\n\nA plugin is a simple object with keys corresponding to hook names, and values of functions.\n\n```js\nimport { registerPlugins } from 'conveyor-mq';\n\n// Create a simple plugin.\nconst myPlugin = {\n  onBeforeEnqueueTask: ({ task }) =\u003e console.log(task),\n  onAfterEnqueueTask: ({ task }) =\u003e console.log(task),\n  onBeforeTaskProcessing: ({ taskId }) =\u003e console.log(taskId),\n  onAfterTaskProcessing: ({ task }) =\u003e console.log(task),\n  onAfterTaskSuccess: ({ task }) =\u003e console.log(task),\n  onAfterTaskError: ({ task }) =\u003e console.log(task),\n  onAfterTailFail: ({ task }) =\u003e console.log(task),\n};\n\n// Register the plugin and unpack new createManager and createWorker functions.\nconst { createManager, createWorker } = registerPlugins(myPlugin);\n\nconst queue = 'my-queue';\nconst redisConfig = { host: 'localhost', port: 6370 };\n\n// Create a manager which is registered with myPlugin.\nconst manager = createManager({ queue, redisConfig });\n\n// Create a worker which is registered with myPlugin.\nconst manager = createManager({\n  queue,\n  redisConfig,\n  handler: ({ task }) =\u003e {\n    // Do processing\n    return 'some-result';\n  },\n});\n```\n\nSee the [Plugins example](#plugins-example) for more information.\n\n### Sharing Redis connections\n\nRedis connections can be shared between a manager, worker and orchestrator as an optimization to reduce the total number of Redis connections used. This is particularly useful to do when your Redis server is hosted and priced based on the number of active connections, such as on Heroku or Compose.\n\nThe functions `createManager`, `createWorker` and `createOrchestrator` each take an optional `redisClient` parameter where a shared Redis client can be passed. The shared Redis client must first be configured with the custom Lua scripts by calling `loadLuaScripts({ client })`.\n\nSee the [Shared redis client example](#shared-redis-client-example) for more details.\n\n### Debugging\n\nConveyor MQ makes use of the [debug](https://www.npmjs.com/package/debug) package for debug logging.\n\nEnable Conveyor MQ debug logging by setting the `DEBUG` environment variable to `conveyor-mq:*` and then executing your project/app in the same shell session:\n\n```bash\nexport DEBUG=conveyor-mq:*\nnode ./my-app.js\n```\n\n## API Reference\n\n### Manager\n\n- [createManager](#createManager)\n- [manager.enqueueTask](#managerenqueueTask)\n- [manager.enqueueTasks](#managerenqueueTasks)\n- [manager.scheduleTask](#managerscheduleTask)\n- [manager.scheduleTasks](#managerscheduleTasks)\n- [manager.onTaskComplete](#manageronTaskComplete)\n- [manager.getTaskById](#managergetTaskById)\n- [manager.getTasksById](#managergetTasksById)\n- [manager.getTaskCounts](#managergetTasksCounts)\n- [manager.getWorkers](#managergetworkers)\n- [manager.removeTaskById](#managerremoveTaskById)\n- [manager.pauseQueue](#managerpauseQueue)\n- [manager.resumeQueue](#managerresumeQueue)\n- [manager.setQueueRateLimit](#managersetqueueratelimit)\n- [manager.getQueueRateLimit](#managergetqueueratelimit)\n- [manager.destroyQueue](#managerdestroyQueue)\n- [manager.quit](#managerquit)\n\n### Worker\n\n- [createWorker](#createWorker)\n\n#### createManager\n\nCreates a manager instance which is responsible for adding tasks to the queue, as well as querying various properties of the queue.\nReturns a promise which resolves with a manager instance.\n\n```js\nimport { createManager } from 'conveyor-mq';\n\nconst manager = createManager({\n  // Queue name.\n  queue: 'my-queue',\n  // Redis configuration\n  redisConfig: {\n    host: 'localhost',\n    port: 6379,\n    db: 0,\n    password: 'abc',\n    url: 'redis://some-password@localhost:6371/0',\n  },\n  // Pass in a shared redis instance.\n  redisClient: sharedRedisInstance,\n});\n```\n\n#### manager.enqueueTask\n\nEnqueues a task on the queue.\nReturns a promise which resolves with a `TaskResponse`.\n\n```js\nconst task = {\n  id: 'my-custom-id', // A custom task id. If omitted, an id (uuid string) will be auto generated by manager.enqueueTask.\n  data: { x: 1, y: 2 }, // Custom task data.\n  errorRetryLimit: 3, // The maximum number of times a task can be retried after due to an error.\n  stallRetryLimit: 3, // The maximum number of times a task can be retried being after having stalled.\n  retryLimit: 5, // The maximum number of times a task can be retired at all (error + stall).\n  executionTimeout: 5000, // The maximum time a task is allowed to execute for after which it will fail with a timeout error.\n  retryBackoff: { strategy: 'linear', factor: 10 }, // Custom retry strategy.\n  expiresAt: new Date('2020-05-06'), // Time after which a task will expire and fail if only picked up by a worker after the time.\n  stallTimeout: 5000, // Time after an acknowledgement after which a task will be considered stalled and re-enqueued by an orchestrator.\n  taskAcknowledgementInterval: 1000, // Frequency at which a task is acknowledged by a worker while being processed.\n};\n\nconst {\n  task: enqueuedTask, // The enqueued task is returned.\n  onTaskComplete, // A function which returns a promise that resolves once the task is complete.\n} = await manager.enqueueTask(task);\n```\n\n#### manager.enqueueTasks\n\nEnqueues multiple tasks in a single transaction.\nReturns a promise which resolves with a list of `TaskResponse`'s.\n\n```js\nconst task1 = { data: { x: 1 } };\nconst task2 = { data: { y: 2 } };\n\nconst [\n  { task: enqueuedTask1 },\n  { task: enqueuedTask2 },\n] = await manager.enqueueTasks([task1, task2]);\n```\n\n#### manager.scheduleTask\n\nSchedules a task to be enqueued at a later time.\nReturns a promise which resolves with a `TaskResponse`.\n\n```js\nconst myScheduledTask = {\n  data: { x: 1, y: 2 },\n  enqueueAfter: new Date('2020-05-30'),\n};\n\nconst {\n  task, // Scheduled task.\n  onTaskComplete, // A function which returns a promise that resolves on task complete.\n} = await manager.scheduleTask(myScheduledTask);\n```\n\n#### manager.scheduleTasks\n\nSchedules a task to be enqueued at a later time.\nReturns a promise which resolves with a list of `TaskResponse`'s.\n\n```js\nconst myScheduledTask = {\n  data: { x: 1, y: 2 },\n  enqueueAfter: new Date('2020-05-30'),\n};\n\nconst [{ task, onTaskComplete }] = await manager.scheduleTasks([\n  myScheduledTask,\n]);\n```\n\n#### manager.onTaskComplete\n\nA function which takes a `taskId` and returns a promise that resolves with the task once the task has completed.\n\n```js\nconst task = await manager.enqueueTask({ data: { x: 1, y: 2 } });\nawait manager.onTaskComplete(task.id);\nconsole.log('Task has completed!');\n```\n\n#### manager.getTaskById\n\nGets a task from the queue. Returns a promise that resolves with the task from the queue.\n\n```js\nconst task = await manager.getTaskById('my-task-id');\n```\n\n#### manager.getTasksById\n\nGets multiple tasks from the queue in a transaction. Returns a promises that resolves with a list of tasks.\n\n```js\nconst tasks = await manager.getTasksById(['task-id-1', 'task-id-2']);\n```\n\n#### manager.getTaskCounts\n\nGets the count of tasks per status. Returns a promise that resolves with the counts of tasks per status.\n\n```js\nconst {\n  scheduledCount,\n  queuedCount,\n  processingCount,\n  successCount,\n  failedCount,\n} = await manager.getTaskCounts();\n```\n\n#### manager.getWorkers\n\nGets the workers connected to the queue. Returns a promise that resolves with a list of workers.\n\n```js\nconst workers = await manager.getWorkers();\n```\n\n#### manager.removeTaskById\n\nRemoves a given task from the queue by id. Returns a promise.\n\n```js\nawait manager.removeTaskById('some-task-id');\n```\n\n#### manager.pauseQueue\n\nPauses a queue.\n\n```js\nawait manager.pauseQueue();\n```\n\n#### manager.resumeQueue\n\nResumes a queue.\n\n```js\nawait manager.resumeQueue();\n```\n\n#### manager.setQueueRateLimit\n\nSets the rate limit of a queue. (100 tasks every 60 seconds)\n\n```js\nawait manager.setQueueRateLimit({ points: 100, duration: 60 });\n```\n\n#### manager.getQueueRateLimit\n\nGets the rate limit of a queue.\n\n```js\nconst rateLimit = await manager.getQueueRateLimit();\n// rateLimit = { points: 100, duration: 60 }\n```\n\n#### manager.destroyQueue\n\nDestroys all queue data and data structures. Returns a promise.\n\n```js\nawait manager.destroyQueue();\n```\n\n#### manager.quit\n\nQuits a manager, disconnecting all redis connections and listeners. Returns a promise.\n\n```js\nawait manager.quit();\n```\n\n### Worker\n\n#### createWorker\n\nA worker is responsible for taking enqueued tasks off of the queue and processing them. Create a worker by calling `createWorker` with at least a `queue`, `redisConfig` and `handler` parameter.\n\nThe `handler` parameter should be a function which receives a task and is responsible for processing the task.\nThe handler should return a promise which should resolve if the task was successful, or reject if failed.\n\n```js\nimport { createWorker } from 'conveyor-mq';\n\n// Create a worker which will start monitoring the queue for tasks and process them:\nconst worker = createWorker({\n  queue: 'my-queue',\n  redisConfig: { host: 'localhost', port: 6379 },\n  // Pass a handler which receives tasks, processes them, and then returns the result of a task:\n  handler: ({ task }) =\u003e {\n    return task.data.x + task.data.y;\n  },\n});\n```\n\nAll worker params:\n\n```js\nimport { createWorker } from 'conveyor-mq';\n\nconst worker = createWorker({\n  // Queue name:\n  queue: 'my-queue',\n\n  // Redis configuration:\n  redisConfig: { host: 'localhost', port: 6379 },\n\n  // A handler function to process tasks.\n  // If the handler function throws an error or returns a promise which rejects, the task will be considered to have errorerd and the thrown or rejected error will be set to the task's `error` field.\n    // If the handler function returns a result, or returns a promise which resolves, the task will be considered successfully processed and the return or resolve value will be set to the task's `result` field.\n  handler: async ({ task, updateTaskProgress }) =\u003e {\n    await updateTaskProgress(100);\n    return 'some-task-result';\n  },\n\n  // The number of concurrent tasks the worker can processes at a time:\n  concurrency: 10,\n\n  // The retry delay when retrying a task after it has errored or stalled:\n  getRetryDelay: ({ task }) =\u003e (task.retries + 1) * 100,\n\n  // Task success callback:\n  onTaskSuccess: ({ task }) =\u003e {\n    console.log('Task processed successfully', result);\n  },\n\n  // Task error callback:\n  onTaskError: ({ task, error }) =\u003e {\n    console.log('Task had an error', error);\n  },\n\n  // Task fail callback:\n  onTaskFailed: ({ task, error }) =\u003e {\n    console.log('Task failed with error', error);\n  },\n\n  // Worker idle callback. Called when the worker becomes idle:\n  onIdle: () =\u003e {\n    console.log('worker is now idle and not processing tasks');\n  },\n\n  // Amount of time since processing a task after which the worker is considered idle and the onIdle callback is called.\n  idleTimeout: 250,\n\n  // Worker ready callback, called once a worker is ready to start processing tasks:\n  onReady: () =\u003e {\n    console.log('Worker is now ready to start processing tasks');\n  },\n\n  // Control whether the worker should start automatically, else worker.start() must be called manually:\n  autoStart: true,\n\n  // Remove tasks once they are processed successfully\n  removeOnSuccess = false,\n\n  // Remove tasks once they are fail to be processed successfully\n  removeOnFailed = false,\n});\n```\n\n## Examples\n\n### [Simple example](https://github.com/conveyor-mq/conveyor-mq/tree/master/examples/simple-example)\n\n### [Express example](https://github.com/conveyor-mq/conveyor-mq/tree/master/examples/express-example)\n\n### [Scheduled task example](https://github.com/conveyor-mq/conveyor-mq/tree/master/examples/scheduled-task-example)\n\n### [Child/sub tasks example](https://github.com/conveyor-mq/conveyor-mq/tree/master/examples/sub-tasks-example)\n\n### [Task types example](https://github.com/conveyor-mq/conveyor-mq/tree/master/examples/task-types-example)\n\n### [Shared redis client example](https://github.com/conveyor-mq/conveyor-mq/tree/master/examples/redis-client-sharing-example)\n\n### [Plugins example](https://github.com/conveyor-mq/conveyor-mq/tree/master/examples/plugins-example)\n\n## Roadmap\n\n- [ ] Improve documentation\n- [ ] Task priorities\n- [ ] Recurring tasks\n- [x] Task rate limiting\n- [ ] Performance optimisations\n- [ ] Child process workers\n- [ ] Web UI\n\n## Contributing\n\nSee [CONTRIBUTING.md](https://github.com/jasrusable/conveyor-mq/blob/master/CONTRIBUTING.md)\n\n## License\n\nSee [LICENSE](https://github.com/jasrusable/conveyor-mq/blob/master/LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconveyor-mq%2Fconveyor-mq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fconveyor-mq%2Fconveyor-mq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconveyor-mq%2Fconveyor-mq/lists"}