{"id":20128016,"url":"https://github.com/arkadiyshuvaev/durable-message-queue","last_synced_at":"2025-10-17T06:50:57.767Z","repository":{"id":47864689,"uuid":"247343988","full_name":"ArkadiyShuvaev/durable-message-queue","owner":"ArkadiyShuvaev","description":"A durable message queue library based on the Redis reliable queue pattern and LUA scripts","archived":false,"fork":false,"pushed_at":"2022-12-05T22:06:36.000Z","size":530,"stargazers_count":0,"open_issues_count":8,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-13T08:26:40.553Z","etag":null,"topics":["fifo-queue","metrics"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ArkadiyShuvaev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-03-14T19:55:34.000Z","updated_at":"2023-01-14T14:22:18.000Z","dependencies_parsed_at":"2023-01-24T07:01:00.959Z","dependency_job_id":null,"html_url":"https://github.com/ArkadiyShuvaev/durable-message-queue","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArkadiyShuvaev%2Fdurable-message-queue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArkadiyShuvaev%2Fdurable-message-queue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArkadiyShuvaev%2Fdurable-message-queue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArkadiyShuvaev%2Fdurable-message-queue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ArkadiyShuvaev","download_url":"https://codeload.github.com/ArkadiyShuvaev/durable-message-queue/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241572210,"owners_count":19984235,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fifo-queue","metrics"],"created_at":"2024-11-13T20:25:03.700Z","updated_at":"2025-10-17T06:50:57.677Z","avatar_url":"https://github.com/ArkadiyShuvaev.png","language":"TypeScript","readme":"# Durable message queue\n[![Build Status](https://travis-ci.org/ArkadiyShuvaev/durable-message-queue.svg?branch=master)](https://travis-ci.org/ArkadiyShuvaev/durable-message-queue)\n\nThe library helps you build a distributed application with decoupled components using a FIFO queue.\n\nIt implements the [Redis Reliable queue](https://redis.io/commands/rpoplpush) pattern and supports moving unprocessed or corrupted messages to a dead-letter queue.\n\nAll important operations are produced with LUA scripts that [guarantee](https://redis.io/commands/eval#atomicity-of-scripts) that they are executed in an atomic way.\n\n## Table of contents\n- [Features](#features)\n- [Quick start](#quick-start)\n- [Components](#components)\n  - [Producer](#producer)\n  - [Consumer](#consumer)\n    - [The processing timeout](#the-processing-timeout)\n    - [The maximum receive count](#the-maximum-receive-count)\n  - [Queue manager](#queue-manager)\n  - [Monitor](#monitor)\n- [Settings](#settings)\n  - [Library related settings](#library-related-settings)\n  - [Ioredis related settings](#ioredis-related-settings)\n- [Metrics](#metrics)\n- [Debug](#debug)\n\n\n## Features\nThe library supports the following features (described below in details):\n1. A FIFO queue for published messages\n1. Monitoring metrics that allow you to integrate the library with your monitoring system\n1. A processing timeout that determines the length of time for which a message is invisible for other consumer when the message begins to be processed\n1. A maximum number of processing attempts before a message is moved to the dead-letter queue\n\n\n## Quick start\n1. Create a new \"quick-start.js\" (or copy the example from the [quick-start.js](examples/quick-start.js) file):\n    ```\n    const dmq = require(\"durable-message-queue\");\n\n    const queueName = \"send-registration-confirmation-email\";\n    const config = {\n        host: \"127.0.0.1\",\n        port: \"6379\"\n    };\n\n    const producer = dmq.Builder.createProducer(queueName, config);\n    const consumer = dmq.Builder.createConsumer(queueName, config);\n\n    consumer.subscribe(async (message) =\u003e console.log(message));\n\n    const message = {\n        userId: \"1\",\n        messageType: \"registrationConfirmation\",\n        to: \"user@company.com\"\n    };\n    producer.send(JSON.stringify(message));\n    ```\n1. Install the package:\n    ```\n    npm i durable-message-queue\n    ```\n1. Start the script:\n    ```\n    node quick-start.js\n    ```\n1. See the output:\n    ```\n    {  \n        id: '2',  \n        receiveCount: '1',  \n        updatedDt: '2020-11-26T21:46:01.870Z',  \n        createdDt: '2020-11-26T21:46:01.827Z',  \n        payload: '{\"userId\":\"1\",\"messageType\":\"registrationConfirmation\",\"to\":\"user@company.com\"}',  \n        receivedDt: '2020-11-26T21:46:01.870Z'  \n    }\n    ```\n\n## Components\n### Producer\nOne or more producers can send messages to a FIFO queue. Messages are stored in the order that they were successfully received by Redis.\n\nBefore being added to the queue each message is updated by metadata. This metadata is used by the [queue manager](#queue-manager) to conduct the message state:\n- a message can be available to process by consumers\n- the message can be invisible for other consumers because it is processed\n- the message is moved to the dead queue because a [maximum receive count](#the-maximum-receive-count) is exceeded\n\n### Consumer\nOne or more consumers can subscribe to the queue's updates but only one of them receives the message to process. The message should be processed  during a limited period of time, otherwise, the message is marked as available and can be processed again by this or another consumer. See the [processing timeout](#the-processing-timeout) section to get more details.\n\nIn case if a message cannot be processed X times, it is moved to the dead queue by the [queue manager](#queue-manager). See all details about this feature in the [maximum receive count](#the-maximum-receive-count) section below.\n\nIf the consumers are started after messages were added to the queue, they check the queue for available unprocessed messages and start processing them. Your Redis infrastructure is responsible for retaining all messages that are added to the queue and the messages are stored for an unlimited period of time until they are not processed by consumers.\n\nThe consumer has access to message metadata and should filter all  outdated messages on its own using a value of the \"createdDt\" or \"updatedDt\" property (see the output of the [quick start](#quick-start) section above).\n\n#### The processing timeout\nWhen a message is added to the queue the consumer starts processing the message and the message is become unavailable for other consumers:\n\n![Processing Timeout](https://raw.githubusercontent.com/ArkadiyShuvaev/durable-message-queue/master/assests/processing-timeout.png)\n\nIf something goes wrong and the message cannot be processed by the consumer due to any reason, a [queue manager](#queue-manager) makes the message visible for processing by another consumer. By default, the processing timeout equals 300 seconds.\n\nYou can specify this period by assigning desired value in seconds to the visibilityTimeout value. See the [Settings](#settings) section to get more details.\n\n#### The maximum receive count\nIf a message cannot be processed X times by a consumer, the message is moved to the dead-letter queue by a queue manager. This period of time is called the maximum receive count and equals 3 by default.\n\n![Dead-letter queue](https://raw.githubusercontent.com/ArkadiyShuvaev/durable-message-queue/master/assests/dead-letter-queue.png)\n\nYou can change the default value by modifying the maxReceiveCount property. See the [Settings](#settings) section to get more details.\n\n### Queue manager\nThe description is under construction.\n\n### Monitor\nThe monitor tool is an additional component that allows us to keep up to date on metrics across all queues.\n\n![Dead-letter queue](https://raw.githubusercontent.com/ArkadiyShuvaev/durable-message-queue/master/assests/monitor-animation.gif)\n\n Please use the [start-monitor.js](examples/start-monitor.js) file as an example to start the monitor tool and see the [Settings](#settings) section to get more details about configuration.\n\n## Settings\n### Library related settings\nThere are three settings that you can use with this library:\n1. visibilityTimeout - the period of time in seconds during which the library prevents other consumers from receiving and processing the message. The default visibility timeout for a message is 300 seconds (5 minutes).\n1. maxReceiveCount - the maximum number of receives that are allowed before the message is moved to the dead-letter queue. If something goes wrong and the number of receives exceeds the MaxReceiveCount value, the queue manager moves the message to the dead-letter queue (see the picture of the [maximum receive count](#the-maximum-receive-count) section to get more details). The default value is 3.\n1. monitorUpdateInterval - the period of time in seconds to update metrics of the queue monitor tool.\n\n### Ioredis related settings\nAs far as the library uses [ioredis](https://github.com/luin/ioredis) to access to Redis, all ioredis-related configuration settings are transmitted to ioredis as is. Visit the ioredis main page to get more details about configuration of the connection.\n\nThe simplest configuration includes only a host property with the server address:\n  ```\n  const config = {\n    host: \"server name or ip address\"\n  };\n  ```\n\n## Metrics\nThe library supports a set of monitoring metrics that indicates the number of:\n1. \"numberOfMessagesSent\" - messages are sent by producers for processing\n1. \"numberOfMessagesReceived\" - messages are received and started processing by consumers\n1. \"numberOfMessagesDead\" - corrupted messages are moved to the dead-letter queue by a queue manager\n1. \"numberOfMessagesReturned\" - unprocessed messages are moved back to the queue because of crashing a consumer or for an undefined reason\n1. \"numberOfMessagesDeleted\" - messages are successfully processed and removed from the queue.\n\n## Debug\nYou can set the DEBUG env to ioredis:* to print debug and error info. Here is the example for Windows and the [quick-start.js](examples/quick-start.js) file:\n```\nset DEBUG=dmq:* \u0026 node quick-start.js\n```","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farkadiyshuvaev%2Fdurable-message-queue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Farkadiyshuvaev%2Fdurable-message-queue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farkadiyshuvaev%2Fdurable-message-queue/lists"}