{"id":23820995,"url":"https://github.com/truesparrowsystems/queue","last_synced_at":"2025-09-07T03:31:25.274Z","repository":{"id":38316505,"uuid":"499027462","full_name":"TrueSparrowSystems/queue","owner":"TrueSparrowSystems","description":"Queue is a node.js package to create background jobs in topic-based RabbitMQ exchanges and process them later.","archived":false,"fork":false,"pushed_at":"2023-01-30T13:17:13.000Z","size":46,"stargazers_count":23,"open_issues_count":0,"forks_count":1,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-12-05T15:46:48.387Z","etag":null,"topics":["message-queue","pubsub","rabbitmq","worker"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TrueSparrowSystems.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-06-02T07:01:27.000Z","updated_at":"2024-01-04T17:09:04.000Z","dependencies_parsed_at":"2023-02-16T07:45:34.294Z","dependency_job_id":null,"html_url":"https://github.com/TrueSparrowSystems/queue","commit_stats":null,"previous_names":["plg-works/queue"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TrueSparrowSystems%2Fqueue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TrueSparrowSystems%2Fqueue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TrueSparrowSystems%2Fqueue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TrueSparrowSystems%2Fqueue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TrueSparrowSystems","download_url":"https://codeload.github.com/TrueSparrowSystems/queue/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":232169055,"owners_count":18482527,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["message-queue","pubsub","rabbitmq","worker"],"created_at":"2025-01-02T08:19:40.096Z","updated_at":"2025-01-02T08:19:40.745Z","avatar_url":"https://github.com/TrueSparrowSystems.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Queue\n\n![npm version](https://img.shields.io/npm/v/@truesparrow/queue.svg?style=flat)\n\nTrue Sparrow Queue helps in publishing and subscribing tasks over [RabbitMQ](https://www.rabbitmq.com/). Internally it uses topic-based exchange.\n\nOne use-case is to publish tasks for asynchronous processing. For example, **API worker process** can publish tasks which will be taken up by **asynchronous worker processes** which have subscribed for such tasks.\n\n## Prerequisites\n- [RabbitMQ](https://www.rabbitmq.com/tutorials/amqp-concepts.html)\n\n## Install\n\n```bash\nnpm install @truesparrow/queue --save\n```\n\n## Initialize\n`configStrategy` is passed to initialize True Sparrow Queue. `configStrategy` is an object with `rabbitmq` as a key. The value of `rabbitmq` is an object with following keys:\n- **username** [string] (mandatory) RabbitMQ connection username\n- **password** [string] (mandatory) RabbitMQ connection password\n- **host** [string] (mandatory) RabbitMQ host\n- **port** [string] (mandatory) RabbitMQ port\n- **heartbeats** [string] (mandatory) [heartbeats](https://www.rabbitmq.com/heartbeats.html) defines after what period of time the peer TCP connection should be considered unreachable.\n- **clusterNodes** [Array] (mandatory) - List of [RMQ cluster](https://www.rabbitmq.com/clustering.html#node-names) hosts.\n- **enableRabbitmq** [integer] (optional) 0 if local usage.\n- **switchHostAfterSec** [integer] (optional) Wait time before switching RMQ host.\n- **connectionTimeoutSec** [integer] (optional) Wait time for connection to establish.\n\nFollowing snippet initializes True Sparrow Queue Manager:\n\n```js\nconst Queue = require('@truesparrow/queue');\n\n// Config Strategy for True Sparrow Queue.\nconfigStrategy = {\n\t'rabbitmq': {\n        'username': 'guest',\n        'password': 'guest',\n        'host': '127.0.0.1',\n        'port': '5672',\n        'heartbeats': '30',\n        'enableRabbitmq': 1\n    }\n};\n\n// Create instance\nconst queueManager = await Queue.getInstance(configStrategy);\n```\n\n## queueManager object methods\n- `queueManager.subscribeEvent.rabbit(topics, options, readCallback, subscribeCallback)`\n\u003cbr\u003eDescription: Subscribe to multiple topics over a queue.\n\u003cbr\u003eParameters:\n  - **topics** [Array] (mandatory) - Array of topics to subscribe to.\n  - **options** [object] (mandatory) Object with following keys:\n    - **queue** [string] (optional) - Name of the queue on which messages with relevant topics will be published. If not passed, a queue with a unique name is created and is deleted when the subscriber gets disconnected.\n    - **ackRequired** [integer] (optional) - The delivered message needs ack if passed 1 ( default 0 ). if 1 passed and ack not done, message will redeliver.\n    - **broadcastSubscription** [integer] (optional) -  Set to 1, when queue needs to be subscribed to broadcasting events. Default 0.\n    - **prefetch** [integer] (optional) - The number of messages released from queue in parallel. In case of ackRequired=1, queue will pause unless delivered messages are acknowledged. Default 1.\n  - **readCallback** [function] (mandatory) - Callback method will be invoked whenever there is a new notification.\n  - **subscribeCallback** [function] (optional) - Callback method to get consumerTag.\n\n- `queueManager.publishEvent.perform(params)`\n\u003cbr\u003eDescription: Publish event to topics.\n\u003cbr\u003eParameters:\n  - **params** [object] (mandatory) Object with following keys:\n    - **topics** [Array] (mandatory) Array of topics to which message to be publish.\n    - **broadcast** [integer] (optional) Set to 1 for broadcasting. Default 0.\n    - **publishAfter** [integer] (optional) Delay in milli-seconds between publish and being available for consumption. Default 0.\n    - **publisher** [string] (mandatory) Name of publisher\n    - **message** [object] (mandatory) Object with following keys:\n      - **kind** [string] (mandatory) Kind of the message.\n      - **payload** [object] (mandatory) Payload to identify message and extra info.\n\n## Examples\n\n### Subscribe\nFollowing snippet subscribes to specific topics over a queue.\n\n```js\nconst Queue = require('@truesparrow/queue');\n\n// Config Strategy for True Sparrow Queue.\nconfigStrategy = {\n\t'rabbitmq': {\n        'username': 'guest',\n        'password': 'guest',\n        'host': '127.0.0.1',\n        'port': '5672',\n        'heartbeats': '30',\n        'enableRabbitmq': 1\n    }\n};\n\nlet unAckCount = 0; // Number of unacknowledged messages.\n\nconst topics = [\"topic.testTopic\"];\n\nconst options = {\n  queue: 'testQueue',\n  ackRequired: 1, // When set to 1, all delivered messages MUST get acknowledge.\n  broadcastSubscription: 1, // When set to 1, it will subscribe to broadcast channel and receive all broadcast messages. \n  prefetch:10\n};\n\nconst processMessage = function(msgContent) {\n  // Process message code here.\n  // msgContent is the message string, which needs to be JSON parsed to get message object.\n};\n\nconst readCallback = function(msgContent) {\n  // Please make sure to return promise in callback function. \n  // On resolving the promise, the message will get acknowledged.\n  // On rejecting the promise, the message will be re-queued (noAck)\n  return new Promise(async function(onResolve, onReject) {\n    // Incrementing unacknowledged message count.\n    unAckCount++;\n\n    // Process the message. Following is a \n    response = await processMessage(msgContent);\n  \n    // Complete the task and in the end of all tasks done\n    if(response == success){\n      // The message MUST be acknowledged here.\n      // To acknowledge the message, call onResolve\n      // Decrementing unacknowledged message count.\n      unAckCount--;\n      onResolve();   \n    } else {\n      //in case of failure to requeue same message.\n      onReject();\n    }\n  })    \n};\n\nconst subscription = {}; // object to store the consumer tag\nconst subscribeCallback = function(consumerTag) {\n  subscription.consumerTag = consumerTag;\n};\n\nconst subscribe = async function() {\n  const queueManager = await Queue.getInstance(configStrategy);\n  queueManager.subscribeEvent.rabbit(\n    topics, // List of topics\n    options,\n    readCallback,\n    subscribeCallback\n   );\n};\n\n// Gracefully handle SIGINT, SIGTERM signals.\n// Once SIGINT/SIGTERM signal is received, programme will stop consuming new messages. \n// But, the current process MUST handle unacknowledged queued messages.\nprocess.on('SIGINT', function () {\n  // Stop the consumption of messages\n  process.emit('CANCEL_CONSUME', subscription.consumerTag);\n\n  console.log('Received SIGINT, checking unAckCount.');\n  const f = function(){\n    if (unAckCount === 0) {\n      process.exit(1);\n    } else {\n      console.log('waiting for open tasks to be done.');\n      setTimeout(f, 1000);\n    }\n  };\n  // Wait for open tasks to be done.\n  setTimeout(f, 1000);\n});\n\nfunction rmqError(err) {\n  console.log('rmqError occured.', err);\n  process.emit('SIGINT');\n}\n// Event published from package in case of internal error.\nprocess.on('rmq_error', rmqError);\n\nsubscribe();\n```\n\n### Publish\n\nFollowing snippet publishes a task for specific topics.\n\n```js\n// Config Strategy for True Sparrow Queue.\nconfigStrategy = {\n\t'rabbitmq': {\n        'username': 'guest',\n        'password': 'guest',\n        'host': '127.0.0.1',\n        'port': '5672',\n        'heartbeats': '30',\n        'enableRabbitmq': 1\n    }\n};\n\nconst topics = [\"topic.testTopic\"];\n\nconst message = {\n   kind: 'testMessageKind',\n   payload: {\n      // Custom payload for message\n   }\n};\n\n// Import the Queue module.\nconst Queue = require('@truesparrow/queue');\nconst publish = async function() {\n  const queueManager = await Queue.getInstance(configStrategy);\n  queueManager.publishEvent.perform(\n    {\n      topics: topics,\n      publisher: 'MyPublisher',\n      publishAfter: 30*1000, // delay in milli-seconds\n      message: message\n    });\n};\n\npublish();\n```\n\n### Publish with delay\n\nIn some use cases, it is required to process certain task with a delay. For example, after one hour of user sign-up, we need to send an email.\nSuch tasks can be published by using the `publishAfter` parameter. Internally, we use [dead letter exchange](https://www.rabbitmq.com/dlx.html) for achieving this functionality.\n\n**Important Note**: Do not use arbitrary values of delays. Internally, the message is stored in a delay specific queue for the waiting duration. As the number of allowed delays increases, so do the number of waiting queues. Having too many queues, can hamper RabbitMQ performance.\n\n```js\n// Config Strategy for True Sparrow Queue.\nconfigStrategy = {\n\t'rabbitmq': {\n        'username': 'guest',\n        'password': 'guest',\n        'host': '127.0.0.1',\n        'port': '5672',\n        'heartbeats': '30',\n        'enableRabbitmq': 1\n    }\n};\n\nconst topics = [\"topic.testTopic\"];\n\nconst message = {\n   kind: 'testMessageKind',\n   payload: {\n      // Custom payload for message\n   }\n};\n\n// Import the Queue module.\nconst Queue = require('@truesparrow/queue');\nconst publish = async function() {\n  const queueManager = await Queue.getInstance(configStrategy);\n  queueManager.publishEvent.perform(\n    {\n      topics: topics,\n      publisher: 'MyPublisher',\n      message: message\n    });\n};\n\npublish();\n```\n\n### Cancel and Resume message consumption\n\nAs seen in the subscribe snippet, cancelling consumption is the first step in SIGINT handling. For cancelling the consumption,\nconsumerTag is needed, which is obtained in subscribeCallback. See subscribe snippet above for more details.\n\nFor cancelling the consumption, emit `CANCEL_CONSUME` event with consumerTag info.\n```js\nprocess.emit('CANCEL_CONSUME', consumerTag);\n```\n\nFor resuming the consumption, emit `RESUME_CONSUME` event with consumerTag info.\n```js\nprocess.emit('RESUME_CONSUME', consumerTag);\n```\n\n## Running test cases\nRun following command to execute test cases.\n```shell script\n./node_modules/.bin/mocha --recursive \"./test/**/*.js\"\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftruesparrowsystems%2Fqueue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftruesparrowsystems%2Fqueue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftruesparrowsystems%2Fqueue/lists"}