{"id":16229240,"url":"https://github.com/morris/mongomq2","last_synced_at":"2025-08-30T17:38:12.578Z","repository":{"id":57301904,"uuid":"456220459","full_name":"morris/mongomq2","owner":"morris","description":"A general-purpose message and event queuing library for MongoDB","archived":false,"fork":false,"pushed_at":"2025-02-16T08:42:52.000Z","size":1642,"stargazers_count":14,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-28T19:01:20.056Z","etag":null,"topics":["event-driven","message-queue","mongodb"],"latest_commit_sha":null,"homepage":"https://morris.github.io/mongomq2/","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"isc","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/morris.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-02-06T17:26:27.000Z","updated_at":"2025-02-16T08:42:56.000Z","dependencies_parsed_at":"2024-02-11T15:29:50.432Z","dependency_job_id":"891fdd20-9304-44cb-a502-f4b8f3d590cb","html_url":"https://github.com/morris/mongomq2","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/morris%2Fmongomq2","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/morris%2Fmongomq2/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/morris%2Fmongomq2/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/morris%2Fmongomq2/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/morris","download_url":"https://codeload.github.com/morris/mongomq2/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243995880,"owners_count":20380898,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["event-driven","message-queue","mongodb"],"created_at":"2024-10-10T12:57:45.838Z","updated_at":"2025-08-30T17:38:12.571Z","avatar_url":"https://github.com/morris.png","language":"TypeScript","funding_links":[],"categories":["TypeScript","Libraries"],"sub_categories":["JavaScript"],"readme":"# MongoMQ2\n\n[![NPM version](https://img.shields.io/npm/v/mongomq2?style=flat-square)](https://www.npmjs.com/package/mongomq2)\n[![Build status](https://img.shields.io/github/actions/workflow/status/morris/mongomq2/pipeline.yml?branch=main\u0026style=flat-square)](https://github.com/morris/mongomq2/actions)\n[![Coverage](https://img.shields.io/codecov/c/github/morris/mongomq2?style=flat-square\u0026token=5GBOZPEJW0)](https://app.codecov.io/gh/morris/mongomq2)\n\nMongoMQ2 is a light-weight Node.js library that turns MongoDB collections into\n**general-purpose message queues** or event logs,\n_without_ additional deployments or infrastructure.\n\nAt an expense of throughput compared to specialized\nmessage queues and brokers like SQS, SNS, RabbitMQ or Kafka, you get:\n\n- Durable message/event logs in MongoDB collections.\n- Real-time, fan-out, at-most-once delivery to **subscribers**.\n- Isolated, acknowledged, at-least-once delivery to **queue consumers**.\n  - Effectively exactly-once if consumer workloads are idempotent.\n- All the capabilities of regular MongoDB collections, e.g.\n  - search indexes,\n  - unique indexes for message/event deduplication,\n  - aggregations,\n  - capped collections,\n  - transactions,\n  - sharding,\n  - and TTL indexes.\n- No chaining of queues required because subscribers and consumers can read from the same queue.\n- Low-cost ops (no additional infrastructure besides a Node.js app and MongoDB)\n\nThere's more:\n\n- Configurable number of retries\n- Configurable visibility timeouts\n- Configurable visibility delays\n- Multiple isolated consumer groups on one queue\n- Batch publishing of messages/events\n\nMongoMQ2 can be an effective and flexible building block for\nmessage- and event-driven architectures,\nespecially if you're already on MongoDB\nand don't want to introduce additional system components to deploy and operate.\n\n## Installation\n\n```sh\nnpm install mongomq2 mongodb\n```\n\n## Quick Start\n\n```ts\nimport { MongoClient, ObjectId } from 'mongodb';\nimport { MessageQueue } from 'mongomq2';\n\nconst mongoClient = new MongoClient('mongodb://localhost:27017');\n\ntype MyMessage = InputMessage | OutputMessage;\n\ninterface InputMessage {\n  _id?: ObjectId;\n  type: 'input';\n  data: string;\n}\n\ninterface OutputMessage {\n  _id?: ObjectId;\n  type: 'output';\n  result: string;\n}\n\n// create MessageQueue\nconst messageCollection = mongoClient.db().collection\u003cMyMessage\u003e('messages');\nconst messageQueue = new MessageQueue(messageCollection);\n\n// Consume \"input\" messages (including past ones)\n// Publish one \"output\" message per \"input\" message\nmessageQueue.consume\u003cInputMessage\u003e(\n  // consumer callback to be executed at least once per message\n  async (message) =\u003e {\n    console.log(`Processing ${message.data}...`);\n\n    await messageQueue.publish({ type: 'output', result: message.data + '!' });\n  },\n  {\n    group: 'handleInput', // group identifier, unique per consumer callback\n    filter: { type: 'input' }, // only consume messages of type \"input\"\n  },\n);\n\n// Subscribe to (future) \"output\" messages\nmessageQueue.subscribe\u003cOutputMessage\u003e(\n  (message) =\u003e {\n    console.log(`Processing done: ${message.result}`);\n  },\n  { filter: { type: 'output' } },\n);\n\n// Publish some messages\nawait messageQueue.publish({ type: 'input', data: 'hello' });\nawait messageQueue.publish({ type: 'input', data: 'world' });\n\n// \u003e Processing xxx... (processed exactly once)\n// \u003e Processing done: xxx! (per active subscriber)\n```\n\n## Usage\n\n(See [API documentation](https://morris.github.io/mongomq2/)\nfor a detailed reference of all configuration and functionalities.)\n\n### Setup\n\n```ts\nconst messageCollection = mongoClient.db().collection\u003cMyMessage\u003e('messages');\nconst messageQueue = new MessageQueue(messageCollection);\n```\n\n### Publishing\n\n```ts\nawait messageQueue.publish({ type: 'input' });\n```\n\n- Publishes the given message to the queue immediately.\n- Message insertion is acknowledged, or an error is thrown.\n\nUseful for:\n\n- Critical messages and events\n- Job ingestion\n- Commands\n\nCan be used inside transactions by passing a session (same as MongoDB `insertOne`).\n\n### Batched Publishing\n\n```ts\nmessageQueue.publishBatched({ type: 'input' });\n```\n\n- Queues the given message for publication in memory.\n- Bulk inserts batched messages after a configurable delay.\n- By default, publishes messages with best effort (`majority` write concern, retries)\n- Can be set to \"fire \u0026 forget\" mode by passing `bestEffort: false` (no write concern, no retries)\n\nUseful for:\n\n- Uncritical messages\n- Uncritical notifications\n\n### Consumers\n\n```ts\nmessageQueue.consume(\n  (message) =\u003e {\n    // handle message\n  },\n  {\n    // consumer group identifier, defaults to collection name\n    group: 'myConsumerGroup',\n    filter: {\n      // optional filter\n    },\n  },\n);\n\nmessageQueue.on('deadLetter', (err, message, group) =\u003e {\n  // handle dead letter, i.e. message that failed repeatedly and exhausted maxRetries\n});\n```\n\n- Consumes future and past matching messages.\n- Order of message consumption is not guaranteed.\n- Per unique `group`, each matching message is consumed by at most one consumer.\n- Messages are consumed at least once per `group`.\n  - Keep the `group` property stable per consumer callback.\n  - Otherwise, messages will be reprocessed (once per unique `group`).\n- Configurable visibility timeout, visibility delay, maximum number of retries, etc.\n\nUseful for:\n\n- Message queues\n- Job queues\n- Event processing\n- Command processing\n\n#### Explicit Retries in Consumer Callbacks\n\nConsumer callbacks receive a context object containing\n\n- the number of retries for the received message so far, and\n- a function to retry consuming the message after a specified number of seconds.\n\nThese can be used, for example, to implement\n[exponential backoff](https://en.wikipedia.org/wiki/Exponential_backoff):\n\n```ts\nmessageQueue.consume(\n  (message, context) =\u003e {\n    try {\n      // handle message\n    } catch (err) {\n      await context.retry(\n        Math.min(Math.pow(2, context.retries + 1), 30), // 2, 4, 8, 16, 30, 30, 30, 30, ...\n      );\n\n      throw err;\n    }\n  },\n  { maxRetries: 10 },\n);\n```\n\nCalling `context.retry()` does not circumvent\nthe configured maximum number of retries;\nit only prevents acknowledgement of the message (even if no error is thrown)\nand extends the visibility timeout (if specified). In the following example,\nretries are triggered without throwing errors:\n\n```ts\nmessageQueue.consume(\n  (message, context) =\u003e {\n    if (isReady(message)) {\n      // handle message\n    } else {\n      // retry after globally configured visibility timeout\n      await context.retry();\n    }\n  },\n  { maxRetries: 10 },\n);\n```\n\n### Subscriptions\n\n```ts\nmessageQueue.subscribe(\n  (message) =\u003e {\n    // handle message\n  },\n  {\n    filter: {\n      // optional local filter applied in memory\n    },\n  },\n);\n```\n\n- Subscribes to matching messages in the future.\n- All active subscribers will receive all future matching messages.\n- Messages are delivered at most once.\n- Messages are delivered in database insertion order.\n- Past messages are ignored.\n- Each `MessageQueue` instance creates one MongoDB change stream.\n  - Change streams occupy one connection,\n  - so you'll usually want only exactly one `MessageQueue` instance,\n  - and multiple `.subscribe(...)` calls with local filters.\n\nUseful for:\n\n- Real-time notifications\n- Cache invalidation\n\n### Additional Notes\n\n- All MongoMQ2 clients are `EventEmitters`.\n- Always attach `.on('error', (err, message?, group?) =\u003e /* report error */)` to monitor errors.\n- Always `.close()` MongoMQ2 clients on shutdown (before closing the MongoClient).\n  - MongoMQ2 will try to finish open tasks with best effort.\n- MongoDB change streams are only supported for MongoDB replica sets.\n  - To start a one-node replica set locally (e.g. for testing), see `docker-compose.yml`.\n- MongoMQ2 relies on the `_id` index which always exists (no other indexes required)\n- MongoMQ2 stores metadata for consumers in a `_c` field per message document (no other metadata is generated)\n\n## Performance\n\nFor common workloads\n(message size ~1 KB, produced and consumed in the same time frame),\nMongoMQ2 should be able to handle **hundreds of messages\nper second** in most environments; plenty for a variety of use cases.\n\nAs discussed earlier, MongoMQ2's trade-offs are\n\n- less infrastructure,\n- more flexibility,\n- but therefore less specialization on queuing (e.g. performance/throughput).\n\nYour mileage may vary.\n\n---\n\nGenerally, MongoMQ2 is bound by the performance and latency\nof the underlying MongoDB.\n\nPublishing/producing messages in MongoMQ2 is bound by insertion time\non the message collection. Insertion time depends on message size\nand number of indexes on the message collection.\nAs stated above, the simplest use cases only need the `_id` index.\n\nConsumers are bound by MongoDB `findOneAndUpdate` performance, which will\nusually perform an index scan (`IXSCAN`) on the `_id` index. This scan is mainly\nbound by the number of messages currently being consumed, as consumers are\nable to seek efficiently based on `_id` via time-based ordering.\n\nAdditionally, `findOneAndUpdate` performs some locking internally,\nwhich may degrade for large numbers of concurrent producers/consumers.\n\nSee `test/benchmarks` for a benchmark suite\n(as of yet, severely lacking - PRs welcome!).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmorris%2Fmongomq2","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmorris%2Fmongomq2","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmorris%2Fmongomq2/lists"}