{"id":18821156,"url":"https://github.com/compwright/batching-queue","last_synced_at":"2025-08-30T16:39:01.525Z","repository":{"id":42810305,"uuid":"268648715","full_name":"compwright/batching-queue","owner":"compwright","description":"A simple batching queue","archived":false,"fork":false,"pushed_at":"2024-06-16T12:55:56.000Z","size":983,"stargazers_count":7,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-27T14:47:46.478Z","etag":null,"topics":["batching","queue"],"latest_commit_sha":null,"homepage":"https://www.npmjs.com/package/batching-queue","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/compwright.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"compwright"}},"created_at":"2020-06-01T22:44:42.000Z","updated_at":"2024-12-04T08:28:24.000Z","dependencies_parsed_at":"2023-02-08T13:16:54.748Z","dependency_job_id":"9986b3e4-ce48-4f28-8e25-0496d0931313","html_url":"https://github.com/compwright/batching-queue","commit_stats":{"total_commits":36,"total_committers":4,"mean_commits":9.0,"dds":"0.33333333333333337","last_synced_commit":"a6b6b3fec6a873f3561e5a1c42ef1c1608c7d86b"},"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/compwright%2Fbatching-queue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/compwright%2Fbatching-queue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/compwright%2Fbatching-queue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/compwright%2Fbatching-queue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/compwright","download_url":"https://codeload.github.com/compwright/batching-queue/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248800113,"owners_count":21163404,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["batching","queue"],"created_at":"2024-11-08T00:34:47.422Z","updated_at":"2025-04-14T00:26:31.261Z","avatar_url":"https://github.com/compwright.png","language":"JavaScript","funding_links":["https://github.com/sponsors/compwright"],"categories":[],"sub_categories":[],"readme":"# batching-queue\n\n[![Download Status](https://img.shields.io/npm/dm/batching-queue.svg?style=flat-square)](https://www.npmjs.com/package/batching-queue)\n[![Sponsor on GitHub](https://img.shields.io/static/v1?label=Sponsor\u0026message=❤\u0026logo=GitHub\u0026link=https://github.com/sponsors/compwright)](https://github.com/sponsors/compwright)\n\nA batching queue for items that need to be enqueued one by one, but dequeued in batches\n\n* Works with multiple storage backends\n* Queues items one by one\n* Dequeues in batches (FIFO)\n* Tracks the number of waiting items\n* Emits an event when the number of waiting items reaches the batch size threshold\n\n## Installation\n\n```\nnpm install batching-queue\n```\n\n## Included Storage Backends\n\n* MemoryStore - stores items in an array (not for production use)\n* AsyncMemoryStore - same as MemoryStore, but with async methods\n* RedisStore - stores items in Redis\n\n## Example Usage\n\n```javascript\nimport { BatchingQueue, MemoryStore } from 'batching-queue'\n\nconst queue = new BatchingQueue({\n  store: new MemoryStore(),\n  batchSize: 12\n})\n\nconst drain = async (batchesWaiting) =\u003e {\n  const result = await batchesWaiting\n  console.log(result, 'batches waiting')\n\n  for (let i = 0; i \u003c result; i++) {\n    const batch = await queue.dequeue()\n    console.log(batch, 'batch')\n  }\n}\n\n(async () =\u003e {\n  // drain remaining batches immediately\n  drain(queue.length)\n\n  // drain batches every 12 items\n  queue.on('drain', drain)\n\n  for (var i = 0; i \u003c 1000; i++) {\n    await queue.enqueue(i)\n  }\n})()\n```\n\n## API\n\n### BatchingQueue\n\n#### constructor(config)\n\nConfig:\n\n* `store` (require) Storage backend for the queue (see src/stores/memory.js for a reference implementation)\n* `batchSize` (required) Batch size\n\n#### enqueue(item)\n\nAdds an item to the queue. Returns `true` if a batch is full, otherwise returns `false`.\n\n#### dequeue()\n\nDequeues the first [batchSize] number of items. If there are not enough items to satisfy batchSize, all the items available will be returned.\n\n#### length\n\nProperty reports the number of batches waiting in the queue.\n\n\u003e Note: if the storage backend is not initialized, this will return null.\n\n### Storage Backend Interface\n\nYou may to implement your own storage backend to interface your database or cache. See the bundled MemoryStore or AsyncMemoryStore for a reference.\n\n#### setup()\n\nInitialize and set up the storage backend. Establish connections, create database collections, etc.\n\nThis will be called before calling `enqueue()` or `dequeue()` when `ready === false`.\n\n#### destroy()\n\nDestroy the storage backend (optional).\n\n#### async enqueue(item)\n\nStore an item in the queue.\n\nThe store must return the *exact* number of items in the store subsequent to the storage operation.\n\n#### async dequeue(batchSize)\n\nRetrieve and delete [batchSize] items from the storage backend.\n\nReturn the set of items retrieved.\n\n#### length\n\nReturn the number of items in the store.\n\n#### ready\n\nReturn `true` if ready and setup, otherwise, return `false`.\n\n### RedisStore\n\nThe RedisStore class has the following additional methods:\n\n#### constructor(config)\n\nConfig:\n\n* `redisClient` (required) Instance of [node-redis](https://npmjs.org/package/redis)\n* `name` Queue list key name, will be randomized if omitted\n\n### IoredisStore\n\nThe RedisStore class has the following additional methods:\n\n#### constructor(config)\n\nConfig:\n\n* `redisClient` (required) Instance of [ioredis](https://npmjs.org/package/ioredis)\n* `name` Queue list key name, will be randomized if omitted\n\n## License\n\nMIT License\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcompwright%2Fbatching-queue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcompwright%2Fbatching-queue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcompwright%2Fbatching-queue/lists"}