{"id":16578037,"url":"https://github.com/glynnbird/qrate","last_synced_at":"2025-04-23T20:23:47.781Z","repository":{"id":47241529,"uuid":"93296123","full_name":"glynnbird/qrate","owner":"glynnbird","description":"A Node.js queue that provides concurrency and rate-limiting (based on async.queue)","archived":false,"fork":false,"pushed_at":"2024-07-12T15:43:35.000Z","size":122,"stargazers_count":21,"open_issues_count":1,"forks_count":3,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-12T06:48:01.679Z","etag":null,"topics":["nodejs","queue","rate-limiting"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/glynnbird.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-06-04T07:02:23.000Z","updated_at":"2024-07-12T15:43:39.000Z","dependencies_parsed_at":"2024-10-26T20:29:02.019Z","dependency_job_id":"43317670-ac29-4ea1-af18-f4db5a062a93","html_url":"https://github.com/glynnbird/qrate","commit_stats":{"total_commits":10,"total_committers":2,"mean_commits":5.0,"dds":"0.19999999999999996","last_synced_commit":"03ef42b3d80506fd74a3e02969a1532c3c70789c"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/glynnbird%2Fqrate","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/glynnbird%2Fqrate/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/glynnbird%2Fqrate/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/glynnbird%2Fqrate/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/glynnbird","download_url":"https://codeload.github.com/glynnbird/qrate/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250507034,"owners_count":21441908,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["nodejs","queue","rate-limiting"],"created_at":"2024-10-11T22:12:57.390Z","updated_at":"2025-04-23T20:23:47.760Z","avatar_url":"https://github.com/glynnbird.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# qrate\n\n[![Build Status](https://travis-ci.org/glynnbird/qrate.svg?branch=master)](https://travis-ci.org/glynnbird/qrate) [![npm version](https://badge.fury.io/js/qrate.svg)](https://badge.fury.io/js/qrate)\n\n## Introduction\n\nThe queue library based on the [async.queue](http://caolan.github.io/async/docs.html#queue) utility but modified to allow a queue's throughput to be controlled in terms of:\n\n- concurrency - the maximum number of workers running at any point in time\n- rateLimit - the maximum number of workers started per second\n\nThe default behaviour is a `concurrency` of 1 (one worker at a time) and a `rateLimit` of null (no rate limiting).\n\nThe *qrate* library can be used as a drop-in replacement for the [async.queue](http://caolan.github.io/async/docs.html#queue) function.\n\n## Installation\n\nInstall with\n\n```sh\nnpm install qrate\n```\n\nor to import it into your Node.js project:\n\n```sh\nnpm install --save qrate\n```\n\n## Usage\n\nA queue is created by calling `qrate` passing in the the worker function you want to operate on each item in the queue. The returned `q` can then be used to push data into the queue.\n\n```js\n\n// require qrate library\nconst qrate = require('qrate');\n\n// mark the start time of this script\nconst start = new Date().getTime();\n\n// worker function that calls back after 100ms\nconst worker = function(data, done) {\n\n  // your worker code goes here\n  // 'data' contains the queue to work on\n  // call 'done' when finished.\n\n\n  // output a message including a timestamp\n  console.log('Processing', data, '@', new Date().getTime() - start, 'ms');\n\n  // call the 'done' function after 100ms\n  setTimeout(done, 100);\n};\n\n// create a queue with default properties (concurrency = 1, rateLimit = null)\n// using our 'worker' function to process each item in the queue\nconst q = qrate(worker);\n\n// add ten things to the queue\nfor (let i = 0; i \u003c 10; i++) {\n  q.push({ i: i });\n}\n```\n\nThe queue has the default `concurrency` of 1, so worker starts after its predecessor finishes:\n\n```sh\nProcessing { i: 0 } @ 21 ms\nProcessing { i: 1 } @ 129 ms\nProcessing { i: 2 } @ 233 ms\nProcessing { i: 3 } @ 338 ms\nProcessing { i: 4 } @ 441 ms\nProcessing { i: 5 } @ 545 ms\nProcessing { i: 6 } @ 650 ms\nProcessing { i: 7 } @ 751 ms\nProcessing { i: 8 } @ 852 ms\nProcessing { i: 9 } @ 958 ms\n```\n\nWe can increase the number of workers running in parallel by passing a `concurrency` value as a second parameter:\n\n```js\n\n// create a queue where up to three workers run at any time\nconst q = qrate(worker, 3);\n```\n\nwhich speeds things up significantly:\n\n```\nProcessing { i: 0 } @ 27 ms\nProcessing { i: 1 } @ 33 ms\nProcessing { i: 2 } @ 35 ms\nProcessing { i: 3 } @ 134 ms\nProcessing { i: 4 } @ 135 ms\nProcessing { i: 5 } @ 135 ms\nProcessing { i: 6 } @ 235 ms\nProcessing { i: 7 } @ 235 ms\nProcessing { i: 8 } @ 236 ms\nProcessing { i: 9 } @ 340 ms\n```\n\nSo far we have not done anything that a normal `async.queue` could do. This is where the third parameter comes in.\n\n## Rate limiting the queue\n\nIf you want to limit the rate of throughput of the queue (e.g. 5 jobs per second), then you can pass a third `rateLimit`  parameter to `qrate`. The `rateLimit` indicates the maximum number workers per second you want the queue to start:\n\n- rateLimit = 1 - one per second\n- rateLimit = 5 - five per second\n- rateLimit = 0.5 - one every two seconds\n- rateLimit = null - as fast as possible (default)\n\n```js\n// concurrency 1, rateLimit 2 workers per second\nconst q = qrate(worker, 1, 2);\n```\n\nwhich produces the output:\n\n```\nProcessing { i: 0 } @ 16 ms\nProcessing { i: 1 } @ 126 ms\nProcessing { i: 2 } @ 1007 ms\nProcessing { i: 3 } @ 1111 ms\nProcessing { i: 4 } @ 2013 ms\nProcessing { i: 5 } @ 2118 ms\nProcessing { i: 6 } @ 3018 ms\nProcessing { i: 7 } @ 3124 ms\nProcessing { i: 8 } @ 4025 ms\nProcessing { i: 9 } @ 4127 ms\n```\n\nNotice how in the early part of each second, two workers are executed in turn, then the queue waits until the next second boundary before resuming work again.\n\nRate-limiting is useful if you want to ensure that the number of API calls your code generates stays below the API provider's quota, e.g. five API calls per second.\n\n## Worker functions with callbacks\n\nYour worker function can be a standard JavaScript function with two parameters\n\n- the payload - the data that your function receives from the queue.\n- a callback - you call this function to indicate that the worker has finished its work.\n\n```js\n// worker function that calls back after 100ms\nconst worker = function(data, done) {\n  // let's imagine we're writing data to a database\n  // This is typically an asynchronous action.\n  db.insert(data, function(err, insertData) {\n    // now we can call the callback function to show that we're finished\n    done(err, insertData)\n  })\n};\n```\n\n## Work functions with Promises\n\nAlternatively, a more modern pattern is to define your worker function as an `async` function. This allows you to deal with asynchronous activity, like database calls, without callbacks. This time the function only accepts one parameter:\n\n```js\nconst worker = async (data) =\u003e {\n  const insertData = await db.insert(data)\n  return {ok: true}\n};\n```\n\n## Detecting that the queue is empty\n\nIf you create a `q.drain` function, it will be called when the queue size reaches zero. This can be used as a trigger to publish results, fetch more work or to kill the queue \u0026 tidy up. (see Killing the queue).\n\n```js\nq.drain = () =\u003e {\n  // all of the queue items have been processed\n  console.log('the queue is empty');\n}\n```\n\n## Killing the queue\n\nA rate-limited `qrate` queue sets up a timer to handle the throttling of a rate-limited queue. The queue can be cleaned up by calling the `q.kill()` function.\n\nIf your application is working through a single list of work, the you can provide a `q.drain` function that is called when a queue is emptied and call `q.kill` in that function:\n\n```js\nq.drain = () =\u003e {\n  console.log('the queue is empty');\n  q.kill();\n};\n```\n\nor, simply tie the `drain` and `kill` functions together:\n\n```js\nq.drain = q.kill;\n```\n\nIn other applications, you may wish to keep the queue alive and periodically feed it with fresh work. \n\n\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fglynnbird%2Fqrate","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fglynnbird%2Fqrate","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fglynnbird%2Fqrate/lists"}