{"id":20229271,"url":"https://github.com/hisco/zeroq","last_synced_at":"2025-04-10T17:51:20.333Z","repository":{"id":57405673,"uuid":"133269258","full_name":"hisco/zeroq","owner":"hisco","description":"High performace in-memory tasks queue for Node.js. Perfect for executing in batches while utilizing resources to the maximum","archived":false,"fork":false,"pushed_at":"2020-05-23T11:51:41.000Z","size":177,"stargazers_count":6,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-04-26T06:04:36.765Z","etag":null,"topics":["async","batches","bulk","concurrency","javascript","nodejs","parallel","queue","tasks","typescript"],"latest_commit_sha":null,"homepage":"https://www.npmjs.com/package/zeroq","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hisco.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-05-13T19:48:38.000Z","updated_at":"2020-09-18T15:33:22.000Z","dependencies_parsed_at":"2022-09-19T11:21:13.512Z","dependency_job_id":null,"html_url":"https://github.com/hisco/zeroq","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hisco%2Fzeroq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hisco%2Fzeroq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hisco%2Fzeroq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hisco%2Fzeroq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hisco","download_url":"https://codeload.github.com/hisco/zeroq/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248262433,"owners_count":21074308,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async","batches","bulk","concurrency","javascript","nodejs","parallel","queue","tasks","typescript"],"created_at":"2024-11-14T07:35:05.949Z","updated_at":"2025-04-10T17:51:20.317Z","avatar_url":"https://github.com/hisco.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Fast queue - ZeroQ\n\n[![Greenkeeper badge](https://badges.greenkeeper.io/hisco/zeroq.svg)](https://greenkeeper.io/)\n\n[![NPM Version][npm-image]][npm-url]\n[![Build Status][travis-image]][travis-url]\n[![Test Coverage][coveralls-image]][coveralls-url]\n\n  High performace in-memory tasks queue for Node.js.\n  Perfect for executing in batches while utilizing resources to the maximum.\n\n## High perormance\nZeroQ was built to endure great tasks load, while preserving the order of the tasks.\nIt surpassing other implementations by:\n\n  * It's in-memory data structure is faster - Linked list instead of Array (I measured 200X latency with `[].shift` comparing to changing a single pointer of linked list)\n  * Super-high test coverage\n  * Out of the box support for TypeScript.\n  * No dependencies.\n\nI currently don't provide other important queue features such as priority queue as I found current popular implementation fast enough with the features these provide.\n\nSpeed was our greatest concern and not a full featured queue.\n## Simple to use\n\n### Using Javascript\n```js\n  //Require all ZeroQ module\n  const ZeroQ = require('zeroq');\n  //Now you can use both `ZeroQ.DataQueue` and `ZeroQ.TasksQueue`\n\n  //Or Directly load the desired class\n  const {DataQueue ,TasksQueue} = require('zeroq');\n\n```\n\n### Using TypeScript\n```ts\n  import {DataQueue , TasksQueue} from 'zeroq';\n  //Now you can use both `ZeroQ.DataQueue` and `ZeroQ.TasksQueue`\n\n```\n\n## API\n\nBoth Queues supports `maxConcurrency` number as first argument.\nThis enables locking/releasing multiple resources with a single queue.\n\n### Tasks queue - TasksQueue\u003cT\u003e(maxConcurrency : number , isSyncMode : boolean )\nThis class was created to distribute tasks in the order these were pushed.\nTasks will be excuted once locked resource will be released.\n\n```js\nconst {TasksQueue} = require('zeroq');\n//Create a sync queue with no task executing in parallel\nconst queue = new TasksQueue(1 , true);\n//Push some task to queue\nqueue.push(()=\u003e{\n    console.log('task 1');\n});\n//=\u003e task 1\nqueue.push(()=\u003e{\n    console.log('task 2');\n});\nqueue.release();\n//=\u003e task 2\n```\n\n### Data queue - DataQueue\u003cT\u003e(maxConcurrency : number , onData : (data:T)=\u003evoid )\nThis class was created to distribute data in the order these were pushed.\nData will be pushed once locked resource will be released.\n\n```js\nconst {DataQueue} = require('zeroq');\n//Create a sync queue with no task executing in parallel\nconst queue = new DataQueue(1 , function onData(data){\n    console.log('just recived ' , data);\n});\n//Any data can be pushed to the queue it will be release into your callback as is\nqueue.push({ eventName : 'click'});\n//=\u003ejust recived { eventName : 'click'}\nqueue.push(\"https://www.example.com\");\nqueue.push(200);\n\nsetTimeout(()=\u003e{\n    queue.release();\n//=\u003ejust recived https://www.example.com\n    queue.release();\n//=\u003ejust recived 200\n} , 2000);\n```\n\n##Examples\n\n### Making multiple parallel request\nUsually when making requests with Noed.JS the best approch is to request I/O operation to execute in parallel and wait for the last one to return.\n\n```js\n  const rp = require('request-promise');\n  Promise.all([\n      rp('https://www.example.com/resource/1'),\n      rp('https://www.example.com/resource/2'),\n      rp('https://www.example.com/resource/3')\n  ])\n  .then(function (results) {\n      //All requests have finished\n  })\n  .catch(function (err) {\n      //One of the requests have failed\n  });\n```\nWhile it is in *most* cases the best approch when the amount of requests required is much higher, requesting from the OS to preform so many requests at once will probably result in random recurring timeouts.\n\nThe best approach is to lock a big number of requests.\nAs long as the number of requests is lower than the maximum concurrency - *all* requests will execute together.\nElse it will be executed in bulks.\n```js\n  const rp = require('request-promise');\n  const requestQueue = new TasksQueue(30);\n\n  const thousendsOfResources = [];\n  //No matter the number in this case only 30 request will be exectured in parllel\n  for (let i =0; i\u003c50 ;i++){\n      thousendsOfResources.push(`https://www.example.com/resource/${i}`);\n  }\n\n  Promise.all(\n      thousendsOfResources.map((requestOptions) =\u003e{\n          return new Promise(function (reoslve , reject){\n              requestQueue.push(function makeRequestTask(){\n                  return rp(requestOptions)\n                      .then((result)=\u003e{\n                          //This single request has finished excuting successfully\n                          //Do something with this single result\n                          //You must not forget to release the request resource\n                          requestQueue.release(); \n                          return result\n                      })\n                      .catch((error)=\u003e{\n                          //This single request has failed excuting\n                          //Do something with this error \n                          //You must not forget to release the request resource\n                          requestQueue.release(); \n                          throw error;\n                      })\n              })\n          })\n      }\n  ))\n  .then((results)=\u003e{\n      //All request have finshed successfully\n  })\n\n```\n### Synchronous writing to an IPC socket\nWhile Node.js doesn't preform any task in parallel, the OS do.\nSome of the following messages will be lost because the OS can't write two messages in parallel.\n```js\nconst net = require('net');\nconst client = net.createConnection({ path: '/tmp/app.sock' }, () =\u003e {\n    for (let i=0;i\u003c10000;i++){\n        client.write(`check out ${i}`)\n    }\n});\n```\nWith tasks queue it's super safe and simple to make sure all your messages would arrive\n```js\nconst net = require('net');\nconst writeQueue = new TasksQueue(1);\nconst client = net.createConnection({ path: '/tmp/app.sock' }, () =\u003e {\n    for (let i=0;i\u003c10000;i++){\n        writeQueue.push(()=\u003e{\n            client.write(`check out ${i}` , writeQueue.release)\n        })\n    }\n});\n\n```\n### Asynchronous proccesing thousands of files\n```js\n  const queue = new TasksQueue(1000);\n  const glob = require('glob');\n  const {readFile} = require('fs')\n\n  const getFileNames = new Promise(()=\u003e{\n      glob('**/*.js', function (er, files) {\n          // files is an array of filenames.\n          if (err)\n              reject(err);\n          else //Assume that you have millions of file names in this array\n              resolve(files);\n      })\n  });\n  function readAndProcessSingleFile(filename){\n      return new Promise((resolve , reject)=\u003e{\n          readFile(fileName , (err , data)=\u003e{\n              if (err)\n                  reject(err);\n              else{\n                  resolve(data);\n                  //doSomething(data);\n              }\n              queue.release();\n          })\n      })\n  }\n\n  getFileNames\n      .then((fileNames)=\u003e \n          Promise.all(\n              fileNames.map((fileName)=\u003e\n                  new Promise((resolve , reject)=\u003e{\n                      queue.push(()=\u003e{\n\n                      });\n                  })\n              )\n          )\n      )\n```\n\n## License\n\n  [MIT](LICENSE)\n\n[npm-image]: https://img.shields.io/npm/v/zeroq.svg\n[npm-url]: https://npmjs.org/package/zeroq\n[travis-image]: https://img.shields.io/travis/hisco/zeroq/master.svg?style=flat-square\n[travis-url]: https://travis-ci.org/hisco/zeroq\n[coveralls-image]: https://coveralls.io/repos/github/hisco/zeroq/badge.svg?branch=master\n[coveralls-url]: https://coveralls.io/github/hisco/zeroq?branch=master\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhisco%2Fzeroq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhisco%2Fzeroq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhisco%2Fzeroq/lists"}