{"id":21258919,"url":"https://github.com/microfleet/distributed-callback-queue","last_synced_at":"2025-09-07T17:45:00.267Z","repository":{"id":29475728,"uuid":"33012510","full_name":"microfleet/distributed-callback-queue","owner":"microfleet","description":"Creates distributed callback queue based on redis pubsub. Pass 2 redis clients (supports redis sentinel as well for HA), manages locks across multiple processes","archived":false,"fork":false,"pushed_at":"2025-01-20T04:02:55.000Z","size":1849,"stargazers_count":9,"open_issues_count":7,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-08-07T14:12:41.503Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/microfleet.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2015-03-27T22:42:42.000Z","updated_at":"2025-01-20T04:00:53.000Z","dependencies_parsed_at":"2024-06-19T22:49:04.920Z","dependency_job_id":"d779c902-73b6-4af8-b8f5-dd38a3a4ebed","html_url":"https://github.com/microfleet/distributed-callback-queue","commit_stats":null,"previous_names":["avvs/distributed-callback-queue"],"tags_count":21,"template":false,"template_full_name":null,"purl":"pkg:github/microfleet/distributed-callback-queue","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microfleet%2Fdistributed-callback-queue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microfleet%2Fdistributed-callback-queue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microfleet%2Fdistributed-callback-queue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microfleet%2Fdistributed-callback-queue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/microfleet","download_url":"https://codeload.github.com/microfleet/distributed-callback-queue/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microfleet%2Fdistributed-callback-queue/sbom","scorecard":{"id":7509,"data":{"date":"2025-08-04","repo":{"name":"github.com/microfleet/distributed-callback-queue","commit":"4ff9c3b65fc9ae4e5d9c07f67fc77f0ca8244cd2"},"scorecard":{"version":"v5.2.1-28-gc1d103a9","commit":"c1d103a9bb9f635ec7260bf9aa0699466fa4be0e"},"score":1.3,"checks":[{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/c1d103a9bb9f635ec7260bf9aa0699466fa4be0e/docs/checks.md#token-permissions"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/c1d103a9bb9f635ec7260bf9aa0699466fa4be0e/docs/checks.md#packaging"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/c1d103a9bb9f635ec7260bf9aa0699466fa4be0e/docs/checks.md#dangerous-workflow"}},{"name":"Code-Review","score":0,"reason":"Found 1/28 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/c1d103a9bb9f635ec7260bf9aa0699466fa4be0e/docs/checks.md#code-review"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/c1d103a9bb9f635ec7260bf9aa0699466fa4be0e/docs/checks.md#maintained"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/c1d103a9bb9f635ec7260bf9aa0699466fa4be0e/docs/checks.md#binary-artifacts"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/c1d103a9bb9f635ec7260bf9aa0699466fa4be0e/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/c1d103a9bb9f635ec7260bf9aa0699466fa4be0e/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/c1d103a9bb9f635ec7260bf9aa0699466fa4be0e/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/c1d103a9bb9f635ec7260bf9aa0699466fa4be0e/docs/checks.md#fuzzing"}},{"name":"License","score":0,"reason":"license file not detected","details":["Warn: project does not have a license file"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/c1d103a9bb9f635ec7260bf9aa0699466fa4be0e/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/c1d103a9bb9f635ec7260bf9aa0699466fa4be0e/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/c1d103a9bb9f635ec7260bf9aa0699466fa4be0e/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 6 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/c1d103a9bb9f635ec7260bf9aa0699466fa4be0e/docs/checks.md#sast"}},{"name":"Vulnerabilities","score":0,"reason":"11 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GHSA-x4c5-c7rf-jjgv","Warn: Project is vulnerable to: GHSA-h5c3-5r3r-rr8q","Warn: Project is vulnerable to: GHSA-rmvr-2pp2-xj38","Warn: Project is vulnerable to: GHSA-xx4v-prfh-6cgc","Warn: Project is vulnerable to: GHSA-v6h2-p8h4-qcjw","Warn: Project is vulnerable to: GHSA-grv7-fg5c-xmjg","Warn: Project is vulnerable to: GHSA-3xgq-45jj-v275","Warn: Project is vulnerable to: GHSA-mg2h-6x62-wpwc","Warn: Project is vulnerable to: GHSA-p6mc-m468-83gw","Warn: Project is vulnerable to: GHSA-952p-6rrq-rcjv","Warn: Project is vulnerable to: GHSA-cxrh-j4jr-qwg3"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/c1d103a9bb9f635ec7260bf9aa0699466fa4be0e/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-14T13:54:00.599Z","repository_id":29475728,"created_at":"2025-08-14T13:54:00.599Z","updated_at":"2025-08-14T13:54:00.599Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272828499,"owners_count":25000036,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-30T02:00:09.474Z","response_time":77,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-21T04:11:26.480Z","updated_at":"2025-09-07T17:45:00.246Z","avatar_url":"https://github.com/microfleet.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Distributed callback queue\n\n[![Codacy Badge](https://www.codacy.com/project/badge/1a90183ad6964bfca54a7ba0f4b9b3a7)](https://www.codacy.com/app/v/distributed-callback-queue)\n\nPurpose of this module to allow only 1 similar action to run at the same time across any amount\nof the machines. This task is solved by the means of acquiring lock before any action\nis performed. Currently redis is used as a backing provider, which is super fast,\nand more than reliable for this task, especially when combined with redis-sentinel, or cluster\nin the future (once it's production ready).\n\nThis library requires you to pass 2 redis instances: 1 will be used for acquiring and releasing locks\nand the other one for pubsub events (and this is how the system becomes distributed)\n\n`npm install dlock -S`\n\n## Usage\n\n[Opts description](https://github.com/AVVS/distributed-callback-queue/blob/master/lib/distributed-callback-queue.js#L8-L23)\n\n```js\nconst CallbackQueue = require('dlock');\n\n// assume we have 2 redis clients ready\n// one called `redis` and the other `pubsub`. They must be different clients\n// technically they can even connect to different redis instances machines,\n// since they are used for 2 separate tasks\n```\n\n### Sample configuration\n\n```js\nconst opts = {\n  client: redis,  // lock acquisition, can be shared with the rest of the app\n  pubsub: pubsub, // pubsub, please do not share unless you know what you are doing\n  pubsubChannel: '{mypubsubchannel}',\n  lock: {\n    timeout: 20000, // in 20000ms lock will expire, defaults to 10000\n    retries: 0, // if we failed to acquire lock for the first time, retry in `delay`. Defaults to 1\n    delay: 500 // retry in 500 ms when acquiring new lock. Default: 100\n  },\n  lockPrefix: '{mylockprefix}', // used for prefixing keys in redis and in local queue, defaults to {dcb}\n  log: false // turn of logging\n};\n\nconst callbackQueue = new CallbackQueue(opts);\n```\n\n### In-flight Request Caching\n\nPerform only 1 request and fan-out results via redis pubsub on the network, so that\nwe never perform more than 1 requests to the same resource in parallel\n\n```js\n/**\n * Method push\n * Accepts following arguments:\n * `id` {String} identificator of the job. Unique across processes between same lockPrefix\n * `onJobcompleted` {Function} called when job has been completed or failed\n * @returns {Promise} called after `onJobCompleterd` has been added to queue\n */\nconst errPredicate = err =\u003e err.name !== 'LockAcquisitionError';\n\ncallbackQueue\n  .push('jobid', onJobCompleted)\n  .then(completed =\u003e {\n    // do work and call completed once it's done\n    // when completed is called, onJobCompleted will be invoked with the args\n    // that were passed from here\n\n    const nastylongcalculations = 1 + 1;\n    completed(null, nastylongcalculations);\n  })\n  // omit that error\n  .catch({ name: 'LockAcquisitionError' }, noop)\n  // handle this one\n  .catch(errPredicate, err =\u003e {\n    // failed to queue\n  });\n\n/**\n * Will be called once job had been completed\n * This function will be called for every queued callback\n */\nfunction onJobCompleted(err, ...args) {\n    if (err) {\n        // first arg is canonical error\n        // you decide what the rest of args are, but make sure that first one\n        // is an error\n        return console.error('Failed to complete job: ', err);\n    }\n\n    // prints 2\n    console.log(args[0]);\n}\n```\n\n### Async/Await in-flight request caching\n\n#### Fanout(jobId: String, [timeout: Number], job: Function, [...args]: any[])\n\nUse `fanout(...)` method for the easiest way to handle job subscriptions where\none actor must perform long-running job, but as soon as it's done - everyone who\nqueued for the results of this job must be notified.\n\n`args` are optional args to be passed on to the job function. If you need to preserve context - use .bind\n\nSample of code is provided to make use of this feature:\n\n```js\nconst jobId = 'xxx';\nconst job = async () =\u003e {\n  await Promise.delay(10000);\n  return 'done';\n}\n\nlet result;\ntry {\n  result = await callbackQueue.fanout(jobId, 2000, job);\n} catch (e) {\n  // handle timeout error - because it will fail\n}\n\ntry {\n  result = await callbackQueue.fanout(jobId, 20000, job);\n  // result will be equal 'done'\n} catch (e) {\n  // no error is expected, but make sure you handle unexpected cases\n}\n```\n\n### Distributed Resource Locking\n\nAllows to acquire lock across multiple processes with redis based lock\n\n```js\n/**\n * Method `once` - means there can only be 1 concurrent task\n * and callers will be rejected if they can't acquire lock\n */\n\ncallbackQueue\n  .once('app:job:processing')\n  .then(lock =\u003e {\n    // perform lengthy calculations here\n    // call `lock.extend()` if needed\n\n    // do not forget to release lock after the job is done\n    // do not care about result\n    return lock.release()\n  })\n  .catch(err =\u003e {\n    // lock could not be acquire\n  })\n```\n\n### Distributed Locking on Multiple Keys\n\nA little more complex lock, which ensures that we can acquire all locks from a list.\nWhen at least one lock is not acquired - we can't proceed further.\nThis can be helpful in cases when partial resource can be altered in a separate action\nand side-effect from such event would affect further actions from a multi lock.\n\n```js\n/**\n * Method `multi` - similar to once, except that it expects to hold\n * multiple locks concurrently. This is useful when you want to perform non-atomic\n * changes on different resource and must ensure that they don't change during the transaction\n */\nconst { MultiLockError } = CallbackQueue;\ncallbackQueue\n  .multi('1', '2', '3', ['4', '5'])\n  .then(multiLock =\u003e {\n    // multiLock has the same interface as normal lock\n    // .release()\n    // .extend()\n\n    return multiLock\n      // will reject promise if it can't extend ALL the locks,\n      // you can catch `MultiLockError`, which extends Promise.AggregateError\n      // and has the same interface + `.locks`. If MultiLockError is thrown\n      // successfully extends locks would be automatically released\n      .extend(1000)\n      .then(() =\u003e longRunningJob())\n      // same as .extend in terms of cleaning up\n      // in case we can't do it - it will throw\n      .then(() =\u003e multiLock.release())\n      .catch(MultiLockError, e =\u003e {\n        // re-throw generic error so that it doesn't end up in the generic error handler\n      });\n  })\n  .catch(MultiLockError, e =\u003e {\n    // could not acquire locks\n  })\n  .catch(e =\u003e {\n    // unexpected error - perhaps redis was offline\n    // or timed out. ioredis-created errors would be defined here\n  });\n```\n\n### Semaphore\n\nEnsures that all requests are processed one by one. It doesn't guarantee FIFO, but ensures that\nnot more than 1 request runs at any given time.\n\n```js\nconst Promise = require('bluebird');\nconst semaphore = callbackQueue.semaphore('job:id');\n\n// option 1 - use disposer that will automatically call semaphore.leave\nPromise.using(semaphore.take(), () =\u003e {\n  // do some work, return promise\n});\n\n// option 2 - without disposer\ntry {\n  await semaphore.take(false);\n  // perform some async work and ensure we call leave afterwards\n} finally {\n  semaphore.leave();\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmicrofleet%2Fdistributed-callback-queue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmicrofleet%2Fdistributed-callback-queue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmicrofleet%2Fdistributed-callback-queue/lists"}