{"id":18427691,"url":"https://github.com/mugli/orkid-node","last_synced_at":"2025-04-07T16:33:43.792Z","repository":{"id":34308045,"uuid":"149886104","full_name":"mugli/orkid-node","owner":"mugli","description":"Reliable and modern Redis Streams based task queue for Node.js 🤖","archived":false,"fork":false,"pushed_at":"2023-01-04T21:57:17.000Z","size":2046,"stargazers_count":63,"open_issues_count":37,"forks_count":10,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-31T20:01:44.398Z","etag":null,"topics":["javascript","job","job-queue","job-scheduler","message-queue","node-js","nodejs","nodejs-library","orkid","queue","redis","redis-queue","redis-stream","redis-streams","task-queue","task-scheduler","typescript","typescript-definitions","typescript-library","worker-queue"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mugli.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-09-22T14:47:07.000Z","updated_at":"2025-02-07T14:44:07.000Z","dependencies_parsed_at":"2023-01-15T06:13:47.161Z","dependency_job_id":null,"html_url":"https://github.com/mugli/orkid-node","commit_stats":null,"previous_names":["mugli/orkid"],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mugli%2Forkid-node","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mugli%2Forkid-node/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mugli%2Forkid-node/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mugli%2Forkid-node/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mugli","download_url":"https://codeload.github.com/mugli/orkid-node/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247688418,"owners_count":20979664,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["javascript","job","job-queue","job-scheduler","message-queue","node-js","nodejs","nodejs-library","orkid","queue","redis","redis-queue","redis-stream","redis-streams","task-queue","task-scheduler","typescript","typescript-definitions","typescript-library","worker-queue"],"created_at":"2024-11-06T05:11:31.494Z","updated_at":"2025-04-07T16:33:43.379Z","avatar_url":"https://github.com/mugli.png","language":"JavaScript","readme":"\u003ch1 align=\"center\"\u003e\n\u003cimg src=\"https://raw.github.com/mugli/orkid-node/master/orkid.svg?sanitize=true\" width=\"150px\" height=\"150px\" /\u003e\u003cbr /\u003e\norkid\u003c/h1\u003e\n\n\u003c!-- [![codecov](https://codecov.io/gh/mugli/orkid-node/branch/master/graph/badge.svg)](https://codecov.io/gh/mugli/orkid-node) --\u003e\n\n[![NPM version](https://img.shields.io/npm/v/orkid.svg)](https://www.npmjs.com/package/orkid)\n[![Build Status](https://img.shields.io/circleci/build/github/mugli/orkid-node/master?token=9e4999a9e95ab359bb1b458bbaed97985308a704)](https://circleci.com/gh/mugli/orkid-node)\n![Code Coverage](https://raw.github.com/mugli/orkid-node/master/badges/badge-lines.svg?sanitize=true)\n![Dependencies](https://img.shields.io/david/mugli/orkid.svg?style=flat)\n![Dev Dependencies](https://img.shields.io/david/dev/mugli/orkid.svg?style=flat)\n![Required Node](https://img.shields.io/node/v/orkid.svg?style=flat)\n![License](https://img.shields.io/npm/l/orkid.svg?style=flat)\n\nReliable and modern [Redis-Streams](https://redis.io/topics/streams-intro) based task queue for Node.js.\n\n---\n\n# Screenshot\n\n![screenshot](https://raw.githubusercontent.com/mugli/orkid-node/master/screenshot.png)\n\n---\n\n# Table of Contents\n\n- [Features](#features)\n- [Requirements](#requirements)\n- [Install](#install)\n- [Examples](#examples)\n- [API Documentation](#api-documentation)\n- [Monitoring and Management UI/Admin Panel](#monitoring-and-management-ui-admin-panel)\n- [FAQ](#faq)\n\n---\n\n# Features\n\n- [x] Orkid lets Redis do the heavy lifting with [Redis-Streams](https://redis.io/topics/streams-intro).\n- [x] **Adjustable concurrency** per consumer instance for scaling task processing. See example code. [See example code](https://github.com/mugli/orkid-node/tree/master/examples/basic).\n- [x] Job **timeouts** and **retries**. All configurable per consumer. [See example code](https://github.com/mugli/orkid-node/tree/master/examples/failure-timeout-retry).\n- [x] Task **Deduplication**. If a task is already waiting in the queue, it can be configured to avoid queueing the same task again. _(Useful for operations like posting database record updates to elasticsearch for re-indexing. Deduplication is a common pattern here to avoid unnecessary updates)_. [See example code](https://github.com/mugli/orkid-node/tree/master/examples/deduplication).\n- [x] Add tasks in **bulk** in one call. The producer will handle chunking and optimize round-trips to redis using pipelining.\n- [x] Monitoring and management **UI** for better visibility.\n  \u003c!-- - [ ] Cron-like **scheduled job** producing. This is different than queueing task now and executing it later. Instead the producer function will be called later at a particular time to produce task. If multiple instances of the application is running, Orkid will ensure that only one producer function gets called. --\u003e\n- [ ] **Rate-limiting** workers. (_work in progress_)\n\n---\n\n# Requirements\n\n- Node.js \u003e= 10\n- Redis \u003e= 5\n\n👏 **Important**: [Redis-Streams](https://redis.io/topics/streams-intro) feature is not available before **Redis version 5**.\n\n---\n\n# Install\n\n```\nnpm install orkid --save\n```\n\n---\n\n# Examples\n\nBasic example of producing and consuming tasks:\n\nProducing tasks:\n\n```js\nconst { Producer } = require('orkid');\n\n// `basic` is the queue name here\n//  We'll use the same name in the consumer to process this task\nconst producer = new Producer('basic');\n\nasync function addTasks() {\n  for (let i = 0; i \u003c 10; i++) {\n    console.log('Adding task ', i);\n    await producer.addTask(i);\n  }\n}\n\naddTasks()\n  .then(() =\u003e {\n    console.log('Done');\n    process.exit(); // To disconnect from redis\n  })\n  .catch(e =\u003e console.error(e));\n```\n\nConsuming tasks:\n\n```js\nconst { Consumer } = require('orkid');\n\n// Worker Function\nasync function workerFn(data, metadata) {\n  let result;\n  /*\n    Do operation on `data` here\n    and store the result in `result` variable\n\n    Anything you return from this function will\n    be saved in redis and can be viewed in the Orkid UI.\n\n    Returning nothing is fine too.\n\n    Throwing error will mark the job as failed,\n    which can be retried too.\n  */\n\n  console.log('Task done!');\n  return result;\n}\n\n// Consume task from the `basic` queue\nconst consumer = new Consumer('basic', workerFn);\n\n// Start processing tasks!\n// Important: Until you call this method, orkid consumer will do nothing.\nconsumer.start();\n```\n\n\u003e 👏 **More examples are available in the [./examples](https://github.com/mugli/orkid-node/tree/master/examples) directory, including how to do task de-duplication, retry on failure, timeout etc.** 👏\n\n---\n\n# API Documentation\n\n**API Documentation is [available here](https://github.com/mugli/orkid-node/blob/master/API.md).**\n\n---\n\n# Monitoring and Management UI/Admin Panel\n\n![screenshot](https://raw.githubusercontent.com/mugli/orkid-node/master/screenshot.png)\n![screenshot](https://raw.githubusercontent.com/mugli/orkid-node/master/screenshot-2.png)\n\n\u003e You need to run `orkid-ui` separately for the dashboard. Detail instructions on how to run `orkid-ui` locally or in production using docker/docker-compose can be found here:\n\u003e https://github.com/mugli/orkid-ui#running-locally\n\n---\n\n# Task/job life-cycle\n\n[TODO: Add a flowchart here]\n\n---\n\n# FAQ\n\n\u003cdetails\u003e\n  \u003csummary\u003eIs this production ready?\u003c/summary\u003e\n  This project is under active development right now. API WILL introduce breaking changes until we reach version 1.0. After that semantic versioning will be followed.\n\u003c/details\u003e\n\n\u003cp\u003e\u003c/p\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eWhy a new job queue for Node.js?\u003c/summary\u003e\n  - All the redis-based solutions were created before Redis-Streams (https://redis.io/topics/streams-intro) became available. They all require a lot of work on the queue-side to ensure durability and atomicity of jobs handling operations. Redis-Streams was specifically designed to made this kind of tasks easier, thus allows simpler core in the queue and more reliable and maintainable operations.\n\u003cbr/\u003e\n- None of existing usable job queues in Node.js offers a monitoring option that we liked.\n\u003cbr/\u003e\n- None of the existing usable task queues support task de-duplication.\n\n\u003c/details\u003e\n\n\u003cp\u003e\u003c/p\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eHow do I set priority in the tasks?\u003c/summary\u003e\n  Redis-Streams isn't a right primitive to make a priority queue efficiently on top of it. Orkid doesn't support priority queues now and probably never will.\n\u003cbr/\u003e\nHowever, as a workaround, you can create a separate queue, keep its workload minimal and use it for high priority jobs with Orkid.\n\n\u003c/details\u003e\n\n\u003cp\u003e\u003c/p\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eWhat is the order of job/task delivery?\u003c/summary\u003e\n  Jobs are processed in the order they are produced. However, if retry option is turned on and is applicable, failed jobs gets enqueued to the queue at once, along with other newly produced jobs.\n\u003c/details\u003e\n\n\u003cp\u003e\u003c/p\u003e\n\n---\n\n## Maintainer(s)\n\n- Mehdi Hasan Khan ([@MehdiHK](https://twitter.com/MehdiHK))\n\n---\n\n## License\n\nMIT\n\n---\n\n### Related Projects\n\n- [orkid-ui](https://github.com/mugli/orkid-ui): Dashboard to monitor and manage Orkid task queue.\n- [orkid-api](https://github.com/mugli/orkid-api): GraphQL API to monitor and manage Orkid task queue (used internally by orkid-ui).\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmugli%2Forkid-node","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmugli%2Forkid-node","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmugli%2Forkid-node/lists"}