{"id":28376595,"url":"https://github.com/apiaryio/ivy","last_synced_at":"2025-07-30T05:33:25.247Z","repository":{"id":12302205,"uuid":"14933144","full_name":"apiaryio/ivy","owner":"apiaryio","description":"A Node.js queue library focused on easy, yet flexible task execution.","archived":true,"fork":false,"pushed_at":"2018-08-27T13:07:45.000Z","size":179,"stargazers_count":9,"open_issues_count":19,"forks_count":1,"subscribers_count":18,"default_branch":"master","last_synced_at":"2025-06-26T08:42:50.218Z","etag":null,"topics":["sre"],"latest_commit_sha":null,"homepage":"https://www.npmjs.com/package/ivy","language":"CoffeeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/apiaryio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-12-04T19:13:58.000Z","updated_at":"2024-11-08T09:04:04.000Z","dependencies_parsed_at":"2022-09-01T23:31:12.673Z","dependency_job_id":null,"html_url":"https://github.com/apiaryio/ivy","commit_stats":null,"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"purl":"pkg:github/apiaryio/ivy","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apiaryio%2Fivy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apiaryio%2Fivy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apiaryio%2Fivy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apiaryio%2Fivy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/apiaryio","download_url":"https://codeload.github.com/apiaryio/ivy/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apiaryio%2Fivy/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267815187,"owners_count":24148356,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-30T02:00:09.044Z","response_time":70,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["sre"],"created_at":"2025-05-30T00:07:20.847Z","updated_at":"2025-07-30T05:33:25.238Z","avatar_url":"https://github.com/apiaryio.png","language":"CoffeeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![wercker status](https://app.wercker.com/status/4c8beab06abca20f3f59243de5d3f9d0/s/master \"wercker status\")](https://app.wercker.com/project/byKey/4c8beab06abca20f3f59243de5d3f9d0)\n\n## Ivy\n\n[Ivy](https://github.com/apiaryio/ivy) is [node.js](http://nodejs.org) queue library focused on easy, yet flexible task execution.\n\n### Installation\n\nInstallation is done via NPM, by running ```npm install ivy```\n\n### Version 2 quick example (called TODO)\n\n```javascript\n\nvar ivy = require('ivy');\n\nvar factorial = function factorial(number, callback) {\n    callback(null, 42);\n}\n\nvar finished  = function resolved(result) {\n    console.log('result is', result);\n}\n\n\n// task must be explicitly registered for now\n// we'd like to change that in the future\n\n// Also, task must be both available and registered on both client\n// and producer\n\n// ...and name must be unique globally. \"package.module.submodule.function\"\n// pattern is highly encouraged.\nivy.registerTask(factorial, finished, {\n   'name':   'testpackage.factorial',\n   'queue':  'testpackage' //,\n//    'route':  'testpackage.route1',\n//    'priority': 0,\n//    retry:    true,\n//    maxRetries: 10\n});\n\nif (process.env.NODE_ENV==='producer') {\n\n  ivy.setupQueue({\n      type: 'ironmq',\n      auth: {\n          token:      process.env.IRONMQ_TOKEN      || 'dummy',\n          project_id: process.env.IRONMQ_PROJECT_ID || 'testpackage'\n      }\n      //, queue: 'testpackage' // optional, inferred from task\n  });\n\n\n  // optional, only if callback is registered\n  ivy.startNotificationConsumer({\n      'type': 'redis',\n      'url':  'redis://name:password@hostname:port'\n  });\n\n  // execute task\n  ivy.delayedCall(factorial, 5, function(err, result) {\n    console.log(\"Factorial result is\", result);\n  });\n\n}\nelseif (process.env.NODE_ENV==='worker') {\n    ivy.startNotificationProducer({\n        'type': 'redis',\n        'url':  'redis://name:password@hostname:port'\n    });\n    ivy.listen({\n        queue: 'testpackage',\n\n        type: 'ironmq',\n        auth: {\n            token:      process.env.IRONMQ_TOKEN   || 'dummy',\n            project_id: process.env.IRONMQ_PROJECT || 'testpackage'\n        },\n\n        // optional\n        messages: {\n            'testpackage.factorial': {\n                reserveTime: 60*60\n            }\n        }\n\n    });\n\n}\n\n```\n\n### Problem solved by Ivy\n\nIvy touches the following workflow:\n\n* Function execution is scheduled from application (\"producer\") in similar way as executing function directly\n* Call is serialized and transferred through queue to worker. Producer subscribes to notifier for completion\n  * Worker job is considered essential. It should be thus delivered through robust, HA queue, such as AMQP, RabbitMQ, SQS, IronMQ or similar.\n  * Arguments are send as a stringifyed JSON. There is no attempt to magically recover original object; called functions should thus rely only on attribute access, not method calls\n* Worker executes function on shared code base, with arguments fetched from queue task\n  * If execution errors, task stays in MQ or is returned there, depending on implementation\n  * If it fails permanently, please beware of JSON.stringify(new Error()) idiosyncrasy\n* Producer is notified back about completion\n  * Speed over robustness is preferred as this should be about notifying client back, not further work\n  * Thus, redis pub/sub is preferred\n  * If non-notification work should follow after execution is done, it should be scheduled as another task in MQ\n\n### Thoughts/assumptions:\n\n* Only tasks/functions with async interface supported. Assumptions:\n  * callback is last argument provided\n  * callback must be present\n  * first argument of callback signature is either Error or null\n\n* Think about context change\n  * Last callback is about placing task in queue as opposed to having direct callback\n  * However, extracting to named function is needed \n  * Multiple callbacks looks strange.\n\n* Explicit is better then implicit\n  * In first version, use explicit task registrations\n  * Leave continuation and function \"backresolve\" to v2\n  * We can implicitly decide whether notifications are producer or consumer: consumer when `listen` is invoked, producer when first `delayedCall` is executed. Make it explicit in v1, we'll see later.\n\n* Task registries must be same on both sides\n  * \"Protocol\" specification for backends in other languages\n\n* Serialization boundaries \n  * There are (mostly) no requirements on payload in queues\n  * Default \"protocol\" is JSON, should be separated into serialization module/package\n  * Protobufs should be neat choice\n\n### Naming and definitions\n\nThere are a lot of parts and components in distributed environment. This is how `Ivy` understands them.\n\n* **Producer**: Process that decides some task should not be processed by itself, but instead delegated to another process through queue.\n* **Caller**: Particular function/code where `ivy.delayedCall` has been called.\n* **Queue**: Service/process designed to dispatch messages between processes or services. It ideally processes them in (prioritized) FIFO with one time delivery. Also known as *broker*.\n* **Queue name**: Inside `Queue` services, `Message`'s are organized into separate, well, queues, identified by name. To avoid naming clashes, those are always referred to as `Queue names` instead of just \"queues\".\n* **Queue backend**: Particular piece of software implementing `Queue`'s role, i.e. `IronMQ`, `SQS`, `RabbitMQ`, ...\n* **Consumer**: Process designed to consume messages from `Queue` and processing them.\n* **Listener**: Part of the `Consumer` that listens to `Queue` and waits for `Message`s\n* **Message**: Structured data format placed in `Queue`, understood on both ends.\n* **Message serialization**: Particular serialization format used for placing `Message` into `Queue`, i.e. JSON.\n* **Message format**: Particular structure used for particular `Message serialization`, i.e. `{\"task\": \"taskname\", \"arguments\": []}` migth be an example `Message format` for JSON `Message serialization`.\n* **Task**: Function to be invoked on `Consumer`. May be parametrized by `Message`'s content.\n* **Scheduled Task**: A way to describe intent of invoking `Task` at some point.\n* **Task status**: A state that describes current state of `Scheduled Task` or `Task`. May be `scheduled` (successfully placed in `Queue`, but not consumed by `Consumer` yet), `running` (processing on consumer), `errored` (some state failed), `successfull` (processing done on consumer and `Notifier` successfully notified) and `done` (`successfull` + `Producer` successfully notified).\n* **Task result**: Data \"returned\" by `Task` upon its completion with the intent of informing `Producer` about it. While the primary purpose might be computation task that produces an output that is stored in database, it is *not* considered `Task result` if it's not intended for `Producer`. Result is an array of arguments given to `Task`'s callback.\n* **Task execution**: The act of running task on `Consumer`.\n* **Task arguments**: Array of arguments for the `Task` *excluding* the last one (that *must* be callback. I.e. for function `factorial = (number, cb)`, the `arguments` are `[number]`, i.e. `[5]`.\n* **Sending task** is an act of creating `ScheduledTask` by serializing original `delayedCall` call into `Message` and putting it in `Queue`.\n* **Consuming tasks** is an act of retrieving `Messages` from `Queue` on `Consumer` done by `Listener`.\n* **Caller resume**: The act of resuming the workflow back on `Producer`, done by calling callback passed to original `delayedCall`.  \n* **Task resolved**: Task has been executed and `Producer` notified -- or there has been an error.\n* **Notifier**: Service/process designed to inform `Producer` about `Task status` and/or `Task result`. Might be same piece of software/service as `Queue`.\n* **Notification channel**: Uniquely-named \"queue\" used to pass `Task result`s from *any* `Consumer` to *particular* `Producer`. \n* **Notifier backend**: Particular piece of software implementing `Notifier`'s role, i.e. `IronMQ`, `Redis`, ...\n\n### Encryption support for IronMQ\n\nIf you can encrypt all messages for better security add encryptionKey as password. We use `aes-256-cbc` algorithm for encrypt and decrypt messages.\n\n    ivy.setupQueue({\n        queue: 'testpackage',\n\n        type: 'ironmq',\n        auth: {\n            token:      process.env.IRONMQ_TOKEN      || 'dummy',\n            project_id: process.env.IRONMQ_PROJECT_ID || 'testpackage'\n        },\n        encryptionKey:  process.env.MESSAGES_ENCRYPTION_KEY\n    });\n\n## Development\n\n## Install grunt\n\n    npm -g install grunt-cli\n\n## Run tests\n\n    grunt\n\n## Release new version using [grunt-bump](https://github.com/vojtajina/grunt-bump)\n\n    grunt bump\n    grunt bump:minor\n    grunt bump:major\n\nand\n\n    npm publish\n\n## Using Docker\n\n### Installation\n\n    docker-compose build\n\n### Running tests\n\n    docker-compose run ivy\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapiaryio%2Fivy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapiaryio%2Fivy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapiaryio%2Fivy/lists"}