{"id":28253229,"url":"https://github.com/science-periodicals/workers","last_synced_at":"2026-07-15T14:02:08.605Z","repository":{"id":38408463,"uuid":"195594688","full_name":"science-periodicals/workers","owner":"science-periodicals","description":"sci.pe (science periodicals) multi-threaded workers processing schema.org Action concurrently","archived":false,"fork":false,"pushed_at":"2023-01-04T05:38:50.000Z","size":34202,"stargazers_count":1,"open_issues_count":25,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-10-26T10:44:51.486Z","etag":null,"topics":["schema-org","scholarly-publishing","workers","zeromq"],"latest_commit_sha":null,"homepage":"https://sci.pe","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/science-periodicals.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-07-07T00:35:08.000Z","updated_at":"2020-06-09T16:20:41.000Z","dependencies_parsed_at":"2023-02-01T23:31:53.465Z","dependency_job_id":null,"html_url":"https://github.com/science-periodicals/workers","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/science-periodicals/workers","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/science-periodicals%2Fworkers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/science-periodicals%2Fworkers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/science-periodicals%2Fworkers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/science-periodicals%2Fworkers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/science-periodicals","download_url":"https://codeload.github.com/science-periodicals/workers/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/science-periodicals%2Fworkers/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35507793,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-15T02:00:06.706Z","response_time":131,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["schema-org","scholarly-publishing","workers","zeromq"],"created_at":"2025-05-19T16:18:03.556Z","updated_at":"2026-07-15T14:02:08.588Z","avatar_url":"https://github.com/science-periodicals.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# `@scipe/workers`\n\n[![CircleCI](https://circleci.com/gh/science-periodicals/workers.svg?style=svg\u0026circle-token=20a8bbea4fe6a650c8b9aa9ddfefcfb3dedee004)](https://circleci.com/gh/science-periodicals/workers)\n\n[![styled with prettier](https://img.shields.io/badge/styled_with-prettier-ff69b4.svg)](https://github.com/prettier/prettier)\n\nMulti-threaded workers processing\nschema.org [`Action`](http://schema.org/Action) concurrently.\n\nNote: this module is auto published to npm on CircleCI. Only run `npm version\npatch|minor|major` and let CI do the rest.\n\n\n![Logo](./test/fixtures/worker/pieuvre.png?raw=true)\n\n## Getting Started\n\nWorkers work with schema.org [`Action`](http://schema.org/Action). Readers not\nfamiliar with [`Action`](http://schema.org/Action) should refer\nto\n[schema.org](http://schema.org) [Actions overview document](http://schema.org/docs/actions.html) for\na quick introduction.\n\nFor an API endpoint receiving actions like:\n\n```js\n{\n  \"@context\": \"http://schema.org\",\n  \"@id-input\": { \"@type\": \"PropertyValueSpecification\", \"valueRequired\": true },\n  \"@type-input\": { \"@type\": \"PropertyValueSpecification\", \"valueRequired\": true },\n  \"actionStatus\": \"PotentialActionStatus\",\n  \"agent-input\": { \"@type\": \"PropertyValueSpecification\", \"valueRequired\": true },\n  \"object-input\": {\n    \"@type\": \"PropertyValueSpecification\",\n    \"valueRequired\": true,\n    \"valueName\": \"objectId\"\n  },\n  \"result\": {\n    \"@id-output\": {\n      \"@type\": \"PropertyValueSpecification\",\n      \"valueRequired\": true,\n      \"valueName\": \"resultId\"\n    },\n    \"@type\": \"UpdateAction\"\n  },\n  \"target\": {\n    \"@type\": \"EntryPoint\",\n    \"httpMethod\": \"PUT\",\n    \"urlTemplate\": \"http://example.com/{objectId}\",\n    \"encodingType\": \"application/ld+json\",\n    \"contentType\": \"application/ld+json\"\n  }\n}\n```\n\n`@scipe/workers` provides everything required to create scalable action\nprocessing pipelines supporting cancellation and real time progress events.\n\n## Worker\n\n### Worker.prototype.handleAction, Worker.prototype.handleExit and life cycle methods\n\n`@scipe/workers` provides a base `Worker` class.  Workers implementors\nmust extend this base class with:\n\n- a `handleAction` method (required)\n- a `handleExit` method (optional)\n- life cycles methods (`onActiveActionStatus`,\n  `onCompletedActionStatus`, `onFailedActionStatus`) (optional).\n\n\n```js\nimport { Worker } from '@scipe/workers';\n\nclass CustomWorker extends Worker {\n  constructor(config) {\n    super(config);\n  }\n\n  handleAction(action, callback) {\n    // Do work\n    callback(err, handledAction, nextAction);\n  }\n\n  handleExit(err) {\n    // err is an error in case of crash or a status code in case of clean exit\n    // Do cleanup things like killing child processes\n  }\n\n  onActiveActionStatus(action, callback) {\n    // Called before the worker starts to emit the first\n    // ActiveActionStatus message. Calling the callback with an error will\n    // abort the work.\n  }\n\n  onCompletedActionStatus(handledAction, callback) {\n    // Called if handleAction succesfully completed and before emitting\n    // CompletedActionStatus message. Calling the callback with an error will\n    // call onFailedActionStatus (passing the error and the handledAction).\n  }\n\n  onCanceledAction(action, callback) {\n    // Called when the user issue a `CancelAction` targetting `action`\n    // Calling the callback with an error will abort the cancellation\n  }\n\n  onFailedActionStatus(err, action, callback) {\n    // Called if handleAction or onCompletedActionStatus failed\n    // Calling the callback with and error with a negative\n    // property will trigger the suicide of the worker. After suicide, a new\n    // worker node will be automatically respawned.\n  }\n}\n```\n\n\nWorkers are spawned (using\nNode.js [cluster module](https://nodejs.org/api/cluster.html)) and\nexpose [ZeroMQ](http://zeromq.org/) sockets so that:\n\n- work (action) can be dispatched to the workers.\n- workers can notify their progress.\n- ongoing work (action) can be canceled.\n\nIf the `handleAction` method calls its completion callback with a\n`nextAction` argument, the next actions will be automatically\ndispatched.\n\nErrors should be instances of\n[Error](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Error).\nErrors may have a `code` property.\n\nErrors with a **code \u003c 0** triggers the suicide of the current\nworker. After suicide, a new worker node will be automatically\nrespawned.\n\n\n### Worker.prototype.listen, Worker.prototype.stop, Worker.prototype.dispatch\n\n**worker**:\n\n```js\nimport { Worker } from '@scipe/workers';\n\nclass CustomWorker extends Worker {\n  constructor(config) {\n    super(config);\n  }\n\n  handleAction(action, callback) {\n    callback(err, processedAction, nextAction);\n  }\n\n  handleExit(err) {\n  }\n\n}\n\nlet w = new CustomWorker({nWorkers: 1});\nw.listen();\nw.stop(() =\u003e {\n  //stopped\n});\n```\n\n**client**:\n\n```js\nimport { Worker } from '@scipe/workers';\n\nlet w = new Worker();\nw.dispatch({\n  '@context': 'http://schema.org',\n  '@id': 'http://example.com/actionId',\n  '@type': 'Action',\n  agent: 'http://example.com/agentId',\n  object: 'http://example.com/objectId',\n  result: {\n    '@id-outptut': {\n      '@type': 'PropertyValueSpecification',\n      valueRequired: true,\n      valueName: 'resultId'\n    }\n  },\n  target: {\n    '@type': 'EntryPoint',\n    httpMethod: 'PUT',\n    urlTemplate: 'http://example.com/{resultId}',\n    encodingType: 'application/ld+json',\n    contentType: 'application/ld+json'\n  }\n}, (err) =\u003e {\n  // the worker acknowledge the dispatch as soon as the action is received by the worker\n});\n```\n\nA broker is needed so that the client can reach the worker. The broker\nwill also ensure proper balancing of the load among the multiple\nconnected workers (using a least recently used strategy).\n\n**Broker**:\n\n```js\nimport { Broker } from '@scipe/workers';\n\nconst broker = new Broker();\nbroker.listen(err =\u003e {\n  if (err) {\n    throw err;\n  }\n});\nbroker.on('change', (data) =\u003e {\n console.log(data);\n})\n```\n\nThe `broker` is an `EventEmitter` and emit `change` event that can be tracked to\nknow:\n- the number of pending requests\n- the number of available workers (in READY state).\n\nThis data can be used to auto-scale the workers based on work load.\n\n\n### Cancellation ([`CancelAction`](http://example.com/CancelAction))\n\nWorkers subscribe to a [ZeroMQ](http://zeromq.org/) SUB socket and\nmessages can be sent to this socket to administrate the workers.\n\nIn particular, work related to a given `action` can be canceled by sending\na [`CancelAction`](http://example.com/CancelAction)\nwhose [`object`](http://schema.org/object) is the `action` `@id` to the\nworker [zeromq](http://zeromq.org/) under the `worker` topic to the pub socket.\n\n\n```js\nimport zmq from 'zmq';\n\nconst pub = zmq.socket('push');\nconst topic = 'worker';\nconst cancelAction = {\n  '@type': CancelAction,\n  actionStatus: 'CompletedActionStatus',\n  object: 'scipe:actionId'\n}\npub.connect(w.PULL_ENDPOINT);\npub.send([topic, JSON.stringify(cancelAction)]);\n```\n\n### Worker status\n\nWorkers publish the status of their work through a\n[ZeroMQ](http://zeromq.org/) SUB socket.\n\n\n```js\nimport zmq from 'zmq';\n\nlet sub = zmq.socket('sub');\nsub.connect(w.XPUB_ENDPOINT);\nsub.subscribe('');\n\nsub.on('message', function(topic, action) {\n  // topic is the action agent['@id']\n  // note that topic and action are Buffers\n});\n\n```\n\n#### `ActiveActionStatus`\n\nWhen a worker starts (and while the job is running), it will re-emit\nthe action sent at a regular interval with an\n[`actionStatus`](http://schema.org/actionStatus) of\n[`ActiveActionStatus`](http://schema.org/ActiveActionStatus).\n\n\n#### `CanceledActionStatus`\n\nIf a user cancel a job, the worker will emit emit the original action with an\n[`actionStatus`](http://schema.org/actionStatus) of\n[`CanceledActionStatus`](http://ns.sci.pe/CanceledActionStatus).\n\n#### `FailedActionStatus`\n\nIf a worker fails, it will emit emit the original action with an\n[`actionStatus`](http://schema.org/actionStatus) of\n[`FailedActionStatus`](http://schema.org/FailedActionStatus) and\nan [`error`](http://schema.org/error) property containing more\ninformation on the cause of the failure.\n\n#### `CompletedActionStatus`\n\nWhen a worker is done processing an action, it will emit the\n`handledAction` returned by the `handleAction` method usually with an\n`actionStatus` of\n[`CompletedActionStatus`](http://schema.org/CompletedActionStatus) .\n\n\n### `Worker.prototype.emitEvent`\n\nWithin a worker, further information can be published to\nthe [ZeroMQ](http://zeromq.org/) PUB socket by calling the `emitEvent(action,\nevent)` method. Calling `emitEvent` will publish\na [`ProgressEvent`](http://ns.science.ai/ProgressEvent) to the PUB socket. The\ntopic (required by ZeroMQ) will be set to the action agent `@id`.\n\n```js\n{\n  \"@context\": \"http://schema.org\",\n  \"@id\": \"scipe:eventId\",\n  \"@type\": \"Event\",\n  \"about\": \"scipe:actionId\",\n  \"description\": \"starting to process the action\",\n  \"startDate\": \"2016-02-29T16:21:32.886Z\"\n}\n```\n\nIn addition to publishing the `ProgressEvent`, the `emitEvent` method\nreturns an object with:\n\n- `emitEndedEvent`, a function returning the same `ProgressEvent` as the\n  one emitted the previous call but, with an added `endDate` property.\n- `emitEvent`, returning a new `ProgressEvent` linked to the previous event\n  through the [superEvent](http://schema.org/superEvent) property.\n- `toJSON`, function returning the emitted `ProgressEvent` JavaScript object (note\n  that this function will be called by JSON.stringify).\n\n```js\nimport { Worker } from '@scipe/workers';\n\nclass CustomWorker extends Worker {\n  handleAction(action, callback) {\n    const superEvent = this.emitEvent(action, 'starting to process the action');\n    const imageConversionEvent = superEvent.emitEvent('starting image conversion');\n    // convert images...\n    imageConversionEvent.emitEndedEvent();\n    superEvent.emitEndedEvent();\n\n    callback(err, handledAction, nextAction);\n  }\n}\n```\n\n## Config\n\nWorkers can be configured by passing a `config` object to their\nconstructor (see worker source code for details).\n\n\n## Specialized workers\n\n```js\nimport { ImageWorker, AudioVideoWorker, DocumentWorker } from '@scipe/workers';\n```\n\n### ImageWorker\n\nThe `ImageWorker` class extends the `Worker` class and\nprocess [Action](http://schema.org/Action)\nwhose [`object`](http://schema.org/object)\nare [`ImageObject`](http://schema.org/ImageObject).\n\n### AudioVideoWorker\n\nThe `AudioVideoWorker` class extends the `Worker` class and\nprocess [`Action`](http://schema.org/Action)\nwhose [`object`](http://schema.org/object)\nare [`VideoObject`](http://schema.org/VideoObject)\nor [`AudioObject`](http://schema.org/AudioObject).\n\n### DocumentWorker\n\nThe `DocumentWorker` class extends the `Worker` class and\nprocesses [Actions](http://schema.org/Action)\nthe [`object`](http://schema.org/object) of which are `DocumentObject`, a\nsubclass of [`MediaObject`](http://schema.org/MediaObject).\n\n\n## CLI\n\nA CLI is available to quickly launch a broker and all the specialized worker.\n\nSee\n\n```sh\nrun-workers --help\n```\n\nFor more details\n\n\n---\n\n## Installation\n\n- Install [graphicsmagick](http://www.graphicsmagick.org/) (```brew install graphicsmagick --with-libtiff``` on OSX).\n- Install [imagemagick](http://www.imagemagick.org/) (```brew install imagemagick --with-libtiff``` on OSX).\n- Install [ffmpeg](https://www.ffmpeg.org/) (```brew install ffmpeg --with-libvpx --with-libvorbis --with-theora --with-aac --with-libx264``` on OSX).\n- Install [LibreOffice](http://www.libreoffice.org/) (it needs to be used headless)\n\n- Run ```npm install```\n\n\n## Tests\n\nRun `npm test`\n\n\n## License\n\n`@scipe/workers` is dual-licensed under commercial and open source licenses\n([AGPLv3](https://www.gnu.org/licenses/agpl-3.0.en.html)) based on the intended\nuse case. Contact us to learn which license applies to your use case.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscience-periodicals%2Fworkers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscience-periodicals%2Fworkers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscience-periodicals%2Fworkers/lists"}