{"id":16630860,"url":"https://github.com/ryanclark/enhanced-farm","last_synced_at":"2025-03-11T18:38:16.313Z","repository":{"id":57225471,"uuid":"162504505","full_name":"ryanclark/enhanced-farm","owner":"ryanclark","description":"A library to distribute processing tasks to child workers concurrently","archived":false,"fork":false,"pushed_at":"2019-01-10T16:47:16.000Z","size":60,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-11T02:05:29.975Z","etag":null,"topics":["concurrency","javascript","node","processes","typescript"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ryanclark.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-12-20T00:11:09.000Z","updated_at":"2020-03-08T02:59:27.000Z","dependencies_parsed_at":"2022-08-24T11:00:20.821Z","dependency_job_id":null,"html_url":"https://github.com/ryanclark/enhanced-farm","commit_stats":null,"previous_names":["rynclark/enhanced-farm"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryanclark%2Fenhanced-farm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryanclark%2Fenhanced-farm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryanclark%2Fenhanced-farm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryanclark%2Fenhanced-farm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ryanclark","download_url":"https://codeload.github.com/ryanclark/enhanced-farm/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243092998,"owners_count":20235278,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["concurrency","javascript","node","processes","typescript"],"created_at":"2024-10-12T04:49:41.092Z","updated_at":"2025-03-11T18:38:16.287Z","avatar_url":"https://github.com/ryanclark.png","language":"TypeScript","readme":"\u003ch1 align=\"center\"\u003e\n  \u003cp align=\"center\"\u003eenhanced-farm\u003c/p\u003e\n\u003c/h1\u003e\n\nA library to distribute processing tasks to child workers concurrently. Heavily based on [worker-farm](https://github.com/rvagg/node-worker-farm). Written in TypeScript.\n\n## Getting Started\n\nInstall `enhanced-farm` using [`yarn`](https://yarnpkg.com/en/package/enhanced-farm):\n\n```bash\nyarn add enhanced-farm\n```\n\nOr via [`npm`](https://www.npmjs.com/package/enhanced-farm):\n\n```bash\nnpm install enhanced-farm\n```\n\n## Use Case\n\nYou can use this module to create child processes and evenly distribute your processing tasks between them. Your processing tasks can also then send back data whilst they are running, for example, progress data when parallel compiling webpack.\n\n## API\n\nFirst of all, you need to create a worker farm. You specify the path to the worker that contains the method you want to repeatedly (and concurrently) call and process in child processes.\n\n#### createFarm(workerPath[, options])\n\nCreates a new `Farm` instance.\n\n### Farm\n\n#### runWorker([...args]): EventEmitter\n\nSchedule your worker to process a task with the given arguments when it can.\n\nThis returns an `EventEmitter` instance for your task. You can listen for `error`, `started`, `data` or `complete`.\n\n```js\nconst worker = runWorker();\n\nworker.on('error', (err) =\u003e console.error(err));\nworker.on('started', (exit) =\u003e exit());\nworker.on('data', (data) =\u003e console.log(data));\nworker.on('complete', (data) =\u003e console.log(data));\n```\n\nIf you listen for the `started` event, you'll receive a callback that you can use to end the worker outside of the worker itself.\n\n#### end()\n\nOnce you've ran all your tasks, you'll want to end your farm and kill all left over child processes, otherwise they'll still be running, awaiting a new task.\n\nNormally you'd compare the amount of tasks run (when you call `runWorker`) vs. the amount of `complete` or `error` events you've received from your workers in total. Each worker will only ever emit the `complete` or `error` events once, but can emit the `data` event as many times as needed.\n\n\n### Creating a worker\n\nYour worker is just a function that is called whenever there is a spare child process to run it. You'll need to export it via `module.exports` unless you specify the exported method name in the options.\n\nThe worker function is called with four methods on it's `this` context.\n\n#### error(err)\n\nReport an error and stop processing.\n\n#### send(data)\n\nSend data back to the parent process. You can call this as many times as you want before the task has completed.\n\n#### complete(data)\n\nComplete the task and pass data back. This will then free the child process up to run the next task.\n\n#### exit()\n\nEnd the process immediately. Useful when listening for `SIGINT` events.\n\n```javascript\nmodule.exports = function() {\n  this.send('send data back!');\n  \n  setTimeout(() =\u003e this.complete('complete the task with this data'));\n};\n```\n\n## Options\n\nThese are the options and their default values that you can pass through to `createFarm`.\n\n```js\n{\n  autoStart: false,\n  exportedMethodName: null,\n  maxCallsPerWorker: Infinity,\n  maxCallTime: Infinity,\n  maxConcurrentWorkers: require('os').cpus().length,\n  maxConcurrentCallsPerWorker: 10,\n  maxConcurrentCalls: Infinity,\n  maxRetries: Infinity,\n}\n```\n\n#### autoStart: boolean\n\nIf set to `true`, once creating the farm, all child processes will be created immediately (up to the amount specified by the `maxConcurrentCallsPerWorker` option). If not, they are created once `runWorker` is called.\n\n#### exportedMethodName: string\n\nThe name of the exported method from your worker file if you are not using `module.exports`.\n\n#### maxCallsPerWorker: number\n\nThe maximum number of calls a worker can make before it is killed and replaced.\n\n#### maxCallTime: number\n\nThe maximum time that your worker can run for.\n\n#### maxConcurrentWorkers: number\n\nThe maximum number of workers that can exist concurrently. Normally it's best to match this with the number of cores the processor has.\n\n#### maxConcurrentCallsPerWorker: number\n\nThe maximum number of calls a worker can concurrently handle.\n\n#### maxConcurrentCalls: number\n\nThe maximum number of calls the entire farm can concurrently handle.\n\n#### maxRetries: number\n\nThe number of times to retry a task if it fails.\n\n## TypeScript Usage\n\n#### Basic Usage\n\n##### index.ts\n```typescript\nimport { createFarm } from 'enhanced-farm';\n\nconst farm = createFarm(require.resolve('./worker'));\n\nfor (let i = 0; i \u003c 10; i++) {\n  const worker = farm.runWorker(i);\n\n  worker.on('error', (error) =\u003e console.error(error));\n  worker.on('data', (data) =\u003e console.log('data received', data));\n  \n  worker.on('complete', (data) =\u003e {\n    if (data === 9) {\n      farm.end();\n    }\n  });\n}\n```\n\nNote that you'll need to make sure your `tsconfig.json` is set to include all `.ts` files (when using the `include` option) or you'll need to specify every worker file in the `files` array to ensure they're outputted.\n\n##### worker.ts\n```typescript\nfunction worker(index: number) {\n  this.send('send back some data');\n\n  setTimeout(() =\u003e this.done(index), 2000);\n}\n\nmodule.exports = worker;\n```\n\n### Advanced Usage\n\nYou can strongly type your worker function to ensure the correct arguments are passed to the workers.\n\n```typescript\nconst farm = createFarm\u003ctypeof import('./worker')\u003e(require.resolve('./worker'));\n```\n\nWill infer the types from the worker when you then call `runWorker`.\n\n```typescript\nfarm.runWorker('string'); // type error\nfarm.runWorker(1); // all good\n```\n\nTo specify the types of data, pass them through after the function type generic.\n\n```typescript\ntype WorkerOutput = number; // or an interface\ntype WorkerData = string; // or an interface\n\ncreateFarm\u003ctypeof import('./worker'), WorkerOutput, WorkerData, Error\u003e(require.resolve('./worker'));\n```\n\n```typescript\nconst worker = farm.runWorker(i);\n  \nworker.on('error', (error) =\u003e console.error(error)); // error inferred to Error\nworker.on('data', (data) =\u003e console.log('data received', data)); // data inferred to WorkerData\n\nworker.on('complete', (data) =\u003e console.log('completed!', data)); // data inferred to WorkerOutput\n```\n\nYou can also specify the context in your workers for when you use the `error`, `send` or `complete` methods.\n\n```typescript\nimport { WorkerContext } from 'enhanced-farm';\n\nfunction worker(this: WorkerContext\u003cWorkerOutput, WorkerData, Error\u003e, index: number) {\n  this.send('send back some data');\n\n  setTimeout(() =\u003e this.done(index), 2000);\n}\n\nmodule.exports = worker;\n```\n\nThis will then type guard those methods.\n\n### Using a named export\n\nYou can also use a named export by specifying the name in the options. If you're type guarding your code, you'll need to change it a little.\n\n##### index.ts\n```typescript\ncreateFarm\u003ctypeof import('./worker').Worker\u003e(require.resolve('./worker'), { exportedMethodName: 'Worker' });\n```\n\n##### worker.ts\n\n```typescript\nimport { WorkerContext } from 'enhanced-farm';\n\nexport function worker(this: WorkerContext\u003cWorkerOutput, WorkerData, Error\u003e, index: number) {\n  this.send('send back some data');\n\n  setTimeout(() =\u003e this.done(index), 2000);\n}\n```\n\n## JavaScript Usage\n\n##### index.js\n\n```js\nconst { createFarm } = require('enhanced-farm');\n\nconst farm = createFarm(require.resolve('./worker'));\n\nfor (let i = 0; i \u003c 10; i++) {\n  const worker = farm.runWorker(i);\n  \n  worker.on('error', (error) =\u003e console.error(error));\n  worker.on('data', (data) =\u003e console.log('data received', data));\n  \n  worker.on('complete', (data) =\u003e {\n    if (data === 9) {\n      farm.end();\n    }\n  });\n}\n```\n\n##### worker.js\n\n```js\nfunction worker(index) {\n  this.send('send back some data');\n\n  setTimeout(() =\u003e this.done(index), 2000);\n}\n\nmodule.exports = worker;\n```\n\n## Credits\n\nThis module was heavily inspired and created off of the work of [worker-farm](https://github.com/rvagg/node-worker-farm).\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fryanclark%2Fenhanced-farm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fryanclark%2Fenhanced-farm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fryanclark%2Fenhanced-farm/lists"}