{"id":16389290,"url":"https://github.com/marcelog/minionpool","last_synced_at":"2025-10-18T17:23:49.747Z","repository":{"id":12976980,"uuid":"15655641","full_name":"marcelog/minionpool","owner":"marcelog","description":"Worker pool for nodejs","archived":false,"fork":false,"pushed_at":"2014-08-26T23:37:52.000Z","size":428,"stargazers_count":7,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2023-12-27T08:03:02.947Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/marcelog.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-01-05T17:47:52.000Z","updated_at":"2018-07-23T05:06:27.000Z","dependencies_parsed_at":"2022-09-17T14:11:38.578Z","dependency_job_id":null,"html_url":"https://github.com/marcelog/minionpool","commit_stats":null,"previous_names":[],"tags_count":9,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marcelog%2Fminionpool","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marcelog%2Fminionpool/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marcelog%2Fminionpool/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marcelog%2Fminionpool/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/marcelog","download_url":"https://codeload.github.com/marcelog/minionpool/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":219862889,"owners_count":16555951,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-11T04:32:16.227Z","updated_at":"2025-10-18T17:23:49.677Z","avatar_url":"https://github.com/marcelog.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# About\n**minionpool** allows you to concurrently process any tasks you need to (similar to\na worker pool). And it's very simple to use.\n\n## Installing it\nThe npm package is called **minionpool**.\n\n# How it works\nFirst things first. To make use of this, you just have to provide a few callbacks,\nthen instantiate a MinionPool and start() it. A *MinionPool* has\n*Minions*, that process *tasks*. Tasks are provided by a *task source*, which\nis called to get one task at a time, one for each minion started, and one for\neach minion that finishes a previous task. \n\nBoth the minions and the task source can keep a *state*, which is useful to keep\ndatabase connections for example. \n\n## Polling for tasks vs Injecting tasks\nYou can either make the pool poll for tasks from the task source, or inject\ntasks asynchronously \"from the outside\". You can also find an example\nof polling from tasks in the mysql example, and one that injects tasks in the\nrabbitmq example. See below. \n\n# Quick Example\n\n## Main code\n```js\nvar minionpoolMod = require('minionpool');\n\nvar minionPool = new minionpoolMod.MinionPool(options);\nminionPool.start();\n```\n\n## Configuring the pool\nLet's see now what can be defined inside the **options** that we're passing to\nthe **MinionPool** constructor:\n\n```js\nvar options = {\n\n  // A name for your minions pool, useful for debugging.\n  name: 'test',\n\n  // Optional. When 'true', some messages are written to console.\n  debug: true,\n\n  // How many minions to run concurrently.\n  concurrency: 5,\n\n  // Optional. Uses the given Function to log messages.\n  logger: console.log,\n\n  // Optional, defaults to true. If false, the pool will stop processing tasks\n  // and will end on the first task that finishes with error.\n  continueOnError: true,\n\n  // Optional. Called to initialize a 'task source'. It should call the callback\n  // with an initial 'state' (like db connections, file descriptors, etc).\n  // See below. The state will be passed when calling the next property.\n  // Use the err variable to signal errors.\n  taskSourceStart: function(callback) {\n    var err = ... ;\n    var state = ... ;\n    callback(err, state); // err === undefined if no errors\n  },\n\n  // Polling for tasks: The task source produce 'tasks', that are assigned to\n  // minions. A task source receives its state from 'taskSourceStart' first, and\n  // then from whatever it returns on subsequen calls (see below). Also, it\n  // should call the callback with the new task (or 'undefined' when none is found).\n  //\n  // If you are not going to poll for tasks, set this one to undefined, and call\n  // MinionPool.injectTask() to inject the tasks into the pool, they will be \n  // assigned to free minions (or rescheduled if none is available at the time,\n  // be careful not to produce too many tasks).\n  // Use the err variable to signal errors. On errors, will return undefined,\n  // which also means \"no more tasks\" and the pool will end.\n  taskSourceNext: function(state, callback) {\n    var task = ...;\n    var err = ...;\n    callback(err, task); // err === undefined if no errors\n    return state;\n  },\n\n  // Optional. Called to do any cleanup for the task generator once it runs out\n  // of tasks. Receives the 'state', doesn't have to pass any arguments to the\n  // callback.\n  taskSourceEnd: function(state, callback) {\n    callback();\n  },\n\n  // The actual code that works on a task. Should call the 'callback' when\n  // done, passing the new state.\n  // Use the err variable to signal errors.\n  minionTaskHandler: function(task, state, callback) {\n    var err = ...;\n    callback(err, state); // err === undefined if no errors\n  },\n\n  // Optional. Called to initialize each one of the minions. Returns the initial\n  // state for each one of them.\n  // Use the err variable to signal errors.\n  minionStart: function(callback) {\n    var err = ...;\n    callback(err, state); // err === undefined if no errors\n  },\n\n  // Optional. Called to cleanup any needed stuff for each minion.\n  minionEnd: function(state, callback) {\n    callback();\n  },\n\n  // Optional. Called when the pool has finished all the available work.\n  poolEnd: function() {\n    process.exit(0);\n  }\n};\n```\n\n## Example using the ArrayMinionPool\nSee [this](https://github.com/marcelog/minions/tree/master/examples/array.js).\n\n## Example using MySQL (will process all rows in a given table).\n\nYou might want to check out [mysql_minionpool](https://github.com/marcelog/mysql_minionpool) if you need to work with mysql, since it's a\nspecialized **minionpool**. In any case, [see this](https://github.com/marcelog/minions/tree/master/examples/mysql.js) for an example using\nplain vanilla **minionpool**.\n\n## Example using RabbitMQ\n\nAs with mysql, there's a specialized minionpool that you might want to check in\ncase you want to work with RabbitMQ, it's called [rabbitmq_minionpool](https://github.com/marcelog/rabbitmq_minionpool).\n\nAlso, see [this](https://github.com/marcelog/minions/tree/master/examples/rabbitmq.js) for an example of with plain vanilla **minionpool**.\n\n## Using multiple cores\n\nIn the case of having rabbitmq and mysql workers, it's very useful to take advantage\nof multicore cpu's. For this, you can use [taskset](http://linuxcommand.org/man_pages/taskset1.html)\nand launch multiple minionpool instances on different cores.\n\n## Into the details: Lifecycle (Polling)\n\n 1. A pool starts.\n 2. All minions in the pool are started. For each minion, *minionStart* is called.\n 3. After all the minions are started, the task source is started, by calling\n *taskSourceStart*. On error, the pool will end.\n 4. After the task source starts one task per minion is requested by calling\n *taskSourceNext*. If taskSourceNext returns error, it will interpreted as\n no more tasks available, and the pool will end.\n 5. When a minion finishes processing a task (*minionTaskHandler*), a new one\n will be assigned to it.\n 6. When a minion finishes a task, but the task source does not have a task for\n it, that minion will become idle, and shutdown, by calling *minionEnd*.\n 7. When all the minions have shutdowned, the task source will shutdown too, by\n calling *taskSourceEnd*.\n 8. When the task source shuts down, the pool will shut down, and *poolEnd* will\n be called.\n\n## Into the details: Lifecycle (Injected tasks)\n\n 1. A pool starts.\n 2. All minions in the pool are started. For each minion, *minionStart* is called.\n 3. After all the minions are started, the task source is started, by calling\n *taskSourceStart*.\n 4. Someone calls *MinionPool.injectTask* from the outside, a free minion is\n picked up and the task is assigned to it. Otherwise, the task is rescheduled\n internally until a free minion is available.\n 5. The pool only shuts down when MinionPool.end() is called. Then all minions\n are finished.\n 6. When all the minions have shutdowned, the task source will shutdown too, by\n calling *taskSourceEnd*.\n 7. When the task source shuts down, the pool will shut down, and *poolEnd* will\n be called.\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarcelog%2Fminionpool","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmarcelog%2Fminionpool","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarcelog%2Fminionpool/lists"}