{"id":16389285,"url":"https://github.com/marcelog/rabbitmq_minionpool","last_synced_at":"2025-04-04T13:28:10.726Z","repository":{"id":13347518,"uuid":"16034723","full_name":"marcelog/rabbitmq_minionpool","owner":"marcelog","description":"minionpool that uses rabbitmq to inject tasks","archived":false,"fork":false,"pushed_at":"2014-05-09T23:31:31.000Z","size":276,"stargazers_count":2,"open_issues_count":2,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-10-30T04:50:10.686Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/marcelog.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-01-18T21:47:39.000Z","updated_at":"2014-10-02T21:13:05.000Z","dependencies_parsed_at":"2022-08-28T20:30:35.432Z","dependency_job_id":null,"html_url":"https://github.com/marcelog/rabbitmq_minionpool","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marcelog%2Frabbitmq_minionpool","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marcelog%2Frabbitmq_minionpool/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marcelog%2Frabbitmq_minionpool/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marcelog%2Frabbitmq_minionpool/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/marcelog","download_url":"https://codeload.github.com/marcelog/rabbitmq_minionpool/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247184291,"owners_count":20897747,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-11T04:32:14.502Z","updated_at":"2025-04-04T13:28:10.701Z","avatar_url":"https://github.com/marcelog.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# About\n\n**rabbitmq_minionpool** is a specialized [minionpool](https://github.com/marcelog/minionpool) that will let you process tasks coming in\nvia RabbitMQ (it uses de [node-amqp](https://github.com/postwait/node-amqp) library).\n\n# How it works\nYou need to provide some key pieces of information:\n * An exchange name (the \"worker's exchange\" from now on)\n * A queue name (the \"worker's queue\" from now on)\n * A routing key (will default to the queue name if missing)\n * A retry timeout for failed operations.\n\nWhen you create a **rabbitmq_minionpool**, 2 exchanges are created in the\nrabbitmq server:\n * The exchange name specified (let's say \"workers\").\n * A [dead letter exchange](http://www.rabbitmq.com/dlx.html) for failed operations, \n automatically named as the original exchange name and with a suffix \".retry\"\n (e.g: \"workers.retry\").\n\nBoth exchanges are created as 'topic', durable', 'not passive'. Also, the channel is\nset in 'confirm' mode (in case you want to publish your own messages).\n\nAlso, some queues are created:\n * The worker's queue name specified in the given worker's exchange, and binded\n to the given routing key. The pool will subscribe to this queue to get messages.\n This queue is created with the arguments:\n  * x-dead-letter-exchange = exchangeName.retry\n  * x-dead-letter-routing-key = queueName.retry\n\n * Another queue in the dead letter exchange, so failed operations can get\n there. This queue is created with the arguments:\n  * x-dead-letter-exchange = exchangeName\n  * x-dead-letter-routing-key = queueName\n  * x-message-ttl = retryTimeout\n\nBoth queue will subscribe to 'routingKey' but also to their respectively queueName. So\nthey will get messages directed to 'routingKey' but will also get dead-lettered\nmessages (and these messages will reach the correct consumer, the one that \nrejected them.)\n\nWhen messages are routed to the specified worker's queue, the minionpool will \ndispatch them to the minions. Each minion will get access to the message and the\nqueue object where it came from. If the minion rejects the message, the message\nwill be routed to the queue in the dead letter exchange with the given TTL. When\nthe TTL expires, the message will go back automatically to the original queue,\nwhere the operation can be retried.\n\n# Example\n\n```js\nvar options = {\n  name: 'test',\n  debug: true,\n  concurrency: 5,\n  logger: console.log,\n  mqOptions: {\n    host: '127.0.0.1',\n    login: 'guest',\n    password: 'guest',\n    authMechanism: 'AMQPLAIN',\n    vhost: '/',\n    reconnect: true,\n    reconnectBackoffStrategy: 'linear',\n    reconnectExponentialLimit: 120000,\n    reconnectBackoffTime: 1000,\n    exchangeName: 'workers',  // Will also create workers.retry\n    queueName: 'myWorkers',   // Will also create myWorkers.retry\n    routingKey: 'myWorkers',  // Optional. Equals to queueName if missing\n    retryTimeout: 20000\n  },\n  minionTaskHandler: function(msg, state, callback) {\n    var payload = msg.payload;\n    var headers = msg.headers;\n    var deliveryInfo = msg.deliveryInfo;\n    var message = msg.message;\n    var queue = msg.queue;\n    console.log('got task: %s', util.inspect(payload));\n    // See the node-amqp doc for more info.\n    message.reject(); // or message.acknowledge();\n    callback(undefined, state);\n  },\n  poolEnd: function() {\n    process.exit(0);\n  }\n};\n\nvar pool = new minionsMod.RabbitMqMinionPool(options);\nprocess.on('SIGINT', function() {\n  pool.end();\n});\npool.start();\n```\n\n## Tips\n * Design your apps and architecture in such a way that operations are [idempotent](http://en.wikipedia.org/wiki/Idempotence) to max the benefits of this.\n\n## Using multiple cores\n\nIn the case of having rabbitmq and mysql workers, it's very useful to take advantage\nof multicore cpu's. For this, you can use [taskset](http://linuxcommand.org/man_pages/taskset1.html)\nand launch multiple minionpool instances on different cores.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarcelog%2Frabbitmq_minionpool","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmarcelog%2Frabbitmq_minionpool","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarcelog%2Frabbitmq_minionpool/lists"}