{"id":13509039,"url":"https://github.com/shinyscorpion/task_bunny","last_synced_at":"2025-04-04T10:09:18.338Z","repository":{"id":16779584,"uuid":"80631449","full_name":"shinyscorpion/task_bunny","owner":"shinyscorpion","description":"TaskBunny is a background processing application written in Elixir and uses RabbitMQ as a messaging backend","archived":false,"fork":false,"pushed_at":"2021-11-09T11:20:01.000Z","size":307,"stargazers_count":202,"open_issues_count":15,"forks_count":30,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-03-28T09:09:13.544Z","etag":null,"topics":["elixir","rabbitmq","workers"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shinyscorpion.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-02-01T15:07:27.000Z","updated_at":"2024-03-05T04:43:29.000Z","dependencies_parsed_at":"2022-07-22T07:17:57.605Z","dependency_job_id":null,"html_url":"https://github.com/shinyscorpion/task_bunny","commit_stats":null,"previous_names":[],"tags_count":21,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shinyscorpion%2Ftask_bunny","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shinyscorpion%2Ftask_bunny/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shinyscorpion%2Ftask_bunny/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shinyscorpion%2Ftask_bunny/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shinyscorpion","download_url":"https://codeload.github.com/shinyscorpion/task_bunny/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247157283,"owners_count":20893220,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elixir","rabbitmq","workers"],"created_at":"2024-08-01T02:01:02.154Z","updated_at":"2025-04-04T10:09:18.315Z","avatar_url":"https://github.com/shinyscorpion.png","language":"Elixir","funding_links":[],"categories":["Queue"],"sub_categories":[],"readme":"# TaskBunny\n\n[![Hex.pm](https://img.shields.io/hexpm/v/task_bunny.svg \"Hex\")](https://hex.pm/packages/task_bunny)\n[![Build Status](https://travis-ci.org/shinyscorpion/task_bunny.svg?branch=master)](https://travis-ci.org/shinyscorpion/task_bunny)\n[![Inline docs](http://inch-ci.org/github/shinyscorpion/task_bunny.svg?branch=master)](http://inch-ci.org/github/shinyscorpion/task_bunny)\n[![Deps Status](https://beta.hexfaktor.org/badge/all/github/shinyscorpion/task_bunny.svg)](https://beta.hexfaktor.org/github/shinyscorpion/task_bunny)\n[![Hex.pm](https://img.shields.io/hexpm/l/task_bunny.svg \"License\")](LICENSE.md)\n\nTaskBunny is a background processing application written in Elixir and uses RabbitMQ as a messaging backend.\n\n[API Reference](https://hexdocs.pm/task_bunny/)\n\n## Use cases\n\nAlthough TaskBunny provides similar features to popular background processing libraries in other languages such as Resque, Sidekiq, RQ etc., you might not need it for a same reason.\nErlang process or GenServer would always be your first choice for background processing in Elixir.\n\nHowever you might want to try out TaskBunny in the following cases:\n\n- You want to separate a background processing concern from your phoenix application\n- You use container based deployment such as Heroku, Docker etc. and each deploy is immutable and disposable\n- You want to have a control on retry and its interval on background job processing\n- You want to schedule the job execution time\n- You want to use a part of functionalities TaskBunny provides to talk to RabbitMQ\n- You want to enqueue jobs from other system via RabbitMQ\n- You want to control the concurrency to avoid making too much traffic\n\n\n## Getting started\n\n### 1. Check requirements\n\n- Elixir 1.4+\n- RabbitMQ 3.6.0 or greater\n\n### 2. Install TaskBunny\n\nEdit `mix.exs` and add `task_bunny` to your list of dependencies and applications:\n\n```elixir\ndef deps do\n  [{:task_bunny, \"~\u003e 0.3.2\"}]\nend\n\ndef application do\n  [applications: [:task_bunny]]\nend\n```\n\nThen run `mix deps.get`.\n\n### 3. Configure TaskBunny\n\nConfigure hosts and queues:\n\n```elixir\nconfig :task_bunny, hosts: [\n  default: [connect_options: \"amqp://localhost?heartbeat=30\"]\n]\n\nconfig :task_bunny, queue: [\n  namespace: \"task_bunny.\",\n  queues: [[name: \"normal\", jobs: :default]]\n]\n```\n\n### 4. Define TaskBunny job\n\nUse `TaskBunny.Job` module in your job module and define `perform/1` that takes a map as an argument.\n\n```elixir\ndefmodule HelloJob do\n  use TaskBunny.Job\n  require Logger\n\n  def perform(%{\"name\" =\u003e name}) do\n    Logger.info(\"Hello #{name}\")\n    :ok\n  end\nend\n```\n\nMake sure you return `:ok` or `{:ok, something}` when the job was successfully processed.\nOtherwise TaskBunny would treat the job as failed and move it to retry queue.\n\n### 5. Enqueueing TaskBunny job\n\nThen enqueue a job\n\n```elixir\nHelloJob.enqueue!(%{\"name\" =\u003e \"Cloud\"})\n```\n\nThe worker invokes the job with `Hello Cloud` in your logger output.\n\n## Queues\n\n#### Worker queue and sub queues\n\nTaskBunny declares four queues for each worker queue on RabbitMQ.\n\n```elixir\nconfig :task_bunny, queue: [\n  namespace: \"task_bunny.\"\n  queues: [\n    [name: \"normal\", jobs: :default]\n  ]\n]\n```\n\nIf have a config like above, TaskBunny will define these four queues on RabbitMQ:\n\n- task_bunny.normal: main worker queue\n- task_bunny.normal.retry: queue for retry\n- task_bunny.normal.rejected: queue that stores jobs failed more than allowed times\n- task_bunny.normal.delay: queue that stores jobs that are performed in the future\n\n\n#### Reset queues\n\nTaskBunny provides a mix task to reset queues.\nThis task deletes the queues and creates them again.\nExisting messages in the queue will be lost so please be aware of this.\n\n```\n% mix task_bunny.queue.reset\n```\n\nYou need to redefine a queue when you want to change the retry interval for a queue.\n\n\n#### Umbrella app\n\nWhen you use TaskBunny under an umbrella app and each apps needs a different queue definition, you can prefix config key like below so that it doesn't overwrite the other configuration.\n\n```elixir\n  config :task_bunny, app_a_queue: [\n    namespace: \"app_a.\",\n    queues: [\n      [name: \"normal\", jobs: \"AppA.*\"]\n    ]\n  ]\n\n  config :task_bunny, app_b_queue: [\n    namespace: \"app_b.\",\n    queues: [\n      [name: \"normal\", jobs: \"AppB.*\"]\n    ]\n  ]\n```\n\n\n## Enqueue job\n\n#### Enqueue\n\n`TaskBunny.Job` will define `enqueue/1` and `enqueue!/1` to your job module.\nLike other Elixir libraries, `enqueue!/1` is similar to `enqueue/1` but raises\nan exception when it gets an error during the enqueue.\n\nYou can also use `TaskBunny.Job.enqueue/2` which takes a module as a first\nargument. The two examples below will give you the same result.\n\n```elixir\nSampleJob.enqueue!()\nTaskBunny.Job.enqueue!(SampleJob)\n```\n\nFirst expression is concise and preferred but the later expression lets you\nenqueue the job without defining job module.\nTaskBunny takes the module just as atom and doesn't check module existence when\nit enqueues.\nIt is useful when you have separate applications for enqueueing and performing.\n\n#### Schedule job\n\nWhen you don't want to perform the job immediately you can use `delay` options\nwhen you enqueue the job.\n\n```elixir\nSampleJob.enqueue!(delay: 10_000)\n```\n\nIt will enqueue the job to the worker queue in 10 seconds.\n\nWhen you use `delay` option it enqueues the job to the delay queue.\nThe job will be moved to the worker queue after the specific time.\n\nThe move between those queues will be handled by RabbitMQ so the job will be enqueued safely even if your application dies after the call.\n\n#### Enqueue job from other system\n\nThe message should be encoded in JSON format and set job and payload(argument).\nFor example:\n\n```javascript\n{\n  \"job\": \"YourApp.HelloJob\",\n  \"payload\": {\"name\": \"Aerith\"}\n}\n```\n\nThen send the message to the worker queue - TaskBunny will process it.\n\n#### Select queue\n\nTaskBunny looks up config and chooses a right queue for the job.\n\n```elixir\nconfig :task_bunny, queue: [\n  queues: [\n    [name: \"default\", jobs: :default],\n    [name: \"fast_track\", jobs: [YourApp.RushJob, YourApp.HurryJob],\n    [name: \"analytics\", jobs: \"Analytics.*\"]\n  ]\n]\n```\n\nYou can configure it with module name(atom), string (support wildcard) or list of them.\nIf the job matches one of them the queue will be chosen.\nIf the doesn't match any the queue with :default will be chosen.\n\n```elixir\nYourApp.RushJob.enqueue(payload) #=\u003e \"fast_track\"\nAnalytics.MiningJob.enqueue(payload) #=\u003e \"analytics\"\nYourApp.HelloJob.enqueue(payload) #=\u003e \"default\"\n```\n\nIf you pass the queue option TaskBunny will use it.\n\n```elixir\nYourApp.HelloJob.enqueue(payload, queue: \"fast_track\") #=\u003e \"fast_track\"\n```\n\n## Workers\n\n#### What is worker?\n\nTaskBunny worker is a GenServer that processes jobs and handles errors.\nA worker listens to a single queue, receives messages(jobs) from it and invokes jobs to perform.\n\n#### Concurrency\n\nBy default a TaskBunny worker runs two jobs concurrently.\nYou can change the concurrency with the config.\n\n```elixir\nconfig :task_bunny, queue: [\n  namespace: \"task_bunny.\"\n  queues: [\n    [name: \"default\", jobs: :default, worker: [concurrency: 1]],\n    [name: \"analytics\", jobs: \"Analytics.*\", worker: [concurrency: 10]]\n  ]\n]\n```\n\nThe concurrency is set per an application.\nIf you run your application on five different hosts with above configuration,\nthere can be 55 jobs performing simultaneously in total.\n\n#### Disable storing rejected jobs in a queue\n\nBy default, if a job fails more than `max_retry` times, the payload is sent to `[namespace].[job_name].rejected` queue.\nYou can disable this behavior in the config by setting `store_rejected_jobs` worker parameter to `false` (defaults to `true`).\nThis might be useful when rejected jobs queue is never consumed, thus making the queue grow infinitely.\n\n```elixir\nconfig :task_bunny, queue: [\n  namespace: \"task_bunny.\"\n  queues: [\n    [name: \"default\", jobs: :default, worker: [concurrency: 1, store_rejected_jobs: false]]\n  ]\n]\n```\n\nWith above, worker does not store rejected jobs. However, `on_reject` callback is still called when a job gets rejected.\n\n#### Disable worker\n\nYou can disable workers starting with your application by setting `1`, `TRUE` or `YES` to `TASK_BUNNY_DISABLE_WORKER` environment variable.\n\n```\n% TASK_BUNNY_DISABLE_WORKER=1 mix phoenix.server\n```\n\nYou can also disable workers in the config.\n\n```elixir\nconfig :task_bunny, disable_worker: true\n```\n\nYou can also disable worker running for a specific queue with the config.\n\n```elixir\nconfig :task_bunny, queue: [\n  namespace: \"task_bunny.\"\n  queues: [\n    [name: \"default\", jobs: :default],\n    [name: \"analytics\", jobs: \"Analytics.*\", worker: false]\n  ]\n]\n```\n\nWith above, TaskBunny starts only a worker for the default queue.\n\n## Control job execution\n\n#### Retry\n\nTaskBunny marks the job failed when:\n\n- job raises an exception or exits during `perform`\n- `perform` doesn't return `:ok` or `{:ok, something}`\n- `perform` times out.\n\nTaskBunny retries the job automatically if the job has failed.\nBy default, it retries 10 times for every 5 minutes.\n\nIf you want to change it, you can override the value on a job module.\n\n```elixir\ndefmodule FlakyJob do\n  use TaskBunny.Job\n  require Logger\n\n  def max_retry, do: 100\n  def retry_interval(_), do: 10_000\n\n  ...\nend\n```\n\nIn this example, it will retry 100 times for every 10 seconds.\nYou can also change the retry_interval by the number of failures.\n\n```elixir\n  def max_retry, do: 5\n\n  def retry_interval(failed_count) do\n    # failed_count will be between 1 and 5.\n    # Gradually have longer retry interval\n    [10, 60, 300, 3_600, 7_200]\n    |\u003e Enum.map(\u0026(\u00261 * 1000))\n    |\u003e Enum.at(failed_count - 1, 1000)\n  end\n```\n\nIf a job fails more than `max_retry` times, the payload is sent to `jobs.[job_name].rejected` queue.\n\nWhen a job gets rejected the `on_reject` callback is called. By default it does nothing but you can override it.\nIt's useful to execute recovery actions when a job fails (like sending an email to a customer for instance)\n\nIt receives the body containing the payload of the rejected job plus the full error trace. It returns :ok\n\n```elixir\ndefmodule FlakyJob do\n  use TaskBunny.Job\n  require Logger\n\n  def on_reject(_body) do\n     ...\n\n     :ok\n  end\n  ...\nend\n```\n\n#### Immediately Reject\n\nTaskBunny can mark a job as rejected without retrying when `perform` returns `:reject` or `{:reject, something}`\n\nIn this case any `max_retry` config is ignored.\n\n#### Timeout\n\nBy default, jobs timeout after 2 minutes.\nIf job doesn't respond for more than 2 minutes, worker kills the process and moves it to retry queue.\n\nYou can change the timeout by overriding `timeout/0` in your job.\n\n```elixir\ndefmodule SlowJob do\n  use TaskBunny.Job\n  def timeout, do: 300_000\n\n  ...\nend\n```\n\n## Connection management\n\nTaskBunny provides an extra layer on top of the [amqp](https://github.com/pma/amqp) connection module.\n\n#### Configuration\n\nTaskBunny automatically connects to RabbitMQ hosts in the config at the start of\nthe application.\n\nTaskBunny forwards `connect_options` to [AMQP.Connection.open/1](https://hexdocs.pm/amqp/AMQP.Connection.html#open/1).\n\n```elixir\nconfig :task_bunny, hosts: [\n  default: [\n    connect_options: \"amqp://rabbitmq.example.com?heartbeat=30\"\n  ],\n  legacy: [\n    connect_options: [\n      host: \"legacy.example.com\",\n      port: 15672,\n      username: \"guest\",\n      password: \"bunny\"\n    ]\n  ]\n]\n\n```\n\n`:default` host has a special meaning on TaskBunny: TaskBunny would select `:default`\nhost when you didn't specify the host.\n\n```elixir\nassert TaskBunny.Connection.get_connection() == TaskBunny.Connection.get_connection(:default)\n```\n\nYou can specify the host to the queue:\n\n```elixir\nconfig :task_bunny, queue: [\n  queues: [\n    [name: \"normal\", jobs: \"MainApp.*\"], # =\u003e :default host\n    [name: \"normal\", jobs: \"Legacy.*\", host: :legacy]\n  ]\n]\n```\n\nIf you don't want to start TaskBunny automatically in a specific environment, set `true` to `disable_auto_start` in the config:\n\n```elixir\nconfig :task_bunny, disable_auto_start: true\n```\n\n#### Get connection\n\nTaskBunny provides two ways to access the connections.\nMost of time you want to use `Connection.get_connection/1` or\n`Connection.get_connection!/1` that returns the connection synchronously.\n\n```elixir\nconn = TaskBunny.Connection.get_connection()\nlegacy = TaskBunny.Connection.get_connection(:legacy)\n```\n\nTaskBunny also provides asynchronous API `Connection.subscribe_connection/1`.\nSee the [API documentation](https://hexdocs.pm/task_bunny/TaskBunny.Connection.html) for more details.\n\n#### Reconnection\n\nTaskBunny automatically tries reconnecting to RabbitMQ if the connection is gone.\nAll workers will restart automatically once the new connection is established.\n\nTaskBunny aims to provide zero hassle and recover automatically regardless how\nlong the host takes to come back and accessible.\n\n## Failure backends\n\nBy default, when the error occurs during the job execution TaskBunny reports it\nto Logger. If you want to report the error to different services, you can configure\nyour custom failure backend.\n\n```elixir\nconfig :task_bunny, failure_backend: [YourApp.CustomFailureBackend]\n```\n\nYou can also report the errors to the multiple backends. For example, if you\nwant to use our default Logger backend with your custom backend you can\nconfigure like below:\n\n```elixir\nconfig :task_bunny, failure_backend: [\n  TaskBunny.FailureBackend.Logger,\n  YourApp.CustomFailureBackend\n]\n```\n\nCheck out the implementation of [TaskBunny.FailureBackend.Logger](https://github.com/shinyscorpion/task_bunny/blob/master/lib/task_bunny/failure_backend/logger.ex) to learn how to write your custom failure backend.\n\n#### Implementations\n\n- [Rollbar backend](https://github.com/shinyscorpion/task_bunny_rollbar)\n- [Sentry backend](https://github.com/Homepolish/task_bunny_sentry)\n\n(Send us a pull request if you want to add other implementation)\n\n## Monitoring\n\n#### RabbitMQ plugins\n\nRabbitMQ supports a variety of [plugins](http://www.rabbitmq.com/plugins.html).\nIf you are not familiar with them we recommend you to look into those.\n\nThe following plugins will help you use RabbitMQ with TaskBunny.\n\n* [Management Plugin](http://www.rabbitmq.com/management.html): provides an HTTP-based API for management and monitoring of your RabbitMQ server, along with a browser-based UI and a command line tool, rabbitmqadmin.\n* [Shovel Plugin](http://www.rabbitmq.com/shovel.html): helps you to move messages(job) from a queue to another queue.\n\n#### Wobserver integration\n\nTaskBunny automatically integrates with [Wobserver](https://github.com/shinyscorpion/wobserver).\nAll worker and connection information will be added as a page on the web interface.\nThe current amount of job runners and job success, failure, and reject totals are added to the `/metrics` endpoint.\n\n\n## Copyright and License\n\nCopyright (c) 2017, SQUARE ENIX LTD.\n\nTaskBunny code is licensed under the [MIT License](LICENSE.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshinyscorpion%2Ftask_bunny","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshinyscorpion%2Ftask_bunny","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshinyscorpion%2Ftask_bunny/lists"}