{"id":13509005,"url":"https://github.com/akira/exq","last_synced_at":"2025-05-14T02:04:48.658Z","repository":{"id":9803468,"uuid":"11782994","full_name":"akira/exq","owner":"akira","description":"Job processing library for Elixir  - compatible with Resque / Sidekiq","archived":false,"fork":false,"pushed_at":"2025-04-30T03:48:36.000Z","size":3093,"stargazers_count":1507,"open_issues_count":30,"forks_count":182,"subscribers_count":23,"default_branch":"master","last_synced_at":"2025-04-30T04:30:26.148Z","etag":null,"topics":["elixir","exq","job-queue","queue","redis","resque","sidekiq"],"latest_commit_sha":null,"homepage":null,"language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"LeaVerou/prefixfree","license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/akira.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2013-07-31T05:10:24.000Z","updated_at":"2025-04-30T03:37:03.000Z","dependencies_parsed_at":"2025-05-14T02:03:23.784Z","dependency_job_id":null,"html_url":"https://github.com/akira/exq","commit_stats":{"total_commits":618,"total_committers":81,"mean_commits":7.62962962962963,"dds":0.5436893203883495,"last_synced_commit":"5f728e7b59fec48d40a2d532df7884c5374af54b"},"previous_names":[],"tags_count":49,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akira%2Fexq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akira%2Fexq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akira%2Fexq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akira%2Fexq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/akira","download_url":"https://codeload.github.com/akira/exq/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254052668,"owners_count":22006716,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elixir","exq","job-queue","queue","redis","resque","sidekiq"],"created_at":"2024-08-01T02:01:01.642Z","updated_at":"2025-05-14T02:04:48.618Z","avatar_url":"https://github.com/akira.png","language":"Elixir","readme":"# Exq\n\n[![CI](https://github.com/akira/exq/actions/workflows/ci.yml/badge.svg)](https://github.com/akira/exq/actions/workflows/ci.yml)\n[![Coveralls Coverage](https://img.shields.io/coveralls/akira/exq.svg)](https://coveralls.io/github/akira/exq)\n[![Module Version](https://img.shields.io/hexpm/v/exq.svg)](https://hex.pm/packages/exq)\n[![Hex Docs](https://img.shields.io/badge/hex-docs-lightgreen.svg)](https://hexdocs.pm/exq/)\n[![Total Download](https://img.shields.io/hexpm/dt/exq.svg)](https://hex.pm/packages/exq)\n[![License](https://img.shields.io/hexpm/l/exq.svg)](https://github.com/akira/exq/blob/master/LICENSE)\n[![Last Updated](https://img.shields.io/github/last-commit/akira/exq.svg)](https://github.com/akira/exq/commits/master)\n\nExq is a job processing library compatible with Resque / Sidekiq for the [Elixir](http://elixir-lang.org) language.\n* Exq uses Redis as a store for background processing jobs.\n* Exq handles concurrency, job persistence, job retries, reliable queueing and tracking so you don't have to.\n* Jobs are persistent so they would survive across node restarts.\n* You can use multiple Erlang nodes to process from the same pool of jobs.\n* Exq uses a format that is Resque/Sidekiq compatible.\n  * This means you can use it to integrate with existing Rails / Django projects that also use a background job that's Resque compatible - typically with little or no changes needed to your existing apps. However, you can also use Exq standalone.\n  * You can also use the Sidekiq UI to view job statuses, as Exq is compatible with the Sidekiq stats format.\n  * You can run both Exq and Toniq in the same app for different workers.\n* Exq supports uncapped amount of jobs running, or also allows a max limit per queue.\n* Exq supports job retries with exponential backoff.\n* Exq supports configurable middleware for customization / plugins.\n* Exq tracks several stats including failed busy, and processed jobs.\n* Exq stores in progress jobs in a backup queue (using the Redis RPOPLPUSH command).\n  This means that if the system or worker is restarted while a job is in progress,\n  the job will be re_enqueued when the node is restarted and not lost.\n* Exq provides an optional web UI that you can use to view several stats as well as rate of job processing.\n* When shutting down Exq will attempt to let workers terminate gracefully,\n  with a configurable timeout.\n* There is no time limit to how long a job can run for.\n\n### Do you need Exq?\n\nWhile you may reach for Sidekiq / Resque / Celery by default when writing apps in other languages, in Elixir there are some good options to consider that are already provided by the language and platform. So before adding Exq or any Redis backed queueing library to your application, make sure to get familiar with OTP and see if that is enough for your needs. Redis backed queueing libraries do add additional infrastructure complexity and also overhead due to serialization / marshalling, so make sure to evaluate whether it is an actual need or not.\n\nSome OTP related documentation to look at:\n\n* GenServer: http://elixir-lang.org/getting-started/mix-otp/genserver.html\n* Task: https://hexdocs.pm/elixir/Task.html\n* GenStage: https://hexdocs.pm/gen_stage/GenStage.html\n* Supervisor: http://elixir-lang.org/getting-started/mix-otp/supervisor-and-application.html\n* OTP: http://erlang.org/doc/\n\nIf you need a durable jobs, retries with exponential backoffs, dynamically scheduled jobs in the future - that are all able to survive application restarts, then an externally backed queueing library such as Exq could be a good fit.\n\n## Getting Started\n\n### Pre-requisite\n\nThis assumes you have an instance of [Redis](http://redis.io/) to use.  The easiest way to install it on OSX is via brew:\n\n```\n\u003e brew install redis\n```\nTo start it:\n```\n\u003e redis-server\n```\n\n### Screencast on elixircasts.io:\n\nIf you prefer video instructions, check out the screencast on elixircasts.io which details how to install and use the Exq library:\nhttps://elixircasts.io/elixir-job-processing-with-exq\n\n### Installation\nAdd `:exq` to your `mix.exs` deps (replace version with the latest hex.pm package version):\n\n```elixir\ndefp deps do\n  [\n    # ... other deps\n    {:exq, \"~\u003e 0.19.0\"}\n  ]\nend\n```\n\nThen run ```mix deps.get```.\n\n\n### Configuration\n\nBy default, Exq will use configuration from your config.exs file.  You can use this to configure your Redis host, port, password, as well as namespace (which helps isolate the data in Redis). If you would like to specify your options as a Redis URL, that is also an option using the `url` config key (in which case you would not need to pass the other Redis options).\n\nConfiguration options may optionally be given in the `{:system, \"VARNAME\"}` format, which will resolve to the runtime environment value.\n\nOther options include:\n* The `queues` list specifies which queues Exq will listen to for new jobs.\n* The `concurrency` setting will let you configure the amount of concurrent workers that will be allowed, or :infinite to disable any throttling.\n* The `name` option allows you to customize Exq's registered name, similar to using `Exq.start_link([name: Name])`. The default is Exq.\n* If the option `start_on_application` is `false`, Exq won't be started automatically when booting up you Application. You can start it with `Exq.start_link/1`.\n* The `shutdown_timeout` is the number of milliseconds to wait for workers to\n  finish processing jobs when the application is shutting down. It defaults to\n  5000 ms.\n* The `mode` option can be used to control what components of Exq are started. This would be useful if you want to only enqueue jobs in one node and run the workers in different node.\n  * `:default` - starts worker, enqueuer and API.\n  * `:enqueuer` - starts only the enqueuer.\n  * `:api` - starts only the api.\n  * `[:api, :enqueuer]` - starts both enqueuer and api.\n* The `backoff` option allows you to customize the backoff time used for retry when a job fails. By default exponential time scaled based on job's retry_count is used. To change the default behavior, create a new module which implements the `Exq.Backoff.Behaviour` and set backoff option value to the module name.\n\n```elixir\nconfig :exq,\n  name: Exq,\n  host: \"127.0.0.1\",\n  port: 6379,\n  password: \"optional_redis_auth\",\n  namespace: \"exq\",\n  concurrency: :infinite,\n  queues: [\"default\"],\n  poll_timeout: 50,\n  scheduler_poll_timeout: 200,\n  scheduler_enable: true,\n  max_retries: 25,\n  mode: :default,\n  shutdown_timeout: 5000\n```\n\n### Concurrency\n\nExq supports concurrency setting per queue.  You can specify the same ```concurrency``` option to apply to each queue or specify it based on a per queue basis.\n\nConcurrency for each queue will be set at ```1000```:\n\n```elixir\nconfig :exq,\n  host: \"127.0.0.1\",\n  port: 6379,\n  namespace: \"exq\",\n  concurrency: 1000,\n  queues: [\"default\"]\n```\n\nConcurrency for ```q1``` is set at ```10_000``` while ```q2``` is set at ```10```:\n\n```elixir\nconfig :exq,\n  host: \"127.0.0.1\",\n  port: 6379,\n  namespace: \"exq\",\n  queues: [{\"q1\", 10_000}, {\"q2\", 10}]\n```\n\n\n### Job Retries\n\nExq will automatically retry failed job. It will use an exponential backoff timing similar to Sidekiq or delayed_job to retry failed jobs. It can be configured via these settings:\n\n```elixir\nconfig :exq,\n  host: \"127.0.0.1\",\n  port: 6379,\n  ...\n  scheduler_enable: true,\n  max_retries: 25\n```\n\nNote that ```scheduler_enable``` has to be set to ```true``` and ```max_retries``` should be greater than ```0```.\n\n### Dead Jobs\n\nAny job that has failed more than ```max_retries``` times will be\nmoved to dead jobs queue. Dead jobs could be manually re-enqueued via\nSidekiq UI. Max size and timeout of dead jobs queue can be configured via\nthese settings:\n\n```elixir\nconfig :exq,\n  dead_max_jobs: 10_000,\n  dead_timeout_in_seconds: 180 * 24 * 60 * 60, # 6 months\n```\n\n\n### OTP Application\n\nYou can add Exq into your OTP application list, and it will start an instance of Exq along with your application startup.  It will use the configuration from your ```config.exs``` file.\n\n```elixir\ndef application do\n  [\n    applications: [:logger, :exq],\n    #other stuff...\n  ]\nend\n```\n\nWhen using Exq through OTP, it will register a process under the name ```Elixir.Exq``` - you can use this atom where expecting a process name in the Exq module.\n\nIf you would like to control Exq startup, you can configure Exq to not start anything on application start. For example, if you are using Exq along with Phoenix, and your workers are accessing the database or other resources, it is recommended to disable Exq startup and manually add it to the supervision tree.\n\nThis can be done by setting `start_on_application` to false and adding it to your supervision tree:\n\n```elixir\nconfig :exq,\n   start_on_application: false\n```\n\n```elixir\ndef start(_type, _args) do\n  children = [\n    # Start the Ecto repository\n    MyApp.Repo,\n    # Start the endpoint when the application starts\n    MyApp.Endpoint,\n    # Start the EXQ supervisor\n    Exq,\n  ]\n```\n\n### Sentinel\n\nExq uses [Redix](https://github.com/whatyouhide/redix) client for\ncommunication with redis server. The client can be configured to use\nsentinel via `redis_options`. Note: you need to have Redix 0.9.0+.\n\n```elixir\nconfig :exq\n  redis_options: [\n    sentinel: [sentinels: [[host: \"127.0.0.1\", port: 6666]], group: \"exq\"],\n    database: 0,\n    password: nil,\n    timeout: 5000,\n    name: Exq.Redis.Client,\n    socket_opts: []\n  ]\n```\n\n\n## Using IEx\n\nIf you'd like to try Exq out on the iex console, you can do this by typing:\n\n```bash\n\u003e mix deps.get\n```\n\nand then:\n\n```bash\n\u003e iex -S mix\n```\n\n### Standalone Exq\n\nYou can run Exq standalone from the command line, to run it:\n\n```bash\n\u003e mix do app.start, exq.run\n```\n\n## Workers\n\n### Enqueuing jobs:\n\nTo enqueue jobs:\n\n```elixir\n{:ok, ack} = Exq.enqueue(Exq, \"default\", MyWorker, [\"arg1\", \"arg2\"])\n\n{:ok, ack} = Exq.enqueue(Exq, \"default\", \"MyWorker\", [\"arg1\", \"arg2\"])\n\n## Don't retry job in per worker\n{:ok, ack} = Exq.enqueue(Exq, \"default\", MyWorker, [\"arg1\", \"arg2\"], max_retries: 0)\n## max_retries = 10, it will override :max_retries in config\n{:ok, ack} = Exq.enqueue(Exq, \"default\", MyWorker, [\"arg1\", \"arg2\"], max_retries: 10)\n\n```\nIn this example, `\"arg1\"` will get passed as the first argument to the `perform` method in your worker, `\"arg2\"` will be second argument, etc.\n\nYou can also enqueue jobs without starting workers:\n\n```elixir\n{:ok, sup} = Exq.Enqueuer.start_link([port: 6379])\n\n{:ok, ack} = Exq.Enqueuer.enqueue(Exq.Enqueuer, \"default\", MyWorker, [])\n```\n\nYou can also schedule jobs to start at a future time. You need to make sure scheduler_enable is set to true.\n\nSchedule a job to start in 5 mins:\n\n```elixir\n{:ok, ack} = Exq.enqueue_in(Exq, \"default\", 300, MyWorker, [\"arg1\", \"arg2\"])\n\n# If using `mode: [:enqueuer]`\n{:ok, ack} = Exq.Enqueuer.enqueue_in(Exq.Enqueuer, \"default\", 300, MyWorker, [\"arg1\", \"arg2\"])\n```\n\nSchedule a job to start at 8am 2015-12-25 UTC:\n\n```elixir\ntime = Timex.now() |\u003e Timex.shift(days: 8)\n{:ok, ack} = Exq.enqueue_at(Exq, \"default\", time, MyWorker, [\"arg1\", \"arg2\"])\n\n# If using `mode: [:enqueuer]`\n{:ok, ack} = Exq.Enqueuer.enqueue_at(Exq.Enqueuer, \"default\", time, MyWorker, [\"arg1\", \"arg2\"])\n```\n\n### Creating Workers\n\nTo create a worker, create an elixir module matching the worker name that will be enqueued. To process a job with \"MyWorker\", create a MyWorker module.  Note that the perform also needs to match the number of arguments as well.\n\nHere is an example of a worker:\n\n```elixir\ndefmodule MyWorker do\n  def perform do\n  end\nend\n```\n\nWe could enqueue a job to this worker:\n\n```elixir\n{:ok, jid} = Exq.enqueue(Exq, \"default\", MyWorker, [])\n```\n\nThe 'perform' method will be called with matching args. For example:\n\n```elixir\n{:ok, jid} = Exq.enqueue(Exq, \"default\", \"MyWorker\", [arg1, arg2])\n```\n\nWould match:\n\n```elixir\ndefmodule MyWorker do\n  def perform(arg1, arg2) do\n  end\nend\n```\n\n### Job data from worker\n\nIf you'd like to get Job metadata information from a worker, you can call `worker_job` from within the worker:\n\n```elixir\ndefmodule MyWorker do\n  def perform(arg1, arg2) do\n    # get job metadata\n    job = Exq.worker_job()\n  end\nend\n```\n\n### Dynamic queue subscriptions\n\nThe list of queues that are being monitored by Exq is determined by the ```config.exs``` file or the parameters passed to Exq.start_link.  However, we can also dynamically add and remove queue subscriptions after Exq has started.\n\nTo subscribe to a new queue:\n\n```elixir\n# last arg is optional and is the max concurrency for the queue\n:ok = Exq.subscribe(Exq, \"new_queue_name\", 10)\n```\n\nTo unsubscribe from a queue:\n\n```elixir\n:ok = Exq.unsubscribe(Exq, \"queue_to_unsubscribe\")\n```\n\nTo unsubscribe from all queues:\n\n```elixir\n:ok = Exq.unsubscribe_all(Exq)\n```\n\n## Middleware Support\n\nIf you'd like to customize worker execution and/or create plugins like Sidekiq/Resque have, Exq supports custom middleware. The first step would be to define the middleware in ```config.exs``` and add your middleware into the chain:\n\n```elixir\nmiddleware: [Exq.Middleware.Stats, Exq.Middleware.Job, Exq.Middleware.Manager, Exq.Middleware.Unique, Exq.Middleware.Logger]\n```\n\nYou can then create a module that implements the middleware behavior and defines `before_work`,  `after_processed_work` and `after_failed_work` functions.  You can also halt execution of the chain as well. For a simple example of middleware implementation, see the [Exq Logger Middleware](https://github.com/akira/exq/blob/master/lib/exq/middleware/logger.ex).\n\n## Using with Phoenix and Ecto\n\nIf you would like to use Exq alongside Phoenix and Ecto, add `:exq` to your mix.exs application list:\n\n```elixir\ndef application do\n  [\n    mod: {Chat, []},\n    applications: [:phoenix, :phoenix_html, :cowboy, :logger, :exq]\n  ]\nend\n```\n\nAssuming you will be accessing the database from Exq workers, you will want to lower the concurrency level for those workers, as they are using a finite pool of connections and can potentially back up and time out. You can lower this through the ```concurrency``` setting, or perhaps use a different queue for database workers that have a lower concurrency just for that queue. Inside your worker, you would then be able to use the Repo to work with the database:\n\n```elixir\ndefmodule Worker do\n  def perform do\n    HelloPhoenix.Repo.insert!(%HelloPhoenix.User{name: \"Hello\", email: \"world@yours.com\"})\n  end\nend\n```\n\n## Using alongside Sidekiq / Resque\n\nTo use alongside Sidekiq / Resque, make sure your namespaces as configured in Exq match the namespaces you are using in Sidekiq. By default, Exq will use the ```exq``` namespace, so you will have to change that.\n\nAnother option is to modify Sidekiq to use the Exq namespace in the sidekiq initializer in your ruby project:\n\n```ruby\nSidekiq.configure_server do |config|\n  config.redis = { url: 'redis://127.0.0.1:6379', namespace: 'exq' }\nend\n\nSidekiq.configure_client do |config|\n  config.redis = { url: 'redis://127.0.0.1:6379', namespace: 'exq' }\nend\n```\n\nFor an implementation example, see sineed's demo app illustrating  [Sidekiq to Exq communication](https://github.com/sineed/exq_sidekiq_demo_app).\n\nIf you would like to exclusively send some jobs from Sidekiq to Exq as your migration strategy, you should create queue(s) that are exclusively listened to only in Exq (and configure those in the queue section in the Exq config). Make sure they are not configured to be listened to in Sidekiq, otherwise Sidekiq will also take jobs off that queue. You can still Enqueue jobs to that queue in Sidekiq even though they are not being monitored:\n\n```ruby\nSidekiq::Client.push('queue' =\u003e 'elixir_queue', 'class' =\u003e 'ElixirWorker', 'args' =\u003e ['foo', 'bar'])\n```\n\n## Security\n\nBy default, your Redis server could be open to the world. As by default, Redis comes with no password authentication, and some hosting companies leave that port accessible to the world.. This means that anyone can read data on the queue as well as pass data in to be run. Obviously this is not desired, please secure your Redis installation by following guides such as the [Digital Ocean Redis Security Guide](https://www.digitalocean.com/community/tutorials/how-to-secure-your-redis-installation-on-ubuntu-14-04).\n\n## Node Recovery\n\nA Node can be stopped unexpectedly while processing jobs due to various reasons like deployment, system crash, OOM, etc. This could leave the jobs in the in-progress state. Exq comes with two mechanisms to handle this situation.\n\n### Same Node Recovery\n\nExq identifies each node using an identifier. By default machine's hostname is used as the identifier. When a node comes back online after a crash, it will first check if there are any in-progress jobs for its identifier. Note that it will only re-enqueue jobs with the same identifier. There are environments like Heroku or Kubernetes where the hostname would change on each deployment. In those cases, the default identifier can be overridden\n\n```elixir\nconfig :exq,\n   node_identifier: MyApp.CustomNodeIdentifier\n```\n\n```elixir\ndefmodule MyApp.CustomNodeIdentifier do\n  @behaviour Exq.NodeIdentifier.Behaviour\n\n  def node_id do\n     # return node ID, perhaps from environment variable, etc\n     System.get_env(\"NODE_ID\")\n  end\nend\n```\n\n### Heartbeat\n\nSame node recovery is straightforward and works well if the number of worker nodes is fixed. There are use cases that need the worker nodes to be autoscaled based on the workload. In those situations, a node that goes down might not come back for a very long period.\n\nHeartbeat mechanism helps in these cases. Each node registers a heartbeat at regular interval. If any node misses 5 consecutive heartbeats, it will be considered dead and all the in-progress jobs belong to that node will be re-enqueued.\n\nThis feature is disabled by default and can be enabled using the following config:\n\n```elixir\nconfig :exq,\n    heartbeat_enable: true,\n    heartbeat_interval: 60_000,\n    missed_heartbeats_allowed: 5\n```\n\n## Unique Jobs\n\nThere are many use cases where we want to avoid duplicate jobs. Exq\nprovides a few job level options to handle these cases.\n\nThis feature is implemented using lock abstraction. When you enqueue a\njob for the first time, a unique lock is created. The lock token is\nderived from the job queue, class and args or from the `unique_token`\nvalue if provided. If you try to enqueue another job with same args\nand the lock has not expired yet, you will get back `{:conflict,\njid}`, here jid refers the first successful job.\n\nThe lock expiration is controlled by two options.\n\n* `unique_for` (seconds), controls the maximum duration a lock can be\nactive. This option is mandatory to create a unique job and the lock\nnever outlives the expiration duration. In cases of scheduled job, the\nexpiration time is calculated as `scheduled_time + unique_for`\n\n* `unique_until` allows you to clear the lock based on job\nlifecycle. Using `:success` will clear the lock on successful\ncompletion of job or if the job is dead, `:start` will clear the lock\nwhen the job is picked for execution for the first time. `:expiry`\nspecifies the lock should be cleared based on the expiration time set\nvia `unique_for`.\n\n```elixir\n{:ok, jid} = Exq.enqueue(Exq, \"default\", MyWorker, [\"arg1\", \"arg2\"], unique_for: 60 * 60)\n{:conflict, ^jid} = Exq.enqueue(Exq, \"default\", MyWorker, [\"arg1\", \"arg2\"], unique_for: 60 * 60)\n```\n\n### Example usages\n\n* Idempotency - Let's say you want to send a welcome email and want to\n  make sure it's never sent more than once, even when the enqueue part\n  might get retried due to timeout etc. Use a reasonable expiration\n  duration (unique_for) that covers the retry period along with\n  `unique_until: :expiry`.\n\n* Debounce - Let's say for any change to user data, you want to sync\n  it to another system. If you just enqueue a job for each change, you\n  might end up with unnecessary duplicate sync calls. Use\n  `unique_until: :start` along with expiration time based on queue\n  load. This will make sure you never have more than one job pending\n  for a user in the queue.\n\n* Batch - Let's say you want to send a notification to user, but want\n  to wait for an hour and batch them together. Schedule a job one hour\n  in the future using `enqueue_in` and set `unique_until:\n  :success`. This will make sure no other job get enqueued till the\n  scheduled job completes successfully.\n\nAlthough Exq provides unique jobs feature, try to make your worker\nidempotent as much as possible. Unique jobs doesn't prevent your job\nfrom getting retried on failure etc. So, unique jobs is **best\neffort**, not a guarantee to avoid duplicate execution. Uniqueness\nfeature depends on `Exq.Middleware.Unique` middleware. If you override\n`:middleware` configuration, make sure to include it.\n\n\n## Enqueuing Many Jobs Atomically\n\nSimilar to database transactions, there are cases where you may want to\nenqueue/schedule many jobs atomically. A common usecase of this would be when you\nhave a computationally heavy job and you want to break it down to multiple smaller jobs so they\ncan be run concurrently. If you use a loop to enqueue/schedule these jobs,\nand a network, connectivity, or application error occurs while passing these jobs to Exq,\nyou will end up in a situation where you have to roll back all the jobs that you may already\nhave scheduled/enqueued which will be a complicated process. In order to avoid this problem,\nExq comes with an `enqueue_all` method which guarantees atomicity.\n\n\n```elixir\n{:ok, [{:ok, jid_1}, {:ok, jid_2}, {:ok, jid_3}]} = Exq.enqueue_all(Exq, [\n  [job_1_queue, job_1_worker, job_1_args, job_1_options],\n  [job_2_queue, job_2_worker, job_2_args, job_2_options],\n  [job_3_queue, job_3_worker, job_3_args, job_3_options]\n])\n```\n\n`enqueue_all` also supports scheduling jobs via `schedule` key in the `options` passed for each job:\n```elixir\n{:ok, [{:ok, jid_1}, {:ok, jid_2}, {:ok, jid_3}]} = Exq.enqueue_all(Exq, [\n  [job_1_queue, job_1_worker, job_1_args, [schedule: {:in, 60 * 60}]],\n  [job_2_queue, job_2_worker, job_2_args, [schedule: {:at, midnight}]],\n  [job_3_queue, job_3_worker, job_3_args, []] # no schedule key is present, it is enqueued immediately\n])\n```\n\n## Web UI\n\nExq has a separate repo, exq_ui which provides with a Web UI to monitor your workers:\n\n![Screenshot](https://i.imgur.com/MmIiv8b.png)\n\nSee https://github.com/akira/exq_ui for more details.\n\n## Community Plugins\n\n* [exq_scheduler](https://github.com/activesphere/exq-scheduler) Exq Scheduler is a cron like job scheduler for Exq, it's also compatible with Sidekiq and Resque.\n* [exq_limit](https://github.com/ananthakumaran/exq_limit) ExqLimit implements different types of rate limiting for Exq queue.\n* [exq_batch](https://github.com/ananthakumaran/exq_batch) ExqBatch provides a building block to create complex workflows using Exq jobs. A batch monitors a group of Exq jobs and creates callback job when all the jobs are processed.\n\n## Starting Exq manually\n\nTypically, Exq will start as part of the application along with the configuration you have set.  However, you can also start Exq manually and set your own configuration per instance.\n\nHere is an example of how to start Exq manually:\n\n```elixir\n{:ok, sup} = Exq.start_link\n```\n\nTo connect with custom configuration options (if you need multiple instances of Exq for example), you can pass in options under start_link:\n\n```elixir\n{:ok, sup} = Exq.start_link([host: \"127.0.0.1\", port: 6379, namespace: \"x\"])\n```\n\nBy default, Exq will register itself under the ```Elixir.Exq``` atom.  You can change this by passing in a name parameter:\n\n```elixir\n{:ok, exq} = Exq.start_link(name: Exq.Custom)\n```\n\n## Testing\n\n`Exq.Mock` module provides few options to test your workers:\n\n```elixir\n# change queue_adapter in config/test.exs\nconfig :exq,\n  queue_adapter: Exq.Adapters.Queue.Mock\n\n# start mock server in your test_helper.exs\nExq.Mock.start_link(mode: :redis)\n```\n\n`Exq.Mock` currently supports three modes. The default mode can provided\non the `Exq.Mock.start_link` call. The mode could be overridden for\neach test by calling `Exq.Mock.set_mode(:fake)`\n\n### redis\n\nThis could be used for integration testing. Doesn't support `async: true` option.\n\n### fake\n\nThe jobs get enqueued in a local queue and never get executed. `Exq.Mock.jobs()` returns all the jobs. Supports `async: true` option.\n\n### inline\n\nThe jobs get executed in the same process. Supports `async: true` option.\n\n## Donation\n\nTo donate, send to:\n\nBitcoin (BTC): `17j52Veb8qRmVKVvTDijVtmRXvTUpsAWHv`\nEthereum (ETH): `0xA0add27EBdB4394E15b7d1F84D4173aDE1b5fBB3`\n\n\n## Questions?  Issues?\n\nFor issues, please submit a Github issue with steps on how to reproduce the problem.\n\n\n## Contributions\n\nContributions are welcome. Tests are encouraged.\n\nTo run tests / ensure your changes have not caused any regressions:\n\n```\nmix test --no-start\n```\n\nTo run the full suite, including failure conditions (can have some false negatives):\n\n```\nmix test --trace --include failure_scenarios:true --no-start\n```\n\n## Maintainers\n\nAnantha Kumaran / @ananthakumaran (Lead)\n\n## Contributors\n\nJustin McNally (j-mcnally) (structtv), zhongwencool (zhongwencool), Joe Webb (ImJoeWebb), Chelsea Robb (chelsea), Nick Sanders (nicksanders), Nick Gal (nickgal), Ben Wilson (benwilson512), Mike Lawlor (disbelief), colbyh (colbyh), Udo Kramer (optikfluffel), Andreas Franzén (triptec),Josh Kalderimis (joshk), Daniel Perez (tuvistavie), Victor Rodrigues (rodrigues), Denis Tataurov (sineed), Joe Honzawa (Joe-noh), Aaron Jensen (aaronjensen), Andrew Vy (andrewvy), David Le (dl103), Roman Smirnov (romul), Thomas Athanas (typicalpixel), Wen Li (wli0503), Akshay (akki91), Rob Gilson (D1plo1d), edmz (edmz), and Benjamin Tan Wei Hao (benjamintanweihao).\n\n## Copyright and License\n\nCopyright (c) 2014 Alex Kira\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at [http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0)\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n","funding_links":[],"categories":["Queue","Elixir"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fakira%2Fexq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fakira%2Fexq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fakira%2Fexq/lists"}