{"id":13508993,"url":"https://github.com/mbuhot/ecto_job","last_synced_at":"2025-05-16T15:04:29.550Z","repository":{"id":45290557,"uuid":"100206436","full_name":"mbuhot/ecto_job","owner":"mbuhot","description":"Transactional job queue with Ecto, PostgreSQL and GenStage","archived":false,"fork":false,"pushed_at":"2021-12-24T00:38:52.000Z","size":220,"stargazers_count":278,"open_issues_count":6,"forks_count":35,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-04-03T11:11:22.750Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://hexdocs.pm/ecto_job/","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mbuhot.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-08-13T21:44:39.000Z","updated_at":"2025-02-19T09:45:30.000Z","dependencies_parsed_at":"2022-08-02T14:15:44.225Z","dependency_job_id":null,"html_url":"https://github.com/mbuhot/ecto_job","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mbuhot%2Fecto_job","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mbuhot%2Fecto_job/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mbuhot%2Fecto_job/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mbuhot%2Fecto_job/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mbuhot","download_url":"https://codeload.github.com/mbuhot/ecto_job/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248564968,"owners_count":21125412,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T02:01:01.502Z","updated_at":"2025-04-12T11:50:12.926Z","avatar_url":"https://github.com/mbuhot.png","language":"Elixir","funding_links":[],"categories":["Queue"],"sub_categories":[],"readme":"# EctoJob\n\n[![Build Status](https://travis-ci.org/mbuhot/ecto_job.svg?branch=master)](https://travis-ci.org/mbuhot/ecto_job)\n[![Module Version](https://img.shields.io/hexpm/v/ecto_job.svg)](https://hex.pm/packages/ecto_job)\n[![Hex Docs](https://img.shields.io/badge/hex-docs-lightgreen.svg)](https://hexdocs.pm/ecto_job/)\n[![Inline docs](http://inch-ci.org/github/mbuhot/ecto_job.svg?branch=master\u0026style=flat)](http://inch-ci.org/github/mbuhot/ecto_job)\n[![Total Download](https://img.shields.io/hexpm/dt/ecto_job.svg)](https://hex.pm/packages/ecto_job)\n[![License](https://img.shields.io/hexpm/l/ecto_job.svg)](https://github.com/mbuhot/ecto_job/blob/master/LICENSE.md)\n[![Last Updated](https://img.shields.io/github/last-commit/mbuhot/ecto_job.svg)](https://github.com/mbuhot/ecto_job/commits/master)\n\n\nA transactional job queue built with Ecto and GenStage.\n\nIt is compatible with PostgreSQL and MySQL with a major difference:\n* PostgreSQL: job queue updates are notified to ecto_job through PostgreSQL\n  notification feature.\n* MySQL: job queue updates are notified through database polling.\n\n## Goals\n\n - Transactional job processing\n - Retries\n - Scheduled jobs\n - Multiple queues\n - Low latency concurrent processing\n - Avoid frequent database polling\n - Library of functions, not a full OTP application\n\n\n## Getting Started\n\nAdd `:ecto_job` to your `dependencies`\n\n```elixir\n  {:ecto_job, \"~\u003e 3.1\"}\n```\n\n## Installation\n\nAdd a migration to install the notification function and create a job queue table:\n\n```\nmix ecto.gen.migration create_job_queue\n```\n\n```elixir\ndefmodule MyApp.Repo.Migrations.CreateJobQueue do\n  use Ecto.Migration\n\n  @ecto_job_version 3\n\n  def up do\n    EctoJob.Migrations.Install.up()\n    EctoJob.Migrations.CreateJobTable.up(\"jobs\", version: @ecto_job_version)\n  end\n\n  def down do\n    EctoJob.Migrations.CreateJobTable.down(\"jobs\")\n    EctoJob.Migrations.Install.down()\n  end\nend\n```\n\nBy default, a job holds a map of arbitrary data (which corresponds to a `jsonb` field in the table).\nIf you want to store an arbitrary Elixir/Erlang term in the job (`bytea` in the table),\nyou can set up the `params_type` option:\n\n```elixir\ndef up do\n  EctoJob.Migrations.Install.up()\n  EctoJob.Migrations.CreateJobTable.up(\"jobs\", version: @ecto_job_version, params_type: :binary)\nend\n```\n\n### Compatibility\n\n`EctoJob` leverages specific PostgreSQL features, like notification mechanism\nwhen inserting a new job into a queue.\n\nHowever, a non-optimized version of `EctoJob` can be used on top of MySQL \u003e=\n8.0.1. Other version of MySQL / MariaDB may not be working because of the use of\nthe following specific syntax:\n* `FOR UPDATE SKIP LOCKED`\n* Default value for datetime column\n\n### Upgrading to version 3.0\n\nTo upgrade your project to 3.0 version of `ecto_job` you must add a migration to update the pre-existent job queue tables:\n\n```bash\nmix ecto.gen.migration update_job_queue\n```\n\n```elixir\ndefmodule MyApp.Repo.Migrations.UpdateJobQueue do\n  use Ecto.Migration\n  @ecto_job_version 3\n\n  def up do\n    EctoJob.Migrations.UpdateJobTable.up(@ecto_job_version, \"jobs\")\n  end\n  def down do\n    EctoJob.Migrations.UpdateJobTable.down(@ecto_job_version, \"jobs\")\n  end\nend\n```\n\nAdd a module for the queue, mix in `EctoJob.JobQueue`.\nThis will declare an `Ecto.Schema` to use with the table created in the migration, and a `start_link` function allowing the worker supervision tree to be started conveniently.\n\n```elixir\ndefmodule MyApp.JobQueue do\n  use EctoJob.JobQueue, table_name: \"jobs\"\nend\n```\n\nFor jobs being Elixir/Erlang terms, you should add the `:params_type` option:\n\n```elixir\ndefmodule MyApp.JobQueue do\n  use EctoJob.JobQueue, table_name: \"jobs\", params_type: :binary\nend\n```\n\nAdd `perform/2` function to the job queue module, this is where jobs from the queue will be dispatched.\n\n```elixir\ndefmodule MyApp.JobQueue do\n  use EctoJob.JobQueue, table_name: \"jobs\"\n\n  def perform(multi = %Ecto.Multi{}, job = %{}) do\n    ... job logic here ...\n  end\nend\n```\n\nAdd your new `JobQueue` module to the application supervision tree to run the worker supervisor:\n\n```elixir\ndef start(_type, _args) do\n  import Supervisor.Spec\n\n  children = [\n    MyApp.Repo,\n    {MyApp.JobQueue, repo: MyApp.Repo, max_demand: 100}\n  ]\n\n  opts = [strategy: :one_for_one, name: MyApp.Supervisor]\n  Supervisor.start_link(children, opts)\nend\n```\n\nIf you want to run the workers on a separate node to the enqueuers, just leave your `JobQueue` module out of the supervision tree.\n\n## Usage\n\n### Enqueueing jobs\n\nJobs are Ecto schemas, with each queue backed by a different table.\nA job can be inserted into the Repo directly by constructing a job with the `new/2` function:\n\n```elixir\n%{\"type\" =\u003e \"SendEmail\", \"address\" =\u003e \"joe@gmail.com\", \"body\" =\u003e \"Welcome!\"}\n|\u003e MyApp.JobQueue.new()\n|\u003e MyApp.Repo.insert()\n```\n\nFor inserting any arbitrary Elixir/Erlang term:\n\n```elixir\n{\"SendEmail\", \"joe@gmail.com\", \"Welcome!\"}\n|\u003e MyApp.JobQueue.new()\n|\u003e MyApp.Repo.insert()\n```\n\nor\n\n```elixir\n|\u003e %MyStruct{}\n|\u003e MyApp.JobQueue.new()\n|\u003e MyApp.Repo.insert()\n```\n\nA job can be inserted with optional params:\n\n- `:schedule` : runs the job at the given `%DateTime{}`. The default value is `DateTime.utc_now()`.\n- `:max_attempts` : the maximum attempts for this job. The default value is `5`.\n- `:priority` (integer): lower numbers run first; default is 0\n\n```elixir\n%{\"type\" =\u003e \"SendEmail\", \"address\" =\u003e \"joe@gmail.com\", \"body\" =\u003e \"Welcome!\"}\n|\u003e MyApp.JobQueue.new(max_attempts: 10)\n|\u003e MyApp.Repo.insert()\n\n%{\"type\" =\u003e \"SendEmail\", \"address\" =\u003e \"mickel@gmail.com\", \"body\" =\u003e \"Welcome!\"}\n|\u003e MyApp.JobQueue.new(priority: 1)\n|\u003e MyApp.Repo.insert()\n\n%{\"type\" =\u003e \"SendEmail\", \"address\" =\u003e \"jonas@gmail.com\", \"body\" =\u003e \"Welcome!\"}\n|\u003e MyApp.JobQueue.new(priority: 2, max_attempts: 2)\n|\u003e MyApp.Repo.insert()\n```\n\nThe primary benefit of `EctoJob` is the ability to enqueue and process jobs transactionally.\nTo achieve this, a job can be added to an `Ecto.Multi`, along with other application updates, using the `enqueue/3` function:\n\n```elixir\nEcto.Multi.new()\n|\u003e Ecto.Multi.insert(:add_user, User.insert_changeset(%{name: \"Joe\", email: \"joe@gmail.com\"}))\n|\u003e MyApp.JobQueue.enqueue(:email_job, %{\"type\" =\u003e \"SendEmail\", \"address\" =\u003e \"joe@gmail.com\", \"body\" =\u003e \"Welcome!\"})\n|\u003e MyApp.Repo.transaction()\n```\n\n### Handling Jobs\n\nAll jobs sent to a queue are eventually dispatched to the `perform/2` function defined in the queue module.\nThe first argument supplied is an `Ecto.Multi` which has been initialized with a `delete` operation, marking the job as complete.\nThe `Ecto.Multi` struct must be passed to the `Ecto.Repo.transaction` function to complete the job, along with any other application updates.\n\n```elixir\ndefmodule MyApp.JobQueue do\n  use EctoJob.JobQueue, table_name: \"jobs\"\n\n  def perform(multi = %Ecto.Multi{}, job = %{\"type\" =\u003e \"SendEmail\", \"recipient\" =\u003e recipient, \"body\" =\u003e body}) do\n    multi\n    |\u003e Ecto.Multi.run(:send, fn _repo, _changes -\u003e EmailService.send(recipient, body) end)\n    |\u003e Ecto.Multi.insert(:stats, %EmailSendStats{recipient: recipient})\n    |\u003e MyApp.Repo.transaction()\n  end\nend\n```\n\nWhen a queue handles multiple job types, it is useful to pattern match on the job and delegate to separate modules:\n\n```elixir\ndefmodule MyApp.JobQueue do\n  use EctoJob.JobQueue, table_name: \"jobs\"\n\n  def perform(multi = %Ecto.Multi{}, job = %{\"type\" =\u003e \"SendEmail\"}),      do: MyApp.SendEmail.perform(multi, job)\n  def perform(multi = %Ecto.Multi{}, job = %{\"type\" =\u003e \"CustomerReport\"}), do: MyApp.CustomerReport.perform(multi, job)\n  def perform(multi = %Ecto.Multi{}, job = %{\"type\" =\u003e \"SyncWithCRM\"}),    do: MyApp.CRMSync.perform(multi, job)\n  ...\nend\n```\n\n### Options\n\nYou can customize how often the table is polled for scheduled jobs.  The default is `60_000` ms.\n\n```elixir\nconfig :ecto_job, :poll_interval, 15_000\n```\n\nControl the time for which the job is reserved while waiting for a worker to pick it up, before the poller will make the job available again for dispatch by the producer.  The default is `60_000` ms.\n\n```elixir\nconfig :ecto_job, :reservation_timeout, 15_000\n```\n\nControl the delay between retries following a job execution failure. Keep in mind, for jobs that are expected to retry quickly, any configured `retry_timeout` will only retry a job as quickly as the `poll_interval`.  The default is `30_000` ms (30 seconds).\n\n```elixir\nconfig :ecto_job, :retry_timeout, 30_000\n```\n\nControl the timeout for job execution before an \"IN_PROGRESS\" job is assumed to have failed. Begins when job is picked up by worker. Similarly to `retry_timeout`, any configured `execution_timeout` will only retry a job as quickly as the `poll_interval`.  The default is `300_000` ms (5 mins).\n\n```elixir\nconfig :ecto_job, :execution_timeout, 300_000\n```\n\nYou can control whether logs are on or off and the log level.  The default is `true` and `:info`.\n\n```elixir\nconfig :ecto_job, log: true,\n                  log_level: :debug\n```\n\nSee `EctoJob.Config` for configuration details.\n\n## How it works\n\nEach job queue is represented as a PostgreSQL table and Ecto schema.\n\nJobs are added to the queue by inserting into the table, using `Ecto.Repo.transaction` to transactionally enqueue jobs with other application updates.\n\nA `GenStage` producer responds to demand for jobs by efficiently pulling jobs from the queue in batches.\nWhen there is insufficient jobs in the queue, the demand for jobs is buffered.\n\nAs jobs are inserted into the queue, `pg_notify` notifies the producer that new work is available,\nallowing the producer to dispatch jobs immediately if there is pending demand.\n\nA `GenStage` `ConsumerSupervisor` subscribes to the producer, and spawns a new `Task` for each job.\n\nThe callback for each job receives an `Ecto.Multi` structure, pre-populated with a `delete`\ncommand to remove the job from the queue.\n\nApplication code then add additional commands to the `Ecto.Multi` and submit it to the\n`Repo` with a call to `transaction`, ensuring that application updates are performed atomically with the job removal.\n\nScheduled jobs and Failed jobs are reactivated by polling the database once per minute.\n\n## Job Lifecycle\n\nJobs scheduled to run at a future time start in the \"SCHEDULED\" state.\nScheduled jobs transition to \"AVAILABLE\" after the scheduled time has passed.\n\nJobs that are intended to run immediately start in an \"AVAILABLE\" state.\n\nThe producer will update a batch of jobs setting the state to \"RESERVED\", with an expiry of 5 minutes unless otherwise configured.\n\nOnce a consumer is given a job, it increments the attempt counter and updates the state to \"IN_PROGRESS\", with an initial timeout configurable as `execution_timeout`, defaulting to 5 minutes.\nIf the job is being retried, the expiry will be initial timeout * the attempt counter.\n\nIf successful, the consumer can delete the job from the queue using the preloaded multi passed to the `perform/2` job handler.\nIf an exception is raised in the worker or a successful processing attempt fails to successfully commit the preloaded multi, the job is transitioned to the \"RETRY\" state, scheduled to run again after `retry_timeout` * attempt counter.\nIf the processes is killed or is otherwise unable to transition to \"RETRY\", it will remain in \"IN_PROGRESS\" until the `execution_timeout` expires.\n\nJobs in the \"RESERVED\" or \"IN_PROGRESS\" state past the expiry time will be returned to the \"AVAILABLE\" state.\n\nExpired jobs in the \"IN_PROGRESS\" state with attempts \u003e= MAX_ATTEMPTS move to a \"FAILED\" state.\nFailed jobs are kept in the database so that application developers can handle the failure.\n\n## Job Timeouts and Transactional Safety\n\nWhen performing long-running jobs or when configuring a short execution timeout, keep in mind that a job may be retried before it has finished and the retry has no proactive mechanism to cancel the running job.\n\nIn the case that the initial job attempts to finish and commit a result, and the commit includes the preloaded multi passed as the first parameter to `perform/2`, the optimistic lock will fail the transaction.\n\nIn the case where the job performs other side effects outside of the transaction such as calls to external APIs or additional database writes, these are suggested to implement other idempotency guarantees, as they will not be rolled back in a failed or duplicated job.\n\n## Copyright and License\n\nCopyright (c) 2017 Mike Buhot\n\nThis library is released under the MIT License. See the [LICENSE.md](./LICENSE.md) file\nfor further details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmbuhot%2Fecto_job","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmbuhot%2Fecto_job","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmbuhot%2Fecto_job/lists"}