{"id":13509031,"url":"https://github.com/sheharyarn/que","last_synced_at":"2025-05-15T10:00:39.278Z","repository":{"id":45725913,"uuid":"77558951","full_name":"sheharyarn/que","owner":"sheharyarn","description":"Simple Job Processing in Elixir with Mnesia :zap:","archived":false,"fork":false,"pushed_at":"2024-02-24T05:28:15.000Z","size":166,"stargazers_count":676,"open_issues_count":13,"forks_count":35,"subscribers_count":15,"default_branch":"master","last_synced_at":"2025-04-14T15:56:54.960Z","etag":null,"topics":["background-jobs","cloud-native","elixir","gen-server","hacktoberfest","in-memory","job-queue","kubernetes","mnesia"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sheharyarn.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-12-28T20:19:05.000Z","updated_at":"2025-04-06T14:58:15.000Z","dependencies_parsed_at":"2024-05-01T17:21:10.248Z","dependency_job_id":"0f03748e-96d8-4c17-beeb-4afa791d1282","html_url":"https://github.com/sheharyarn/que","commit_stats":{"total_commits":264,"total_committers":4,"mean_commits":66.0,"dds":"0.018939393939393923","last_synced_commit":"8a3dc1dc9ee8c52525c999eea2652a819e195d35"},"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sheharyarn%2Fque","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sheharyarn%2Fque/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sheharyarn%2Fque/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sheharyarn%2Fque/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sheharyarn","download_url":"https://codeload.github.com/sheharyarn/que/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254310520,"owners_count":22049470,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["background-jobs","cloud-native","elixir","gen-server","hacktoberfest","in-memory","job-queue","kubernetes","mnesia"],"created_at":"2024-08-01T02:01:02.019Z","updated_at":"2025-05-15T10:00:38.171Z","avatar_url":"https://github.com/sheharyarn.png","language":"Elixir","funding_links":[],"categories":["Queue"],"sub_categories":[],"readme":"[\u003cimg src='https://i.imgur.com/Eec71eh.png' alt='Que' width='200px' /\u003e][docs]\n=============================================================================\n\n[![Build Status][shield-travis]][travis-ci]\n[![Coverage Status][shield-inch]][docs]\n[![Version][shield-version]][hexpm]\n[![Downloads][shield-downloads]][hexpm]\n[![License][shield-license]][hexpm]\n\n\u003e Simple Background Job Processing in Elixir :zap:\n\nQue is a job processing library backed by [`Mnesia`][mnesia], a distributed\nreal-time database that comes with Erlang / Elixir. That means it doesn't\ndepend on any external services like `Redis` for persisting job state. This\nmakes it really easy to use since you don't need to install anything other\nthan Que itself.\n\nSee the [Documentation][docs].\n\n\u003cbr\u003e\n\n\n\n\n## Installation\n\nAdd `que` to your project dependencies in `mix.exs`:\n\n```elixir\ndef deps do\n  [{:que, \"~\u003e 0.10.1\"}]\nend\n```\n\nand then add it to your list of `applications`:\n\n```elixir\ndef application do\n  [applications: [:que]]\nend\n```\n\n\n### Mnesia Setup\n\nQue runs out of the box, but by default all jobs are stored in-memory.\nTo persist jobs across application restarts, specify the DB path in\nyour `config.exs`:\n\n```elixir\nconfig :mnesia, dir: 'mnesia/#{Mix.env}/#{node()}'        # Notice the single quotes\n```\n\nAnd run the following mix task:\n\n```bash\n$ mix que.setup\n```\n\nThis will create the Mnesia schema and job database for you. For a\ndetailed guide, see the [Mix Task Documentation][docs-mix]. For\ncompiled releases where `Mix` is not available\n[see this][docs-setup-prod].\n\n\u003cbr\u003e\n\n\n\n\n## Usage\n\nQue is very similar to other job processing libraries such as Ku and\nToniq. Start by defining a [`Worker`][docs-worker] with a `perform/1`\ncallback to process your jobs:\n\n```elixir\ndefmodule App.Workers.ImageConverter do\n  use Que.Worker\n\n  def perform(image) do\n    ImageTool.save_resized_copy!(image, :thumbnail)\n    ImageTool.save_resized_copy!(image, :medium)\n  end\nend\n```\n\nYou can now add jobs to be processed by the worker:\n\n```elixir\nQue.add(App.Workers.ImageConverter, some_image)\n#=\u003e {:ok, %Que.Job{...}}\n```\n\n\n### Pattern Matching\n\nThe argument here can be any term from a Tuple to a Keyword List\nor a Struct. You can also pattern match and use guard clauses like\nany other method:\n\n```elixir\ndefmodule App.Workers.NotificationSender do\n  use Que.Worker\n\n  def perform(type: :like, to: user, count: count) do\n    User.notify(user, \"You have #{count} new likes on your posts\")\n  end\n\n  def perform(type: :message, to: user, from: sender) do\n    User.notify(user, \"You received a new message from #{sender.name}\")\n  end\n\n  def perform(to: user) do\n    User.notify(user, \"New activity on your profile\")\n  end\nend\n```\n\n\n### Concurrency\n\nBy default, all workers process one Job at a time, but you can\ncustomize that by passing the `concurrency` option:\n\n```elixir\ndefmodule App.Workers.SignupMailer do\n  use Que.Worker, concurrency: 4\n\n  def perform(email) do\n    Mailer.send_email(to: email, message: \"Thank you for signing up!\")\n  end\nend\n```\n\n\n### Job Success / Failure Callbacks\n\nThe worker can also export optional `on_success/1` and `on_failure/2`\ncallbacks that handle appropriate cases.\n\n```elixir\ndefmodule App.Workers.ReportBuilder do\n  use Que.Worker\n\n  def perform({user, report}) do\n    report.data\n    |\u003e PDFGenerator.generate!\n    |\u003e File.write!(\"reports/#{user.id}/report-#{report.id}.pdf\")\n  end\n\n  def on_success({user, _}) do\n    Mailer.send_email(to: user.email, subject: \"Your Report is ready!\")\n  end\n\n  def on_failure({user, report}, error) do\n    Mailer.send_email(to: user.email, subject: \"There was a problem generating your report\")\n    Logger.error(\"Could not generate report #{report.id}. Reason: #{inspect(error)}\")\n  end\nend\n```\n\n\n### Setup and Teardown\n\nYou can similarly export optional `on_setup/1` and `on_teardown/1` callbacks\nthat are respectively run before and after the job is performed (successfully\nor not). But instead of the job arguments, they pass the job struct as an\nargument which holds a lot more internal details that can be useful for custom\nfeatures such as logging, metrics, requeuing and more.\n\n```elixir\ndefmodule MyApp.Workers.VideoProcessor do\n  use Que.Worker\n\n  def on_setup(%Que.Job{} = job) do\n    VideoMetrics.record(job.id, :start, process: job.pid, status: :starting)\n  end\n\n  def perform({user, video, options}) do\n    User.notify(user, \"Your video is processing, check back later.\")\n    FFMPEG.process(video.path, options)\n  end\n\n  def on_teardown(%Que.Job{} = job) do\n    {user, video, _options} = job.arguments\n    link = MyApp.Router.video_path(user.id, video.id)\n\n    VideoMetrics.record(job.id, :end, status: job.status)\n    User.notify(user, \"We've finished processing your video. See the results.\", link)\n  end\nend\n```\n\n\nHead over to Hexdocs for detailed [`Worker` documentation][docs-worker].\n\n\u003cbr\u003e\n\n\n\n\n## Roadmap\n\n - [x] Write Documentation\n - [x] Write Tests\n - [x] Persist Job State to Disk\n    - [x] Provide an API to interact with Jobs\n - [x] Add Concurrency Support\n    - [x] Make jobs work in Parallel\n    - [x] Allow customizing the number of concurrent jobs\n - [x] Success/Failure Callbacks\n - [x] Find a more reliable replacement for Amnesia\n - [ ] Delayed Jobs\n - [ ] Allow job cancellation\n - [ ] Job Priority\n - [ ] Support running in a multi-node enviroment\n    - [ ] Recover from node failures\n - [ ] Support for more Persistence Adapters\n    - [ ] Redis\n    - [ ] Postgres\n - [x] Mix Task for creating Mnesia Database\n - [ ] Better Job Failures\n    - [ ] Option to set timeout on workers\n    - [ ] Add strategies to automatically retry failed jobs\n - [ ] Web UI\n\n\u003cbr\u003e\n\n\n\n\n## Contributing\n\n - [Fork][github-fork], Enhance, Send PR\n - Lock issues with any bugs or feature requests\n - Implement something from Roadmap\n - Spread the word :heart:\n\n\u003cbr\u003e\n\n\n\n\n## License\n\nThis package is available as open source under the terms of the [MIT License][license].\n\n\u003cbr\u003e\n\n\n\n\n  [logo]:             https://i.imgur.com/Eec71eh.png\n  [shield-version]:   https://img.shields.io/hexpm/v/que.svg\n  [shield-license]:   https://img.shields.io/hexpm/l/que.svg\n  [shield-downloads]: https://img.shields.io/hexpm/dt/que.svg\n  [shield-travis]:    https://img.shields.io/travis/sheharyarn/que/master.svg\n  [shield-inch]:      https://inch-ci.org/github/sheharyarn/que.svg?branch=master\n\n  [travis-ci]:        https://travis-ci.org/sheharyarn/que\n  [inch-ci]:          https://inch-ci.org/github/sheharyarn/que\n\n  [license]:          https://opensource.org/licenses/MIT\n  [mnesia]:           http://erlang.org/doc/man/mnesia.html\n  [hexpm]:            https://hex.pm/packages/que\n\n  [docs]:             https://hexdocs.pm/que\n  [docs-worker]:      https://hexdocs.pm/que/Que.Worker.html\n  [docs-mix]:         https://hexdocs.pm/que/Mix.Tasks.Que.Setup.html\n  [docs-setup-prod]:  https://hexdocs.pm/que/Que.Persistence.Mnesia.html#setup!/0\n\n  [github-fork]:      https://github.com/sheharyarn/que/fork\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsheharyarn%2Fque","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsheharyarn%2Fque","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsheharyarn%2Fque/lists"}