{"id":13509044,"url":"https://github.com/edgurgel/verk","last_synced_at":"2025-05-15T06:03:03.686Z","repository":{"id":44454005,"uuid":"41664621","full_name":"edgurgel/verk","owner":"edgurgel","description":"A job processing system that just verks! 🧛‍","archived":false,"fork":false,"pushed_at":"2021-09-14T07:10:11.000Z","size":474,"stargazers_count":727,"open_issues_count":12,"forks_count":62,"subscribers_count":17,"default_branch":"master","last_synced_at":"2025-05-15T06:02:22.081Z","etag":null,"topics":["elixir","redis","resque","sidekiq","workers"],"latest_commit_sha":null,"homepage":"https://hex.pm/packages/verk","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/edgurgel.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-08-31T08:29:56.000Z","updated_at":"2025-04-22T19:26:19.000Z","dependencies_parsed_at":"2022-08-28T21:21:46.760Z","dependency_job_id":null,"html_url":"https://github.com/edgurgel/verk","commit_stats":null,"previous_names":[],"tags_count":51,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edgurgel%2Fverk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edgurgel%2Fverk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edgurgel%2Fverk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edgurgel%2Fverk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/edgurgel","download_url":"https://codeload.github.com/edgurgel/verk/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254283336,"owners_count":22045140,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elixir","redis","resque","sidekiq","workers"],"created_at":"2024-08-01T02:01:02.210Z","updated_at":"2025-05-15T06:03:03.617Z","avatar_url":"https://github.com/edgurgel.png","language":"Elixir","funding_links":[],"categories":["Queue","Elixir"],"sub_categories":[],"readme":"![Verk](https://i.imgur.com/unSd0Zr.png)\n\n[![Build Status](https://travis-ci.org/edgurgel/verk.svg?branch=master)](https://travis-ci.org/edgurgel/verk)\n[![Coverage Status](https://coveralls.io/repos/edgurgel/verk/badge.svg?branch=master\u0026service=github)](https://coveralls.io/github/edgurgel/verk?branch=master)\n[![Module Version](https://img.shields.io/hexpm/v/verk.svg)](https://hex.pm/packages/verk)\n[![Hex Docs](https://img.shields.io/badge/hex-docs-lightgreen.svg)](https://hexdocs.pm/verk/)\n[![Total Download](https://img.shields.io/hexpm/dt/verk.svg)](https://hex.pm/packages/verk)\n[![License](https://img.shields.io/hexpm/l/verk.svg)](https://github.com/edgurgel/verk/blob/master/LICENSE)\n[![Last Updated](https://img.shields.io/github/last-commit/edgurgel/verk.svg)](https://github.com/edgurgel/verk/commits/master)\n\nVerk is a job processing system backed by Redis. It uses the same job definition of Sidekiq/Resque.\n\nThe goal is to be able to isolate the execution of a queue of jobs as much as possible.\n\nEvery queue has its own supervision tree:\n\n* A pool of workers;\n* A `QueueManager` that interacts with Redis to get jobs and enqueue them back to be retried if necessary;\n* A `WorkersManager` that will interact with the `QueueManager` and the pool to execute jobs.\n\nVerk will hold one connection to Redis per queue plus one dedicated to the `ScheduleManager` and one general connection for other use cases like deleting a job from retry set or enqueuing new jobs.\n\nThe `ScheduleManager` fetches jobs from the `retry` set to be enqueued back to the original queue when it's ready to be retried.\n\nIt also has one GenStage producer called `Verk.EventProducer`.\n\nThe image below is an overview of Verk's supervision tree running with a queue named `default` having 5 workers.\n\n![Supervision Tree](https://i.imgur.com/1vzAVfZ.png)\n\nFeature set:\n\n* Retry mechanism with exponential backoff\n* Dynamic addition/removal of queues\n* Reliable job processing (RPOPLPUSH and Lua scripts to the rescue)\n* Error and event tracking\n\n## Installation\n\nFirst, add `:verk` to your `mix.exs` dependencies:\n\n```elixir\ndef deps do\n  [\n    {:verk, \"~\u003e 1.0\"}\n  ]\nend\n```\n\nand run `$ mix deps.get`.\n\nAdd `:verk` to your applications list if your Elixir version is 1.3 or lower:\n\n```elixir\ndef application do\n  [\n    applications: [:verk]\n  ]\nend\n```\n\nAdd `Verk.Supervisor` to your supervision tree:\n\n```elixir\ndefmodule Example.App do\n  use Application\n\n  def start(_type, _args) do\n    import Supervisor.Spec\n    tree = [supervisor(Verk.Supervisor, [])]\n    opts = [name: Simple.Sup, strategy: :one_for_one]\n    Supervisor.start_link(tree, opts)\n  end\nend\n```\n\nFinally we need to configure how Verk will process jobs.\n\n## Configuration\n\nExample configuration for Verk having 2 queues: `default` and `priority`\n\nThe queue `default` will have a maximum of 25 jobs being processed at a time and `priority` just 10.\n\n```elixir\nconfig :verk, queues: [default: 25, priority: 10],\n              max_retry_count: 10,\n              max_dead_jobs: 100,\n              poll_interval: 5000,\n              start_job_log_level: :info,\n              done_job_log_level: :info,\n              fail_job_log_level: :info,\n              node_id: \"1\",\n              redis_url: \"redis://127.0.0.1:6379\"\n```\n\nVerk supports the convention `{:system, \"ENV_NAME\", default}` for reading environment configuration at runtime using [Confex](https://hexdocs.pm/confex/readme.html):\n\n```elixir\nconfig :verk, queues: [default: 25, priority: 10],\n              max_retry_count: 10,\n              max_dead_jobs: 100,\n              poll_interval: {:system, :integer, \"VERK_POLL_INTERVAL\", 5000},\n              start_job_log_level: :info,\n              done_job_log_level: :info,\n              fail_job_log_level: :info,\n              node_id: \"1\",\n              redis_url: {:system, \"VERK_REDIS_URL\", \"redis://127.0.0.1:6379\"}\n```\n\nNow Verk is ready to start processing jobs! :tada:\n\n## Workers\n\nA job is defined by a module and arguments:\n\n```elixir\ndefmodule ExampleWorker do\n  def perform(arg1, arg2) do\n    arg1 + arg2\n  end\nend\n```\n\nThis job can be enqueued using `Verk.enqueue/1`:\n\n```elixir\nVerk.enqueue(%Verk.Job{queue: :default, class: \"ExampleWorker\", args: [1,2], max_retry_count: 5})\n```\n\nThis job can also be scheduled using `Verk.schedule/2`:\n\n```elixir\nperform_at = Timex.shift(Timex.now, seconds: 30)\nVerk.schedule(%Verk.Job{queue: :default, class: \"ExampleWorker\", args: [1,2]}, perform_at)\n```\n\n### Retry at\n\nA job can define the function `retry_at/2` for custom retry time delay:\n\n```elixir\ndefmodule ExampleWorker do\n  def perform(arg1, arg2) do\n    arg1 + arg2\n  end\n\n  def retry_at(failed_at, retry_count) do\n    failed_at + retry_count\n  end\nend\n```\n\nIn this example, the first retry will be scheduled a second later,\nthe second retry will be scheduled two seconds later, and so on.\n\nIf `retry_at/2` is not defined the default exponential backoff is used.\n\n### Keys in arguments\n\nBy default, Verk will decode keys in arguments to binary strings.\nYou can change this behavior for jobs enqueued by Verk with the following configuration:\n```elixir\nconfig :verk, :args_keys, value\n```\n\nThe following values are valid:\n\n* `:strings` (default) - decodes keys as binary strings\n* `:atoms` - keys are converted to atoms using `String.to_atom/1`\n* `:atoms!` - keys are converted to atoms using `String.to_existing_atom/1`\n\n\n## Queues\n\nIt's possible to dynamically add and remove queues from Verk.\n\n```elixir\nVerk.add_queue(:new, 10) # Adds a queue named `new` with 10 workers\n```\n\n```elixir\nVerk.remove_queue(:new) # Terminate and delete the queue named `new`\n```\n\n## Deployment\n\nThe way Verk currently works, there are two pitfalls to pay attention to:\n\n1. **Each worker node's `node_id` MUST be unique.** If a node goes online with a\n  `node_id`, which is already in use by another running node, then the second\n  node will re-enqueue all jobs currently in progress on the first node, which\n  results in jobs executed multiple times.\n\n2. **Take caution around removing nodes.** If a node with jobs in progress is\n  killed, those jobs will not be restarted until another node with the same\n  `node_id` comes online. If another node with the same `node_id` never comes\n  online, the jobs will be stuck forever. This means you should not use dynamic\n  `node_id`s such as Docker container ids or Kubernetes Deployment pod names.\n\n### On Heroku\n\nHeroku provides\n[an experimental environment variable](https://devcenter.heroku.com/articles/dynos#local-environment-variables)\nnamed after the type and number of the dyno.\n\n```elixir\nconfig :verk,\n  node_id: {:system, \"DYNO\", \"job.1\"}\n```\n\n_It is possible that two dynos with the same name could overlap for a short time\nduring a dyno restart._ As the Heroku documentation says:\n\n\u003e [...] $DYNO is not guaranteed to be unique within an app. For example, during\n\u003e a deploy or restart, the same dyno identifier could be used for two running\n\u003e dynos. It will be eventually consistent, however.\n\nThis means that you are still at risk of violating the first rule above on\n`node_id` uniqueness. A slightly naive way of lowering the risk would be to\nadd a delay in your application before the Verk queue starts.\n\n### On Kubernetes\n\nWe recommend using a\n[StatefulSet](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/)\nto run your pool of workers. StatefulSets add a label,\n`statefulset.kubernetes.io/pod-name`, to all its pods with the value\n`{name}-{n}`, where `{name}` is the name of your StatefulSet and `{n}` is a\nnumber from 0 to `spec.replicas - 1`. StatefulSets maintain a sticky identity\nfor its pods and guarantee that two identical pods are never up simultaneously.\nThis way it satisfies both of our deployment rules mentioned above.\n\nDefine your worker like this:\n\n```yaml\n# StatefulSets require a service, even though we don't use it directly for anything\napiVersion: v1\nkind: Service\nmetadata:\n name: my-worker\n labels:\n   app: my-worker\nspec:\n clusterIP: None\n selector:\n   app: my-worker\n\n---\n\napiVersion: apps/v1\nkind: StatefulSet\nmetadata:\n name: my-worker\n labels:\n   app: my-worker\nspec:\n selector:\n   matchLabels:\n     app: my-worker\n serviceName: my-worker\n # We run two workers in this example\n replicas: 2\n # The workers don't depend on each other, so we can use Parallel pod management\n podManagementPolicy: Parallel\n template:\n   metadata:\n     labels:\n       app: my-worker\n   spec:\n     # This should probably match up with the setting you used for Verk's :shutdown_timeout\n     terminationGracePeriodSeconds: 30\n     containers:\n       - name: my-worker\n         image: my-repo/my-worker\n         env:\n           - name: VERK_NODE_ID\n             valueFrom:\n               fieldRef:\n                 fieldPath: metadata.labels['statefulset.kubernetes.io/pod-name']\n```\n\nNotice how we use a `fieldRef` to expose the pod's\n`statefulset.kubernetes.io/pod-name` label as the `VERK_NODE_ID` environment\nvariable. Instruct Verk to use this environment variable as `node_id`:\n\n```elixir\nconfig :verk,\n  node_id: {:system, \"VERK_NODE_ID\"}\n```\n\nBe careful when scaling the number of `replicas` down. Make sure that the pods\nthat will be stopped and never come back do not have any jobs in progress.\nScaling up is always safe.\n\nDon't use Deployments for pods that will run Verk. If you hardcode `node_id`\ninto your config, multiple pods with the same `node_id`will be online at the\nsame time, violating the first rule. If you use a non-sticky environment\nvariable, such as `HOSTNAME`, you'll violate the second rule and cause jobs to\nget stuck every time you deploy.\n\nIf your application serves as e.g. both an API and Verk queue, then it may be\nwise to run a separate Deployment for your API, which does not run Verk. In that\ncase you can configure your application to check an environment variable,\n`VERK_DISABLED`, for whether it should handle any Verk queues:\n\n```elixir\n# In your config.exs\nconfig :verk,\n  queues: {:system, {MyApp.Env, :verk_queues, []}, \"VERK_DISABLED\"}\n\n# In some other file\ndefmodule MyApp.Env do\n  def verk_queues(\"true\"), do: {:ok, []}\n  def verk_queues(_), do: {:ok, [default: 25, priority: 10]}\nend\n```\n\nThen set `VERK_DISABLED=true` in your Deployment's spec.\n\n### EXPERIMENTAL - Generate Node ID\n\nSince Verk 1.6.0 there is a new experimental optional configuration `generate_node_id`. Node IDs are completely controlled automatically by Verk if this configuration is set to `true`.\n\n\n#### Under the hood\n\n* Each time a job is moved to the list of jobs inprogress of a queue this node is added to `verk_nodes` (`SADD verk_nodes node_id`) and the queue is added to `verk:node:#{node_id}:queues` (`SADD verk:node:123:queues queue_name`)\n\n* Each frequency milliseconds we set the node key to expire in 2 * frequency\n`PSETEX verk:node:#{node_id} 2 * frequency alive`\n\n* Each frequency milliseconds check for all the keys of all nodes (`verk_nodes`). If the key expired it means that this node is dead and it needs to have its jobs restored.\n\nTo restore we go through all the running queues (`verk:node:#{node_id}:queues`) of that node and enqueue them from inprogress back to the queue. Each \"enqueue back from in progress\" is atomic (\u003c3 Lua) so we won't have duplicates.\n\n#### Configuration\n\nThe default `frequency` is 30_000 milliseconds but it can be changed by setting the configuration key `heartbeat`.\n\n```elixir\nconfig :verk,\n  queues: [default: 5, priority: 5],\n  redis_url: \"redis://127.0.0.1:6379\",\n  generate_node_id: true,\n  heartbeat: 30_000,\n```\n\n## Reliability\n\nVerk's goal is to never have a job that exists only in memory. It uses Redis as the single source of truth to retry and track jobs that were being processed if some crash happened.\n\nVerk will re-enqueue jobs if the application crashed while jobs were running. It will also retry jobs that failed keeping track of the errors that happened.\n\nThe jobs that will run on top of Verk should be idempotent as they may run more than once.\n\n## Error tracking\n\nOne can track when jobs start and finish or fail. This can be useful to build metrics around the jobs. The `QueueStats` handler does some kind of metrics using these events: https://github.com/edgurgel/verk/blob/master/lib/verk/queue_stats.ex\n\nVerk has an Event Manager that notifies the following events:\n\n* `Verk.Events.JobStarted`\n* `Verk.Events.JobFinished`\n* `Verk.Events.JobFailed`\n* `Verk.Events.QueueRunning`\n* `Verk.Events.QueuePausing`\n* `Verk.Events.QueuePaused`\n\nOne can define an error tracking handler like this:\n\n```elixir\ndefmodule TrackingErrorHandler do\n  use GenStage\n\n  def start_link() do\n    GenStage.start_link(__MODULE__, :ok)\n  end\n\n  def init(_) do\n    filter = fn event -\u003e event.__struct__ == Verk.Events.JobFailed end\n    {:consumer, :state, subscribe_to: [{Verk.EventProducer, selector: filter}]}\n  end\n\n  def handle_events(events, _from, state) do\n    Enum.each(events, \u0026handle_event/1)\n    {:noreply, [], state}\n  end\n\n  defp handle_event(%Verk.Events.JobFailed{job: job, failed_at: failed_at, stacktrace: trace}) do\n    MyTrackingExceptionSystem.track(stacktrace: trace, name: job.class)\n  end\nend\n```\n\nNotice the selector to get just the type JobFailed. If no selector is set every event is sent.\n\nThen adding the consumer to your supervision tree:\n\n```elixir\ndefmodule Example.App do\n  use Application\n\n  def start(_type, _args) do\n    import Supervisor.Spec\n    tree = [supervisor(Verk.Supervisor, []),\n            worker(TrackingErrorHandler, [])]\n    opts = [name: Simple.Sup, strategy: :one_for_one]\n    Supervisor.start_link(tree, opts)\n  end\nend\n```\n\n## Dashboard ?\n\nCheck [Verk Web](https://github.com/edgurgel/verk_web)!\n\n![Dashboard](http://i.imgur.com/LsDKIVT.png)\n\n## Metrics ?\n\nCheck [Verk Stats](https://github.com/edgurgel/verk-stats)\n\n## License\n\nCopyright (c) 2013 Eduardo Gurgel Pinho\n\nVerk is released under the MIT License. See the [LICENSE.md](./LICENSE.md) file\nfor further details.\n\n## Sponsorship\n\nInitial development sponsored by [Carnival.io](http://carnival.io)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fedgurgel%2Fverk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fedgurgel%2Fverk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fedgurgel%2Fverk/lists"}