{"id":13879707,"url":"https://github.com/bia-technologies/lowkiq","last_synced_at":"2025-04-05T13:10:07.573Z","repository":{"id":35146212,"uuid":"212767013","full_name":"bia-technologies/lowkiq","owner":"bia-technologies","description":"Ordered background jobs processing","archived":false,"fork":false,"pushed_at":"2023-07-03T00:22:31.000Z","size":1287,"stargazers_count":142,"open_issues_count":3,"forks_count":11,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-03-29T12:17:15.299Z","etag":null,"topics":["background-jobs","lowkiq","redis","ruby"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bia-technologies.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-10-04T08:16:14.000Z","updated_at":"2024-04-22T13:21:44.000Z","dependencies_parsed_at":"2024-01-05T22:00:13.125Z","dependency_job_id":null,"html_url":"https://github.com/bia-technologies/lowkiq","commit_stats":{"total_commits":51,"total_committers":5,"mean_commits":10.2,"dds":"0.11764705882352944","last_synced_commit":"79aa8c496b45893567426c13bfc5d7e751a5c0e2"},"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bia-technologies%2Flowkiq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bia-technologies%2Flowkiq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bia-technologies%2Flowkiq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bia-technologies%2Flowkiq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bia-technologies","download_url":"https://codeload.github.com/bia-technologies/lowkiq/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247339158,"owners_count":20923014,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["background-jobs","lowkiq","redis","ruby"],"created_at":"2024-08-06T08:02:29.803Z","updated_at":"2025-04-05T13:10:07.546Z","avatar_url":"https://github.com/bia-technologies.png","language":"Ruby","funding_links":[],"categories":["Ruby","Queues and Messaging"],"sub_categories":[],"readme":"[![Gem Version](https://badge.fury.io/rb/lowkiq.svg)](https://badge.fury.io/rb/lowkiq)\n\n# Lowkiq\n\nOrdered background jobs processing\n\n![dashboard](doc/dashboard.png)\n\n* [Rationale](#rationale)\n* [Description](#description)\n* [Sidekiq comparison](#sidekiq-comparison)\n* [Queue](#queue)\n  + [Calculation algorithm for `retry_count` and `perform_in`](#calculation-algorithm-for-retry_count-and-perform_in)\n  + [Job merging rules](#job-merging-rules)\n* [Install](#install)\n* [Api](#api)\n* [Ring app](#ring-app)\n* [Configuration](#configuration)\n* [Performance](#performance)\n* [Execution](#execution)\n* [Shutdown](#shutdown)\n* [Debug](#debug)\n* [Development](#development)\n* [Exceptions](#exceptions)\n* [Rails integration](#rails-integration)\n* [Splitter](#splitter)\n* [Scheduler](#scheduler)\n* [Recommendations on configuration](#recommendations-on-configuration)\n  + [`SomeWorker.shards_count`](#someworkershards_count)\n  + [`SomeWorker.max_retry_count`](#someworkermax_retry_count)\n* [Changing of worker's shards amount](#changing-of-workers-shards-amount)\n* [Extended error info](#extended-error-info)\n\n## Rationale\n\nWe've faced some problems using Sidekiq while processing messages from a side system.\nFor instance, the message is the data of an order at a particular time.\nThe side system will send new data of an order on every change.\nOrders are frequently updated and a queue contains some closely located messages of the same order.\n\nSidekiq doesn't guarantee a strict message order, because a queue is processed by multiple threads.\nFor example, we've received 2 messages: M1 and M2.\nSidekiq handlers begin to process them parallel,\nso M2 can be processed before M1.\n\nParallel processing of such kind of messages can result in:\n\n+ deadlocks\n+ overwriting new data with an old one\n\nLowkiq has been created to eliminate such problems by avoiding parallel task processing within one entity.\n\n## Description\n\nLowkiq's queues are reliable i.e.,\nLowkiq saves information about a job being processed\nand returns uncompleted jobs to the queue on startup.\n\nJobs in queues are ordered by preassigned execution time, so they are not FIFO queues.\n\nEvery job has its identifier. Lowkiq guarantees that jobs with equal IDs are processed by the same thread.\n\nEvery queue is divided into a permanent set of shards.\nA job is placed into a particular shard based on an id of the job.\nSo jobs with the same id are always placed into the same shard.\nAll jobs of the shard are always processed with the same thread.\nThis guarantees the sequential processing of jobs with the same ids and excludes the possibility of locks.\n\nBesides the id, every job has a payload.\nPayloads are accumulated for jobs with the same id.\nSo all accumulated payloads will be processed together.\nIt's useful when you need to process only the last message and drop all previous ones.\n\nA worker corresponds to a queue and contains a job processing logic.\n\nThe fixed number of threads is used to process all jobs of all queues.\nAdding or removing queues or their shards won't affect the number of threads.\n\n## Sidekiq comparison\n\nIf Sidekiq is good for your tasks you should use it.\nBut if you use plugins like\n[sidekiq-grouping](https://github.com/gzigzigzeo/sidekiq-grouping),\n[sidekiq-unique-jobs](https://github.com/mhenrixon/sidekiq-unique-jobs),\n[sidekiq-merger](https://github.com/dtaniwaki/sidekiq-merger)\nor implement your own lock system, you should look at Lowkiq.\n\nFor example, sidekiq-grouping accumulates a batch of jobs then enqueues it and accumulates the next batch.\nWith this approach, a queue can contain two batches with data of the same order.\nThese batches are parallel processed with different threads, so we come back to the initial problem.\n\nLowkiq was designed to avoid any type of locking.\n\nFurthermore, Lowkiq's queues are reliable. Only Sidekiq Pro or plugins can add such functionality.\n\nThis [benchmark](examples/benchmark) shows overhead on Redis usage.\nThese are the results for 5 threads, 100,000 blank jobs:\n\n+ lowkiq: 155 sec or 1.55 ms per job\n+ lowkiq +hiredis: 80 sec or 0.80 ms per job\n+ sidekiq: 15 sec or 0.15 ms per job\n\nThis difference is related to different queues structure.\nSidekiq uses one list for all workers and fetches the job entirely for O(1).\nLowkiq uses several data structures, including sorted sets for keeping ids of jobs.\nSo fetching only an id of a job takes O(log(N)).\n\n## Queue\n\nPlease, look at [the presentation](https://docs.google.com/presentation/d/e/2PACX-1vRdwA2Ck22r26KV1DbY__XcYpj2FdlnR-2G05w1YULErnJLB_JL1itYbBC6_JbLSPOHwJ0nwvnIHH2A/pub?start=false\u0026loop=false\u0026delayms=3000).\n\nEvery job has the following attributes:\n\n+ `id` is a job identifier with string type.\n+ `payloads` is a sorted set of payloads ordered by its score. A payload is an object. A score is a real number.\n+ `perform_in` is planned execution time. It's a Unix timestamp with a real number type.\n+ `retry_count` is amount of retries. It's a real number.\n\nFor example, `id` can be an identifier of a replicated entity.\n`payloads` is a sorted set ordered by a score of payload and resulted by grouping a payload of the job by its `id`.\n`payload` can be a ruby object because it is serialized by `Marshal.dump`.\n`score` can be `payload`'s creation date (Unix timestamp) or it's an incremental version number.\nBy default, `score` and `perform_in` are current Unix timestamp.\n`retry_count` for new unprocessed job equals to `-1`,\nfor one-time failed is `0`, so the planned retries are counted, not the performed ones.\n\nJob execution can be unsuccessful. In this case, its `retry_count` is incremented, the new `perform_in` is calculated with determined formula, and it moves back to a queue.\n\nIn case of `retry_count` is getting `\u003e=` `max_retry_count` an element of `payloads` with less (oldest) score is moved to a morgue,\nrest elements are moved back to the queue, wherein `retry_count` and `perform_in` are reset to `-1` and `now()` respectively.\n\n### Calculation algorithm for `retry_count` and `perform_in`\n\n0. a job's been executed and failed\n1. `retry_count++`\n2. `perform_in = now + retry_in (try_count)`\n3. `if retry_count \u003e= max_retry_count` the job will be moved to a morgue.\n\n| type                      | `retry_count` | `perform_in`          |\n| ---                       | ---           | ---                   |\n| new haven't been executed | -1            | set or `now()`        |\n| new failed                | 0             | `now() + retry_in(0)` |\n| retry failed              | 1             | `now() + retry_in(1)` |\n\nIf `max_retry_count = 1`, retries stop.\n\n### Job merging rules\n\nThey are applied when:\n\n+ a job has been in a queue and a new one with the same id is added\n+ a job is failed, but a new one with the same id has been added\n+ a job from a morgue is moved back to a queue, but the queue has had a job with the same id\n\nAlgorithm:\n\n+ payloads are merged, the minimal score is chosen for equal payloads\n+ if a new job and queued job is merged, `perform_in` and `retry_count` is taken from the job from the queue\n+ if a failed job and queued job is merged, `perform_in` and `retry_count` is taken from the failed one\n+ if morgue job and queued job is merged, `perform_in = now()`, `retry_count = -1`\n\nExample:\n\n```\n# v1 is the first version and v2 is the second\n# #{\"v1\": 1} is a sorted set of a single element, the payload is \"v1\", the score is 1\n\n# a job is in a queue\n{ id: \"1\", payloads: #{\"v1\": 1, \"v2\": 2}, retry_count: 0, perform_in: 1536323288 }\n# a job which is being added\n{ id: \"1\", payloads: #{\"v2\": 3, \"v3\": 4}, retry_count: -1, perform_in: 1536323290 }\n\n# a resulted job in the queue\n{ id: \"1\", payloads: #{\"v1\": 1, \"v2\": 3, \"v3\": 4}, retry_count: 0, perform_in: 1536323288 }\n```\n\nA morgue is a part of a queue. Jobs in a morgue are not processed.\nA job in a morgue has the following attributes:\n\n+ id is the job identifier\n+ payloads\n\nA job from morgue can be moved back to the queue, `retry_count` = 0 and `perform_in = now()` would be set.\n\n## Install\n\n```\n# Gemfile\n\ngem 'lowkiq'\n```\n\nRedis \u003e= 3.2\n\n## Api\n\n```ruby\nmodule ATestWorker\n  extend Lowkiq::Worker\n\n  self.shards_count = 24\n  self.batch_size = 10\n  self.max_retry_count = 5\n\n  def self.retry_in(count)\n    10 * (count + 1) # (i.e. 10, 20, 30, 40, 50)\n  end\n\n  def self.retries_exhausted(batch)\n    batch.each do |job|\n      Rails.logger.info \"retries exhausted for #{name} with error #{job[:error]}\"\n    end\n  end\n\n  def self.perform(payloads_by_id)\n    # payloads_by_id is a hash map\n    payloads_by_id.each do |id, payloads|\n      # payloads are sorted by score, from old to new (min to max)\n      payloads.each do |payload|\n        do_some_work(id, payload)\n      end\n    end\n  end\nend\n```\n\nAnd then you have to add it to Lowkiq in your initializer file due to problems with autoloading:\n\n```ruby\nLowkiq.workers = [ ATestWorker ]\n```\n\nDefault values:\n\n```ruby\nself.shards_count = 5\nself.batch_size = 1\nself.max_retry_count = 25\nself.queue_name = self.name\n\n# i.e. 15, 16, 31, 96, 271, ... seconds + a random amount of time\ndef retry_in(retry_count)\n  (retry_count ** 4) + 15 + (rand(30) * (retry_count + 1))\nend\n```\n\n```ruby\nATestWorker.perform_async [\n  { id: 0 },\n  { id: 1, payload: { attr: 'v1' } },\n  { id: 2, payload: { attr: 'v1' }, score: Time.now.to_f, perform_in: Time.now.to_f },\n]\n# payload by default equals to \"\"\n# score and perform_in by default equals to Time.now.to_f\n```\n\nIt is possible to redefine `perform_async` and calculate `id`, `score` и `perform_in` in a worker code:\n\n```ruby\nmodule ATestWorker\n  extend Lowkiq::Worker\n\n  def self.perform_async(jobs)\n    jobs.each do |job|\n      job.merge! id: job[:payload][:id]\n    end\n    super\n  end\n\n  def self.perform(payloads_by_id)\n    #...\n  end\nend\n\nATestWorker.perform_async 1000.times.map { |id| { payload: {id: id} } }\n```\n\n## Ring app\n\n`Lowkiq::Web` - a ring app.\n\n+ `/` - a dashboard\n+ `/api/v1/stats` - queue length, morgue length, lag for each worker and total result\n\n## Configuration\n\nOptions and their default values are:\n\n+ `Lowkiq.workers = []`- list of workers to use. Since 1.1.0.\n+ `Lowkiq.poll_interval = 1` - delay in seconds between queue polling for new jobs.\n   Used only if a queue was empty in a previous cycle or an error occurred.\n+ `Lowkiq.threads_per_node = 5` - threads per node.\n+ `Lowkiq.redis = -\u003e() { Redis.new url: ENV.fetch('REDIS_URL') }` - redis connection options\n+ `Lowkiq.client_pool_size = 5` - redis pool size for queueing jobs\n+ `Lowkiq.pool_timeout = 5` - client and server redis pool timeout\n+ `Lowkiq.server_middlewares = []` - a middleware list, used when job is processed\n+ `Lowkiq.client_middlewares = []` - a middleware list, used when job is enqueued\n+ `Lowkiq.on_server_init = -\u003e() {}` - a lambda is being executed when server inits\n+ `Lowkiq.build_scheduler = -\u003e() { Lowkiq.build_lag_scheduler }` is a scheduler\n+ `Lowkiq.build_splitter = -\u003e() { Lowkiq.build_default_splitter }` is a splitter\n+ `Lowkiq.last_words = -\u003e(ex) {}` is an exception handler of descendants of `StandardError` caused the process stop\n+ `Lowkiq.dump_payload = Marshal.method :dump`\n+ `Lowkiq.load_payload = Marshal.method :load`\n\n+ `Lowkiq.format_error = -\u003e (error) { error.message }` can be used to add error backtrace. Please see [Extended error info](#extended-error-info)\n+ `Lowkiq.dump_error = -\u003e (msg) { msg }` can be used to implement a custom compression logic for errors. Recommended when using `Lowkiq.format_error`.\n+ `Lowkiq.load_error = -\u003e (msg) { msg }` can be used to implement a custom decompression logic for errors.\n\n```ruby\n$logger = Logger.new(STDOUT)\n\nLowkiq.server_middlewares \u003c\u003c -\u003e (worker, batch, \u0026block) do\n  $logger.info \"Started job for #{worker} #{batch}\"\n  block.call\n  $logger.info \"Finished job for #{worker} #{batch}\"\nend\n\nLowkiq.server_middlewares \u003c\u003c -\u003e (worker, batch, \u0026block) do\n  begin\n    block.call\n  rescue =\u003e e\n    $logger.error \"#{e.message} #{worker} #{batch}\"\n    raise e\n  end\nend\n\nLowkiq.client_middlewares \u003c\u003c -\u003e (worker, batch, \u0026block) do\n  $logger.info \"Enqueueing job for #{worker} #{batch}\"\n  block.call\n  $logger.info \"Enqueued job for #{worker} #{batch}\"\nend\n\n```\n\n## Performance\n\nUse [hiredis](https://github.com/redis/hiredis-rb) for better performance.\n\n```ruby\n# Gemfile\n\ngem \"hiredis\"\n```\n\n```ruby\n# config\n\nLowkiq.redis = -\u003e() { Redis.new url: ENV.fetch('REDIS_URL'), driver: :hiredis }\n```\n\n## Execution\n\n`lowkiq -r ./path_to_app`\n\n`path_to_app.rb` must load app. [Example](examples/dummy/lib/app.rb).\n\nThe lazy loading of worker modules is unacceptable.\nFor preliminarily loading modules use\n`require`\nor [`require_dependency`](https://api.rubyonrails.org/classes/ActiveSupport/Dependencies/Loadable.html#method-i-require_dependency)\nfor Ruby on Rails.\n\n## Shutdown\n\nSend TERM or INT signal to the process (Ctrl-C).\nThe process will wait for executed jobs to finish.\n\nNote that if a queue is empty, the process sleeps `poll_interval` seconds,\ntherefore, the process will not stop until the `poll_interval` seconds have passed.\n\n## Debug\n\nTo get trace of all threads of an app:\n\n```\nkill -TTIN \u003cpid\u003e\ncat /tmp/lowkiq_ttin.txt\n```\n\n## Development\n\n```\ndocker-compose run --rm --service-port app bash\nbundle\nrspec\ncd examples/dummy ; bundle exec ../../exe/lowkiq -r ./lib/app.rb\n\n# open localhost:8080\n```\n\n```\ndocker-compose run --rm --service-port frontend bash\nnpm run dumb\n# open localhost:8081\n\n# npm run build\n# npm run web-api\n```\n\n## Exceptions\n\n`StandardError` thrown by a worker are handled with middleware. Such exceptions don't lead to process stops.\n\nAll other exceptions cause the process to stop.\nLowkiq will wait for job execution by other threads.\n\n`StandardError` thrown outside of worker are passed to `Lowkiq.last_words`.\nFor example, it can happen when Redis connection is lost or when Lowkiq's code has a bug.\n\n## Rails integration\n\n```ruby\n# config/routes.rb\n\nRails.application.routes.draw do\n # ...\n mount Lowkiq::Web =\u003e '/lowkiq'\n # ...\nend\n```\n\n```ruby\n# config/initializers/lowkiq.rb\n\n# configuration:\n# Lowkiq.redis = -\u003e { Redis.new url: ENV.fetch('LOWKIQ_REDIS_URL') }\n# Lowkiq.threads_per_node = ENV.fetch('LOWKIQ_THREADS_PER_NODE').to_i\n# Lowkiq.client_pool_size = ENV.fetch('LOWKIQ_CLIENT_POOL_SIZE').to_i\n# ...\n\n# since 1.1.0\nLowkiq.workers = [\n  ATestWorker,\n  OtherCoolWorker\n]\n\nLowkiq.server_middlewares \u003c\u003c -\u003e (worker, batch, \u0026block) do\n  logger = Rails.logger\n  tag = \"#{worker}-#{Thread.current.object_id}\"\n\n  logger.tagged(tag) do\n    time_start = Time.now\n    logger.info \"#{time_start} Started job, batch: #{batch}\"\n    begin\n      block.call\n    rescue =\u003e e\n      logger.error e.message\n      raise e\n    ensure\n      time_end = Time.now\n      logger.info \"#{time_end} Finished job, duration: #{time_end - time_start} sec\"\n    end\n  end\nend\n\n# Sentry integration\nLowkiq.server_middlewares \u003c\u003c -\u003e (worker, batch, \u0026block) do\n  opts = {\n    extra: {\n      lowkiq: {\n        worker: worker.name,\n        batch: batch,\n      }\n    }\n  }\n\n  Raven.capture opts do\n    block.call\n  end\nend\n\n# NewRelic integration\nif defined? NewRelic\n  class NewRelicLowkiqMiddleware\n    include NewRelic::Agent::Instrumentation::ControllerInstrumentation\n\n    def call(worker, batch, \u0026block)\n      opts = {\n        category: 'OtherTransaction/LowkiqJob',\n        class_name: worker.name,\n        name: :perform,\n      }\n\n      perform_action_with_newrelic_trace opts do\n        block.call\n      end\n    end\n  end\n\n  Lowkiq.server_middlewares \u003c\u003c NewRelicLowkiqMiddleware.new\nend\n\n# Rails reloader, responsible for cleaning of ActiveRecord connections\nLowkiq.server_middlewares \u003c\u003c -\u003e (worker, batch, \u0026block) do\n  Rails.application.reloader.wrap do\n    block.call\n  end\nend\n\nLowkiq.on_server_init = -\u003e() do\n  [[ActiveRecord::Base, ActiveRecord::Base.configurations[Rails.env]]].each do |(klass, init_config)|\n    klass.connection_pool.disconnect!\n    config = init_config.merge 'pool' =\u003e Lowkiq.threads_per_node\n    klass.establish_connection(config)\n  end\nend\n```\n\nNote: In Rails 7, the worker files wouldn't be loaded by default in the initializers since they are managed by the `main` autoloader. To solve this, we can wrap setting the workers around the `to_prepare` configuration.\n\n```ruby\nRails.application.config.to_prepare do\n  Lowkiq.workers = [\n    ATestWorker,\n    OtherCoolWorker\n  ]\nend\n```\n\nExecution: `bundle exec lowkiq -r ./config/environment.rb`\n\n\n## Splitter\n\nEach worker has several shards:\n\n```\n# worker: shard ids\nworker A: 0, 1, 2\nworker B: 0, 1, 2, 3\nworker C: 0\nworker D: 0, 1\n```\n\nLowkiq uses a fixed number of threads for job processing, therefore it is necessary to distribute shards between threads.\nSplitter does it.\n\nTo define a set of shards, which is being processed by a thread, let's move them to one list:\n\n```\nA0, A1, A2, B0, B1, B2, B3, C0, D0, D1\n```\n\nDefault splitter evenly distributes shards by threads of a single node.\n\nIf `threads_per_node` is set to 3, the distribution will be:\n\n```\n# thread id: shards\nt0: A0, B0, B3, D1\nt1: A1, B1, C0\nt2: A2, B2, D0\n```\n\nBesides Default Lowkiq has the ByNode splitter. It allows dividing the load by several processes (nodes).\n\n```\nLowkiq.build_splitter = -\u003e () do\n  Lowkiq.build_by_node_splitter(\n    ENV.fetch('LOWKIQ_NUMBER_OF_NODES').to_i,\n    ENV.fetch('LOWKIQ_NODE_NUMBER').to_i\n  )\nend\n```\n\nSo, instead of a single process, you need to execute multiple ones and to set environment variables up:\n\n```\n# process 0\nLOWKIQ_NUMBER_OF_NODES=2 LOWKIQ_NODE_NUMBER=0 bundle exec lowkiq -r ./lib/app.rb\n\n# process 1\nLOWKIQ_NUMBER_OF_NODES=2 LOWKIQ_NODE_NUMBER=1 bundle exec lowkiq -r ./lib/app.rb\n```\n\nSummary amount of threads are equal product of `ENV.fetch('LOWKIQ_NUMBER_OF_NODES')` and `Lowkiq.threads_per_node`.\n\nYou can also write your own splitter if your app needs an extra distribution of shards between threads or nodes.\n\n## Scheduler\n\nEvery thread processes a set of shards. The scheduler selects shard for processing.\nEvery thread has its own instance of the scheduler.\n\nLowkiq has 2 schedulers for your choice.\n`Seq` sequentially looks over shards.\n`Lag`  chooses shard with the oldest job minimizing the lag. It's used by default.\n\nThe scheduler can be set up through settings:\n\n```\nLowkiq.build_scheduler = -\u003e() { Lowkiq.build_seq_scheduler }\n# or\nLowkiq.build_scheduler = -\u003e() { Lowkiq.build_lag_scheduler }\n```\n\n## Recommendations on configuration\n\n### `SomeWorker.shards_count`\n\nSum of `shards_count` of all workers shouldn't be less than `Lowkiq.threads_per_node`\notherwise, threads will stay idle.\n\nSum of `shards_count` of all workers can be equal to `Lowkiq.threads_per_node`.\nIn this case, a thread processes a single shard. This makes sense only with a uniform queue load.\n\nSum of `shards_count` of all workers can be more than `Lowkiq.threads_per_node`.\nIn this case, `shards_count` can be counted as a priority.\nThe larger it is, the more often the tasks of this queue will be processed.\n\nThere is no reason to set `shards_count` of one worker more than `Lowkiq.threads_per_node`,\nbecause every thread will handle more than one shard from this queue, so it increases the overhead.\n\n### `SomeWorker.max_retry_count`\n\nFrom `retry_in` and `max_retry_count`, you can calculate the approximate time that a payload of a job will be in a queue.\nAfter `max_retry_count` is reached a payload with a minimal score will be moved to a morgue.\n\nFor default `retry_in` we receive the following table.\n\n```ruby\ndef retry_in(retry_count)\n  (retry_count ** 4) + 15 + (rand(30) * (retry_count + 1))\nend\n```\n\n| `max_retry_count` | amount of days of job's life |\n| ---               | ---                          |\n| 14                | 1                            |\n| 16                | 2                            |\n| 18                | 3                            |\n| 19                | 5                            |\n| 20                | 6                            |\n| 21                | 8                            |\n| 22                | 10                           |\n| 23                | 13                           |\n| 24                | 16                           |\n| 25                | 20                           |\n\n`(0...25).map{ |c| retry_in c }.sum / 60 / 60 / 24`\n\n\n## Changing of worker's shards amount\n\nTry to count the number of shards right away and don't change it in the future.\n\nIf you can disable adding of new jobs, wait for queues to get empty, and deploy the new version of code with a changed amount of shards.\n\nIf you can't do it, follow the next steps:\n\nA worker example:\n\n```ruby\nmodule ATestWorker\n  extend Lowkiq::Worker\n\n  self.shards_count = 5\n\n  def self.perform(payloads_by_id)\n    some_code\n  end\nend\n```\n\nSet the number of shards and the new queue name:\n\n```ruby\nmodule ATestWorker\n  extend Lowkiq::Worker\n\n  self.shards_count = 10\n  self.queue_name = \"#{self.name}_V2\"\n\n  def self.perform(payloads_by_id)\n    some_code\n  end\nend\n```\n\nAdd a worker moving jobs from the old queue to the new one:\n\n```ruby\nmodule ATestMigrationWorker\n  extend Lowkiq::Worker\n\n  self.shards_count = 5\n  self.queue_name = \"ATestWorker\"\n\n  def self.perform(payloads_by_id)\n    jobs = payloads_by_id.each_with_object([]) do |(id, payloads), acc|\n      payloads.each do |payload|\n        acc \u003c\u003c { id: id, payload: payload }\n      end\n    end\n\n    ATestWorker.perform_async jobs\n  end\nend\n```\n\n## Extended error info\nFor failed jobs, lowkiq only stores `error.message` by default. This can be configured by using `Lowkiq.format_error` setting.\n`Lowkiq.dump` and `Lowkiq.load_error` can be used to compress and decompress the error messages respectively.\nExample:\n```ruby\nLowkiq.format_error = -\u003e (error) { error.full_message(highlight: false) }\n\nLowkiq.dump_error = Proc.new do |msg|\n  compressed = Zlib::Deflate.deflate(msg.to_s)\n  Base64.encode64(compressed)\nend\n\nLowkiq.load_error = Proc.new do |input|\n  decoded = Base64.decode64(input)\n  Zlib::Inflate.inflate(decoded)\nrescue\n  input\nend\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbia-technologies%2Flowkiq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbia-technologies%2Flowkiq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbia-technologies%2Flowkiq/lists"}