{"id":13747487,"url":"https://github.com/socialpandas/sidekiq-superworker","last_synced_at":"2025-05-16T03:05:33.033Z","repository":{"id":8797671,"uuid":"10490955","full_name":"socialpandas/sidekiq-superworker","owner":"socialpandas","description":"Directed acyclic graphs of Sidekiq jobs","archived":false,"fork":false,"pushed_at":"2017-10-04T18:46:12.000Z","size":152,"stargazers_count":439,"open_issues_count":20,"forks_count":33,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-04-19T04:33:52.437Z","etag":null,"topics":["background-jobs","dependency-graph","directed-graph","ruby","sidekiq","sidekiq-monitor","sidekiq-superworker"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/socialpandas.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"MIT-LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-06-04T23:30:29.000Z","updated_at":"2025-03-21T20:49:57.000Z","dependencies_parsed_at":"2022-08-30T09:40:51.991Z","dependency_job_id":null,"html_url":"https://github.com/socialpandas/sidekiq-superworker","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/socialpandas%2Fsidekiq-superworker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/socialpandas%2Fsidekiq-superworker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/socialpandas%2Fsidekiq-superworker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/socialpandas%2Fsidekiq-superworker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/socialpandas","download_url":"https://codeload.github.com/socialpandas/sidekiq-superworker/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254090391,"owners_count":22013099,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["background-jobs","dependency-graph","directed-graph","ruby","sidekiq","sidekiq-monitor","sidekiq-superworker"],"created_at":"2024-08-03T06:01:30.910Z","updated_at":"2025-05-16T03:05:28.026Z","avatar_url":"https://github.com/socialpandas.png","language":"Ruby","funding_links":[],"categories":["Ruby","ruby"],"sub_categories":[],"readme":"Sidekiq Superworker\n===================\nDefine dependency graphs of Sidekiq jobs\n\nOverview\n--------\n\nSidekiq Superworker lets you create superworkers, which are simple or complex graphs of Sidekiq workers.\n\nFor example, you can define complex graphs of workers and use both serial and parallel worker configurations:\n\n[![](https://raw.github.com/socialpandas/sidekiq-superworker/master/doc/diagram-complex.png)](https://raw.github.com/socialpandas/sidekiq-superworker/master/doc/diagram-complex.png)\n\n*(Worker10 will run after Worker5, Worker7, Worker8, and Worker9 have all completed.)*\n\n```ruby\nSuperworker.define(:MySuperworker, :user_id, :comment_id) do\n  Worker1 :user_id\n  Worker2 :user_id do\n    parallel do\n      Worker3 :comment_id do\n        Worker4 :comment_id\n        Worker5 :comment_id\n      end\n      Worker6 :user_id do\n        parallel do\n          Worker7 :user_id\n          Worker8 :user_id\n          Worker9 :user_id\n        end\n      end\n    end\n    Worker10 :comment_id\n  end\nend\n```\n\nAnd you can run it like any other worker:\n\n```ruby\nMySuperworker.perform_async(23, 852)\n```\n\nYou can also define simple serial sequences of workers:\n\n[![](https://raw.github.com/socialpandas/sidekiq-superworker/master/doc/diagram-simple.png)](https://raw.github.com/socialpandas/sidekiq-superworker/master/doc/diagram-simple.png)\n\n```ruby\nSuperworker.define(:MySuperworker, :user_id, :comment_id) do\n  Worker1 :user_id, :comment_id\n  Worker2 :comment_id\n  Worker3 :user_id\nend\n```\n\nInstallation\n------------\n\nInclude it in your Gemfile:\n\n    gem 'sidekiq-superworker'\n\nUsage\n-----\n\nFirst, define a superworker in a file that's included during the initialization of the app. If you're using Rails, you might do this in an initializer:\n\n```ruby\n# config/initializers/superworkers.rb\nDir['./app/superworkers/*'].each { |f| require f }\n\n# app/superworkers/my_superworker.rb\nSuperworker.define(:MySuperworker, :user_id, :comment_id) do\n  Worker1 :user_id, :comment_id\n  Worker2 :comment_id\nend\n\n# app/superworkers/my_other_superworker.rb\nSuperworker.define(:MyOtherSuperworker, :comment_id) do\n  Worker2 :comment_id\n  Worker3 :comment_id\nend\n```\n\nTo run a superworker, call perform_async:\n\n```ruby\nMySuperworker.perform_async(23, 852)\n```\n\n### Arguments\n\nYou can define any number of arguments for the superworker and pass them to different subworkers as you see fit:\n\n```ruby\nSuperworker.define(:MySuperworker, :user_id, :comment_id) do\n  Worker1 :user_id, :comment_id\n  Worker2 :comment_id\n  Worker3 :user_id\nend\n```\n\nIf you want to set any static arguments for the subworkers, you can do that by using any values that are not symbols (e.g. strings, integers, etc):\n\n```ruby\nSuperworker.define(:MySuperworker, :user_id, :comment_id) do\n  Worker1 100, :user_id, :comment_id\n  Worker2 'all'\nend\n```\n\nIf a subworker doesn't take any arguments, you'll need to include parentheses after it:\n\n```ruby\nSuperworker.define(:MySuperworker, :user_id, :comment_id) do\n  Worker1 :user_id, :comment_id\n  Worker2()\nend\n```\n\n### Namespaced Workers\n\nTo refer to a namespaced worker (e.g. `MyModule::Worker1`), replace the two colons with two underscores:\n\n```ruby\nSuperworker.define(:MySuperworker, :user_id, :comment_id) do\n  MyModule__Worker1 :user_id, :comment_id\nend\n```\n\n### Options\n\n#### Delete subjobs after their superjob completes\n\nWhen a superjob is queued, records for all of its subjobs are created. By default, these records are deleted after the superjob completes. This can be changed by setting the following option to false:\n\n```ruby\n# config/initializers/superworker.rb\nSidekiq::Superworker.options[:delete_subjobs_after_superjob_completes] = false\n```\n\n#### Superjob expiration\n\nWhen a subjob dies due to too many retries, the jobs that depend on it will never run and the superjob will never be completed. Consequently, the subjob Redis keys will never be removed. You can set `superjob_expiration` to *N* to make the subjobs keys expire in *N* seconds. The default value is `nil` (the keys will never expire).\n\n```ruby\n# config/initializers/superworker.rb\nSidekiq::Superworker.options[:superjob_expiration] = 2592000 # 1 Month\n```\n\n### Logging\n\nTo make debugging easier, Sidekiq Superworker provides detailed log messages when its logger is set to the DEBUG level:\n\n```ruby\n# config/initializers/superworker.rb\nlogger = Logger.new(Rails.root.join('log', 'superworker.log'))\nlogger.level = Logger::DEBUG\nSidekiq::Superworker::Logging.logger = logger\n```\n\n### Monitoring\n\nUsing [sidekiq_monitor](https://github.com/socialpandas/sidekiq_monitor) with Sidekiq Superworker is encouraged, as it lets you easily monitor when a superjob is running, when it has finished, whether it has encountered errors, and the status of all of its subjobs.\n\n### Batch Jobs\n\nBy using a `batch` block, you can create batches of subjobs that are all associated with the superjob. The following will run Worker1 and Worker2 in serial for every user ID in the array passed to perform_async.\n\n```ruby\nSuperworker.define(:MyBatchSuperworker, :user_ids) do\n  batch user_ids: :user_id do\n    Worker1 :user_id\n    Worker2 :user_id\n  end\nend\n\nMyBatchSuperworker.perform_async([30, 31, 32, 33, 34, 35])\n```\n\nYou can also use multiple arguments:\n\n```ruby\nSuperworker.define(:MyBatchSuperworker, :user_ids, :comment_ids) do\n  batch user_ids: :user_id, comment_ids: :comment_id do\n    Worker1 :user_id, :comment_id\n    Worker2 :user_id\n  end\nend\n\nMyBatchSuperworker.perform_async([10, 11, 12], [20, 21, 22])\n```\n\nThe above produces a sequence equivalent to this (the workers run serially):\n\n```ruby\nWorker1.new.perform(10, 20)\nWorker2.new.perform(10)\nWorker1.new.perform(11, 21)\nWorker2.new.perform(11)\nWorker1.new.perform(12, 22)\nWorker2.new.perform(12)\n```\n\nGrouping jobs into batches greatly improves your ability to audit them and determine when batches have finished.\n\n### Superjob Names\n\nIf you're using sidekiq_monitor and want to set a name for a superjob, you can set it in an additional argument, like so:\n\n```ruby\n# Unnamed\nMySuperworker.perform_async(23)\n\n# Named\nMySuperworker.perform_async(23, name: 'My job name')\n```\n\n### Errors\n\nIf a subjob encounters an exception, the subjobs that depend on it won't run, but the rest of the subjobs will continue as usual.\n\nIf sidekiq_monitor is being used, the exception will be bubbled up to the superjob, which lets you easily see when your superjobs have failed.\n\nUpgrade Notes\n-------------\n\n### Upgrading from 1.0.x to 1.1.x while using Sidekiq Monitor\n\nIf you're using Sidekiq Monitor, were previously using Sidekiq Superworker 1.0.x, and are upgrading to 1.1.x, you should be aware that the strategy for storing subjob IDs has changed. Before upgrading, you should let all of your superworkers finish, then upgrade, then resume running your superworkers. For superjobs that ran before the upgrade, the relationship between superjobs and subjobs will no longer be shown in some parts of the UI. If you're not using Sidekiq Monitor, you can upgrade without any interruption.\n\n### Upgrading from 0.x to 1.x\n\nIf you were previously using Sidekiq Superworker 0.x and are upgrading to 1.x, there are some changes to be aware of:\n\n#### Redis replaced ActiveRecord\n\nActiveRecord was used as the datastore in 0.x due to application-specific requirements, but Redis is a far better choice for many reasons, especially given that Sidekiq uses Redis. When upgrading to 1.x, you'll need to let all of your superjobs complete, then upgrade to 1.x, then resume running superjobs. You can drop the 'sidekiq_superworker_subjobs' table, if you like.\n\n#### Superworker.define replaced Superworker.create\n\nThe name of the `Superworker.create` method caused confusion, as some users would call it multiple times. Since it defines a class, it's been renamed to `Superworker.define`. You'll need to replace it accordingly.\n\nTesting\n-------\n\nSidekiq Superworker is tested against multiple sets of gem dependencies (currently: no gems, Rails 3, and Rails 4), so please run the tests with [Appraisal](https://github.com/thoughtbot/appraisal) before submitting a PR. Thanks!\n\n```bash\nappraisal rspec\n```\n\nLicense\n-------\n\nSidekiq Superworker is released under the MIT License. Please see the MIT-LICENSE file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsocialpandas%2Fsidekiq-superworker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsocialpandas%2Fsidekiq-superworker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsocialpandas%2Fsidekiq-superworker/lists"}