{"id":13878258,"url":"https://github.com/doximity/simplekiq","last_synced_at":"2025-07-18T04:05:38.382Z","repository":{"id":39655355,"uuid":"447332773","full_name":"doximity/simplekiq","owner":"doximity","description":"Sidekiq-based workflow orchestration library","archived":false,"fork":false,"pushed_at":"2023-10-24T09:59:02.000Z","size":6055,"stargazers_count":95,"open_issues_count":12,"forks_count":7,"subscribers_count":59,"default_branch":"master","last_synced_at":"2025-06-27T11:59:48.681Z","etag":null,"topics":["sidekiq"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/doximity.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-01-12T18:44:19.000Z","updated_at":"2025-06-14T09:01:32.000Z","dependencies_parsed_at":"2024-11-13T16:48:08.011Z","dependency_job_id":null,"html_url":"https://github.com/doximity/simplekiq","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/doximity/simplekiq","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doximity%2Fsimplekiq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doximity%2Fsimplekiq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doximity%2Fsimplekiq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doximity%2Fsimplekiq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/doximity","download_url":"https://codeload.github.com/doximity/simplekiq/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doximity%2Fsimplekiq/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265698011,"owners_count":23813123,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["sidekiq"],"created_at":"2024-08-06T08:01:44.302Z","updated_at":"2025-07-18T04:05:38.361Z","avatar_url":"https://github.com/doximity.png","language":"Ruby","funding_links":[],"categories":["Ruby"],"sub_categories":[],"readme":"# Simplekiq\n\nAny time that you find yourself needing to string together a long chain of jobs, particularly when there are multiple stages of Sidekiq-pro batches and callbacks involved, come home instead to the simple flavor of orchestrated job flow with Simplekiq.\n\n## Installation\n\nAdd this line to your application's Gemfile:\n\n```ruby\ngem \"simplekiq\"\n```\n\nNote that this gem requires you be a Sidekiq Pro paid subscriber to be able to use it, so after following the installation docs for getting the private gem configured with your system, ensure you have `sidekiq-pro` at version `~\u003e 5.0.0` or higher (need at least version `5.2.1` if you want to capture `on_death` callbacks [percolating up to parent batches](https://github.com/mperham/sidekiq/blob/main/Pro-Changes.md#521) - a supported feature which is not required for typical orchestration behavior) and that it's being required:\n\n```ruby\ngem \"sidekiq-pro\", \"~\u003e 5.0.0\"\n```\n\nAnd then execute:\n\n    $ bundle install\n\nOr install it yourself as:\n\n    $ gem install simplekiq\n\n## Usage\n\nThere are currently two primary components of the system which were designed to work in harmony:\n\n* [Simplekiq::OrchestrationJob](./lib/simplekiq/orchestration_job.rb) - A mixin for a Sidekiq jobs to be able to orchestrate a flow of jobs in one place. It makes long complicated flows between jobs easier to understand, iterate on, and test. It eliminates the need to hop between dozens of files to determine when, where, and why a particular job gets called.\n* [Simplekiq::BatchingJob](./lib/simplekiq/batching_job.rb) - A mixin designed to make breaking a large job into a batched process dead simple and contained within a single class while still being trivially composable in orchestrations.\n\n## Tool Drilldown\n\n### Simplekiq::OrchestrationJob\n\nMixing in the [Simplekiq::Orchestration](./lib/simplekiq/orchestration_job.rb) module lets you define a human-readable workflow of jobs in a single file with almost* no special requirements or restrictions on how the child jobs are designed. In most cases, Sidekiq jobs not designed for use in orchestrations should be compatible for use in orchestrations. A job implementing `OrchestrationJob` might look like:\n\n```ruby\nclass SomeOrchestrationJob\n  include Simplekiq::OrchestrationJob\n\n  def perform_orchestration(some_id)\n    @some_model = SomeModel.find(some_id) # 1.\n\n    run SomeInitialSetupJob, some_model.id # 2.\n\n    in_parallel do\n      some_related_models.each do |related_model|\n        run SomeParallelizableJob, related_model.id # 3.\n      end\n    end\n\n    run SomeFinalizationJob, some_model.id # 4.\n  end\n\n  def on_death(status, options) # 5.\n    SomeModel.find(options[\"args\"].first).failure_happened!\n  end\n\n  def on_complete(status, options) # 6.\n    failures = Array(status\u0026.failure_info) # sidekiq-pro batch status api\n    return if failures.empty?\n\n    SomeModel.find(options[\"args\"].first).it_was_these_failures(failures)\n  end\n\n  private\n\n  attr_reader :some_model\n\n  def some_related_models\n    @some_related_models ||= some_model.some_relation\n  end\nend\n```\n\nLet's use the above example to describe some specifics of how the flow works.\n\n1. `SomeOrchestrationJob` pulls up some instance of parent model `SomeModel`.\n2. It does some initial work in `SomeInitialSetupJob`, which blocks the rest of the workflow until it completes successfully.\n3. Then it will run a `SomeParallelizableJob` for each of some number of associated models `some_related_models`. These jobs will all run parallel to each other independently.\n4. Finally, after all of the parallel jobs from #3 complete successfully, `SomeFinalizationJob` will run and then after it finishes the orchestration will be complete.\n5. If it ran into an error at some point, `on_death` will get fired with the first failure. (please use `sidekiq-pro` of at least `5.2.1` for this feature)\n6. It will call `on_complete` at the end of the orchestration no matter what, this is the place to collect all the failures and persist them somewhere.\n\n**Note** - it's fine to add utility methods and `attr_accessor`s to keep the code tidy and maintainable.\n\nWhen `SomeOrchestrationJob` itself gets called though, the first thing it does it turn these directives into a big serialized structure indicating which job will be called under what conditions (eg, serial or in parallel) and with what arguments, and then keeps passing that between the simplekiq-internal jobs that actually conduct the flow.\n\nThis means when you want to deploy a change to this flow all previous in-flight workflows will continue undisturbed because the workflow is frozen in sidekiq job arguments and will remain frozen until the workflow completes. This is generally a boon, but note that if you remove a job from a workflow you'll need to remember to either keep the job itself (eg, the `SomeFinalizationJob` class file from our above example) in the codebase or replace it with a stub so that any in-flight workflows won't crash due to not being able to pull up the prior-specified workflow.\n\n\"almost* no special requirements or restrictions on how the child jobs are designed\" - The one thing you'll want to keep in mind when feeding arbitrary jobs into orchestrations is that if the job creates any new sidekiq batches then those new sidekiq batches should be added as child sidekiq batches of the parent sidekiq batch of the job. The parent sidekiq batch of the job is the sidekiq batch that drives the orchestration from step to step, so if you don't do this it will move onto the next step in the orchestration once your job finishes even if the new sidekiq batches it started didn't finish. This sounds more complicated than it is, you can see an example of code that does this in [`BatchingJob#perform`](./lib/simplekiq/batching_job.rb):\n\n```ruby\nif batch # is there a parent batch?\n  batch.jobs do # open the parent batch back up\n    create_a_new_batch_and_add_jobs_to_it_to_run # make our new batch as a child batch of the parent batch\n  end # close the parent batch again\nelse # there's no parent batches, this job was run directly outside of an orchestration\n  create_a_new_batch_and_add_jobs_to_it_to_run # make our new batch without a parent batch\nend\n```\n\n### Simplekiq::BatchingJob\n\nSee the [Simplekiq::BatchingJob](./lib/simplekiq/batching_job.rb) module itself for a description and example usage in the header comments. Nutshell is that you should use this if you're planning on making a batched asynchronous process as it shaves off a lot of ceremony and unexpressive structure. eg - Instead of having `BeerBottlerJob` which queues some number of `BeerBottlerBatchJob`s to handle the broken down sub-tasks you can just have `BeerBottlerJob` with a method for batching, executing individual batches, and a callback that gets run after all batches have completed successfully.\n\n## History\n\nSimplekiq was initially released for private use within Doximity applications in Oct 2020 where it continued to be iterated on towards stability and general use until Jan 2022 when it was deemed settled enough for public release.\n\nThe primary driving factor that inspired this work was a series of over a dozen differently defined and structured jobs part of a single workflow of which the logical flow was extraordinarily difficult to cognitively trace. This led to exteme difficulty in debugging and following problematic instances of the workflow in production as well as needlessly high cost to refactoring and iterative adjustments.\n\nThe crux of the problem was that each job was highly coupled to its position in the overall flow as well as the absence of any central mechanism to indicate what the overall flow was. After building Simplekiq and implementing it into the flow, significant changes to the flow became quick adjustments requiring only a couple lines of code to change and folks unfamiliar with the system could quickly get up to speed by reading through the orchestration job.\n\n## Versioning\n\nThis project follows semantic versioning. See https://semver.org/ for details.\n\n## Development\n\nAfter checking out the repo, run `bin/setup` to install dependencies. Note that this depends on `sidekiq-pro` which requires a [commercial license](https://sidekiq.org/products/pro.html) to install and use.\n\nThen, run `rake ci:specs` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.\n\nTo install this gem onto your local machine, run `bundle exec rake install`.\n\nTo get a new release cut, please either open a PR or an issue with your ask with as much context as possible and someone from Doximity will consider your request. If it makes sense for the direction of the project we'll get it done and inform you of when a release has been made available with the changes.\n\nFor internal employees: consult the company wiki on the current standard process for conducting releases for our public gems.\n\n## Contributing\n\n1. See [CONTRIBUTING.md](./CONTRIBUTING.md)\n2. Fork it\n3. Create your feature branch (`git checkout -b my-new-feature`)\n4. Commit your changes (`git commit -am 'Add some feature'`)\n5. Push to the branch (`git push origin my-new-feature`)\n6. Create a new Pull Request\n\n## License\n\nThe gem is licensed under an Apache 2 license. Contributors are required to sign an contributor license agreement. See LICENSE.txt and CONTRIBUTING.md for more information.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdoximity%2Fsimplekiq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdoximity%2Fsimplekiq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdoximity%2Fsimplekiq/lists"}