{"id":15562132,"url":"https://github.com/mrkamel/distributed_job","last_synced_at":"2026-02-08T14:03:02.099Z","repository":{"id":43483464,"uuid":"419382705","full_name":"mrkamel/distributed_job","owner":"mrkamel","description":"Keep track of distributed jobs spanning multiple workers using redis","archived":false,"fork":false,"pushed_at":"2023-02-03T15:52:03.000Z","size":54,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-06-21T17:18:33.564Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mrkamel.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-10-20T15:15:21.000Z","updated_at":"2021-11-03T16:25:17.000Z","dependencies_parsed_at":"2023-02-18T08:45:20.045Z","dependency_job_id":null,"html_url":"https://github.com/mrkamel/distributed_job","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/mrkamel/distributed_job","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrkamel%2Fdistributed_job","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrkamel%2Fdistributed_job/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrkamel%2Fdistributed_job/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrkamel%2Fdistributed_job/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mrkamel","download_url":"https://codeload.github.com/mrkamel/distributed_job/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrkamel%2Fdistributed_job/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267368943,"owners_count":24076098,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-27T02:00:11.917Z","response_time":82,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-02T16:11:55.028Z","updated_at":"2026-02-08T14:02:57.064Z","avatar_url":"https://github.com/mrkamel.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DistributedJob\n\n[![Build](https://github.com/mrkamel/distributed_job/workflows/test/badge.svg)](https://github.com/mrkamel/distributed_job/actions?query=workflow%3Atest+branch%3Amaster)\n[![Gem Version](https://badge.fury.io/rb/distributed_job.svg)](http://badge.fury.io/rb/distributed_job)\n\nEasily keep track of distributed jobs consisting of an arbitrary number of\nparts spanning multiple workers using redis. Can be used with any kind of\nbackround job processing queue.\n\n## Installation\n\nAdd this line to your application's Gemfile:\n\n```ruby\ngem 'distributed_job'\n```\n\nAnd then execute:\n\n    $ bundle install\n\nOr install it yourself as:\n\n    $ gem install distributed_job\n\n## Usage\n\nGetting started is very easy. A `DistributedJob` allows to keep track of a\ndistributed job, i.e. a job which is split into multiple units running in\nparallel and in multiple workers.\n\nFirst, create a `DistributedJob::Client`:\n\n```ruby\n  DistributedJobClient = DistributedJob::Client.new(redis: Redis.new)\n```\n\nYou can specify a `namespace` to be additionally used for redis keys and set a\n`default_ttl` for keys (Default is `86_400`, i.e. one day), which will be used\nevery time when keys in redis are updated to guarantee that the distributed\njob metadata is cleaned up properly from redis at some point in time.\n\nAfterwards, you have two options to add parts, i.e. units of work, to the\ndistributed job. The first option is to use `#push_all` and pass an enum:\n\n```ruby\n  distributed_job = DistributedJobClient.build(token: SecureRandom.hex)\n  distributed_job.push_all([\"job1\", \"job2\", \"job3\"])\n\n  Job1.perform_async(distributed_job.token)\n  Job2.perform_async(distributed_job.token)\n  Job3.perform_async(distributed_job.token)\n\n  distributed_job.token # can be used to query the status of the distributed job\n```\n\nHere, 3 parts named `job1`, `job2` and `job3` are added to the distributed job\nand then 3 corresponding background jobs are enqueued. It is important to push\nthe parts before the background jobs are enqueued. Otherwise the background\njobs maybe can't find them. The `token` must be passed to the background jobs,\nsuch that the background job can update the status of the distributed job by\nmarking the respective part as done. The token can also be used to query the\nstatus of the distributed job, e.g. on a job summary page or similar. You can\nalso show some progress bar in the browser or in the terminal, etc.\n\n```ruby\n# token is given via URL or via some other means\ndistributed_job = DistributedJobClient.build(token: params[:token])\n\ndistributed_job.total # total number of parts\ndistributed_job.count # number of unfinished parts\ndistributed_job.finished? # whether or not all parts are finished\ndistributed_job.open_parts # returns all not yet finished part id's\n\ndistributed_job.done('job1') # marks the respective part as done\n```\n\nThe second option is to use `#push_each`:\n\n```ruby\n  distributed_job = DistributedJobClient.build(token: SecureRandom.hex)\n\n  distributed_job.push_each(Date.parse('2021-01-01')..Date.today) do |date, part|\n    SomeBackgroundJob.perform_async(date, distributed_job.token, part)\n  end\n\n  distributed_job.token # again, can be used to query the status of the distributed job\n```\n\nHere, the part name is automatically generated to be some id and passed as\n`part` to the block. The part must also be passed to the respective background\njob for it be able to mark the part as finished after it has been successfully\nprocessed. Therefore, when all those background jobs have successfully\nfinished, all parts will be marked as finished, such that the distributed job\nwill finally be finished as well.\n\nWithin the background job, you must use the passed `token` and `part` to query\nand update the status of the distributed job and part accordingly. Please note\nthat you can use whatever background job processing tool you like most.\n\n```ruby\nclass SomeBackgroundJob\n  def perform(whatever, token, part)\n    distributed_job = DistributedJobClient.build(redis: Redis.new, token: token)\n\n    return if distributed_job.stopped?\n\n    # ...\n\n    if distributed_job.done(part)\n      # perform e.g. cleanup or the some other job\n    end\n  rescue\n    distributed_job.stop\n\n    raise\n  end\nend\n```\n\nThe `#stop` and `#stopped?` methods can be used to globally stop a distributed\njob in case of errors. Contrary, the `#done` method tells the distributed job\nthat the specified part has successfully finished. The `#done` method returns\ntrue when all parts of the distributed job have finished, which is useful to\nstart cleanup jobs or to even start another subsequent distributed job.\n\nThat's it.\n\n## Reference docs\n\nPlease find the reference docs at\n[http://www.rubydoc.info/github/mrkamel/distributed_job](http://www.rubydoc.info/github/mrkamel/distributed_job)\n\n## Development\n\nAfter checking out the repo, run `bin/setup` to install dependencies. Then, run\n`bundle exec rspec` to run the tests. You can also run `bin/console` for an\ninteractive prompt that will allow you to experiment.\n\n## Contributing\n\nBug reports and pull requests are welcome on GitHub at\nhttps://github.com/mrkamel/distributed_job.\n\n## License\n\nThe gem is available as open source under the terms of the [MIT\nLicense](https://opensource.org/licenses/MIT).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmrkamel%2Fdistributed_job","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmrkamel%2Fdistributed_job","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmrkamel%2Fdistributed_job/lists"}