{"id":13879775,"url":"https://github.com/reidmorrison/rocketjob","last_synced_at":"2025-04-06T16:14:39.765Z","repository":{"id":34900665,"uuid":"38934038","full_name":"reidmorrison/rocketjob","owner":"reidmorrison","description":"Ruby's missing background and batch processing system","archived":false,"fork":false,"pushed_at":"2023-11-10T11:38:52.000Z","size":5771,"stargazers_count":289,"open_issues_count":0,"forks_count":23,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-03-30T15:08:20.695Z","etag":null,"topics":["batch","batch-file","batch-framework","batch-job","batch-processing","compression","delayed-jobs","encryption","mongodb","mongoid","resque","sidekiq"],"latest_commit_sha":null,"homepage":"http://rocketjob.io/","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/reidmorrison.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":"docs/support.md","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-07-11T17:17:13.000Z","updated_at":"2024-11-25T16:20:39.000Z","dependencies_parsed_at":"2023-11-09T01:44:52.284Z","dependency_job_id":"7ba7dcba-e068-4e42-b6ed-97e5f42c8ea1","html_url":"https://github.com/reidmorrison/rocketjob","commit_stats":null,"previous_names":[],"tags_count":80,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reidmorrison%2Frocketjob","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reidmorrison%2Frocketjob/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reidmorrison%2Frocketjob/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reidmorrison%2Frocketjob/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/reidmorrison","download_url":"https://codeload.github.com/reidmorrison/rocketjob/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247509237,"owners_count":20950232,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["batch","batch-file","batch-framework","batch-job","batch-processing","compression","delayed-jobs","encryption","mongodb","mongoid","resque","sidekiq"],"created_at":"2024-08-06T08:02:32.652Z","updated_at":"2025-04-06T16:14:39.744Z","avatar_url":"https://github.com/reidmorrison.png","language":"Ruby","readme":"# Rocket Job\n[![Gem Version](https://img.shields.io/gem/v/rocketjob.svg)](https://rubygems.org/gems/rocketjob) [![Downloads](https://img.shields.io/gem/dt/rocketjob.svg)](https://rubygems.org/gems/rocketjob) [![License](https://img.shields.io/badge/license-Apache%202.0-brightgreen.svg)](http://opensource.org/licenses/Apache-2.0) ![](https://img.shields.io/badge/status-Production%20Ready-blue.svg) \n\nRuby's missing batch system\n\nCheckout https://rocketjob.io/\n\n![Rocket Job](https://rocketjob.io/images/rocket/rocket-icon-512x512.png)\n\n## Documentation\n\n* [Guide](http://rocketjob.io/)\n* [API Reference](http://www.rubydoc.info/gems/rocketjob/)\n\n## Support\n\n* Ask questions in [Rocket Job Discussions](https://github.com/reidmorrison/rocketjob/discussions)\n* [Report bugs](https://github.com/rocketjob/rocketjob/issues)\n\n## Rocket Job v6\n\n- Support for Ruby v3 and Rails 6.\n- Major enhancements in Batch job support:\n    - Direct built-in Tabular support for all input and output categories.\n    - Multiple output file support, each with its own settings for:\n        - Compression\n            - GZip, Zip, BZip2 (Chunked for much faster loading into Apache Spark).\n        - Encryption\n            - PGP, Symmetric Encryption.\n        - File format\n            - CSV, PSV, JSON, Fixed Format, xlsx.\n- Significant error handling improvements, especially around throttle failures\n  that used to result in \"hanging\" jobs.\n- Support AWS DocumentDB in addition to MongoDB as the data store.\n- Removed use of Symbols to meet Symbol deprecation in MongoDB and Mongoid.\n\n### Upgrading to Rocket Job v6\n\nThe following plugins have been deprecated and are no longer loaded by default.\n- `RocketJob::Batch::Tabular::Input`\n- `RocketJob::Batch::Tabular::Output`\n\nIf your code relies on these plugins and you still want to upgrade to Rocket Job v6,\nadd the following require statement to any jobs that still use them:\n\n~~~ruby\nrequire \"rocket_job/batch/tabular\"\n~~~\n\nIt is important to migrate away from these plugins, since they will be removed in a future release.\n\n#### Scheduled Jobs\n\nFor any scheduled jobs that include the `RocketJob::Plugins::Cron` plugin, the default behavior has changed\nso that the scheduled job instance is created immediately after the currently scheduled instance starts.\n\nTo maintain the old behavior of creating the job when it fails, aborts, or completes, add the following line\nto each of the applicable jobs:\n\n~~~ruby\nself.cron_after_start = false\n~~~\n\nAdditionally, scheduled jobs will now prevent a new one from being created when another scheduled instance \nof the same job is already queued, or running with the _same_ `cron_schedule`.\n\nTo maintain the old behavior of allowing multiple instances with the same cron schedule, add the following\nline to each of the applicable jobs:\n\n~~~ruby\nself.cron_singleton = false\n~~~\n\n##### Singleton\n\nSince Scheduled jobs now implement their own singleton logic, remove the singleton plugin from any scheduled jobs.\n\n#### Upgrading Batch Jobs to Rocket Job v6\n\nRocket Job v6 replaces the array of symbol type for `input_categories` and `output_categories`\nwith an array of `RocketJob::Category::Input` and `RocketJob::Category::Output`.\n\nJobs that added or modified the input or output categories need to be upgraded. For example:\n~~~ruby\nclass MyJob \u003c RocketJob::Job\n  include RocketJob::Batch\n  \n  self.output_categories = [:main, :errors, :ignored]\nend\n~~~\n\nNeeds to be changed to:\n~~~ruby\nclass MyJob \u003c RocketJob::Job\n  include RocketJob::Batch\n  \n  output_category name: :main\n  output_category name: :errors\n  output_category name: :ignored\nend\n~~~\n\n##### slice_size, encrypt, compress\n\nThese fields have been removed from the job itself:\n~~~ruby\nclass MyJob \u003c RocketJob::Job\n  include RocketJob::Batch\n\n  self.slice_size = 1_000\n  self.encrypt    = true\n  self.compress   = true\nend\n~~~\n\nThey are now specified on the `input_category` as follows:\n- `slice_size` just moves under `input_category`. \n- `encrypt` becomes an option to `serializer`.\n- `compress` is now the default for all batch jobs so is not needed.\n\nIf the serializer is set to `encrypt` then it is automatically compressed.\n\n~~~ruby\nclass MyJob \u003c RocketJob::Job\n  include RocketJob::Batch\n\n  input_category slice_size: 1_000, serializer: :encrypt\nend\n~~~\n\n##### collect_output, collect_nil_output\n\nThe following fields have been moved from the job itself:\n~~~ruby\nclass MyJob \u003c RocketJob::Job\n  include RocketJob::Batch\n  \n  self.collect_output     = true\n  self.collect_nil_output = true\nend\n~~~\n\nInto the corresponding `output_category`:\n- `collect_output` no longer has any meaning. Output is collected anytime an `output_category` is defined.\n- `collect_nil_output` is now the option `nils` on the `output_category. \n  It defaults to `false` so that by default any `nil` output from the `perform` method is not collected. \n~~~ruby\nclass MyJob \u003c RocketJob::Job\n  include RocketJob::Batch\n\n  output_category nils: true\nend\n~~~\n\n##### name\n\nFor both `input_category` and `output_category`, when the `name` argument is not supplied\nit defaults to `:main`.\n\nFor Example:\n~~~ruby\nclass MyJob \u003c RocketJob::Job\n  include RocketJob::Batch\n\n  input_category name: :main, serializer: :encrypt\n  output_category name: :main\nend\n~~~\n\nIs the same as:\n~~~ruby\nclass MyJob \u003c RocketJob::Job\n  include RocketJob::Batch\n\n  input_category serializer: :encrypt\n  output_category\nend\n~~~\n\n##### Existing and inflight jobs\n\nWhen migrating to Rocket Job 6, it is recommended to load every job and then save it back again as part of the \ndeployment. When the job loads it will automatically convert itself from the old schema to the new v6 schema.\n\nIn flight jobs should not be affected, other than it is important to shutdown all running batch \nservers _before_ running any new instances.\n\n## Rocket Job v4\n\nRocket Job Pro is now fully open source and included in Rocket Job under the Apache License. \n\nThe `RocketJob::Batch` plugin now adds batch processing capabilities to break up a single task into many\nconcurrent workers processing slices of the entire job at the same time. \n\n\nExample:\n\n```ruby\nclass MyJob \u003c RocketJob::Job\n  include RocketJob::Batch\n  \n  self.description         = \"Reverse names\"\n  self.destroy_on_complete = false\n  \n  # Collect the output for this job in the default output category: `:main`\n  output_category\n\n  # Method to call by all available workers at the same time.\n  # Reverse the characters for each line: \n  def perform(line)\n    line.reverse\n  end\nend\n```\n\nUpload a file for processing, for example `names.csv` which could contain:\n\n```\njack\njane\nbill\njohn\nblake\nchris\ndave\nmarc\n```\n\nTo queue the above job for processing:\n\n```ruby\njob = MyJob.new\njob.upload('names.csv')\njob.save!\n```\n\nOnce the job has completed, download the results into a file:\n\n```ruby\njob.download('names_reversed.csv')\n```\n\n## Contributing to the documentation\n\nTo contribute to the documentation it is as easy as forking the repository\nand then editing the markdown pages directly via the github web interface.\n\nFor more complex documentation changes checkout the source code locally.\n\n#### Local checkout\n\n* Fork the repository in github.\n* Checkout your fork of the source code locally.\n* Install Jekyll\n~~~\n    cd docs\n    bundle update\n~~~\n* Run Jekyll web server:\n~~~\n    jekyll serve\n~~~\n* Open a web browser to view the local documentation:\n    [http://127.0.0.1:4000](http://127.0.0.1:4000)\n* Edit the files in the `/docs` folder.\n* Refresh the page to see the changes.\n\nOnce the changes are complete, submit a github pull request.\n\n## Upgrading to V3\n\nV3 replaces MongoMapper with Mongoid which supports the latest MongoDB Ruby client driver.\n\n### Upgrading Mongo Config file\nReplace `mongo.yml` with `mongoid.yml`.\n\nStart with the sample [mongoid.yml](https://github.com/rocketjob/rocketjob/blob/feature/mongoid/test/config/mongoid.yml).\n \nFor more information on the new [Mongoid config file](https://docs.mongodb.com/ruby-driver/master/tutorials/5.1.0/mongoid-installation/).\n\nNote: The `rocketjob` and `rocketjob_slices` clients in the above `mongoid.yml` file are required.\n\n### Other changes\n\n* Arguments are no longer supported, use fields for defining all named arguments for a job.\n\n* Replace usages of `rocket_job do` to set default values:\n\n~~~ruby\n  rocket_job do |job|\n    job.priority = 25\n  end\n~~~\n\nWith:\n\n~~~ruby\n  self.priority = 25\n~~~\n\n* Replace `key` with `field` when adding attributes to a job:\n\n~~~ruby\n  key :inquiry_defaults, Hash\n~~~\n\nWith:\n\n~~~ruby\n  field :inquiry_defaults, type: Hash, default: {}\n~~~\n\n* Replace usage of `public_rocket_job_properties` with the `user_editable` option:\n\n~~~ruby\nfield :priority, type: Integer, default: 50, user_editable: true\n~~~\n\n## Ruby Support\n\nRocket Job is tested and supported on the following Ruby platforms:\n- Ruby 2.1, 2.2, 2.3, 2.4, and above\n- JRuby 9.0.5 and above\n\n## Dependencies\n\n* [MongoDB](https://www.mongodb.org)\n    * Persists job information.\n    * Version 2.7 or greater.\n* [Semantic Logger](https://rocketjob.github.io/semantic_logger)\n    * Highly concurrent scalable logging.\n\n## Versioning\n\nThis project uses [Semantic Versioning](http://semver.org/).\n\n## Author\n\n[Reid Morrison](https://github.com/reidmorrison)\n\n## Contributors\n\n[Contributors](https://github.com/rocketjob/rocketjob/graphs/contributors)\n\n","funding_links":[],"categories":["Ruby"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freidmorrison%2Frocketjob","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Freidmorrison%2Frocketjob","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freidmorrison%2Frocketjob/lists"}