{"id":18544548,"url":"https://github.com/instructure/inst-jobs","last_synced_at":"2025-05-07T13:02:12.710Z","repository":{"id":21331635,"uuid":"24648452","full_name":"instructure/inst-jobs","owner":"instructure","description":"Instructure-maintained fork of delayed_job","archived":false,"fork":false,"pushed_at":"2025-03-24T14:37:30.000Z","size":813,"stargazers_count":25,"open_issues_count":7,"forks_count":15,"subscribers_count":22,"default_branch":"main","last_synced_at":"2025-05-01T06:48:01.428Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/instructure.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"MIT-LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-09-30T17:43:04.000Z","updated_at":"2025-03-24T14:36:48.000Z","dependencies_parsed_at":"2024-04-23T19:04:24.480Z","dependency_job_id":null,"html_url":"https://github.com/instructure/inst-jobs","commit_stats":{"total_commits":341,"total_committers":31,"mean_commits":11.0,"dds":0.563049853372434,"last_synced_commit":"6a82b5880e39b328fc567815ce5ca30b7213ef86"},"previous_names":[],"tags_count":94,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/instructure%2Finst-jobs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/instructure%2Finst-jobs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/instructure%2Finst-jobs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/instructure%2Finst-jobs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/instructure","download_url":"https://codeload.github.com/instructure/inst-jobs/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252883219,"owners_count":21819157,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T20:16:54.181Z","updated_at":"2025-05-07T13:02:12.668Z","avatar_url":"https://github.com/instructure.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Instructure Delayed Jobs\n\n[![Build\nStatus](https://travis-ci.org/instructure/inst-jobs.svg?branch=master)](https://travis-ci.org/instructure/inst-jobs)\n\nThis gem was forked from\n[delayed_job](https://github.com/collectiveidea/delayed_job) in late 2010. While\nwe have tried to maintain compatibility with delayed_job where possible, so much\ncode has been added and rewritten that you should approach this as a distinct\nlibrary.\n\nIt's still useful to highlight the primary differences with delayed_job, for\nthose familiar with it:\n\n* `inst-jobs` was adapted for [Canvas\n  LMS](https://github.com/instructure/canvas-lms), where it has been\n  battle-hardened over the last 5+ years, scaling with Canvas from zero to tens\n  of millions of jobs run per day.\n  * To achieve this we are using some PostgreSQL specific features, which means\n    support for MySQL and other ActiveRecord backends has been dropped.\n  * Pushing and popping from the queue is very highly optimized, for a SQL-based\n    queue. A typical PostgreSQL database running on a c3.4xlarge EC2 instance\n    can handle queueing and running more than 11 million jobs per day while\n    staying below 30% CPU.\n  * The architecture is designed to support a mix of long-running (even\n    multi-hour) jobs alongside large numbers of very short (less than one\n    second) jobs.\n* Daemon management is highly reliable.\n  * Dead workers will be restarted, and any jobs they were working on will go\n    through the normal failure handling code.\n* Reliable and distributed \"Cron\" style jobs through the built-in [periodic\n  jobs](#periodic-jobs) functionality.\n* [Strands](#strands), allowing for ordered sequences of jobs based on ad-hoc\n  name tags.\n* [N-Strands](#n-strands), building on strands, allowing one to throttle how\n  many jobs on a strand can be running concurrently.\n* [Singleton jobs](#singleton-jobs), where at most one job is running, and one\n  queued at a time.\n* A simple [jobs admin UI](#web-ui), usable by any Rails or Rack application.\n* A separate `failed_jobs` table for tracking failed jobs.\n* Automatic tracking of what code enqueued each job, if\n  [Marginalia](https://github.com/basecamp/marginalia) is enabled.\n\n## Installation\n\ninst-jobs requires Rails 3.2 or above, and Ruby 2.1 or above. It is\ntested through Rails 5.0 and Ruby 2.3.\n\nAdd this line to your Rails application's Gemfile:\n\n```ruby\ngem 'inst-jobs'\n```\n\nAnd then execute:\n\n    $ bundle\n\nOr install it yourself as:\n\n    $ gem install inst-jobs\n\n## Setup\n\n### ActiveRecord Backend\n\nIf you are using the ActiveRecord backend, you'll need to install and\nrun the migrations:\n\n    $ rake delayed_engine:install:migrations\n    $ rake db:migrate\n\nTo use a separate database connection, specify it in an initializer:\n\n```ruby\nDelayed::Backend::ActiveRecord::Job.establish_connection(my_db_queue_config)\n```\n\nWhen upgrading `inst-jobs`, make sure to run `rake\ndelayed_engine:install:migrations` again to add any new migrations.\n\nThe ActiveRecord backend only supports PostgreSQL.\n\n### Worker Configuration\n\nWorker and queue information defaults to read from `config/delayed_jobs.yml`,\nthis can be overridden using the `--config` option from the command line.\n\n```yaml\ndevelopment:\n  workers:\n  - workers: 2\n\nproduction:\n  workers:\n  - workers: 10\n```\n\nAn initializer can also be used to set preferred values for any\nsettings that control specific interal delayed job behavior:\n\n```ruby\nDelayed::Settings.max_attempts              = 1\nDelayed::Settings.queue                     = \"canvas_queue\"\nDelayed::Settings.sleep_delay               = -\u003e{ 2.0 }\n```\n\nYou can find a list of available settings in `lib/delayed_job/settings.rb`.\n\n## Usage\n\n### Signal Handling\n\nInst-jobs makes an attempt at being well behaved with respect to how child\nprocesses are handled. When the pool receives SIGQUIT it will pass that on\nand wait `Settings.slow_exit_timeout` seconds (default 20) for all children\nto finish their currently active task and exit. If they take longer than\nthis a SIGTERM will be sent telling them to clean up and bail quickly, if\nthat doesn't happen within 2 seconds SIGKILL is then sent. This graceful exit\ncan be expedited by sending SIGTERM/SIGINT to the pool, this will still allow\nthe `slow_exit_timeout` period for the workers to exit but they should exit\nalmost immediately.\n\nThe old behavior of the pool exiting and leaving the child processes orphaned\ncan be preserved by setting `kill_workers_on_exit` to false. This will cause\nthe first signal sent to the pool to be propagated to all of the child\nprocesses after which the pool will exit.\n\n### Running Workers\n\n    $ inst_jobs # display help\n    $ inst_jobs start # start a worker in the background\n    $ inst_jobs run # start a worker in the foreground\n\n### Queueing Jobs\n\nIn the simplest form, this means just calling `delay` on any\nobject before calling the actual method:\n\n```ruby\n@user.delay.activate!\n```\n\nIf a method should always be run in the background, you can call\n`#handle_asynchronously` after the method declaration:\n\n```ruby\nclass Device\n  def deliver\n    # long running method\n  end\n  handle_asynchronously :deliver\nend\n\ndevice = Device.new\ndevice.deliver\n```\n\n#### Job Parameters\n\nTo pass parameters to the jobs engine, send them to the  `delay` method:\n\n```ruby\n@user.delay(max_attempts: 1, priority: 50).activate!(other_user)\n```\n\n`handle_asynchronously` and `delay` take these parameters:\n\n- `:priority` (number): lower numbers run first; default is 10 but can be\n  reconfigured.\n- `:run_at` (Time): run the job on or after this time; default is now.\n- `:queue` (string): named queue to put this job in, if using separate queues.\n- `:max_attempts` (number): the max number of attempts to make before\n  permanently failing the job; default is 1. For a job to be retried, it should\n  raise an error. By default, an exponential backoff algorithm is used to determine\n  the run_at for retried jobs (5 + attempt**4 seconds later).\n- `:strand` (string): [strand](#strands) to assign this job to; default is not\n  to assign to a strand.\n- `:n_strand` (string): [n_strand](#n-strands) to assign this job to; default is\n  none.\n- `:singleton` (string): [singleton](#singleton-jobs) to assign this job\n  to; default is none.\n- `:on_conflict` (:use_earliest|:overwrite|:loose): option for how to handle the\n  new job if a singleton[#singleton-jobs] job of the same type already exists.\n- `:on_failure` (symbol): method name to call on failure. The method should accept\n   a single argument (the error). Note that if max_attempts is greater than 1, this\n   method will be invoked each time the job fails, prior to rescheduling. Also note that\n   upon permanent failure (see :max_attempts for definition of 'permanent failure'), both\n   the :on_failure and :on_permanent_failure methods will be invoked.\n- `:on_permanent_failure` (symbol): method name to call on permanent failure (see :max_attempts\n   for definition of 'permanent failure'). The method should accept a single argument (the error).\n\n## Features\n\n### Strands\n\nA strand is a set of jobs that must be run in queue order. When a job is\nassigned to a strand, it will not start running until all previous jobs assigned\nto that strand have either completed or failed permanently. This is very useful\nwhen you have sequences of jobs that need to run in order.\n\nAn example use case is the \"ZIP file import\" functionality in [Canvas\nLMS](https://github.com/instructure/canvas-lms). Each job queued up processes an\nuploaded ZIP file and updates the specified course's files. It's important to\nmake sure that only one import job is ever running for a course, but we don't\nwant to globally serialize the ZIP imports, we only want to serialize them\nper-course.\n\nStrands make this simple. We simply use the course's unique identifier as part\nof the strand name, and we get the desired behavior. The (simplified) code is:\n\n```ruby\nzip_file_import.delay(strand: \"zip_file_import:#{course.uuid}\").process\n```\n\nStrand names are just freeform strings, and don't need to be created in advance.\nThe system is designed to perform well with any number of active strands.\n\n#### N Strands\n\nStrands are also useful when not required for correctness, but to avoid one\nparticular set of jobs monopolizing too many job workers. This can also be done\nby using a different `:queue` parameter for the jobs, and setting up a separate\npool of workers for that queue. But this is often overkill, and can result in\nwasted, idle workers for less-frequent jobs.\n\nAnother option is to use the `n_strand` parameter. This uses the same strand\nfunctionality to cap the number of jobs that can run in parallel for the\nspecified `n_strand`. The limit can be changed at runtime, as well.\n\n```ruby\n# The given proc will be called each time inst-jobs queues an n_strand job, to\n# determine how many jobs with this strand will be allowed to run in parallel.\nDelayed::Settings.num_strands = proc do |strand_name|\n  if strand_name == \"external_api_call\"\n    3\n  else\n    1\n  end\nend\n\nmy_api.delay(n_strand: \"external_api_call\").make_call\n```\n\n### Singleton Jobs\n\nSingleton jobs don't queue another job if a job is already queued with the\ngiven strand name:\n\n```ruby\n# If a job is already queued on the strand with this name, this job will not be\n# queued. It doesn't matter if previous jobs were queued on this strand but have\n# already completed, it only matters what is currently on the queue.\ngrader.delay(singleton: \"grade_student:#{student.uuid}\").grade_student\n```\n\nIf a job is currently running, it doesn't count as being in the queue for the\npurposes of singleton jobs. This is usually the desired behavior to avoid race\nconditions.\n\nYou can also pass an `on_conflict` option. The default of `:use_earliest` means\nthat the queued job will be updated to the earliest `run_at` of the existing and\nthe new job. Assuming you're using the default `run_at` of now, that means the\nnew job will simply be dropped. It can also be used if you run the singleton on\na schedule (like a periodic job), but occasionally want it to run now.\n\nThe second option is `:overwrite` and will always update the pending job to\nuse the `run_at` of the new job. This is useful for \"debouncing\" - you have some\ncleanup that needs to run after a trigger action, but there are many of that\ntrigger action and it's not useful to run the single cleanup job until the\ntrigger action calms down. This is also useful if the arguments to the job\nmight change, and you want it to run with the latest version of those\narguments.\n\nThe last option is `:loose`, which will drop the new job if there is already a\njob queued.\n\nAfter enqueuing a job, the job object will have a hint indicating if a new\njob was inserted, an existing job was updated, or the new job was dropped:\n- `:inserted` - a new job was inserted into the queue\n- `:updated` - an existing job was updated (both `:use_earliest` and `:overwrite`\n  can update the existing job's `run_at`, but only `:overwrite` will update the\n  handler)\n- `:dropped` - the new job was dropped due to an existing job in the queue\n\n#### Mixing N-Strand and Singleton\n\nIt is allowed to use both n_strand and singleton at the same time. This could\nbe useful if you want to throttle the number of jobs hitting an external service,\nbut also only allow one instance of the job along another axis. For example,\na job to refresh the state of a particular user with an external service:\n\n```\nuser.delay(singleton: \"service_update:#{user.id}\", n_strand: \"service_updates\").update_service_data\n```\n\nThe rules for singletons and N-strands are intersected - a job is ready to run\nonly if there are no more than max_concurrent running for the strand, _and_ if\nthere is not another job running for its singleton. In a strand with three jobs,\nand max concurrency of two, if the first two jobs are on one singleton, but the\nthird is on another, the third job will run, even though the second one cannot\nyet (due to waiting for the first job on the same singleton to complete).\n\n### Periodic Jobs\n\nPeriodic jobs are a reliable, distributed way of running recurring tasks, in\nother words \"distributed fault-tolerant Cron\".\n\nPeriodic jobs need to be configured during application startup, so that workers\nhave access to the schedules. For instance, in a Rails app it's suggested to\ncreate a `config/initializers/periodic_jobs.rb` file:\n\n```ruby\n# The first argument is a name tag for the job, and must be unique. The 2nd\n# argument is the run schedule, in Cron syntax.\nDelayed::Periodic.cron 'My Periodic Task', '30 11 * * *' do\n  MyApp::SomeTask.run()\nend\n```\n\nPeriodic Jobs are queued just like normal jobs, and run by your same pool of\nworkers. Jobs are configured with `max_attempts` set to 1, so if the job fails,\nit will not run again until the next scheduled interval.\n\nThis works by storing a registry of periodic jobs with their intervals,\nenqueueing each job immediately for the next time it should be run\nat, and then having an extra step after the job is performed\nthat re-enqueues it for the NEXT time it will run.  It's expected\nthat every periodic job will be in the queue all the time (either executing\nor queued for the next time it will execute).\n\nPeriodic jobs are singletons (see docs above on singleton jobs).\n\n### Lifecycle Events\n\nThere are several callbacks you can hook into from outside\nthe library, find them at the top of the \"lifecycle.rb\" class.\n\nTo hook into a callback, write something that looks like this in\nan initializer:\n\n```ruby\nDelayed::Worker.lifecycle.before(:error) do |worker, exception|\n  ErrorThingy.notify(exception)\nend\n```\n\n#### :error\nruns whenever a job throws an exception during invocation\n\n#### :retry\nruns when a job throws a Delayed::RetriableError during invocation\n(usually within the rescue block of an \"expected\" exception so\nthe proximate cause is the inner exception).  If the job\nCAN retry/reschedule, it will attempt to do so, otherwise\nit will run the :error callback instead.  Think of this as a way\nto handle infrastructural or other \"transient\" exceptions without\nfiring your error notification process.\n\n### Work Queue\n\nBy default, each Worker process will independently query and lock jobs in the\nqueue. There is an experimental ParentProcess WorkQueue implementation that has\neach Worker on a server communicate to a separate process on the server that\ncentrally handles querying and locking jobs. This can be enabled in the yml\nconfig:\n\n```yaml\nproduction:\n  work_queue: parent_process\n```\n\nThis will cut down on DB lock contention drastically, at the cost of potentially\ntaking a bit longer to find new jobs. It also enables another lifecycle callback\nthat can be used by plugins for added functionality. This may become the default\nor only implementation, eventually.\n\n### Sentry Error Reporting\n\nThe [standard delayed_job\nintegration](https://github.com/getsentry/raven-ruby/blob/master/lib/raven/integrations/delayed_job.rb)\nwill work with inst-jobs as well. Just add in an initializer along with your\nother raven-ruby configuration:\n\n```ruby\nrequire 'raven/integrations/delayed_job'\n```\n\n### Worker Health Checking and Unlocking Orphaned Jobs\n\nOccasionally a worker will unexpectedly terminate without being allowed to run\nany cleanup code, when this happens it causes any jobs that process had locked\nto remain locked indefinitely. To alleviate this each worker can register itself\nwith a locally running consul agent which will watch that each process is still\nalive, when a process is found to be dead it will automatically be deregistered\nfrom the agent causing another process to come along and reschedule the locked job.\n\n\n#### Configuring the Consul health check\n\nIn order to use the Consul health check you must include the `diplomat` gem,\nversion 2.5.x, in your application's Gemfile. It is not included in the\ndefault dependencies because it is an optional feature.\n\n```ruby\n# Enable the consul health check\nDelayed::Settings.worker_health_check_type = :consul\n\n# Configure the health check\nDelayed::Settings.worker_health_check_config = {\n  service_name: 'canvas-worker', # Optional, defaults to 'inst-jobs_worker'\n  check_interval: '7m', # Optional, defaults to 5m\n}\n\n# allow the worker pool to fork off a process\n# periodically to kill any jobs that are failing their\n# health checks\nDelayed::Settings.disable_abandoned_job_cleanup = false\n```\n\n## Testing\n\nTo write tests that interact with inst-jobs, you'll need to configure\nan actual database.\n### Locally\n\nBy default, if you have postgres running on its default ports,\nand if you have run:\n\n```\n$\u003e createdb inst-jobs-test-1\n```\n\nThen you should be able to run the tests that come with the library with:\n\n```\n$\u003e bundle exec rspec spec\n```\n\n### In Docker\n\nAlternatively, if you have `docker-compose` set up, you can run the CI\nbuild, which spins up the necessary services in docker:\n\n```\n$\u003e ./build.sh\n```\n\n#### Running individual tests in Docker\n\nFirst, you'll want a persistent gems volume, which you can get by:\n\n```\n$\u003e cp docker-compose.override.yml.example docker-compose.override.yml\n```\n\nThen you can install bundler and gems, which you'll want to do in your ruby version of choice:\n\n```\n$\u003e docker-compose run --rm app bash -lc \"bundle install\"\n```\n\nNow, to run an individual spec:\n\n```\n$\u003e docker-compose run --rm app bash -lc \"bundle exec rspec spec/delayed/worker_spec.rb\"\n```\n\nYou can also run the whole suite, but under just one rvm context, with:\n\n```\n$\u003e docker-compose run --rm app bash -lc \"bundle exec rake spec\"\n```\n\n### Writing Tests\n\nThere are a few basic testing helpers available:\n\n```ruby\nrequire 'delayed/testing'\n\nDelayed::Testing.drain # run all queued jobs\nDelayed::Testing.run_job(job) # run a single job\n\nbefore(:each) do\n  Delayed::Testing.clear_all! # delete all queued jobs\nend\n```\n\n## Web UI\n\ninst-jobs has a built-in web ui that allows users to view running jobs.\nOptionally, this web ui can support basic job management as well (hold, unhold,\nand delete operations are supported).  To enable this feature, pass a hash\ncontaining `update: true` into the `Delayed::Server` constructor.  You probably\nwant to ensure that the jobs endpoint requires authentication before enabling\nthis feature.\n\n### For Rails Apps\nTo use the web UI in your existing Rails application there are two options,\nfirst \"The Rails Way\" as shown just below this text or the Rack way shown\nat the very end of this section.\n\nFor \"The Rails Way\" to work there are two changes that need to be made to your\napplication.\n\nFirst you'll need to add Sinatra and `sinatra-contrib` to your\nGemfile (these dependencies are excluded from the default list so those who\naren't using this feature don't get the extra gems). For Rails 5.x applications,\nyou'll need to use Sinatra 2.x, which is in beta at the time of this writing.\n\nSecond, you'll need to add the following to your routes file:\n\n```ruby\nrequire 'delayed/server'\n\nRails.application.routes.draw do\n  # The delayed jobs server can mounted at any route you desire, delayed_jobs is\n  # just for this example\n  mount Delayed::Server.new =\u003e '/delayed_jobs'\nend\n```\n\nAdditionally, if you wish to restrict who has access to this route it is\nrecommended that users wrap this route in a constraint.\n\n### For Rack and Sinatra Apps\n\nTo use the web UI in your Rack app you can simply mount the app just like any\nother Rack app in your config.ru file:\n\n```ruby\nrequire 'delayed/server'\n\n# The delayed jobs server can mounted at any route you desire, delayed_jobs is\n# just for this example\nmap '/delayed_jobs' do\n  run Delayed::Server.new\nend\n\nrun MyApp\n```\n\n## Contributing\n\n1. Fork it ( https://github.com/instructure/inst-jobs/fork )\n2. Create your feature branch (`git checkout -b my-new-feature`)\n3. Commit your changes (`git commit -am 'Add some feature'`)\n4. Push to the branch (`git push origin my-new-feature`)\n5. Create a new Pull Request\n\n\n## Publishing\n\nReady to release a new version of inst-jobs?\nMake sure you're an owner (https://rubygems.org/gems/inst-jobs)\n\nIf your rubygems credentials are already set in `~/.gem/credentials`,\nyou can just run the release task:\n`bundle exec rake release`\n\nIf they are not, you can do this manually for now, and it will\ncache your credentials as part of the process:\n\n```bash\nbundle exec rake build\n# -\u003e inst-jobs VERSION built to pkg/inst-jobs-VERSION.gem\ngem push pkg/inst-jobs-VERSION.gem\n# -\u003e follow prompts to enter your login information\n```\n\nFuture releases you can now just use the release rake task,\nalthough if you have MFA enabled (and you should!) and your\nMFA valid period expires, you'll have to do the gem push\nmanually to enter a new MFA code.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finstructure%2Finst-jobs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Finstructure%2Finst-jobs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finstructure%2Finst-jobs/lists"}