{"id":19380326,"url":"https://github.com/drivy/checker_jobs","last_synced_at":"2025-04-06T14:12:30.592Z","repository":{"id":29950513,"uuid":"115118718","full_name":"drivy/checker_jobs","owner":"drivy","description":"Regression testing for data","archived":false,"fork":false,"pushed_at":"2024-06-26T15:55:20.000Z","size":121,"stargazers_count":85,"open_issues_count":22,"forks_count":4,"subscribers_count":18,"default_branch":"main","last_synced_at":"2025-03-30T12:08:21.184Z","etag":null,"topics":["data","regression-testing","ruby","sidekiq"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/drivy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-12-22T13:41:36.000Z","updated_at":"2024-11-19T15:25:41.000Z","dependencies_parsed_at":"2024-05-31T14:43:32.451Z","dependency_job_id":"5d21fa0a-a919-4609-9dcf-05e2a6d483f2","html_url":"https://github.com/drivy/checker_jobs","commit_stats":{"total_commits":71,"total_committers":11,"mean_commits":6.454545454545454,"dds":"0.49295774647887325","last_synced_commit":"5e3a55bd2fce11504c404cbd9ccb1dffdcb4c982"},"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/drivy%2Fchecker_jobs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/drivy%2Fchecker_jobs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/drivy%2Fchecker_jobs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/drivy%2Fchecker_jobs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/drivy","download_url":"https://codeload.github.com/drivy/checker_jobs/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247492559,"owners_count":20947545,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data","regression-testing","ruby","sidekiq"],"created_at":"2024-11-10T09:13:40.164Z","updated_at":"2025-04-06T14:12:30.565Z","avatar_url":"https://github.com/drivy.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CheckerJobs\n\nThis gems provides a small DSL to check your data for inconsistencies.\n\n[![Gem Version](https://badge.fury.io/rb/checker_jobs.svg)](https://badge.fury.io/rb/checker_jobs)\n[![Maintainability](https://api.codeclimate.com/v1/badges/7972bd0e4dc65329f5c6/maintainability)](https://codeclimate.com/github/drivy/checker_jobs/maintainability)\n[![Test Coverage](https://api.codeclimate.com/v1/badges/7972bd0e4dc65329f5c6/test_coverage)](https://codeclimate.com/github/drivy/checker_jobs/test_coverage)\n\n## Introduction\n\nTo ensure database integrity, DBMS provides some tools: foreign keys, triggers,\nstrored procedures, ... Those tools aren't the easiests to maintain unless your\nproject is based on them. Also, you may want to avoid to duplicate your business\nrules from your application to another language and add operational complexity\naround deployment.\n\nThis gem doesn't aim to replace those tools but provides something else that\ncould serve a close purpose: _ensure that you work with the data you expect_.\n\nThis gem helps you schedule some verifications on your data and get alerts when\nsomething is unexpected. You declare checks that could contain any business code\nyou like and then those checks are run by your application, in background jobs.\n\nA small DSL is provided to helps you express your predicates:\n\n- `ensure_no` will check that the result of a given block is `zero?`, `empty?` or `false`\n- `ensure_more` will check that the result of a given block is `\u003e=` than a given number\n- `ensure_fewer` will check that the result of a given block is `\u003c=` than a given number\n\nand an easy way to configure notifications.\n\nFor instance, at Drivy we don't expect users to get a negative credit amount. It\nisn't easy to get all the validation right because many rules are in play here.\nBecause of those rules the credit isn't just a column in our database yet but\nneeds to be computed based on various parameters. What we would like to ensure is\nthat _no one ends up with a negative credit_. We could write something like:\n\n``` ruby\nclass UsersChecker\n  include CheckerJobs::Base\n\n  options sidekiq: { queue: :slow }\n\n  notify :email, to: \"oss@drivy.com\"\n\n  ensure_no :negative_rental_credit do\n    # The following code is an over-simplification\n    # Real code is more performance oriented...\n\n    user_ids_with_negative_rental_credit = []\n\n    User.find_each do |user|\n      if user.credit_amount \u003c 0\n        user_ids_with_negative_rental_credit \u003c\u003c user.id\n      end\n    end\n\n    user_ids_with_negative_rental_credit\n  end\nend\n```\n\nThen when something's wrong, [you'll get alerted](https://cl.ly/3l2b3T3n0o2a).\n\nYou'll find more use cases and tips in the [wiki](https://github.com/drivy/checker_jobs/wiki).\n\n## Installation\n\nAdd this line to your application's Gemfile:\n\n```ruby\ngem 'checker_jobs'\n```\n\n## Usage\n\nHave a look at the examples directory of the repository to get a clearer idea about how to use and the gem is offering.\n\n### Configure\n\n``` ruby\nrequire \"checker_jobs\"\n\nCheckerJobs.configure do |c|\n  c.jobs_processor = :sidekiq\n\n  c.notifier :bugsnag do |options|\n    options[:formatter_class] = CheckerJobs::Notifiers::BugsnagDefaultFormatter\n  end\n\n  c.notifier :email do |options|\n    options[:formatter_class] = CheckerJobs::Notifiers::EmailDefaultFormatter\n    options[:email_options] = {\n      from: \"oss@drivy.com\",\n      reply_to: \"no-reply@drivy.com\",\n    }\n  end\n\n  c.notifier :logger do |options|\n    options[:logdev] = STDOUT\n    options[:level] = Logger::INFO\n  end\n\n  c.repository_url = { github: \"drivy/checker_jobs\" }\nend\n\n```\n\nThis piece of code usually goes into the `config/initializers/checker_jobs.rb`\nfile in a rails application. It relies on the fact that ActionMailer and sidekiq\nare already configured.\n\nIf you're on a different stack and you'll like to add a new job processor or\nnotification backend in this gem, [drop us a line][d-jobs].\n\n### Job Processor\n\nAt the moment, only [Sidekiq][gh-sidekiq] is supported as a job processor to asynchronously check for data inconsitencies.\nThe gem is designed to supports more processor.\nPRs are appreciated 🙏\n\n### Notifiers\n\nWe support different kind of notifiers, as of today we have the following:\n\n- `:bugsnag`: uses `Bugsnag` to send notifications. It takes the global Bugsnag configuration.\n- `:email`: uses `ActionMailer` to send emails. You can pass it any `ActionMailer` options.\n- `:logger`: Uses `Logger` to output inconsitencies in the log. Takes the following params:\n  - `logdev`: The log device. This is a filename (String) or IO object (typically STDOUT, STDERR, or an open file).\n  - `level`: Logging severity threshold (e.g. Logger::INFO)\n\n### Write checkers\n\nA checker is a class that inherits `CheckerJobs::Base` and uses the\n[DSL](wiki/DSL) to declare checks.\n\n``` ruby\nclass UserChecker\n  include CheckerJobs::Base\n\n  options sidekiq: { queue: :fast }\n\n  notify :email, to: \"oss@drivy.com\"\n\n  ensure_no :user_without_email do\n    UserRepository.missing_email.size\n  end\nend\n```\n\nThe `UserChecker` will have the same interface as your usual jobs. In this\nexample, `UserChecker` will be a `Sidekiq::Worker`. Its `#perform` method will\nrun the check named `:user_without_email` and if\n`UserRepository.missing_email.size` is greater than 0 then an email will be\nfired through ActionMailer to `oss@drivy.com`.\n\n### Schedule checks\n\nOnce you have checker jobs, you'll need to run them. There are many task\nschedulers out there and it isn't really relevant what you'll be using.\n\nYou have to enqueue your job as often as you like and that's it.\n\n``` ruby\nUserChecker.perform_async\n```\n\n## Contributing\n\nBug reports and pull requests are welcome on GitHub at https://github.com/drivy/checker_jobs.\n\nYou'll find out that the CI is setup to run test coverage and linting.\n\n## License\n\nThe gem is available as open source under the terms of the [MIT License][licence].\n\n[licence]: https://github.com/drivy/checker_jobs/blob/master/LICENSE.txt\n[d-jobs]: https://drivy.engineering/jobs/\n[gh-sidekiq]: https://github.com/mperham/sidekiq\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdrivy%2Fchecker_jobs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdrivy%2Fchecker_jobs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdrivy%2Fchecker_jobs/lists"}