{"id":13521346,"url":"https://github.com/skroutz/rspecq","last_synced_at":"2025-05-15T15:09:36.138Z","repository":{"id":37868939,"uuid":"274994424","full_name":"skroutz/rspecq","owner":"skroutz","description":"Distribute and run RSpec suites among parallel workers; for faster CI builds","archived":false,"fork":false,"pushed_at":"2025-03-18T15:31:14.000Z","size":152,"stargazers_count":163,"open_issues_count":28,"forks_count":23,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-03-31T20:08:28.922Z","etag":null,"topics":["ci","ci-tools","rspec","rspec-runner","rspec-suite","rspec-testing","test-runner","test-runners"],"latest_commit_sha":null,"homepage":"https://rubygems.org/gems/rspecq","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/skroutz.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-06-25T19:05:07.000Z","updated_at":"2025-03-23T00:42:49.000Z","dependencies_parsed_at":"2024-02-23T15:44:53.768Z","dependency_job_id":"6e1624c0-02dc-4573-81a8-4bd3272e498a","html_url":"https://github.com/skroutz/rspecq","commit_stats":{"total_commits":123,"total_committers":13,"mean_commits":9.461538461538462,"dds":"0.39837398373983735","last_synced_commit":"b8c52a9576258292df228d0f63e8e10c88f65ba4"},"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skroutz%2Frspecq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skroutz%2Frspecq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skroutz%2Frspecq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skroutz%2Frspecq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/skroutz","download_url":"https://codeload.github.com/skroutz/rspecq/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247730068,"owners_count":20986404,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ci","ci-tools","rspec","rspec-runner","rspec-suite","rspec-testing","test-runner","test-runners"],"created_at":"2024-08-01T06:00:33.058Z","updated_at":"2025-04-07T21:11:06.794Z","avatar_url":"https://github.com/skroutz.png","language":"Ruby","funding_links":[],"categories":["Ruby","Parallel execution"],"sub_categories":[],"readme":"RSpec Queue\n=========================================================================\n![Build status](https://github.com/skroutz/rspecq/actions/workflows/build.yml/badge.svg)\n[![Gem Version](https://badge.fury.io/rb/rspecq.svg)](https://badge.fury.io/rb/rspecq)\n\nRSpec Queue (RSpecQ) distributes and executes RSpec suites among parallel\nworkers. It uses a centralized queue that workers connect to and pop off\ntests from. It ensures optimal scheduling of tests based on their run time,\nfacilitating faster CI builds.\n\nRSpecQ is inspired by [test-queue](https://github.com/tmm1/test-queue)\nand [ci-queue](https://github.com/Shopify/ci-queue).\n\n## Features\n\n- Run an RSpec suite among many workers\n  (potentially located in different hosts) in a distributed fashion,\n  facilitating faster CI builds.\n- Consolidated, real-time reporting of a build's progress.\n- Optimal scheduling of test execution by using timings statistics from previous runs and\n  automatically scheduling slow spec files as individual examples. See\n  [*Spec file splitting*](#spec-file-splitting).\n- Automatic retry of test failures before being considered legit, in order to\n  rule out flakiness. Additionally, flaky tests are detected and provided to\n  the user. See [*Requeues*](#requeues).\n- Handles intermittent worker failures (e.g. network hiccups, faulty hardware etc.)\n  by detecting non-responsive workers and requeing their jobs. See [*Worker failures*](#worker-failures)\n- Sentry integration for monitoring build-level events. See [*Sentry integration*](#sentry-integration).\n  See [#2](https://github.com/skroutz/rspecq/issues/2).\n- Automatic termination of builds after a certain amount of failures. See [*Fail-fast*](#fail-fast).\n\n## Usage\n\nA worker needs to be given a name and the build it will participate in.\nAssuming there's a Redis instance listening at `localhost`, starting a worker\nis as simple as:\n\n```shell\n$ rspecq --build=123 --worker=foo1 spec/\n```\n\nTo start more workers for the same build, use distinct worker IDs but the same\nbuild ID:\n\n```shell\n$ rspecq --build=123 --worker=foo2\n```\n\nTo view the progress of the build use `--report`:\n\n```shell\n$ rspecq --build=123 --report\n```\n\nFor detailed info use `--help`:\n\n```\nNAME:\n    rspecq - Optimally distribute and run RSpec suites among parallel workers\n\nUSAGE:\n    rspecq [\u003coptions\u003e] [spec files or directories]\n\nOPTIONS:\n    -b, --build ID                   A unique identifier for the build. Should be common among workers participating in the same build.\n    -w, --worker ID                  An identifier for the worker. Workers participating in the same build should have distinct IDs.\n        --seed SEED                  The RSpec seed. Passing the seed can be helpful in many ways i.e reproduction and testing.\n    -r, --redis HOST                 --redis is deprecated. Use --redis-host or --redis-url instead. Redis host to connect to (default: 127.0.0.1).\n        --redis-host HOST            Redis host to connect to (default: 127.0.0.1).\n        --redis-url URL              Redis URL to connect to (e.g.: redis://127.0.0.1:6379/0).\n        --update-timings             Update the global job timings key with the timings of this build. Note: This key is used as the basis for job scheduling.\n        --file-split-threshold N     Split spec files slower than N seconds and schedule them as individual examples.\n        --report                     Enable reporter mode: do not pull tests off the queue; instead print build progress and exit when it's finished.\n                                     Exits with a non-zero status code if there were any failures.\n        --report-timeout N           Fail if build is not finished after N seconds. Only applicable if --report is enabled (default: 3600).\n        --max-requeues N             Retry failed examples up to N times before considering them legit failures (default: 3).\n        --queue-wait-timeout N       Time to wait for a queue to be ready before considering it failed (default: 30).\n        --fail-fast N                Abort build with a non-zero status code after N failed examples.\n        --reproduction               Enable reproduction mode: Publish files and examples in the exact order given in the command. Incompatible with --timings.\n        --tag TAG                    Run examples with the specified tag, or exclude examples by adding ~ before the tag.  - e.g. ~slow  - TAG is always converted to a symbol.\n    -h, --help                       Show this message.\n    -v, --version                    Print the version and exit.\n```\n\nYou can set most options using ENV variables:\n\n```shell\n$ RSPECQ_BUILD=123 RSPECQ_WORKER=foo1 rspecq spec/\n```\n\n### Supported ENV variables\n\n| Name | Desc |\n| --- | --- |\n| `RSPECQ_BUILD` | Build ID |\n| `RSPECQ_WORKER` | Worker ID |\n| `RSPECQ_SEED` | RSpec seed |\n| `RSPECQ_REDIS` | Redis HOST |\n| `RSPECQ_UPDATE_TIMINGS` | Timings |\n| `RSPECQ_FILE_SPLIT_THRESHOLD` | File split threshold |\n| `RSPECQ_REPORT` | Report |\n| `RSPECQ_REPORT_TIMEOUT` | Report Timeout |\n| `RSPECQ_MAX_REQUEUES` | Max requests |\n| `RSPECQ_QUEUE_WAIT_TIMEOUT` | Queue wait timeout |\n| `RSPECQ_REDIS_URL` | Redis URL |\n| `RSPECQ_FAIL_FAST` | Fail fast |\n| `RSPECQ_REPORTER_RERUN_COMMAND_SKIP` | Do not report flaky test's rerun command |\n\n### Sentry integration\n\nRSpecQ can optionally emit build events to a\n[Sentry](https://sentry.io) project by setting the\n`SENTRY_DSN` environment variable.\n\nThis is convenient for monitoring important warnings/errors that may impact\nbuild times, such as the fact that no previous timings were found and\ntherefore job scheduling was effectively random for a particular build.\n\n## How it works\n\nThe core design is almost identical to ci-queue so please refer to its\n[README](https://github.com/Shopify/ci-queue/blob/master/README.md) instead.\n\n### Terminology\n\n- **Job**: the smallest unit of work, which is usually a spec file\n  (e.g. `./spec/models/foo_spec.rb`) but can also be an individual example\n  (e.g. `./spec/models/foo_spec.rb[1:2:1]`) if the file is too slow.\n- **Queue**: a collection of Redis-backed structures that hold all the necessary\n  information for an RSpecQ build to run. This includes timing statistics,\n  jobs to be executed, the failure reports and more.\n- **Build**: a particular test suite run. Each build has its own **Queue**.\n- **Worker**: an `rspecq` process that, given a build id, consumes jobs off the\n  build's queue and executes them using RSpec\n- **Reporter**: an `rspecq` process that, given a build id, waits for the build's\n  queue to be drained and prints the build summary report\n\n### Spec file splitting\n\nParticularly slow spec files may set a limit to how fast a build can be.\nFor example, a single file may need 10 minutes to run while all other\nfiles finish after 8 minutes. This would cause all but one workers to be\nsitting idle for 2 minutes.\n\nTo overcome this issue, RSpecQ can split files which their execution time is\nabove a certain threshold (set with the `--file-split-threshold` option)\nand instead schedule them as individual examples.\n\nNote: In the future, we'd like for the slow threshold to be calculated and set\ndynamically (see #3).\n\n### Requeues\n\nAs a mitigation technique against flaky tests, if an example fails it will be\nput back to the queue to be picked up by another worker. This will be repeated\nup to a certain number of times (set with the `--max-requeues` option), after\nwhich the example will be considered a legit failure and printed as such in the\nfinal report.\n\nFlaky tests are also detected and printed as such in the final report. They are\nalso emitted to Sentry (see [Sentry integration](#sentry-integration)).\n\n### Fail-fast\n\nIn order to prevent large suites running for a long time with a lot of\nfailures, a threshold can be set to control the number of failed examples that\nwill render the build unsuccessful. This is in par with RSpec's\n[--fail-fast](https://relishapp.com/rspec/rspec-core/docs/command-line/fail-fast-option).\n\nThis feature is disabled by default, and can be controlled via the\n`--fail-fast` command line option.\n\n### Worker failures\n\nIt's not uncommon for CI processes to encounter unrecoverable failures for\nvarious reasons: faulty hardware, network hiccups, segmentation faults in\nMRI etc.\n\nFor resiliency against such issues, workers emit a heartbeat after each\nexample they execute, to signal\nthat they're healthy and performing jobs as expected. If a worker hasn't\nemitted a heartbeat for a given amount of time (set by `WORKER_LIVENESS_SEC`)\nit is considered dead and its reserved job will be put back to the queue, to\nbe picked up by another healthy worker.\n\n\n## Rationale\n\n### Why didn't you use ci-queue?\n\n**Update**: ci-queue [deprecated support for RSpec](https://github.com/Shopify/ci-queue/pull/149).\n\nWhile evaluating ci-queue we experienced slow worker boot\ntimes (up to 3 minutes in some cases) combined with disk IO saturation and\nincreased memory consumption. This is due to the fact that a worker in\nci-queue has to load every spec file on boot. In applications with a large\nnumber of spec files this may result in a significant performance hit and\nin case of cloud environments, increased costs.\n\nWe also observed slower build times compared to our previous solution which\nscheduled whole spec files (as opposed to individual examples), due to\nbig differences in runtimes of individual examples, something common in big\nRSpec suites.\n\nWe decided for RSpecQ to use whole spec files as its main unit of work (as\nopposed to ci-queue which uses individual examples). This means that an RSpecQ\nworker only loads the files needed and ends up with a subset of all the suite's\nfiles.  (Note: RSpecQ also schedules individual examples, but only when this is\ndeemed necessary, see [Spec file splitting](#spec-file-splitting)).\n\nThis kept boot and test run times considerably fast. As a side benefit, this\nallows suites to keep using `before(:all)` hooks (which ci-queue explicitly\nrejects).\n\nThe downside of this design is that it's more complicated, since the scheduling\nof spec files happens based on timings calculated from previous runs. This\nmeans that RSpecQ maintains a key with the timing of each job and updates it\non every run (if the `--update-timings` option was used). Also, RSpecQ has a\n\"slow file threshold\" which, currently has to be set manually (but this can be\nimproved in the future).\n\n\n## Development\n\nInstall the required dependencies:\n\n```\n$ bundle install\n```\n\nThen you can execute the tests after spinning up a Redis instance at\n`127.0.0.1:6379`:\n\n```\n$ bundle exec rake\n```\n\nTo enable verbose output in the tests:\n\n```\n$ RSPECQ_DEBUG=1 bundle exec rake\n```\n\n## Redis\n\nRSpecQ by design doesn't expire its keys from Redis. It is left to the user\nto configure the Redis server to do so; see\n[Using Redis as an LRU cache](https://redis.io/topics/lru-cache) for more info.\n\nYou can do this from a configuration file or with `redis-cli`.\n\n## License\n\nRSpecQ is licensed under MIT. See [LICENSE](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fskroutz%2Frspecq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fskroutz%2Frspecq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fskroutz%2Frspecq/lists"}