{"id":13463362,"url":"https://github.com/tmm1/test-queue","last_synced_at":"2025-05-15T04:04:03.499Z","repository":{"id":8200754,"uuid":"9632648","full_name":"tmm1/test-queue","owner":"tmm1","description":"parallel test runner for CI environments","archived":false,"fork":false,"pushed_at":"2023-12-30T15:43:16.000Z","size":318,"stargazers_count":575,"open_issues_count":5,"forks_count":68,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-04-14T04:59:03.835Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tmm1.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2013-04-23T20:30:57.000Z","updated_at":"2025-03-27T12:25:18.000Z","dependencies_parsed_at":"2023-12-30T16:46:50.868Z","dependency_job_id":null,"html_url":"https://github.com/tmm1/test-queue","commit_stats":{"total_commits":255,"total_committers":21,"mean_commits":"12.142857142857142","dds":0.6627450980392157,"last_synced_commit":"2681c6aca24bbc1da4beb70283e419efdc1d5be7"},"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tmm1%2Ftest-queue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tmm1%2Ftest-queue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tmm1%2Ftest-queue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tmm1%2Ftest-queue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tmm1","download_url":"https://codeload.github.com/tmm1/test-queue/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254270640,"owners_count":22042858,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T13:00:52.024Z","updated_at":"2025-05-15T04:04:03.473Z","avatar_url":"https://github.com/tmm1.png","language":"Ruby","readme":"# test-queue\n\n[![Gem Version](https://badge.fury.io/rb/test-queue.svg)](https://rubygems.org/gems/test-queue)\n[![CI](https://github.com/tmm1/test-queue/actions/workflows/test.yml/badge.svg)](https://github.com/tmm1/test-queue/actions/workflows/test.yml)\n\nYet another parallel test runner, built using a centralized queue to ensure\noptimal distribution of tests between workers.\n\nSpecifically optimized for CI environments: build statistics from each run\nare stored locally and used to sort the queue at the beginning of the\nnext run.\n\n## Usage\n\ntest-queue bundles `testunit-queue`, `minitest-queue`, and `rspec-queue` binaries which can be used directly:\n\n```console\n$ minitest-queue $(find test/ -name \\*_test.rb)\n$ rspec-queue --format progress spec\n```\n\nBut the underlying `TestQueue::Runner::TestUnit`, `TestQueue::Runner::Minitest`, and `TestQueue::Runner::RSpec` are\nbuilt to be subclassed by your application. I recommend checking a new\nexecutable into your project using one of these superclasses.\n\n```console\n$ vim script/test-queue\n$ chmod +x script/test-queue\n$ git add script/test-queue\n```\n\nSince test-queue uses `fork(2)` to spawn off workers, you must ensure each worker\nruns in an isolated environment. Use the `after_fork` hook with a custom\nrunner to reset any global state.\n\n``` ruby\n#!/usr/bin/env ruby\n\nclass MyAppTestRunner \u003c TestQueue::Runner::Minitest\n  def after_fork(num)\n    # Use separate mysql database (we assume it exists and has the right schema already)\n    ActiveRecord::Base.configurations.configs_for(env_name: 'test', name: 'primary').database \u003c\u003c num.to_s\n    ActiveRecord::Base.establish_connection(:test)\n\n    # Use separate redis database\n    $redis.client.db = num\n    $redis.client.reconnect\n  end\n\n  def prepare(concurrency)\n    # Create mysql databases exists with correct schema\n    concurrency.times do |i|\n      # ...\n    end\n\n    # If this is a remote master, tell the central master something about us\n    @remote_master_message = \"Output for remote master 123: http://myhost.com/build/123\"\n  end\n\n  def around_filter(suite)\n    $stats.timing(\"test.#{suite}.runtime\") do\n      yield\n    end\n  end\nend\n\nMyAppTestRunner.new.execute\n```\n\n## Environment variables\n\n- `TEST_QUEUE_WORKERS`: Number of workers to use per master (default: all available cores)\n- `TEST_QUEUE_VERBOSE`: Show results as they are available (default: `0`)\n- `TEST_QUEUE_SOCKET`: Unix socket `path` (or TCP `address:port` pair) used for communication (default: `/tmp/test_queue_XXXXX.sock`)\n- `TEST_QUEUE_RELAY`: Relay results back to a central master, specified as TCP `address:port`\n- `TEST_QUEUE_STATS`: `path` to cache build stats in-build CI runs (default: `.test_queue_stats`)\n- `TEST_QUEUE_FORCE`: Comma separated list of suites to run\n- `TEST_QUEUE_RELAY_TIMEOUT`: When using distributed builds, the amount of time a remote master will try to reconnect to start work\n- `TEST_QUEUE_RELAY_TOKEN`: When using distributed builds, this must be the same on remote masters and the central master for remote masters to be able to connect.\n- `TEST_QUEUE_REMOTE_MASTER_MESSAGE`: When using distributed builds, set this on a remote master and it will appear in that master's connection message on the central master.\n- `TEST_QUEUE_SPLIT_GROUPS`: Split tests up by example rather than example group. Faster for tests with short setup time such as selenium. RSpec only. Add the `:no_split` tag to `ExampleGroups` you don't want split.\n\n## Design\n\ntest-queue uses a simple master + pre-fork worker model. The master\nexposes a Unix domain socket server which workers use to grab tests off\nthe queue.\n\n```console\n─┬─ 21232 minitest-queue master\n ├─── 21571 minitest-queue worker [3] - AuthenticationTest\n ├─── 21568 minitest-queue worker [2] - ApiTest\n ├─── 21565 minitest-queue worker [1] - UsersControllerTest\n └─── 21562 minitest-queue worker [0] - UserTest\n```\n\ntest-queue also has a distributed mode, where additional masters can share\nthe workload and relay results back to a central master.\n\n## Distributed mode\n\nTo use distributed mode, the central master must listen on a TCP port. Additional masters can be booted\nin relay mode to connect to the central master. Remote masters must provide a `TEST_QUEUE_RELAY_TOKEN`\nto match the central master's.\n\n```console\n$ TEST_QUEUE_RELAY_TOKEN=123 TEST_QUEUE_SOCKET=0.0.0.0:12345 bundle exec minitest-queue ./test/example_test.rb\n$ TEST_QUEUE_RELAY_TOKEN=123 TEST_QUEUE_RELAY=0.0.0.0:12345  bundle exec minitest-queue ./test/example_test.rb\n$ TEST_QUEUE_RELAY_TOKEN=123 ./test-multi.sh\n```\n\nSee the [Parameterized Trigger Plugin](https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Trigger+Plugin)\nfor a simple way to do this with Jenkins.\n\n## See also\n\n- https://github.com/Shopify/rails_parallel\n- https://github.com/grosser/parallel_tests\n","funding_links":[],"categories":["Testing","Ruby","Parallel execution"],"sub_categories":["Distributed Testing"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftmm1%2Ftest-queue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftmm1%2Ftest-queue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftmm1%2Ftest-queue/lists"}