{"id":31271781,"url":"https://github.com/ruby/ruby-bench","last_synced_at":"2025-09-23T20:05:36.275Z","repository":{"id":39585812,"uuid":"308080502","full_name":"ruby/ruby-bench","owner":"ruby","description":"Set of benchmarks for the Ruby programming language","archived":false,"fork":false,"pushed_at":"2025-09-11T23:11:40.000Z","size":36406,"stargazers_count":96,"open_issues_count":5,"forks_count":28,"subscribers_count":256,"default_branch":"main","last_synced_at":"2025-09-12T01:02:50.029Z","etag":null,"topics":["benchmark","cruby","jit","ruby"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ruby.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2020-10-28T16:42:20.000Z","updated_at":"2025-09-11T23:11:43.000Z","dependencies_parsed_at":"2025-09-12T01:02:53.934Z","dependency_job_id":null,"html_url":"https://github.com/ruby/ruby-bench","commit_stats":null,"previous_names":["ruby/yjit-bench","ruby/ruby-bench"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ruby/ruby-bench","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ruby%2Fruby-bench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ruby%2Fruby-bench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ruby%2Fruby-bench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ruby%2Fruby-bench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ruby","download_url":"https://codeload.github.com/ruby/ruby-bench/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ruby%2Fruby-bench/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":276639739,"owners_count":25678179,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-23T02:00:09.130Z","response_time":73,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","cruby","jit","ruby"],"created_at":"2025-09-23T20:02:18.054Z","updated_at":"2025-09-23T20:05:36.268Z","avatar_url":"https://github.com/ruby.png","language":"Ruby","readme":"ruby-bench\n==========\n\nSmall set of benchmarks and scripts for the Ruby programming language.\n\nThe benchmarks are found in the `benchmarks` directory. Individual Ruby files\nin `benchmarks` are microbenchmarks. Subdirectories under `benchmarks` are\nlarger macrobenchmarks. Each benchmark relies on a harness found in\n[./harness/harness.rb](harness/harness.rb). The harness controls the number of times a benchmark is\nrun, and writes timing values into an output file.\n\nThe `run_benchmarks.rb` script (optional) traverses the `benchmarks` directory and\nruns the benchmarks in there. It reads the\noutput file written by the benchmarking harness. The output is written to\nmultiple files at the end -- CSV, text and JSON -- so that results can be easily viewed or\ngraphed in any spreadsheet editor.\n\n## Installation\n\nClone this repository:\n```\ngit clone https://github.com/ruby/ruby-bench\n```\n\n### Benchmarking YJIT\n\nruby-bench supports benchmarking any Ruby implementation. But if you want to benchmark YJIT,\nfollow [these instructions](https://github.com/ruby/ruby/blob/master/doc/yjit/yjit.md#building-yjit)\nto build and install YJIT.\n\nIf you install it with the name `ruby-yjit` on [chruby](https://github.com/postmodern/chruby),\nyou should enable it before running `./run_benchmarks.rb`:\n\n```\nchruby ruby-yjit\n```\n\n## Usage\n\nTo run all the benchmarks and record the data:\n```\ncd ruby-bench\n./run_benchmarks.rb\n```\n\nThis runs for a few minutes and produces a table like this in the console (results below not up to date):\n```\n-------------  -----------  ----------  ---------  ----------  -----------  ------------\nbench          interp (ms)  stddev (%)  yjit (ms)  stddev (%)  interp/yjit  yjit 1st itr\n30k_ifelse     2372.0       0.0         447.6      0.1         5.30         4.16\n30k_methods    6328.3       0.0         963.4      0.0         6.57         6.25\nactiverecord   171.7        0.8         144.2      0.7         1.19         1.15\nbinarytrees    445.8        2.1         389.5      2.5         1.14         1.14\ncfunc_itself   105.7        0.2         58.7       0.7         1.80         1.80\nfannkuchredux  6697.3       0.1         6714.4     0.1         1.00         1.00\nfib            245.3        0.1         77.1       0.4         3.18         3.19\ngetivar        97.3         0.9         44.3       0.6         2.19         0.98\nlee            1269.7       0.9         1172.9     1.0         1.08         1.08\nliquid-render  204.5        1.0         172.4      1.3         1.19         1.18\nnbody          121.9        0.1         121.6      0.3         1.00         1.00\noptcarrot      6260.2       0.5         4723.1     0.3         1.33         1.33\nrailsbench     3827.9       0.9         3581.3     1.3         1.07         1.05\nrespond_to     259.0        0.6         197.1      0.4         1.31         1.31\nsetivar        73.1         0.2         53.3       0.7         1.37         1.00\n-------------  -----------  ----------  ---------  ----------  -----------  ------------\n```\n\nThe `interp/yjit` column is the ratio of the average time taken by the interpreter over the\naverage time taken by YJIT after a number of warmup iterations. Results above 1 represent\nspeedups. For instance, 1.14 means \"YJIT is 1.14 times as fast as the interpreter\".\n\n### Specific categories\n\nBy default, `run_benchmarks.rb` runs all three [benchmark categories](./benchmarks.yml),\n`--category headline,other,micro`. You can run only benchmarks with specific categories:\n\n```\n./run_benchmarks.rb --category micro\n```\n\nYou can also only the headline benchmarks with the `--headline` option:\n\n```\n./run_benchmarks.rb --headline\n```\n\n### Specific benchmarks\n\nTo run one or more specific benchmarks and record the data:\n```\n./run_benchmarks.rb fib lee optcarrot\n```\n\n### Running a single benchmark\n\nThis is the easiest way to run a single benchmark.\nIt requires no setup at all and assumes nothing about the Ruby you are benchmarking.\nIt's also convenient for profiling, debugging, etc, especially since all benchmarked code runs in that process.\n\n```\nruby benchmarks/some_benchmark.rb\n```\n\n## Ruby options\n\nBy default, ruby-bench benchmarks the Ruby used for `run_benchmarks.rb`.\nIf the Ruby has `--yjit` option, it compares two Ruby commands, `-e \"interp::ruby\"` and `-e \"yjit::ruby --yjit`.\nHowever, if you specify `-e` yourself, you can override what Ruby is benchmarked.\n\n```sh\n# \"xxx::\" prefix can be used to specify a shorter name/alias, but it's optional.\n./run_benchmarks.rb -e \"ruby\" -e \"yjit::ruby --yjit\"\n\n# You could also measure only a single Ruby\n./run_benchmarks.rb -e \"3.1.0::/opt/rubies/3.1.0/bin/ruby\"\n\n# With --chruby, you can easily specify rubies managed by chruby\n./run_benchmarks.rb --chruby \"3.1.0\" --chruby \"3.1.0+YJIT::3.1.0 --yjit\"\n\n# \";\" can be used to specify multiple executables in a single option\n./run_benchmarks.rb --chruby \"3.1.0;3.1.0+YJIT::3.1.0 --yjit\"\n```\n\n### YJIT options\n\nYou can use `--yjit_opts` to specify YJIT command-line options:\n\n```\n./run_benchmarks.rb --yjit_opts=\"--yjit-version-limit=10\" fib lee optcarrot\n```\n\n### Running pre-init code\n\nIt is possible to use `run_benchmarks.rb` to run arbitrary code before\neach benchmark run using the `--with-pre-init` option.\n\nFor example: to run benchmarks with `GC.auto_compact` enabled a\n`pre-init.rb` file can be created, containing `GC.auto_compact=true`,\nand this can be passed into the benchmarks in the following way:\n\n```\n./run_benchmarks.rb --with-pre-init=./pre-init.rb\n```\n\nThis file will then be passed to the underlying Ruby interpreter with\n`-r`.\n\n## Harnesses\n\nYou can find several test harnesses in this repository:\n\n* harness - the normal default harness, with duration controlled by warmup iterations and time/count limits\n* harness-bips - a harness that measures iterations/second until stable\n* harness-continuous - a harness that adjusts the batch sizes of iterations to run in stable iteration size batches\n* harness-once - a simplified harness that simply runs once\n* harness-perf - a simplified harness that runs for exactly the hinted number of iterations\n* harness-stackprof - a harness to profile the benchmark with stackprof\n* harness-stats - count method calls and loop iterations\n* harness-vernier - a harness to profile the benchmark with vernier\n* harness-warmup - a harness which runs as long as needed to find warmed up (peak) performance\n\nTo use it, run a benchmark script directly, specifying a harness directory with `-I`:\n\n```\nruby -Iharness benchmarks/railsbench/benchmark.rb\n```\n\nThere is also a robust but complex CI harness in [the yjit-metrics repo](https://github.com/Shopify/yjit-metrics).\n\n### Iterations and duration\n\nWith the default harness, the number of iterations and duration\ncan be controlled by the following environment variables:\n\n* `WARMUP_ITRS`: The number of warm-up iterations, ignored in the final comparison (default: 15)\n* `MIN_BENCH_ITRS`: The minimum number of benchmark iterations (default: 10)\n* `MIN_BENCH_TIME`: The minimum seconds for benchmark (default: 10)\n\nYou can also use `--warmup`, `--bench`, or `--once` to set these environment variables:\n\n```sh\n# same as: WARMUP_ITRS=2 MIN_BENCH_ITRS=3 MIN_BENCH_TIME=0 ./run_benchmarks.rb railsbench\n./run_benchmarks.rb railsbench --warmup=2 --bench=3\n\n# same as: WARMUP_ITRS=0 MIN_BENCH_ITRS=1 MIN_BENCH_TIME=0 ./run_benchmarks.rb railsbench\n./run_benchmarks.rb railsbench --once\n```\n\nThere is also a handy script for running benchmarks just once using\n`WARMUP_ITRS=0 MIN_BENCH_ITRS=1 MIN_BENCH_TIME=0`, for example\nwith the `--yjit-stats` command-line option:\n\n```\n./run_once.sh --yjit-stats benchmarks/railsbench/benchmark.rb\n```\n\n### Using perf\n\nThere is also a harness to use Linux perf. By default, it only runs a fixed number of iterations.\nIf `PERF` environment variable is present, it starts the perf subcommand after warmup.\n\n```sh\n# Use `perf record` for both warmup and benchmark\nperf record ruby --yjit-perf=map -Iharness-perf benchmarks/railsbench/benchmark.rb\n\n# Use `perf record` only for benchmark\nPERF=record ruby --yjit-perf=map -Iharness-perf benchmarks/railsbench/benchmark.rb\n```\n\nThis is the only harness that uses `run_benchmark`'s argument, `num_itrs_hint`.\n\n### Printing YJIT stats\n\nThe `--yjit-stats` option of `./run_benchmarks.rb` allows you to print the diff of YJIT stats counters\nafter each iteration with the default harness.\n\n```\n./run_benchmarks.rb --yjit-stats=code_region_size,yjit_alloc_size\n```\n\n## Measuring memory usage\n\n`--rss` option of `run_benchmarks.rb` allows you to measure RSS after benchmark iterations.\n\n```\n./run_benchmarks.rb --rss\n```\n\n## Rendering a graph\n\n`--graph` option of `run_benchmarks.rb` allows you to render benchmark results as a graph.\n\n```bash\n# Write a graph at data/output_XXX.png (it will print the path)\n./run_benchmarks.rb --graph\n```\n\n### Installation\n\nBefore using this option, you might need to install the dependencies of [Gruff](https://github.com/topfunky/gruff):\n\n```bash\n# macOS\nbrew install imagemagick\n\n# Ubuntu\nsudo apt-get install libmagickwand-dev\n```\n\n### Changing font size\n\nYou can regenerate a graph with `misc/graph.rb`, changing its font size.\n\n```\nUsage: misc/graph.rb [options] CSV_PATH\n        --title SIZE                 title font size\n        --legend SIZE                legend font size\n        --marker SIZE                marker font size\n```\n\n## Disabling CPU Frequency Scaling\n\nTo disable CPU frequency scaling with an Intel CPU, edit `/etc/default/grub` or `/etc/default/grub.d/50-cloudimg-settings.cfg` and add `intel_pstate=no_hwp` to `GRUB_CMDLINE_LINUX_DEFAULT`. It’s a space-separated list.\n\nThen:\n```bash\nsudo update-grub\nsudo reboot\nsudo sh -c 'echo 1 \u003e /sys/devices/system/cpu/intel_pstate/no_turbo'\n```\n\nTo verify things worked:\n - `cat /proc/cmdline` to see the `intel_pstate=no_hwp` parameter is in there\n - `ls /sys/devices/system/cpu/intel_pstate/` and `hwp_dynamic_boost` should not exist\n - `cat /sys/devices/system/cpu/intel_pstate/no_turbo` should say `1`\n\nHelpful docs:\n - https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/processor_state_control.html#baseline-perf\n","funding_links":[],"categories":["Ruby"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fruby%2Fruby-bench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fruby%2Fruby-bench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fruby%2Fruby-bench/lists"}