{"id":19933741,"url":"https://github.com/bazelbuild/bazel-bench","last_synced_at":"2025-05-03T11:33:49.794Z","repository":{"id":37597162,"uuid":"175792560","full_name":"bazelbuild/bazel-bench","owner":"bazelbuild","description":"Benchmarking tool for bazel","archived":false,"fork":false,"pushed_at":"2024-09-11T13:04:21.000Z","size":261,"stargazers_count":32,"open_issues_count":10,"forks_count":13,"subscribers_count":30,"default_branch":"master","last_synced_at":"2025-04-07T15:46:50.036Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bazelbuild.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-03-15T09:47:16.000Z","updated_at":"2025-03-18T21:49:03.000Z","dependencies_parsed_at":"2023-01-25T09:15:58.894Z","dependency_job_id":"a261f079-3fa3-4b84-bd44-3b6e6fd72ddc","html_url":"https://github.com/bazelbuild/bazel-bench","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bazelbuild%2Fbazel-bench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bazelbuild%2Fbazel-bench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bazelbuild%2Fbazel-bench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bazelbuild%2Fbazel-bench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bazelbuild","download_url":"https://codeload.github.com/bazelbuild/bazel-bench/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252185926,"owners_count":21708230,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T23:14:43.756Z","updated_at":"2025-05-03T11:33:49.451Z","avatar_url":"https://github.com/bazelbuild.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Bazel Performance Benchmarking\n\n[![Build Status](https://badge.buildkite.com/1499c911d1faf665b9f6ba28d0a61e64c26a8586321b9d63a8.svg)](https://buildkite.com/bazel/bazel-bench)\n\n**Status**: WIP\n\n![logo](bb-icon.png)\n\n# Setup\n\nPre-requisites: `git` and `bazel`.\n\n```\n# Clone bazel-bench.\n$ git clone https://github.com/bazelbuild/bazel-bench.git\n$ cd bazel-bench\n```\n\nTo do a test run, run the following command (if you're on Windows, populate\n`--data_directory` with an appropriate Windows-style path):\n\n```shell\n$ bazel run :benchmark \\\n-- \\\n--bazel_commits=b8468a6b68a405e1a5767894426d3ea9a1a2f22f,ad503849e78b98d762f03168de5a336904280150 \\\n--project_source=https://github.com/bazelbuild/rules_cc.git \\\n--data_directory=/tmp/bazel-bench-data \\\n--verbose \\\n-- build //:all\n```\n\nThe Bazel commits might be too old and no longer buildable by your local Bazel. Replace them with the more recent commits from [bazelbuild/bazel](https://github.com/bazelbuild/bazel). The above command would print a result table on the terminal and outputs a csv\nfile to the specified `--data_directory`.\n\n## Syntax\n\nBazel-bench has the following syntax:\n\n```shell\n$ bazel run :benchmark -- \u003cbazel-bench-flags\u003e -- \u003cargs to pass to bazel binary\u003e\n\n```\n\nFor example, to benchmark the performance of 2 bazel commits A and B on the same\ncommand `bazel build --nobuild //:all` of `rules_cc` project, you'd do:\n\n```shell\n$ bazel run :benchmark \\\n-- \\\n--bazel_commits=A,B \\\n--project_source=https://github.com/bazelbuild/rules_cc.git \\\n-- build --nobuild //:all\n```\n\nNote the double-dash `--` before the command arguments. You can pass any\narguments that you would normally run on Bazel to the script. The performance of\ncommands other than `build` can also be benchmarked e.g. `query`, ...\n\n### Config-file Interface\n\nThe flag-based approach does not support cases where the benchmarked Bazel\ncommands differ. The most common use case for this: As a rule developer, I want\nto verify the effect of my flag on Bazel performance. For that, we'd need the\nconfig-file interface. The example config file would look like this:\n\n```yaml\n# config.yaml\nglobal_options:\n  project_commit: 595a730\n  runs: 5\n  collect_profile: false\n  project_source: /path/to/project/repo\nunits:\n - bazel_binary: /usr/bin/bazel\n   command: --startup_option1 build --nomy_flag //:all\n - bazel_binary: /usr/bin/bazel\n   command: --startup_option2 build --my_flag //:all\n```\n\nTo launch the benchmark:\n\n```shell\n$ bazel run :benchmark -- --benchmark_config=/absolute/path/to/config.yaml\n```\n\nThe above config file would benchmark 2 \"units\". A unit is defined as a set of \nconditions that describes a scenario to be benchmarked. This setup allows\nmaximum flexibility, as the conditions are independent between units. It's even \npossible to benchmark a `bazel_commit` against a pre-built `bazel_binary`.\n\n`global_options` is the list of options applied to every units. These global options are overridden by local options.\n\nFor the list of currently supported flags/attributes and their default values,\nrefer to [utils/benchmark_config.py](utils/benchmark_config.py).\n\n#### Known Limitations:\n\n- `project_source` should be a global option, as we don't support benchmarking\nmultiple projects in 1 benchmark. Though, `project_commit` can differ between units.\n- Incremental benchmarks isn't available.\n- Commands have to be in canonical form (next section).\n\n\n### Bazel Arguments Interpretation\n\nBazel arguments are parsed manually. It\nis _important_ that the supplied arguments in the command line strictly follows\nthe canonical form:\n\n```\n\u003ccommand\u003e \u003ccanonical options\u003e \u003cexpressions\u003e\n```\n\nExample of non-canonical command line arguments that could result in wrong\ninterpretation:\n\n```\nGOOD: (correct order, options in canonical form)\n  build --nobuild --compilation_mode=opt //:all\n\nBAD: (non-canonical options)\n  build --nobuild -c opt //:all\n\nBAD: (wrong order)\n  build --nobuild //:all --compilation_mode=opt\n```\n\n## Available flags\n\nTo show all the available flags:\n\n```\n$ bazel run :benchmark -- --helpshort\n```\n\nSome useful flags are:\n\n```\n  --bazel_binaries: The pre-built bazel binaries to benchmark.\n    (a comma separated list)\n  --bazel_commits: The commits at which bazel is built.\n    (default: 'latest')\n    (a comma separated list)\n  --bazel_source: Either a path to the local Bazel repo or a https url to a GitHub repository.\n    (default: 'https://github.com/bazelbuild/bazel.git')\n  --bazelrc: The path to a .bazelrc file.\n  --csv_file_name: The name of the output csv, without the .csv extension\n  --data_directory: The directory in which the csv files should be stored.\n  --[no]prefetch_ext_deps: Whether to do an initial run to pre-fetch external dependencies.\n    (default: 'true')\n  --project_commits: The commits from the git project to be benchmarked.\n    (default: 'latest')\n    (a comma separated list)\n  --project_source: Either a path to the local git project to be built or a https url to a GitHub repository.\n  --runs: The number of benchmark runs.\n    (default: '5')\n    (an integer)\n  --[no]verbose: Whether to include git/Bazel stdout logs.\n    (default: 'false')\n  --[no]collect_profile: Whether to collect JSON profile for each run.\n    Requires --data_directory to be set.\n    (default: 'false')\n```\n\n## Collecting JSON Profile\n\n[Bazel's JSON Profile](https://docs.bazel.build/versions/master/skylark/performance.html#json-profile)\nis a useful tool to investigate the performance of Bazel. You can configure\n`bazel-bench` to export these JSON profiles on runs using the\n`--collect_profile` flag.\n\n### JSON Profile Aggregation\n\nFor each pair of `project_commit` and `bazel_commit`, we produce a couple JSON\nprofiles, based on the number of runs. To have a better overview of the\nperformance of each phase and events, we can aggregate these profiles and\nproduce the median duration of each event across them.\n\nTo run the tool:\n\n```\nbazel run utils:json_profiles_merger \\\n-- \\\n--bazel_source=\u003csome commit or path\u003e \\\n--project_source=\u003csome url or path\u003e \\\n--project_commit=\u003csome_commit\u003e \\\n--output_path=/tmp/outfile.csv \\\n-- /tmp/my_json_profiles_*.profile\n```\n\nYou can pass the pattern that selects the input profiles into the positional\nargument of the script, like in the above example\n(`/tmp/my_json_profiles_*.profile`).\n\n## Output Directory Layout\n\nBy default, bazel-bench will store the measurement results and other required\nfiles (project clones, built binaries, ...) under the `~/.bazel-bench`\ndirectory.\n\nThe layout is:\n\n```\n~/.bazel-bench/                         \u003c= The root of bazel-bench's output dir.\n  bazel/                                \u003c= Where bazel's repository is cloned.\n  bazel-bin/                            \u003c= Where the built bazel binaries are stored.\n    fba9a2c87ee9589d72889caf082f1029/   \u003c= The bazel commit hash.\n      bazel                             \u003c= The actual bazel binary.\n  project-clones/                       \u003c= Where the projects' repositories are cloned.\n    7ffd56a6e4cb724ea575aba15733d113/   \u003c= Each project is stored under a project hash,\n                                           computed from its source.\n  out/                                  \u003c= This is the default output root. But\n                                           the output root can also be set via --data_directory.\n```\n\nTo clear the caches, simply `rm -rf` where necessary.\n\n## Uploading to BigQuery \u0026 Storage\n\nAs an important part of our bazel-bench daily pipeline, we upload the csv output\nfiles to BigQuery and Storage, using separate targets.\n\nTo upload the output to BigQuery \u0026 Storage you'll need the GCP credentials and\nthe table details. Please contact leba@google.com.\n\nBigQuery:\n\n```\nbazel run utils:bigquery_upload \\\n-- \\\n--upload_to_bigquery=\u003cproject_id\u003e:\u003cdataset_id\u003e:\u003ctable_id\u003e:\u003clocation\u003e \\\n-- \\\n\u003cfile1\u003e \u003cfile2\u003e ...\n```\n\nStorage:\n\n```\nbazel run utils:storage_upload \\\n-- \\\n--upload_to_storage=\u003cproject_id\u003e:\u003cbucket_id\u003e:\u003csubdirectory\u003e \\\n-- \\\n\u003cfile1\u003e \u003cfile2\u003e ...\n```\n\n## Performance Report\n\nWe generate a performance report with BazelCI. The generator script can be found\nunder the `/report` directory.\n\nExample Usage: `$ python3 report/generate_report.py --date=2019-01-01\n--project=dummy --storage_bucket=dummy_bucket`\n\nFor more detailed usage information, run: `$ python3 report/generate_report.py\n--help`\n\n## Tests\n\nThe tests for each module are found in the same directory. To run the test,\nsimply:\n\n```\n$ bazel test ...\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbazelbuild%2Fbazel-bench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbazelbuild%2Fbazel-bench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbazelbuild%2Fbazel-bench/lists"}