{"id":13450642,"url":"https://github.com/Tinder/bazel-diff","last_synced_at":"2025-03-23T16:31:56.169Z","repository":{"id":38173762,"uuid":"295491525","full_name":"Tinder/bazel-diff","owner":"Tinder","description":"Performs Bazel Target Diffing between two revisions in Git, allowing for Test Target Selection and Selective Building","archived":false,"fork":false,"pushed_at":"2024-10-23T16:54:18.000Z","size":778,"stargazers_count":397,"open_issues_count":15,"forks_count":59,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-10-23T22:59:07.023Z","etag":null,"topics":["bazel","target-selection","test-selection"],"latest_commit_sha":null,"homepage":"","language":"Kotlin","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Tinder.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-09-14T17:37:18.000Z","updated_at":"2024-10-23T17:06:30.000Z","dependencies_parsed_at":"2024-03-13T02:23:52.728Z","dependency_job_id":"a28be94d-0f35-4e7b-b387-37e689363bb8","html_url":"https://github.com/Tinder/bazel-diff","commit_stats":null,"previous_names":[],"tags_count":90,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tinder%2Fbazel-diff","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tinder%2Fbazel-diff/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tinder%2Fbazel-diff/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tinder%2Fbazel-diff/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Tinder","download_url":"https://codeload.github.com/Tinder/bazel-diff/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221856421,"owners_count":16892439,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bazel","target-selection","test-selection"],"created_at":"2024-07-31T07:00:36.912Z","updated_at":"2025-03-23T16:31:56.142Z","avatar_url":"https://github.com/Tinder.png","language":"Kotlin","readme":"# bazel-diff\n\n[![Build status](https://github.com/Tinder/bazel-diff/actions/workflows/ci.yaml/badge.svg?branch=master)](https://github.com/Tinder/bazel-diff/actions/workflows/ci.yaml)\n\n`bazel-diff` is a command line tool for Bazel projects that allows users to determine the exact affected set of impacted targets between two Git revisions. Using this set, users can test or build the exact modified set of targets.\n\n`bazel-diff` offers several key advantages over rolling your own target diffing solution\n\n1. `bazel-diff` is designed for very large Bazel projects. We use Java Protobuf's `parseDelimitedFrom` method alongside Bazel Query's `streamed_proto` output option. These two together allow you to parse Gigabyte or larger protobuf messages. We have tested it with projects containing tens of thousands of targets.\n2. We avoid usage of large command line query lists when interacting with Bazel, [issue here](https://github.com/bazelbuild/bazel/issues/8609). When you interact with Bazel with thousands of query parameters you can reach an upper maximum limit, seeing this error `bash: /usr/local/bin/bazel: Argument list too long`. `bazel-diff` is smart enough to avoid these errors.\n3. `bazel-diff` has been tested with file renames, deletions, and modifications. Works on `bzl` files, `WORKSPACE` files, `BUILD` files and regular files\n\nTrack the feature request for target diffing in Bazel [here](https://github.com/bazelbuild/bazel/issues/7962)\n\nThis approach was inspired by the [following BazelConf talk](https://www.youtube.com/watch?v=9Dk7mtIm7_A) by Benjamin Peterson.\n\n\u003e There are simpler and faster ways to approximate the affected set of targets.\n\u003e However an incorrect solution can result in a system you can't trust,\n\u003e because tests could be broken at a commit where you didn't select to run them.\n\u003e Then you can't rely on green-to-red (or red-to-green) transitions and\n\u003e lose much of the value from your CI system as breakages can be discovered\n\u003e later on unrelated commits.\n\n## Prerequisites\n\n* Git\n* Bazel 3.3.0 or higher\n* Java 8 JDK or higher (Bazel requires this)\n\n## Getting Started\n\nTo start using `bazel-diff` immediately, simply clone down the repo and then run the example shell script:\n\n```terminal\ngit clone https://github.com/Tinder/bazel-diff.git\ncd bazel-diff\n./bazel-diff-example.sh WORKSPACE_PATH BAZEL_PATH START_GIT_REVISION END_GIT_REVISION\n```\n\nHere is a breakdown of those arguments:\n\n* `WORKSPACE_PATH`: Path to directory containing your `WORKSPACE` file in your Bazel project.\n  * Note: Your project must use Git for `bazel-diff` to work!\n* `BAZEL_PATH`: Path to your Bazel executable\n* `START_GIT_REVISION`: Starting Git Branch or SHA for your desired commit range\n* `END_GIT_REVISION`: Final Git Branch or SHA for your desired commit range\n\nYou can see the example shell script in action below:\n\n![Demo](demo.gif)\n\nOpen `bazel-diff-example.sh` to see how this is implemented. This is purely an example use-case, but it is a great starting point to using `bazel-diff`.\n\n## How it works\n\n`bazel-diff` works as follows\n\n* The previous revision is checked out, then we run `generate-hashes`. This gives us the hashmap representation for the entire Bazel graph, then we write this JSON to a file.\n\n* Next we checkout the initial revision, then we run `generate-hashes` and write that JSON to a file. Now we have our final hashmap representation for the Bazel graph.\n\n* We run `bazel-diff` on the starting and final JSON hash filepaths to get our impacted set of targets. This impacted set of targets is written to a file.\n\n## Build Graph Distance Metrics\n\n`bazel-diff` can optionally compute build graph distance metrics between two revisions. This is\nuseful for understanding the impact of a change on the build graph. Directly impacted targets are\ntargets that have had their rule attributes or source file dependencies changed. Indirectly impacted\ntargets are that are impacted only due to a change in one of their target dependencies.\n\nFor each target, the following metrics are computed:\n\n* `target_distance`: The number of dependency hops that it takes to get from an impacted target to a directly impacted target.\n* `package_distance`: The number of dependency hops that cross a package boundary to get from an impacted target to a directly impacted target.\n\nBuild graph distance metrics can be used by downstream tools to power features such as:\n\n* Only running sanitizers on impacted tests that are in the same package as a directly impacted target.\n* Only running large-sized tests that are within a few package hops of a directly impacted target.\n* Only running computationally expensive jobs when an impacted target is within a certain distance of a directly impacted target.\n\nTo enable this feature, you must generate a dependency mapping on your final revision when computing hashes, then pass it into the `get-impacted-targets` command.\n\n```bash\ngit checkout BASE_REV\nbazel-diff generate-hashes [...]\n\ngit checkout FINAL_REV\nbazel-diff generate-hashes --depEdgesFile deps.json [...]\n\nbazel-diff get-impacted-targets --depEdgesFile deps.json [...]\n```\n\nThis will produce an impacted targets json list with target label, target distance, and package distance:\n\n```text\n[\n  {\"label\": \"//foo:bar\", \"targetDistance\": 0, \"packageDistance\": 0},\n  {\"label\": \"//foo:baz\", \"targetDistance\": 1, \"packageDistance\": 0},\n  {\"label\": \"//bar:qux\", \"targetDistance\": 1, \"packageDistance\": 1}\n]\n```\n\n## CLI Interface\n\n`bazel-diff` Command\n\n```terminal\nUsage: bazel-diff [-hvV] [COMMAND]\nWrites to a file the impacted targets between two Bazel graph JSON files\n  -h, --help      Show this help message and exit.\n  -v, --verbose   Display query string, missing files and elapsed time\n  -V, --version   Print version information and exit.\nCommands:\n  generate-hashes       Writes to a file the SHA256 hashes for each Bazel\n                          Target in the provided workspace.\n  get-impacted-targets  Command-line utility to analyze the state of the bazel\n                          build graph\n```\n\n### `generate-hashes` command\n\n```terminal\nUsage: bazel-diff generate-hashes [-hkvV] [--[no-]useCquery] [-b=\u003cbazelPath\u003e]\n                                  [--[no-]excludeExternalTargets]\n                                  [--[no-]includeTargetType]\n                                  [--contentHashPath=\u003ccontentHashPath\u003e]\n                                  [-s=\u003cseedFilepaths\u003e] -w=\u003cworkspacePath\u003e\n                                  [-co=\u003cbazelCommandOptions\u003e]...\n                                  [--cqueryCommandOptions=\u003ccqueryCommandOptions\u003e\n                                  ]...\n                                  [--fineGrainedHashExternalRepos=\u003cfineGrainedHa\n                                  shExternalRepos\u003e]...\n                                  [-so=\u003cbazelStartupOptions\u003e]... \u003coutputPath\u003e\nWrites to a file the SHA256 hashes for each Bazel Target in the provided\nworkspace.\n      \u003coutputPath\u003e        The filepath to write the resulting JSON to.\n                            If not specified, the JSON will be written to STDOUT.\n\n                            By default the JSON schema is a dictionary of target =\u003e SHA-256 values.\n                            Example:\n                                 {\n                                    \"//cli:bazel-diff_deploy.jar\":  \"4ae310f8ad2bc728934e3509b6102ca658e828b9cd668f79990e95c6663f9633\"\n                                    ...\n                                 }\n\n                            If --includeTargetType is specified, the JSON schema will include the target type (SourceFile/Rule/GeneratedFile)\n                            Example:\n                                {\n                                  \"//cli:src/test/resources/fixture/integration-test-1.zip\": \"SourceFile#c259eba8539f4c14e4536c61975457c2990e090067893f4a2981e7bb5f4ef4cf\",\n                                  \"//external:android_gmaven_r8\": \"Rule#795f583449a40814c05e1cc5d833002afed8d12bce5b835579c7f139c2462d61\",\n                                  \"//cli:bazel-diff_deploy.jar\": \"GeneratedFile#4ae310f8ad2bc728934e3509b6102ca658e828b9cd668f79990e95c6663f9633\",\n                                  ...\n                                }\n      --[no-]excludeExternalTargets\n                          If true, exclude external targets. This must be\n                            switched on when using `--noenable_workspace` Bazel\n                            command line option. Defaults to `false`.\n      --[no-]includeTargetType\n                          Whether include target type in the generated JSON or not.\n                            If false, the generate JSON schema is: {\"\u003ctarget\u003e\": \"\u003csha256\u003e\"}\n                            If true, the generate JSON schema is: {\"\u003ctarget\u003e\": \"\u003ctype\u003e#\u003csha256\u003e\"\n      -tt, --targetType=\u003ctargetType\u003e\n                          The type of targets to filter, available options are SourceFile/Rule/GeneratedFile\n                          Only works if the JSON was generated with `--includeTargetType` enabled.\n                          If not specified, all types of impacted targets will be returned.\n      -b, --bazelPath=\u003cbazelPath\u003e\n                          Path to Bazel binary. If not specified, the Bazel\n                            binary available in PATH will be used.\n      -co, --bazelCommandOptions=\u003cbazelCommandOptions\u003e\n                          Additional space separated Bazel command options used\n                            when invoking `bazel query`\n      --contentHashPath=\u003ccontentHashPath\u003e\n                          Path to content hash json file. It's a map which maps\n                            relative file path from workspace path to its\n                            content hash. Files in this map will skip content\n                            hashing and use provided value\n      --cqueryCommandOptions=\u003ccqueryCommandOptions\u003e\n                          Additional space separated Bazel command options used\n                            when invoking `bazel cquery`. This flag is has no\n                            effect if `--useCquery`is false.\n      --fineGrainedHashExternalRepos=\u003cfineGrainedHashExternalRepos\u003e\n                          Comma separate list of external repos in which\n                            fine-grained hashes are computed for the targets.\n                            By default, external repos are treated as an opaque\n                            blob. If an external repo is specified here,\n                            bazel-diff instead computes the hash for individual\n                            targets. For example, one wants to specify `@maven`\n                            here if they use rules_jvm_external so that\n                            individual third party dependency change won't\n                            invalidate all targets in the mono repo. Note that\n                            if `--useCquery` is true; or `--useCquery` is false but\n                            `--bazelCommandOptions=--consistent_labels` is provided,\n                            the canonical repo name must be provided,\n                            e.g. `@@maven` or `@@rules_jvm_external~~maven~maven` (bzlmod)\n                            instead of apparent name `@maven`\n  -h, --help              Show this help message and exit.\n      --ignoredRuleHashingAttributes=\u003cignoredRuleHashingAttributes\u003e\n                          Attributes that should be ignored when hashing rule\n                            targets.\n  -k, --[no-]keep_going   This flag controls if `bazel query` will be executed\n                            with the `--keep_going` flag or not. Disabling this\n                            flag allows you to catch configuration issues in\n                            your Bazel graph, but may not work for some Bazel\n                            setups. Defaults to `true`\n  -s, --seed-filepaths=\u003cseedFilepaths\u003e\n                          A text file containing a newline separated list of\n                            filepaths, each of these filepaths will be read and\n                            used as a seed for all targets.\n      -so, --bazelStartupOptions=\u003cbazelStartupOptions\u003e\n                          Additional space separated Bazel client startup\n                            options used when invoking Bazel\n      --[no-]useCquery    If true, use cquery instead of query when generating\n                            dependency graphs. Using cquery would yield more\n                            accurate build graph at the cost of slower query\n                            execution. When this is set, one usually also wants\n                            to set `--cqueryCommandOptions` to specify a\n                            targeting platform. Note that this flag only works\n                            with Bazel 6.2.0 or above because lower versions\n                            does not support `--query_file` flag.\n  -v, --verbose           Display query string, missing files and elapsed time\n  -V, --version           Print version information and exit.\n  -w, --workspacePath=\u003cworkspacePath\u003e\n                          Path to Bazel workspace directory.\n```\n\n**Note**: `--useCquery` flag may not work with very large repos due to limitation\nof Bazel. You may want to fallback to use normal query mode in that case.\nSee \u003chttps://github.com/bazelbuild/bazel/issues/17743\u003e for more details.\n\n### What does the SHA256 value of `generate-hashes` represent?\n\n`generate-hashes` is a canonical SHA256 value representing all attributes and inputs into a target. These inputs\nare the summation of the rule implementation hash, the SHA256 value\nfor every attribute of the rule and then the summation of the SHA256 value for\nall `rule_inputs` using the same exact algorithm. For source_file inputs the\ncontent of the file are converted into a SHA256 value.\n\n### `get-impacted-targets` command\n\n```terminal\nUsage: bazel-diff get-impacted-targets [-v] -fh=\u003cfinalHashesJSONPath\u003e\n                                       -o=\u003coutputPath\u003e\n                                       -tt=\u003ctargetType\u003e\n                                       -sh=\u003cstartingHashesJSONPath\u003e\nCommand-line utility to analyze the state of the bazel build graph\n      -fh, --finalHashes=\u003cfinalHashesJSONPath\u003e\n                  The path to the JSON file of target hashes for the final\n                    revision. Run 'generate-hashes' to get this value.\n      -o, --output=\u003coutputPath\u003e\n                  Filepath to write the impacted Bazel targets to, newline\n                    separated\n      -sh, --startingHashes=\u003cstartingHashesJSONPath\u003e\n                  The path to the JSON file of target hashes for the initial\n                    revision. Run 'generate-hashes' to get this value.\n      -tt, --targetType=\u003ctargetType\u003e\n                  The type of targets to filter, available options are SourceFile/Rule/GeneratedFile\n                  Only works if the JSON was generated with `--includeTargetType` enabled.\n                  If not specified, all types of impacted targets will be returned.\n      -v, --verbose\n                  Display query string, missing files and elapsed time\n```\n\n## Installing\n\n### Integrate into your project (recommended)\n\nFirst, add the following snippet to your project:\n\n#### WORKSPACE Snippet\n\n```bazel\nload(\"@bazel_tools//tools/build_defs/repo:http.bzl\", \"http_jar\")\n\nhttp_jar(\n    name = \"bazel_diff\",\n    urls = [\n        \"https://github.com/Tinder/bazel-diff/releases/download/6.1.0/bazel-diff_deploy.jar\",\n    ],\n    sha256 = \"5d90de4561afd1e711bc62956560a9dfcbb4454bd6b209d6e68272b65c3cb50a\",\n)\n```\n\n#### MODULE.bazel Snippet (if you are using Bzlmod)\n\n```bazel\nhttp_jar = use_repo_rule(\"@bazel_tools//tools/build_defs/repo:http.bzl\", \"http_jar\")\nhttp_jar(\n    name = \"bazel_diff\",\n    urls = [\n        \"https://github.com/Tinder/bazel-diff/releases/download/7.0.0/bazel-diff_deploy.jar\"\n    ],\n    sha256 = \"0b9e32f9c20e570846b083743fe967ae54d13e2a1f7364983e0a7792979442be\",\n)\n```\n\n#### WORKSPACE snippet\n\n```bazel\nhttp_jar = use_repo_rule(\"@bazel_tools//tools/build_defs/repo:http.bzl\", \"http_jar\")\nhttp_jar(\n    name = \"bazel_diff\",\n    urls = [\n        \"https://github.com/Tinder/bazel-diff/releases/download/7.0.0/bazel-diff_deploy.jar\"\n    ],\n    sha256 = \"0b9e32f9c20e570846b083743fe967ae54d13e2a1f7364983e0a7792979442be\",\n)\n\nload(\"//@bazel_diff:repositories.bzl\", \"bazel_diff_dependencies\")\n\nbazel_diff_dependencies()\n```\n\nSecond, add in your root `BUILD.bazel` file:\n\n```bazel\nload(\"@rules_java//java:defs.bzl\", \"java_binary\")\n\njava_binary(\n    name = \"bazel-diff\",\n    main_class = \"com.bazel_diff.Main\",\n    runtime_deps = [\"@bazel_diff//jar\"],\n)\n```\n\nThat's it! You can now run the tool with:\n\n```terminal\nbazel run //:bazel-diff\n```\n\n\u003e Note, in releases prior to 2.0.0 the value for the `main_class` attribute is just `BazelDiff`\n\n### Run Via JAR Release\n\n```terminal\ncurl -Lo bazel-diff.jar https://github.com/Tinder/bazel-diff/releases/latest/download/bazel-diff_deploy.jar\njava -jar bazel-diff.jar -h\n```\n\n### Build from Source\n\nAfter cloning down the repo, you are good to go, Bazel will handle the rest\n\nTo run the project\n\n```terminal\nbazel run :bazel-diff -- bazel-diff -h\n```\n\n#### Debugging (when running from source)\n\nTo run `bazel-diff` with debug logging, run your commands with the `verbose` config like so:\n\n```terminal\nbazel run :bazel-diff --config=verbose -- bazel-diff -h\n```\n\n### Build your own deployable JAR\n\n```terminal\nbazel build //cli:bazel-diff_deploy.jar\njava -jar bazel-bin/cli/bazel-diff_deploy.jar # This JAR can be run anywhere\n```\n\n### Build from source in your Bazel Project\n\nAdd the following to your `WORKSPACE` file to add the external repositories, replacing the `RELEASE_ARCHIVE_URL` with the archive url of the bazel-diff release you wish to depend on:\n\n```bazel\nload(\"@bazel_tools//tools/build_defs/repo:http.bzl\", \"http_archive\")\n\nhttp_archive(\n  name = \"bazel_diff\",\n  urls = [\n        \"RELEASE_ARCHIVE_URL\",\n    ],\n    sha256 = \"UPDATE_ME\",\n    strip_prefix = \"UPDATE_ME\"\n)\n\nload(\"@bazel_diff//:repositories.bzl\", \"bazel_diff_dependencies\")\n\nbazel_diff_dependencies()\n\nload(\"@rules_jvm_external//:defs.bzl\", \"maven_install\")\nload(\"@bazel_diff//:artifacts.bzl\", \"BAZEL_DIFF_MAVEN_ARTIFACTS\")\n\nmaven_install(\n    name = \"bazel_diff_maven\",\n    artifacts = BAZEL_DIFF_MAVEN_ARTIFACTS,\n    repositories = [\n        \"http://uk.maven.org/maven2\",\n        \"https://jcenter.bintray.com/\",\n    ],\n)\n```\n\nNow you can simply run `bazel-diff` from your project:\n\n```terminal\nbazel run @bazel_diff//:bazel-diff -- bazel-diff -h\n```\n\n## Learn More\n\nTake a look at the following bazelcon talks to learn more about `bazel-diff`:\n\n* [BazelCon 2023: Improving CI efficiency with Bazel querying and bazel-diff](https://www.youtube.com/watch?v=QYAbmE_1fSo)\n* [BazelCon 2024: Not Going the Distance: Filtering Tests by Build Graph Distance](https://youtu.be/Or0o0Q7Zc1w?si=nIIkTH6TP-pcPoRx)\n\n## Running the tests\n\nTo run the tests simply run\n\n```terminal\nbazel test //...\n```\n\n## Versioning\n\nWe use [SemVer](http://semver.org/) for versioning. For the versions available,\nsee the [tags on this repository](https://github.com/Tinder/bazel-diff/tags).\n\n## License\n\n---\n\n```text\nCopyright (c) 2020, Match Group, LLC\nAll rights reserved.\n\nRedistribution and use in source and binary forms, with or without\nmodification, are permitted provided that the following conditions are met:\n    * Redistributions of source code must retain the above copyright\n      notice, this list of conditions and the following disclaimer.\n    * Redistributions in binary form must reproduce the above copyright\n      notice, this list of conditions and the following disclaimer in the\n      documentation and/or other materials provided with the distribution.\n    * Neither the name of Match Group, LLC nor the names of its contributors\n      may be used to endorse or promote products derived from this software\n      without specific prior written permission.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\nANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\nWARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\nDISCLAIMED. IN NO EVENT SHALL MATCH GROUP, LLC BE LIABLE FOR ANY\nDIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\n(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\nLOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\nON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\nSOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n```\n","funding_links":[],"categories":["Kotlin","Tooling"],"sub_categories":["General"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FTinder%2Fbazel-diff","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FTinder%2Fbazel-diff","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FTinder%2Fbazel-diff/lists"}