{"id":13450590,"url":"https://github.com/spotify/bazel-tools","last_synced_at":"2025-03-23T16:31:46.011Z","repository":{"id":66066360,"uuid":"110121462","full_name":"spotify/bazel-tools","owner":"spotify","description":"Tools for dealing with very large Bazel-managed repositories","archived":true,"fork":false,"pushed_at":"2020-02-20T18:53:12.000Z","size":591,"stargazers_count":165,"open_issues_count":2,"forks_count":17,"subscribers_count":16,"default_branch":"master","last_synced_at":"2024-07-31T07:15:43.257Z","etag":null,"topics":["bazel-rules","build-tool"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/spotify.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-11-09T13:52:36.000Z","updated_at":"2024-07-09T08:20:21.000Z","dependencies_parsed_at":null,"dependency_job_id":"656f8e93-5076-4119-8d3d-e30cda72cc28","html_url":"https://github.com/spotify/bazel-tools","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spotify%2Fbazel-tools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spotify%2Fbazel-tools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spotify%2Fbazel-tools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spotify%2Fbazel-tools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/spotify","download_url":"https://codeload.github.com/spotify/bazel-tools/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221856417,"owners_count":16892439,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bazel-rules","build-tool"],"created_at":"2024-07-31T07:00:36.446Z","updated_at":"2024-10-28T16:31:34.846Z","avatar_url":"https://github.com/spotify.png","language":"Java","readme":"# bazel-tools [![Build Status](https://travis-ci.org/spotify/bazel-tools.svg?branch=master)](https://travis-ci.org/spotify/bazel-tools)\n\nThis repository contains a collection of tools for working with Bazel workspaces; mostly tailored\ntowards writing JVM backend services.\n\n  - [depfuzz](depfuzz) - A tool for removing unused dependencies with a fuzzing strategy.\n  - [expand-macros](expand-macros) - A tool for expanding Bazel macros into the rules that they\n    generate.\n  - [format](format) - A tool for formatting all files in the repository according to common style\n    guides.\n  - [unused](unused) - A tool for showing source files that are not used in the build.\n  - [sync-deps](sync-deps) - A tool for synchronizing third-party dependencies.\n  - [sync-repos](sync-repos) - A tool for synchronizing third-party repositories.\n\nThese tools are being used in production at Spotify but have been built for very specific use-cases.\nThey will continue to evolve and cover more use-cases as they mature.\n\n# Usage\n\nUsing these tools has some setup cost, but is hopefully worth it in the long run since it only has\nto be done once for a large/mono-repository.\n\nAdd a `WORKSPACE` dependency on this repository by including the following Skylark rule:\n\n```python\n# current SHA1 of branch master\nspotify_bazel_tools_version=\"\u003cfill in SHA1\u003e\"\nhttp_archive(\n    name = \"spotify_bazel_tools\",\n    sha256 = \"\u003cfill in SHA256\u003e\",\n    strip_prefix = \"bazel-tools-%s\" % spotify_bazel_tools_version,\n    url = \"https://github.com/spotify/bazel-tools/archive/%s.zip\" % spotify_bazel_tools_version,\n)\n```\n\nYou additionally need to add Go and Scala support by adding the official plugins for that, e.g.:\n\n```python\n# current SHA1 of branch master\nrules_go_version=\"\u003cfill in SHA1\u003e\"\nhttp_archive(\n    name = \"io_bazel_rules_go\",\n    sha256 = \"\u003cfill in SHA256\u003e\",\n    strip_prefix = \"rules_go-%s\" % rules_go_version,\n    url = \"https://github.com/bazelbuild/rules_go/archive/%s.zip\" % rules_go_version,\n)\n\n# current SHA1 of branch master\nrules_scala_version=\"\u003cfill in SHA1\u003e\"\nhttp_archive(\n    name = \"io_bazel_rules_scala\",\n    sha256 = \"\u003cfill in SHA256\u003e\",\n    strip_prefix = \"rules_scala-%s\" % rules_scala_version,\n    url = \"https://github.com/bazelbuild/rules_scala/archive/%s.zip\" % rules_scala_version,\n)\n\nload(\"@io_bazel_rules_go//go:deps.bzl\", \"go_rules_dependencies\", \"go_register_toolchains\")\n\ngo_rules_dependencies()\n\ngo_register_toolchains()\n\nload(\"@io_bazel_rules_scala//scala:scala.bzl\", \"scala_repositories\")\n\nscala_repositories()\n```\n\nNow you need to declare various tool dependencies.  If you want to use the pre-packaged ones, use\nthe bundled repository declaration:\n\n```python\nload(\"@spotify_bazel_tools//:tools.bzl\", \"bazel_tools_repositories\")\n\nbazel_tools_repositories()\n```\n\nIf you want to customize the dependencies, you can instead bind your own versions:\n\n```python\nbind(\n    name = \"spotify_bazel_tools/dependency/buildtools/buildifier\",\n    actual = \"@io_bazel_buildtools//buildifier\",\n)\n\nbind(\n    name = \"spotify_bazel_tools/dependency/buildtools/buildozer\",\n    actual = \"@io_bazel_buildtools//buildozer\",\n)\n\nbind(\n    name = \"spotify_bazel_tools/dependency/buildtools/unused_deps\",\n    actual = \"@io_bazel_buildtools//unused_deps\",\n)\n\n# ...and declare your own maven dependencies for Java dependencies, maybe by copy-pasting the\n# 3rdparty/dependencies.yaml file from this repository.\n```\n\nNow you can start leveraging the tools in this repository by creating a wrapper script, e.g.:\n\n```bash\n#!/bin/sh\nexec bazel run @spotify_bazel_tools//sync-deps -- -w \"$PWD\" \"$@\"\n```\n\n...or if you don't want to hold the Bazel lock, but instead want to create a temporary standalone\ntool:\n\n```bash\n#!/bin/sh\nscript=$(mktemp)\nbazel run --script_path=\"$script\" @spotify_bazel_tools//sync-deps\nexec \"$script\" -w \"$PWD\" \"$@\"\n```\n\nHere are guides for the most useful tools:\n\n## Dependency management\n\nThere are a few tools that are useful for managing dependencies of a large repository.  The\ndependency management assumes that you are using the same versions of all external dependencies\nacross the repository.\n\nAll dependencies should be listed in `3rdparty/dependencies.yaml`.  For now only maven-managed\ndependencies are supported.\n\nThe structure of that file is as follows:\n\n```yaml\n# Options concerning the entire dependency graph\noptions:\n  # Resolvers used during dependency resolution (usually Maven servers). Does not need to match your\n  # WORKSPACE maven_servers!\n  mavenResolvers:\n    - id: \"default\"\n      url: https://repo.maven.apache.org/maven2/\n  # The Scala ABI to be used for dependencies with `kind: scala*`\n  scalaAbi: \"2.11\"\n  # Dependencies to completely remove from the dependency graph, because they are not needed.\n  excludedDependencies:\n    - com.google.guava:guava-jdk5\n    - org.slf4j:slf4j-log4j12\n  # Treat the specific versions in this file as \"pinned\" (do not use later versions)\n  # See this for more info: https://github.com/bazelbuild/rules_jvm_external#resolving-user-specified-and-transitive-dependency-version-conflicts\n  versionConflictPolicy: \"pinned\"\n\n# Dependencies to be fetched from GitHub repositories\ngithub:\n  # The Bazel target name used to refer to the repo\n  # repo in combination with either commit, branch, tag or tag+release(+stripPrefix)\n  io_bazel:\n    # The org/repo identifier\n    repo: bazelbuild/bazel\n    # The git commit\n    commit: dbc504c8a033f06041ba42c219b983f475972583\n    # The git branch\n    branch: master\n    # The git tag\n    tag: 2.1.0\n\n    # The release artifact name (has to be used together with a tag)\n    release: bazel-2.1.0-dist.zip\n    # Any top level directory to strip out after extracting the release\n    #stripPrefix: bazel-2.1.0-dist\n\n# Dependencies to be fetched from Maven repositories\nmaven:\n  # The groupId prefix of dependencies to download\n  com.fasterxml.jackson:\n    # The artifactId prefix of dependencies to download\n    jackson:\n      # For Maven artifacts, it's quite common to have a lot of related artifacts with the same\n      # version. We chose not to support the Bill Of Manifests (BOM) pattern, and instead use a more\n      # concise way to list modules.\n      #\n      # The modules block is optional and by default this block simply includes the artifact\n      # groupId:artifactId.\n      modules:\n        # \u003cfoo\u003e:\u003cbar\u003e will append \".\u003cfoo\u003e\" to the groupId and \"-\u003cbar\u003e\" to the artifactId of this\n        # block.  In the end, this becomes \"com.fasterxml.jackson.core:jackson-annotations\"\n        - core:annotations\n        # Only writing \u003cbar\u003e will append \"-\u003cbar\u003e\" to the artifactId.  In the end, this becomes\n        # \"com.fasterxml.jackson:jackson-core\" (which doesn't actually exist).\n        - core\n        # The empty string will not append anything, so the resulting artifact would become\n        # \"com.fasterxml.jackson:jackson\" in this case (which doesn't actually exist).\n        - \"\"\n\n      # The version for all of the artifacts/modules in this group.\n      version: \"2.9.1\"\n      # The dependency kind; can be java, scala, or scala-macro\n      kind: java\n```\n\nTo add dependencies, modify the YAML file and then run the `sync-deps` tool.  This will download all\nof the dependencies, and output a few files:\n\n  - `3rdparty/workspace.bzl` - This file exports a Skylark function for each type of dependency that\n    is being managed.\n\n    For now only Maven dependencies are supported, so there is a function called\n    `maven_dependencies` that is exported.  This function takes a callback argument that will be\n    called for every resolved dependency.  The callback will be called with the following `kwargs`:\n\n      - `artifact` - specifies the Maven coordinates of the artifact as\n        `\u003cgroupId\u003e:\u003cartifactId\u003e:\u003cversion\u003e` etc.\n      - `name` - specifies a name suitable for use in the `WORKSPACE` e.g. for a `maven_jar` rule.\n      - `jar` - specifies the path to the resolved ijar.\n      - `file` - specifies the path to the resolved JAR file.\n      - `bind_jar` - the system expects this to be re-bound to `jar`.\n      - `bind_file` - the system expects this to be re-bound to `file`.\n      - `sha1` - (optional) specifies the SHA1 of the JAR if it is known.\n\n    You are expected to call this function with a callback similar to the following:\n\n    ```python\n    def declare_maven(name, artifact, jar, file, bind_jar, bind_file, sha1=None):\n      if sha1 == None:\n        # You can also fail here, if preferred\n        print(\"%s does not have a sha1 checksum; integrity cannot be verified\" % artifact)\n        native.maven_jar(name=name, artifact=artifact)\n      else:\n        native.maven_jar(name=name, artifact=artifact, sha1=sha1)\n\n      native.bind(name=bind_jar, actual=jar)\n      native.bind(name=bind_file, actual=file)\n    ```\n  - `3rdparty/jvm` - This directory contains `BUILD` files that group together artifacts with their\n    transitive dependencies in an user-friendly manner.  This lets you refer to dependencies as e.g.\n    `//3rdparty/jvm/com/google/guava` in your actual `java_library` dependencies etc.  The directory\n    structure is derived from the Maven `groupId` and the rule names from the Maven `artifactId` in\n    the case of Maven artifacts.\n  - `3rdparty/dependencies.lock` - Contains checksums for verifying the integrity of the dependency\n    tree.  Use `sync-deps --verify` to check that everything is in sync with the\n    `dependencies.yaml`, for example that a developer has added a dependency while forgetting to run\n    `sync-deps`.\n\n# Code of Conduct\n\nThis project adheres to the [Open Code of Conduct][code-of-conduct]. By participating, you are\nexpected to honor this code.\n\n[code-of-conduct]: https://github.com/spotify/code-of-conduct/blob/master/code-of-conduct.md\n","funding_links":[],"categories":["Java","Tooling"],"sub_categories":["General"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspotify%2Fbazel-tools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspotify%2Fbazel-tools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspotify%2Fbazel-tools/lists"}