Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/spotify/bazel-tools

Tools for dealing with very large Bazel-managed repositories
https://github.com/spotify/bazel-tools

bazel-rules build-tool

Last synced: about 2 months ago
JSON representation

Tools for dealing with very large Bazel-managed repositories

Awesome Lists containing this project

README 

        
# bazel-tools [![Build Status](https://travis-ci.org/spotify/bazel-tools.svg?branch=master)](https://travis-ci.org/spotify/bazel-tools)



This repository contains a collection of tools for working with Bazel workspaces; mostly tailored

towards writing JVM backend services.



  - [depfuzz](depfuzz) - A tool for removing unused dependencies with a fuzzing strategy.

  - [expand-macros](expand-macros) - A tool for expanding Bazel macros into the rules that they

    generate.

  - [format](format) - A tool for formatting all files in the repository according to common style

    guides.

  - [unused](unused) - A tool for showing source files that are not used in the build.

  - [sync-deps](sync-deps) - A tool for synchronizing third-party dependencies.

  - [sync-repos](sync-repos) - A tool for synchronizing third-party repositories.



These tools are being used in production at Spotify but have been built for very specific use-cases.

They will continue to evolve and cover more use-cases as they mature.



# Usage



Using these tools has some setup cost, but is hopefully worth it in the long run since it only has

to be done once for a large/mono-repository.



Add a `WORKSPACE` dependency on this repository by including the following Skylark rule:



```python

# current SHA1 of branch master

spotify_bazel_tools_version=""

http_archive(

    name = "spotify_bazel_tools",

    sha256 = "",

    strip_prefix = "bazel-tools-%s" % spotify_bazel_tools_version,

    url = "https://github.com/spotify/bazel-tools/archive/%s.zip" % spotify_bazel_tools_version,

)

```



You additionally need to add Go and Scala support by adding the official plugins for that, e.g.:



```python

# current SHA1 of branch master

rules_go_version=""

http_archive(

    name = "io_bazel_rules_go",

    sha256 = "",

    strip_prefix = "rules_go-%s" % rules_go_version,

    url = "https://github.com/bazelbuild/rules_go/archive/%s.zip" % rules_go_version,

)



# current SHA1 of branch master

rules_scala_version=""

http_archive(

    name = "io_bazel_rules_scala",

    sha256 = "",

    strip_prefix = "rules_scala-%s" % rules_scala_version,

    url = "https://github.com/bazelbuild/rules_scala/archive/%s.zip" % rules_scala_version,

)



load("@io_bazel_rules_go//go:deps.bzl", "go_rules_dependencies", "go_register_toolchains")



go_rules_dependencies()



go_register_toolchains()



load("@io_bazel_rules_scala//scala:scala.bzl", "scala_repositories")



scala_repositories()

```



Now you need to declare various tool dependencies.  If you want to use the pre-packaged ones, use

the bundled repository declaration:



```python

load("@spotify_bazel_tools//:tools.bzl", "bazel_tools_repositories")



bazel_tools_repositories()

```



If you want to customize the dependencies, you can instead bind your own versions:



```python

bind(

    name = "spotify_bazel_tools/dependency/buildtools/buildifier",

    actual = "@io_bazel_buildtools//buildifier",

)



bind(

    name = "spotify_bazel_tools/dependency/buildtools/buildozer",

    actual = "@io_bazel_buildtools//buildozer",

)



bind(

    name = "spotify_bazel_tools/dependency/buildtools/unused_deps",

    actual = "@io_bazel_buildtools//unused_deps",

)



# ...and declare your own maven dependencies for Java dependencies, maybe by copy-pasting the

# 3rdparty/dependencies.yaml file from this repository.

```



Now you can start leveraging the tools in this repository by creating a wrapper script, e.g.:



```bash

#!/bin/sh

exec bazel run @spotify_bazel_tools//sync-deps -- -w "$PWD" "$@"

```



...or if you don't want to hold the Bazel lock, but instead want to create a temporary standalone

tool:



```bash

#!/bin/sh

script=$(mktemp)

bazel run --script_path="$script" @spotify_bazel_tools//sync-deps

exec "$script" -w "$PWD" "$@"

```



Here are guides for the most useful tools:



## Dependency management



There are a few tools that are useful for managing dependencies of a large repository.  The

dependency management assumes that you are using the same versions of all external dependencies

across the repository.



All dependencies should be listed in `3rdparty/dependencies.yaml`.  For now only maven-managed

dependencies are supported.



The structure of that file is as follows:



```yaml

# Options concerning the entire dependency graph

options:

  # Resolvers used during dependency resolution (usually Maven servers). Does not need to match your

  # WORKSPACE maven_servers!

  mavenResolvers:

    - id: "default"

      url: https://repo.maven.apache.org/maven2/

  # The Scala ABI to be used for dependencies with `kind: scala*`

  scalaAbi: "2.11"

  # Dependencies to completely remove from the dependency graph, because they are not needed.

  excludedDependencies:

    - com.google.guava:guava-jdk5

    - org.slf4j:slf4j-log4j12

  # Treat the specific versions in this file as "pinned" (do not use later versions)

  # See this for more info: https://github.com/bazelbuild/rules_jvm_external#resolving-user-specified-and-transitive-dependency-version-conflicts

  versionConflictPolicy: "pinned"



# Dependencies to be fetched from GitHub repositories

github:

  # The Bazel target name used to refer to the repo

  # repo in combination with either commit, branch, tag or tag+release(+stripPrefix)

  io_bazel:

    # The org/repo identifier

    repo: bazelbuild/bazel

    # The git commit

    commit: dbc504c8a033f06041ba42c219b983f475972583

    # The git branch

    branch: master

    # The git tag

    tag: 2.1.0



    # The release artifact name (has to be used together with a tag)

    release: bazel-2.1.0-dist.zip

    # Any top level directory to strip out after extracting the release

    #stripPrefix: bazel-2.1.0-dist



# Dependencies to be fetched from Maven repositories

maven:

  # The groupId prefix of dependencies to download

  com.fasterxml.jackson:

    # The artifactId prefix of dependencies to download

    jackson:

      # For Maven artifacts, it's quite common to have a lot of related artifacts with the same

      # version. We chose not to support the Bill Of Manifests (BOM) pattern, and instead use a more

      # concise way to list modules.

      #

      # The modules block is optional and by default this block simply includes the artifact

      # groupId:artifactId.

      modules:

        # : will append "." to the groupId and "-" to the artifactId of this

        # block.  In the end, this becomes "com.fasterxml.jackson.core:jackson-annotations"

        - core:annotations

        # Only writing  will append "-" to the artifactId.  In the end, this becomes

        # "com.fasterxml.jackson:jackson-core" (which doesn't actually exist).

        - core

        # The empty string will not append anything, so the resulting artifact would become

        # "com.fasterxml.jackson:jackson" in this case (which doesn't actually exist).

        - ""



      # The version for all of the artifacts/modules in this group.

      version: "2.9.1"

      # The dependency kind; can be java, scala, or scala-macro

      kind: java

```



To add dependencies, modify the YAML file and then run the `sync-deps` tool.  This will download all

of the dependencies, and output a few files:



  - `3rdparty/workspace.bzl` - This file exports a Skylark function for each type of dependency that

    is being managed.



    For now only Maven dependencies are supported, so there is a function called

    `maven_dependencies` that is exported.  This function takes a callback argument that will be

    called for every resolved dependency.  The callback will be called with the following `kwargs`:



      - `artifact` - specifies the Maven coordinates of the artifact as

        `::` etc.

      - `name` - specifies a name suitable for use in the `WORKSPACE` e.g. for a `maven_jar` rule.

      - `jar` - specifies the path to the resolved ijar.

      - `file` - specifies the path to the resolved JAR file.

      - `bind_jar` - the system expects this to be re-bound to `jar`.

      - `bind_file` - the system expects this to be re-bound to `file`.

      - `sha1` - (optional) specifies the SHA1 of the JAR if it is known.



    You are expected to call this function with a callback similar to the following:



    ```python

    def declare_maven(name, artifact, jar, file, bind_jar, bind_file, sha1=None):

      if sha1 == None:

        # You can also fail here, if preferred

        print("%s does not have a sha1 checksum; integrity cannot be verified" % artifact)

        native.maven_jar(name=name, artifact=artifact)

      else:

        native.maven_jar(name=name, artifact=artifact, sha1=sha1)



      native.bind(name=bind_jar, actual=jar)

      native.bind(name=bind_file, actual=file)

    ```

  - `3rdparty/jvm` - This directory contains `BUILD` files that group together artifacts with their

    transitive dependencies in an user-friendly manner.  This lets you refer to dependencies as e.g.

    `//3rdparty/jvm/com/google/guava` in your actual `java_library` dependencies etc.  The directory

    structure is derived from the Maven `groupId` and the rule names from the Maven `artifactId` in

    the case of Maven artifacts.

  - `3rdparty/dependencies.lock` - Contains checksums for verifying the integrity of the dependency

    tree.  Use `sync-deps --verify` to check that everything is in sync with the

    `dependencies.yaml`, for example that a developer has added a dependency while forgetting to run

    `sync-deps`.



# Code of Conduct



This project adheres to the [Open Code of Conduct][code-of-conduct]. By participating, you are

expected to honor this code.



[code-of-conduct]: https://github.com/spotify/code-of-conduct/blob/master/code-of-conduct.md