https://github.com/filmil/bazel_local_nix
Use a bazel-specific nix repository for hermetic builds
https://github.com/filmil/bazel_local_nix
Last synced: about 1 month ago
JSON representation
Use a bazel-specific nix repository for hermetic builds
- Host: GitHub
- URL: https://github.com/filmil/bazel_local_nix
- Owner: filmil
- Created: 2024-04-13T06:15:20.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2025-02-18T04:19:53.000Z (2 months ago)
- Last Synced: 2025-02-18T05:24:11.779Z (2 months ago)
- Language: Shell
- Size: 80.4 MB
- Stars: 8
- Watchers: 2
- Forks: 1
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Hermetic, ephemeral and reproducible ("HER") Bazel builds [](https://github.com/filmil/bazel_local_nix/workflows/Test/badge.svg)
This is an experiment in fully hermetic, but also self-installing [nix][nx]
based hermetic bazel build.
I call it a HER build.
[nx]: https://nixos.org
Sadly it only works on Linux today, since some essential parts are only work
under Linux. So if you are on Windows or Mac, ... or Fuchsia for that matter,
sorry.
## The appeal of using nix in bazel builds
Ostensibly, nix allows you to have a predictable development
environment in which to run bazel.
## The problem with nix
If you are not ready to commit to using nix everywhere, you are stuck.
To elaborate a bit. Thanks to the work by tweag.io, it is possible to use nix
packages within bazel. The build rules at https://github.com/tweag/rules_nixpkgs
allow bazel to bring in dependencies from [nixpkgs][nxp].
But, it requires having a `/nix/store` on
your machine. In turn, that means you need to have a pre-existing system-wide
nix installation.
If you do not, or you can not commit to doing that, then you can not use this.
Which is a bummer.
"Let's remove that problem!", I said one day, thinking it was going to be easy.
(It wasn't, but now it's done.)
[nxp]: https://github.com/NixOS/nixpkgs
## How I try to solve this
This repository goes one step further than what tweag.io has offered us:
with the changes described here, `bazel` will install an **ephemeral**
nix store and pull in the appropriate dependencies from nixpkgs.
This means that
you **do not** need to install nix on your system to use the packages from nixpkgs.
`bazel` will do that for you. Moreover, the resulting nix installation will be
**ephemeral** and will only take effect within your bazel workspace.
Once the ephemeral nix is installed, you can use the tweag rules to bring in
the toolchains that are installed by nix. Since both the bazel and the nix parts
of this setup are reproducible, and since the installation is ephemeral,
you also get a fully reproducible build as well.
And you don't need to stop at tweag-provided rules. You can depend on any binaries
you may need from the nix installation.
## So, what do I get from this exercise?
You get a source code repository that will self-install its development environment
when you attempt to build it for the first time.
This self-installed environment will not pollute your existing system at all. It does
not require `root` privileges to be installed, or to be run.
## Wait, doesn't `bazel` do this already?
Yes, *if* you only ever use dependencies that already build with bazel, or you know how
to bring them in line with bazel's expectations.
However, the real world is different: most dependencies you might want to use are *not*
aware of bazel, which means additional work to make them usable in a bazel build. Worse
yet, some dependencies may be outright hostile to operating within a bazel sandbox, which is its own can of worms.
Nix works around this wrinkle by allowing bazel to use a predictable dev environment
set up by nix.
This repo, in turn, removes the need for nix to be preinstalled on your machine for this
to work.
Taken together, it's a HER build.
## Does this work on remote builds?
I have not tried. I think that it could be made to work. If you are curious to try
making it work for remote builds, do let me know.
## References
See how this is used in the [integration test repo][itr].
[itr]: https://github.com/filmil/bazel_local_nix/tree/main/integration
Read [the article describing the approach][ta].
[ta]: https://hdlfactory.com/post/2024/04/20/nix-bazel-%EF%B8%8F/
I think it is important to note that the contribution of this repository is just in
using a handful of pre-existing tools in what seems to be a novel way, to a useful effect.
## Installation
Installation is done once per repository. Once the ephemeral nix changes have been committed, any new checkouts will use the nix installation automatically.
This means that whoever makes the changes first will need to know a tad bit more
than any later users of the repository.
> **Note**: the setup is not production ready yet. While it should work, many sharp
> corners may still exist. File a bug with a repro case if you want me to take a look
> and perhaps help. Proper documentation is also a bit wanting.
### Add the nix files
The HER configuration depends on a small number of nix files that we place in the
root directory of the repo. (You could also place them elsewhere by changing the obvious things such as references to `//:flake.lock` in the config files below.)
`flake.lock` contains the data used to reproduce the nix installation.
```
{
"nodes": {
"flake-compat": {
// ...
}
},
"root": "root",
"version": 7
}
```
`flake.nix`: creates the ephemeral environment that you want to
use. You want to add to the `packages = [ ... ]` list any
packages that you need in your build. Use the [nixos package
search][nps] to figure out the names of the packages.
```
{
description = "C++ environment using Nix flakes";
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.05";
flake-compat = {
url = "github:edolstra/flake-compat";
flake = false;
};
flake-utils.url = "github:numtide/flake-utils";
};
outputs = { nixpkgs, flake-utils, ... }:
flake-utils.lib.eachDefaultSystem (system:
let
pkgs = import nixpkgs { inherit system; };
in
{
devShells.default = with pkgs; mkShell {
name = "flake-example-shell";
packages = [ nix gcc gnumake bazel_6 ];
};
});
}
```
[nps]: https://search.nixos.org/packages
`shell.nix`:
```
(import
(
let lock = builtins.fromJSON (builtins.readFile ./flake.lock); in
fetchTarball {
url = "https://github.com/edolstra/flake-compat/archive/${lock.nodes.flake-compat.locked.rev}.tar.gz";
sha256 = lock.nodes.flake-compat.locked.narHash;
}
)
{ src = ./.; }
).shellNix
```
### Workspace setup
Add the following to your WORKSPACE file:
```
load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "rules_cc",
integrity = "sha256-IDeHW5pEVtzkp50RKorohbvEqtlo5lh9ym5k86CQDN8=",
urls = [
"https://github.com/bazelbuild/rules_cc/releases/download/0.0.9/rules_cc-0.0.9.tar.gz",
],
strip_prefix = "rules_cc-0.0.9",
)
http_archive(
name = "io_tweag_rules_nixpkgs",
strip_prefix = "rules_nixpkgs-126e9f66b833337be2c35103ce46ab66b4e44799",
urls = ["https://github.com/tweag/rules_nixpkgs/archive/126e9f66b833337be2c35103ce46ab66b4e44799.tar.gz"],
sha256 = "480df4a7777a5e3ee7a755ab38d18ecb3ddb7b2e2435f24ad2037c1b084faa65",
)
load("@io_tweag_rules_nixpkgs//nixpkgs:repositories.bzl", "rules_nixpkgs_dependencies")
rules_nixpkgs_dependencies()
load("@io_tweag_rules_nixpkgs//nixpkgs:nixpkgs.bzl", "nixpkgs_local_repository")
nixpkgs_local_repository(
name = "nixpkgs",
nix_flake_lock_file = "//:flake.lock",
nix_file_deps = ["//:flake.lock"],
)
# Configure the C++ toolchain
load("@io_tweag_rules_nixpkgs//nixpkgs:nixpkgs.bzl", "nixpkgs_cc_configure")
nixpkgs_cc_configure(
name = "nixpkgs_config_cc",
repository = "@nixpkgs",
attribute_path = "clang_13",
)
load("@rules_cc//cc:repositories.bzl", "rules_cc_dependencies", "rules_cc_toolchains")
rules_cc_dependencies()
rules_cc_toolchains()
git_repository(
name = "bazel_local_nix",
remote = "https://github.com/filmil/bazel_local_nix",
commit = "1658ed1563b6862abac349b407234ceee0a57ae0",
)
load("@bazel_local_nix//:repositories.bzl", "bazel_local_nix_dependencies")
bazel_local_nix_dependencies()
```
### Modify your `.bazelrc` to add a nix-specific configuration
Add the following to your `.bazelrc`. If you do not have a file named `.bazelrc` in
the root of your repository, create it.
```
common:nix --host_platform=@rules_nixpkgs_core//platforms:host
common:nix --incompatible_enable_cc_toolchain_resolution
```
This does two things:
1. Sets the host platform to be compatible with the tweag nix tooling. This is essential to allow nix-based toolchains to be used in a build.
2. Turns on the new C++ toolchain resolution rules. These instruct bazel to use `--host_platform` flags instead of `--crosstool_top`. If you still use the latter,
you should stop using the latter.
### Install `//tools/bazel`
Finally, install the script `//tools/bazel`, which bazel will use to wrap itself every time it is invoked:
```
bazel --max_idle_secs=1 run @bazel_local_nix//:install
```
You can now set up the rest of this project. Note that if you are setting
up ephemeral nix for the entire project, you may need to turn off any early
toolchain checks. Place such checks under an env-protected flag, place
it under `//tools/bazel_local_nix.config.sh`. This will be used to bootstrap
installation for everyone else checking out the source.
### Compile
Once all of the above is done and done, the following should work:
```
bazel build --config=nix //...
```
If this works, you are done installing. Any subsequent users will not need
to do anything special, except remember to use `--config=nix` in their build
commands -- or add to their `user.bazelrc` or some such.
Similarly, any continuous integration builds will need this flag. But this is
usually a one-time setup, so I don't expect it to be a challenge.
## Maintenance
### Updating `flake.lock`
The `flake.lock` file can be updated, but this must be done outside of bazel for
now. I suppose we could have a build rule that does that as well.
For the time being, use the following command in the directory where the file
`flake.lock` is located.
```
nix-portable nix flake update
```
## Troubleshooting
### The built binaries can not find shared libraries
The shared libraries paths will be something like
`/nix/store/...`. This means that a binary built inside a HER
build might not work at all outside of the bazel repo.
This is not very useful. What to do?
Fortunately, the nice people at tweag.io have got you covered. Please see their project `clodl` at: https://github.com/tweag/clodl.
This project allows you to build an archive with a transitive
closure of the libraries you need. Check the licensing, though.
I am also not quite sure what you would need to do to make truly
portable packaging. I suspect some `readelf` tricks would be
necessary, but I haven't done the legwork. It is always hard to
provide a self-contained binary when shared libraries are
involved.
### Other
Did you find a bug? Do you have a question?
Make a minimal repro case. Then [file an issue][ff].
[ff]: https://github.com/filmil/bazel_local_nix/issues