Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/Mic92/nix-fast-build
Combine the power of nix-eval-jobs with nix-output-monitor to speed-up your evaluation and building process.
https://github.com/Mic92/nix-fast-build
build-with-buildbot managed-by-renovate
Last synced: about 2 months ago
JSON representation
Combine the power of nix-eval-jobs with nix-output-monitor to speed-up your evaluation and building process.
- Host: GitHub
- URL: https://github.com/Mic92/nix-fast-build
- Owner: Mic92
- License: mit
- Created: 2023-09-08T08:11:43.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-10-09T07:36:09.000Z (2 months ago)
- Last Synced: 2024-10-11T23:34:43.753Z (2 months ago)
- Topics: build-with-buildbot, managed-by-renovate
- Language: Python
- Homepage:
- Size: 102 KB
- Stars: 231
- Watchers: 3
- Forks: 10
- Open Issues: 7
-
Metadata Files:
- Readme: README.md
- License: LICENSE.md
Awesome Lists containing this project
- stars - Mic92/nix-fast-build - eval-jobs with nix-output-monitor to speed-up your evaluation and building process. (HarmonyOS / Windows Manager)
README
# nix-fast-build 🚀 (previously known as nix-ci-build)
Combine the power of `nix-eval-jobs` with `nix-output-monitor` to speed-up your
evaluation and building process. `nix-fast-build` an also integrates with remote
machines by uploading the current flake, performing the evaluation/build
remotely, and then transferring the resultant store paths back to you.
## Why `nix-fast-build`?
**Problem**: Evaluating and building big flakes i.e. with numerous NixOS
machines can be painfully slow. For instance, rebuilding the already-compiled
[disko integration test suite](https://github.com/nix-community/disko) demands
1:50 minutes on an AMD Ryzen 9 7950X3D. But, it only takes a
[10 seconds](https://github.com/Mic92/nix-fast-build/issues/1) with
`nix-fast-build`.
**Solution**: `nix-fast-build` makes builds faster by evaluating and building
your nix packages concurrently, reducing the overall time.
## How Does It Work?
Under the hood:
1. It leverages the output from `nix-eval-jobs` to evaluate flake attributes in
parallel.
2. As soon as attributes complete evaluation, `nix-fast-build` initiates their
build, even if the overall evaluation is ongoing.
3. Lastly, `nix-output-monitor` to show the build progress nicely.
4. (Optional) Once a build finishes, `nix-fast-build` can initiate its upload to
a designated remote binary cache.
## Usage
To get started, run:
```console
$ nix-fast-build
```
or:
```
$ nix run github:Mic92/nix-fast-build
```
This command will concurrently evaluate all systems in `.#checks` and build the
attributes in `.#checks.$currentSystem`.
---
Enjoy faster and more efficient NixOS builds with `nix-fast-build`!
## Remote building
When leveraging the remote-builder protocol, uploading pre-built paths or
sources from the local machine can often turn into a bottleneck.
`nix-fast-build` does not use the remote-builder protocol. Instead it uploads
only the flake and executes all evaluation/build operations on the remote end.
At the end `nix-fast-build` will download the finished builds to the local
machine while not having to download all build dependencies in between.
Here is how to use it:
```
nix run github:Mic92/nix-fast-build -- --remote youruser@yoursshhostname
```
Replace `youruser@yoursshhostname` with your SSH login credentials for the
target machine. Please note that as of now, you must be recognized as a trusted
user on the remote endpoint to access this feature.
## CI-Friendly Output
By default, `Nix-output-monitor` (abbreviated as `nom`) updates its output every
0.5 seconds. In standard terminal environments, this frequent update is
unnoticeable, as `nom` erases the previous output before displaying the new one.
However, in Continuous Integration (CI) systems, each update appears as a
separate line of output.
To make output more concise for CI environments, use the `--no-nom` flag. This
replaces `nom` with a streamlined status reporter, which updates only when
there's a change in the number of pending builds, uploads, or downloads.
## Avoiding Redundant Package Downloads
By default, `nix build` will download pre-built packages, leading to needless
downloads even when there are no changes to any package. This can be especially
burdensome for CI environments without a persistent Nix store, such as GitHub
Actions.
To optimize this, use the `--skip-cached` flag with `nix-fast-build`. This
ensures that only those packages missing from the binary caches will be built.
## Specifying Build Systems
By default, `nix-fast-build` evaluates all architectures but only initiates
builds for the current system. You can modify this behavior with the `--systems`
flag. For instance, using `--systems "aarch64-linux x86_64-linux"` will prompt
builds for both `aarch64-linux` and `x86_64-linux` architectures. Ensure that
your system is capable of building for the specified architectures, either
locally or through the remote builder protocol.
## Building different flake attributes
`nix-fast-build` by default builds `.#checks.$currentSystem`, which refers to
all checks for the current flake. You can modify this default behavior by using
the `--flake` flag to specify a different attribute path.
Example:
```console
$ nix run github:Mic92/nix-fast-build -- --flake github:NixOS/nixpkgs#legacyPackages.x86_64-linux.hello
```
**Note:** Always provide the complete flake path. Unlike `nix build`,
`nix-fast-build` does not iterate over different attributes; the full path must
be explicitly stated.
## Only evaluate the current system
By default nix-fast-build will evaluate all systems in `.#checks`, you can limit
it to the current system by using this command:
```console
$ nix run github:Mic92/nix-fast-build -- --skip-cached --no-nom --flake ".#checks.$(nix eval --raw --impure --expr builtins.currentSystem)"
```
## Cachix support
nix-fast-build can upload to cachix like this:
```console
$ nix-fast-build --cachix-cache mic92
```
nix-fast-build assumes that your current machine is either logged in to cachix
or has the environment variables `CACHIX_SIGNING_KEY` or `CACHIX_AUTH_TOKEN`
set. These environment variables are currently not propagated to ssh when using
the `--remote` flag, instead the user is expected that cachix credentials are
configured on the remote machine.
## Machine-readable builds results
nix-fast-build supports both its own json format and junit:
Example for json output:
```console
nix-fast-build --result-file result.json
cat ./result.json
{
"results": [
{
"attr": "riscv64-linux.package-default",
"duration": 0.0,
"error": null,
"success": true,
"type": "EVAL"
},
# ...
```
Example for junit result output:
```console
nix-fast-build --result-format junit --result-file result.xml
```
```console
nix-shell -p python3Packages.junit2html --run 'junit2html result.xml result.html'
```
## Reference
```console
usage: nix-fast-build [-h] [-f FLAKE] [-j MAX_JOBS] [--option name value]
[--remote-ssh-option name value]
[--cachix-cache CACHIX_CACHE] [--no-nom]
[--systems SYSTEMS] [--retries RETRIES] [--no-link]
[--out-link OUT_LINK] [--remote REMOTE]
[--always-upload-source] [--no-download] [--skip-cached]
[--copy-to COPY_TO] [--debug]
[--eval-max-memory-size EVAL_MAX_MEMORY_SIZE]
[--eval-workers EVAL_WORKERS]
[--result-file RESULT_FILE]
[--result-format {json,junit}]
options:
-h, --help show this help message and exit
-f FLAKE, --flake FLAKE
Flake url to evaluate/build (default: .#checks
-j MAX_JOBS, --max-jobs MAX_JOBS
Maximum number of build jobs to run in parallel (0 for
unlimited)
--option name value Nix option to set
--remote-ssh-option name value
ssh option when accessing remote
--cachix-cache CACHIX_CACHE
Cachix cache to upload to
--no-nom Don't use nix-output-monitor to print build output
(default: false)
--systems SYSTEMS Space-separated list of systems to build for (default:
current system)
--retries RETRIES Number of times to retry failed builds
--no-link Do not create an out-link for builds (default: false)
--out-link OUT_LINK Name of the out-link for builds (default: result)
--remote REMOTE Remote machine to build on
--always-upload-source
Always upload sources to remote machine. This is
needed if the remote machine cannot access all sources
(default: false)
--no-download Do not download build results from remote machine
--skip-cached Skip builds that are already present in the binary
cache (default: false)
--copy-to COPY_TO Copy build results to the given path (passed to nix
copy, i.e. file:///tmp/cache?compression=none)
--debug debug logging output
--eval-max-memory-size EVAL_MAX_MEMORY_SIZE
Maximum memory size for nix-eval-jobs (in MiB) per
worker. After the limit is reached, the worker is
restarted.
--eval-workers EVAL_WORKERS
Number of evaluation threads spawned
--result-file RESULT_FILE
File to write build results to
--result-format {json,junit}
Format of the build result file
```