https://github.com/l3mon4d3/copy-untracked-tarballs
A tool to help preserve non-nixpkgs FOD dependencies.
https://github.com/l3mon4d3/copy-untracked-tarballs
Last synced: about 1 month ago
JSON representation
A tool to help preserve non-nixpkgs FOD dependencies.
- Host: GitHub
- URL: https://github.com/l3mon4d3/copy-untracked-tarballs
- Owner: L3MON4D3
- Created: 2025-05-18T20:39:00.000Z (6 months ago)
- Default Branch: main
- Last Pushed: 2025-07-26T09:38:04.000Z (3 months ago)
- Last Synced: 2025-07-26T15:41:51.346Z (3 months ago)
- Language: Shell
- Size: 7.81 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Motivation
The goal of `copy-untracked-tarballs` is to aid in finding and preserving
non-nixpkgs dependencies a flake or some other nix derivation depends on.
These dependencies are especially crucial because they are usually exempt from
being preserved on `cache.nixos.org` and `tarballs.nixos.org`, so once their
source disappears, they have to be added manually, or the derivation that
depends on them can not be built.
My personal usecase for this are nightly builds of `zig`, which can be
downloaded from the official server using [this flake](https://github.com/mitchellh/zig-overlay),
but since the official server removes non-release builds older than a few weeks,
this quickly leads to (naively) unbuildable derivations.
# Functionality
`copy-untracked-tarballs` requires a nix derivation and a rclone-compatible
destination as inputs. The derivations' dependencies are searched for fixed
output derivations (the results of eg. a `fetchurl`), and those are filtered to
those FODs that are not cached on `cache.nixos.org` or `tarballs.nixos.org`
(this is a useful property because the nix team [seems concerned with keeping
FODs accessible
indefinitely](https://discourse.nixos.org/t/upcoming-garbage-collection-for-cache-nixos-org/39078#garbage-collection-policy-and-implications-2),
so those don't have to be preserved manually).
These FODs are then either downloaded or copied from the local nix store to the
rclone-compatible destination in a format compatible with the content-addressed
tarball-archive at [tarballs.nixos.org](tarballs.nixos.org), which `fetchurl`
can fall back to if all urls are inaccessible.
# Usage
The first argument is a nix derivation, the second the destination which will
contain the tarballs, and the third, optional, argument contains flags
passed to `rclone`.
```bash
copy-untracked-tarballs .#default /srv/http/tarballs
```
or
```bash
copy-untracked-tarballs /run/current-system s3-bucket:/bucketname '-P --config ./rclone.conf'
```
# Using the tarballs
In order to use the archived tarballs in a build, the `impureEnvVar`
`NIX_HASHED_MIRRORS` has to be passed to the `nix build` that builds the
derivation. If the archive exists at `tarballs.myhostname.com`, this looks like
```bash
sudo NIX_HASHED_MIRRORS="tarballs.myhostname.com" nix build ".#default"
```
The sudo is likely necessary because otherwise the build is relegated to the
`nix-daemon`, which does not have `NIX_HASHED_MIRRORS` in its environment. An
alternative, of course, is to modify the service to include this variable.
# Completeness
This tool supports FODs with hash method `nar`, `flat`, and `text` (the latter
of which is handled exactly like `flat`, which seems like the right thing to
do?). `nar` only supports sources that are single archives which can be
extracted with `tar`.