{"id":51226832,"url":"https://github.com/numtide/bob","last_synced_at":"2026-06-28T12:31:32.996Z","repository":{"id":353117276,"uuid":"1210465246","full_name":"numtide/bob","owner":"numtide","description":"Bob the Builder — fast incremental builds on top of Nix","archived":false,"fork":false,"pushed_at":"2026-04-22T15:14:31.000Z","size":259,"stargazers_count":2,"open_issues_count":1,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-22T15:31:00.306Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/numtide.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-14T12:51:37.000Z","updated_at":"2026-04-22T15:14:34.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/numtide/bob","commit_stats":null,"previous_names":["numtide/bob"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/numtide/bob","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/numtide%2Fbob","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/numtide%2Fbob/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/numtide%2Fbob/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/numtide%2Fbob/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/numtide","download_url":"https://codeload.github.com/numtide/bob/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/numtide%2Fbob/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34889047,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-28T02:00:05.809Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-28T12:31:32.089Z","updated_at":"2026-06-28T12:31:32.990Z","avatar_url":"https://github.com/numtide.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Bob the Builder\n\nFast incremental builds on top of Nix. Replays fine-grained `buildRustCrate` derivations outside the Nix sandbox with a content-addressed artifact cache, persistent stdenv workers, and rustc incremental compilation.\n\n**Status: experimental.** Currently targets Rust workspaces built via [cargo-nix-plugin] / `buildRustCrate`, and C/C++ projects built with cmake or meson. The core (drv parser, scheduler, cache, path rewriter) is language-agnostic; other backends (Go via go2nix) are planned.\n\n[cargo-nix-plugin]: https://github.com/Mic92/cargo-nix-plugin\n\n## How it works\n\n1. **Resolve** — translate a workspace member name to a `.drv` path via `nix-instantiate` (cached on `Cargo.lock` hash)\n2. **Graph** — parse `.drv` files directly (ATerm) to build the crate dependency DAG\n3. **Build** — replay each crate's configure/build/install phases in parallel, in persistent bash workers with `$stdenv/setup` pre-sourced; non-crate inputs (toolchain, C libs, fetchers) are realised once via `nix-store --realise`\n4. **Cache** — registry/untracked units key on `blake3(drv_path)` (Nix has already hashed all their inputs); workspace units key on `blake3(own_src ‖ dep_output_hashes)` so a rebuild that produces an identical artifact doesn't move dependents' keys (see [Early cutoff](#early-cutoff))\n5. **Pipeline** — a `rustc` wrapper emits `metadata,link`, signals `__META_READY__` on fd 3 once the fat `.rmeta` exists, and the scheduler unblocks dependents before codegen finishes (cargo-style pipelining)\n\nOn repeat builds only changed crates rebuild; `-C incremental` makes each rebuild fast, and early cutoff stops the rebuild from cascading past the point where outputs actually differ.\n\n## Early cutoff\n\nCargo's freshness check is *input-mtime*: edit a deep crate → its `.rmeta` mtime bumps → every reverse-dep's check fails → rustc runs on each → their mtimes bump → all transitive revdeps rebuild. `-C incremental` makes each call cheap, but you still pay one rustc spawn per revdep, plus the leaf relinks.\n\nbob's tracked-unit cache key is *output-addressed*: `eff(c) = blake3(own_src(c) ‖ prop(d) for tracked d ∈ deps(c))`, where `prop(d)` is the hash of `d`'s **built output**, not its inputs. The scheduler computes `eff(c)` at the moment `c` becomes ready (once each `prop(d)` is known), and if `artifacts/\u003ceff(c)\u003e/` exists `c` is skipped entirely.\n\nFor an edit at the bottom of a 20-deep revdep chain:\n\n1. The edited crate rebuilds.\n2. Its rmeta is hashed. If the public interface didn't change (comment, private body, formatting), the rmeta is byte-identical → every lib dependent's `eff` key is unchanged → all 19 intermediate crates cache-hit without spawning rustc.\n3. The leaf cdylib/bin re-links (its key folds in the edited crate's *rlib* bytes, which did change).\n\nIf the edit *does* change the interface, the cascade runs until rmeta stabilises — typically one or two layers, not the full reachable set.\n\n### Two-tier propagation\n\n`prop(d)` is per-edge:\n\n- **lib→lib** uses `early_hash(d)` = `blake3(rmeta)`, taken at `__META_READY__`. rmeta is rustc's interface artifact and is byte-stable for unchanged inputs even under `-C incremental`, so cutoff fires for non-interface edits *and* the edge stays early-gated (pipelining preserved).\n- **→link** (cdylib/staticlib/bin/proc-macro) uses `out_hash(d)` = `blake3(full output)`, taken at commit. rlibs are *not* byte-stable across `-C incremental` session states, so keying the link on rmeta would be unsound — a stale `.so` could be served against a changed rlib. These edges are done-gated.\n\ncc units have no early signal yet, so cc→anything is done-gated on `out_hash`.\n\n### Trade-offs\n\n- **Hash on the critical path.** Each built unit's rmeta and full output are blake3'd before dependents can compute their key. ~3 GB/s; tens of ms on fat rlibs.\n- **Relies on rmeta determinism.** rustc gives no stability guarantee for `.rmeta`. Today it's byte-stable for equal inputs; if a future rustc embeds a nonce, lib→lib cutoff stops firing. The result is *slow*, not *wrong* (dependents rebuild and `-C incremental` does the work).\n- **Link targets always rebuild if any transitive rlib did.** rlibs aren't reproducible under `-C incremental`, so every leaf bin/cdylib re-links whenever anything upstream rebuilt. One fat `.so` is fine; many leaf binaries pay this per leaf.\n- **Precise invalidation = precise input model.** Cargo's blanket rebuild masks build scripts that read untracked state. `eff(c)` covers own sources, dep outputs, and the drv env (which already hashes declared `buildInputs`/flags); it does **not** cover ambient env a `build.rs` reads via `cargo:rerun-if-env-changed` — see [When to invalidate](#when-to-invalidate-manually).\n- **No sandbox, no remote.** Replay runs in your worktree with your env; out-hashes aren't portable across machines, and outputs aren't store-registered. This is a dev-loop accelerator; `nix build` stays the source of truth.\n\n## Setup\n\nbob needs two things from the target repo:\n\n1. **Per-crate derivations** — a Cargo workspace wired through cargo-nix-plugin's `buildRustCrate`, so each crate is its own `.drv`.\n2. **A `bob.nix` at the repo root** with one top-level attr per backend. The Rust backend reads `rust.workspaceMembers.\u003cname\u003e.build`:\n\n   ```nix\n   # bob.nix\n   { pkgs ? import \u003cnixpkgs\u003e {} }:\n   let\n     cargoNix = pkgs.callPackage ./Cargo.nix {};  # or however your repo wires cargo-nix-plugin\n   in {\n     rust = { inherit (cargoNix) workspaceMembers; };\n   }\n   ```\n\nIf your `bob.nix` needs `builtins.resolveCargoWorkspace`, point bob at a patched `nix-instantiate`:\n\n```bash\nexport BOB_NIX_INSTANTIATE=/path/to/patched/nix-instantiate\n```\n\n### Eval-cache invalidation\n\nbob caches the `nix-instantiate` result so the ~1–2s eval is paid once, not per build. The cache key always covers `bob.nix` and `Cargo.lock`. If `bob.nix` imports other files (crate overrides, `flake.lock` for pins), declare them so edits invalidate the cache — either in `Cargo.toml`:\n\n```toml\n[workspace.metadata.bob]\neval-inputs = [\"flake.lock\", \"nix/overrides/*.nix\"]\n```\n\nor, if you can't put bob config into the upstream manifest, in a `bob.toml` next to `bob.nix`:\n\n```toml\neval-inputs = [\"flake.lock\", \"nix/overrides/*.nix\"]\n```\n\nBoth lists are additive. Globs use `*`/`?`/`[…]`/`**`; `*` matches within a single path component (so `nix/*.nix` matches `nix/a.nix` but not `nix/sub/b.nix`), `**` recurses (`nix/**/*.nix` matches both).\n\n## Commands\n\n```bash\nbob build \u003cname\u003e             # build a workspace member\nbob build .                  # auto-detect from nearest Cargo.toml\nbob build /nix/store/….drv   # raw drv path (skips resolve)\nbob clean [--all|\u003cname\u003e]     # drop cached artifacts\nbob status                   # cache stats\nbob graph \u003cdrv\u003e              # print dependency DAG\n```\n\nOptions: `-j N` (jobs, default nproc), `--repo-root \u003cpath\u003e` (default: walk up to `bob.nix`, or `$BOB_REPO_ROOT`), `-o/--out-link \u003cpath\u003e` (result symlink prefix, default `result`), `--no-out-link`, `--print-out-paths` (artifact paths on stdout).\n\nResult symlinks follow nix-build: `result` → `$out`, `result-lib` → `$lib`; for multiple targets the second and onward get `-2`, `-3`, … suffixes.\n\n## Cache\n\nAll state lives under `$XDG_CACHE_HOME/bob/`:\n\n- `artifacts/\u003ckey\u003e/{out,lib,.out-hash,.early-hash}` — committed outputs plus the propagated hashes dependents key on. `\u003ckey\u003e` is `blake3(drv_path)` for untracked units, `eff(c)` for tracked ones (so a tracked unit accumulates one entry per distinct source state it's been built at).\n- `incremental/\u003cblake3(drv_path)\u003e/` — rustc `-C incremental` session / cc build dir. Drv-path-keyed so source edits reuse it; toolchain/flag changes (which move the drv path) cold-start it.\n- `tmp/\u003cblake3(drv_path)\u003e/` — in-flight `$out`. Drv-path-keyed (not eff-keyed) so `$out` is stable across source edits — cmake/pkg-config/rpaths embed it, and `-C incremental`'s session inputs include it.\n- `eval/` — `nix-instantiate` results, keyed on `bob.nix` + lockfile + `eval-inputs`.\n- `rmeta/`, `build/` — in-flight pipelining state.\n\n### When to invalidate manually\n\nIn normal use, never: source edits change `own_src` → new `eff` key; dep edits change `prop(d)` → new `eff` key; toolchain/flag/override changes change the drv path → new key for both tracked and untracked units *and* a fresh `incremental/` dir.\n\nThe cases that need a manual `bob clean`:\n\n- **`build.rs` reads ambient state.** `cargo:rerun-if-env-changed=FOO` where `FOO` comes from your shell, not the drv env. Change `FOO` → bob serves the old artifact. `bob clean \u003ccrate\u003e` (drops its incremental dir; next build re-runs `build.rs`) or set `FOO` via a crate override so it lands in the drv env and keys correctly.\n- **Non-hermetic cc unit.** A `CMakeLists.txt` that does `find_package` against a system path, or reads an env var the drv doesn't set. Same remedy.\n- **`-C incremental` corruption.** Rare rustc bug where the session state produces bad codegen after certain edits; symptoms are link errors or wrong behaviour that `nix build` doesn't reproduce. `bob clean \u003ccrate\u003e` or `bob clean --incremental`.\n- **Disk pressure.** `artifacts/` grows by one entry per (tracked unit × distinct source state). `bob clean --all`.\n\nWhat the subcommands actually remove:\n\n| | `artifacts/` | `incremental/` | `eval/` |\n|---|:---:|:---:|:---:|\n| `bob clean \u003cmember\u003e` | only the drv-keyed entry¹ | that member's | — |\n| `bob clean --incremental` | — | all | — |\n| `bob clean --all` | all | all | — |\n\n¹ Tracked units' eff-keyed `artifacts/` entries aren't individually addressable (there's one per source-hash, and the name→key mapping needs the source). They're harmless to keep; use `--all` to reclaim disk. The `eval/` cache self-invalidates on `bob.nix`/lockfile/`eval-inputs` changes; `rm -rf ~/.cache/bob/eval` if you need to force a re-instantiate without touching those.\n\n## Crate layout\n\n```\ncrates/\n├── core/   bob-core  — language-agnostic .drv replay engine: ATerm parser,\n│                     unit DAG, content-addressed cache, path rewriter,\n│                     persistent stdenv workers, .attrs.{json,sh} emission,\n│                     two-tier (early-signal/done) scheduler, Backend trait\n├── rust/   bob-rust  — Rust backend: buildRustCrate/cargo-nix-plugin drvs,\n│                     rmeta pipelining via the __rustc-wrap shim,\n│                     -C incremental injection, Cargo workspace introspection\n├── cc/     bob-cc    — C/C++ backend: cmake/meson stdenv drvs marked via\n│                     lib/cc.nix, persistent out-of-tree build dir for\n│                     ninja-level per-TU incrementality (no pipelining yet)\n└── cli/    bob       — the binary; registers backends and wires the CLI\n```\n\n## Adding a backend\n\nImplement `bob_core::Backend` in a new `crates/\u003clang\u003e/` crate and append it\nto `BACKENDS` in `crates/cli/src/main.rs`. The minimum is:\n\n- `is_unit(drv)` — e.g. `drv.env.contains_key(\"goPackagePath\")`\n- `unit_name(drv)` — progress display\n- `resolve_attr(target, root)` — attr path under `(import bob.nix {}).\u003cid()\u003e`\n- `lock_hash(root)` — e.g. `blake3(go.sum)`\n- `build_script_hooks(ctx)` — e.g. `export GOCACHE=…`\n- `output_populated(tmp, drv)`\n\n`pipeline()` and `dispatch_internal()` default to no-ops; backends without\nan early-artifact analogue (Go) get correct done-gated scheduling for free.\nA `core-leakage` flake check enforces that `bob-core` stays free of\nbackend-specific identifiers.\n\n## C/C++ backend\n\nA cc unit is a plain `stdenv.mkDerivation` (cmake or meson, out-of-tree)\ndeclared in `bob.nix`:\n\n```nix\n# bob.nix\nlet bobCc = import \"${bob}/lib/cc.nix\"; in\n{\n  rust = { inherit (cargoNix) workspaceMembers; };\n  cc = bobCc.units {\n    libfoo = { drv = pkgs.libfoo; src = \"path/to/libfoo\"; };\n  };\n}\n```\n\n`bobCc.unit` attaches `bobCcSrc` as a Nix-level attribute (`drv // { … }`),\nso `drvPath` is **unchanged** — if `pkgs.libfoo` also appears in some Rust\ncrate's `buildInputs`, bob's graph walk from a Rust root finds the same drv\nas a unit and a C edit cascades through to the `.so`. The cc backend\nevaluates `(import bob.nix {}).cc` once to get the drvPath→src map; nothing\nis written into the drv env.\n\n`bob build libfoo` keeps a drv-path-keyed build directory under\n`~/.cache/bob/incremental/` so reconfigure is warm and `ninja` rebuilds only\nthe TUs whose `.d` depfiles changed. The drv still `nix build`s normally —\n`dontUnpack`/`cmakeBuildDir` are injected only at replay time.\n\nCaveats: unpack/patch are skipped (the build runs against the live worktree),\nso patched derivations are not supported; cc edges are done-gated (no early\nsignal yet — see `crates/cc/src/lib.rs` for what's needed).\n\n## Limitations\n\n- Outputs are not registered in the Nix store — downstream Nix consumers can't use them. Use `nix-build` for that.\n- No file watcher; re-run `bob build` after edits.\n- No `buildTests = true` support yet.\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnumtide%2Fbob","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnumtide%2Fbob","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnumtide%2Fbob/lists"}