{"id":28954622,"url":"https://github.com/polygon/nix-buildproxy","last_synced_at":"2025-10-28T13:32:33.397Z","repository":{"id":223347226,"uuid":"759381038","full_name":"polygon/nix-buildproxy","owner":"polygon","description":"Providing reproducible HTTP/HTTPS responders to builds that just can not live without","archived":false,"fork":false,"pushed_at":"2024-02-29T13:18:06.000Z","size":26,"stargazers_count":107,"open_issues_count":0,"forks_count":2,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-06-23T19:11:32.273Z","etag":null,"topics":["build-tool","cmake","mitmproxy","nix","nixos","proxy"],"latest_commit_sha":null,"homepage":"","language":"Nix","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/polygon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-02-18T12:45:26.000Z","updated_at":"2025-06-01T02:20:08.000Z","dependencies_parsed_at":"2024-02-29T14:45:12.089Z","dependency_job_id":null,"html_url":"https://github.com/polygon/nix-buildproxy","commit_stats":null,"previous_names":["polygon/nix-buildproxy"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/polygon/nix-buildproxy","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/polygon%2Fnix-buildproxy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/polygon%2Fnix-buildproxy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/polygon%2Fnix-buildproxy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/polygon%2Fnix-buildproxy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/polygon","download_url":"https://codeload.github.com/polygon/nix-buildproxy/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/polygon%2Fnix-buildproxy/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265489081,"owners_count":23775247,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["build-tool","cmake","mitmproxy","nix","nixos","proxy"],"created_at":"2025-06-23T19:10:53.346Z","updated_at":"2025-10-28T13:32:33.309Z","avatar_url":"https://github.com/polygon.png","language":"Nix","readme":"# Nix Buildproxy\n\nProviding reproducible HTTP/HTTPS responders to builds that just can not live without.\n\n## Motivation\n\nWhen building Nix packages in the sandbox, internet access is usually not available. However, some packages insist on loading content from the internet. The motivation to build this tool came from a CMake build that loaded additional cmake-files that, in turn, would trigger further package downloads.\n\nUnwilling to go through multi-level patching during the build, I wondered if it's possible to capture, nixify, and later serve HTTP/HTTPS requests from the Nix store to create an escape hatch when the proper solution is just too much effort. Turns out, this is possible, with some caveats.\n\n## Usage\n\nA quick example on how this package works with the included `example/evil_build.sh` example.\n\n### Overlay\n\nThe flake provides an overlay that will make the `buildproxy-capture` package available as well as extend `lib` to contain `lib.mkBuildproxy`. It is recommended to use this overlay. For example, when importing `nixpkgs` in a flake, the overlay is applied as follows:\n\n```nix\n{\n    inputs = {\n        nixpkgs.url = \"github:NixOS/nixpkgs/nixos-unstable\";\n        nix-buildproxy.url = \"github:polygon/nix-buildproxy\";\n    };\n\n    outputs = inputs@{ self, nixpkgs, nix-buildproxy, ... }:\n    let\n        system = \"x64_64-linux\";\n        pkgs = import nixpkgs {\n            inherit system;\n            overlays = [ nix-buildproxy.overlays.default ];\n        };\n    in\n    {\n        ...\n    }\n}\n```\n\n### Capturing requests\n\nBefore starting to bring in `nix-buildproxy`, you should be able to build your project (e.g. in a devShell) and downloads during the build are preventing a proper sandboxed nix build. Run `buildproxy-capture` by either adding the `buildproxy-capture` program to your environment or directly through `nix run github:polygon/nix-buildproxy#buildproxy-capture`. This will launch `mitmproxy` and a subshell that has `HTTPS_PROXY` and `HTTP_PROXY` set. This is fine for CMake, since it respects these variables and does not check certificates. Other build systems might require more convincing.\n\nThen, run your build and exit the subshell when done. This will generate a `proxy_content.nix` file with all the requests.\n\n\u003cdetails\u003e\n\u003csummary\u003eHere is how the session might look like:\u003c/summary\u003e\n\n```bash\nnixbrett ➜ nix/nix-buildproxy/example (main ✗) buildproxy-capture\nEntering proxy capture shell, run your build now, exit shell when done\nnixbrett ➜ nix/nix-buildproxy/example (main ✗) ./evil_build.sh\n  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n100  1237  100  1237    0     0   2811      0 --:--:-- --:--:-- --:--:--  2817\n  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n100   165  100   165    0     0    553      0 --:--:-- --:--:-- --:--:--   551\nnixbrett ➜ nix/nix-buildproxy/example (main ✗) \u003cCtrl+D\u003e\nSaving captured requests to proxy_content.nix\nnixbrett ➜ nix/nix-buildproxy/example (main ✗) cat proxy_content.nix\n{ fetchurl }: [\n  {\n    url = \"https://raw.githubusercontent.com/NixOS/nixpkgs/ba563a6ec1cd6b3b82ecb7787f9ea2cb4b536a1e/pkgs/by-name/he/hello/package.nix\";\n    file = fetchurl {\n      url = \"https://raw.githubusercontent.com/NixOS/nixpkgs/ba563a6ec1cd6b3b82ecb7787f9ea2cb4b536a1e/pkgs/by-name/he/hello/package.nix\";\n      hash = \"sha256-dFkeANLBJW1FWfL0d8ciS4siWP7B4z0vGsj9revgWGw=\";\n    };\n  }\n  {\n    url = \"https://raw.githubusercontent.com/NixOS/nixpkgs/ba563a6ec1cd6b3b82ecb7787f9ea2cb4b536a1e/pkgs/by-name/he/hello/test.nix\";\n    file = fetchurl {\n      url = \"https://raw.githubusercontent.com/NixOS/nixpkgs/ba563a6ec1cd6b3b82ecb7787f9ea2cb4b536a1e/pkgs/by-name/he/hello/test.nix\";\n      hash = \"sha256-fg+tJQ4+U2G/9lqvOnakIJ2VBgKJoteewT2LHUV6sP4=\";\n    };\n  }\n]\n```\n\u003c/details\u003e\n\n### Replaying responses\nIn order to reply to responses, you need to create a buildproxy recipe that serves your `proxy_content.nix`. You can use `lib.mkBuildproxy \u003cpath-to-proxy_content.nix\u003e` for this. To enable the buildproxy in your build, run `source ${buildproxy}` early in your build (before any downloads are attempted, `prePatch` is a good candidate). This will start `mitmproxy` in replay mode and set the `HTTP_PROXY` and `HTTPS_PROXY` variables. A basic scaffold:\n\n```nix\n{ stdenv, lib, ... }:\nlet\n  buildproxy = lib.mkBuildproxy ./proxy_content.nix;\nin\nstdenv.mkDerivation {\n    # ...\n    prePatch = ''\n      source ${buildproxy}\n    '';\n    # ...\n}\n```\n\n## How it works\n\n`nix-buildproxy` uses [mitmproxy](https://mitmproxy.org/) under the hood to do the heavy lifting of providing local proxy functionality. Python addons are used to intercept requests and will either create the proxy content library or serve it. Building the proxy content library works as follows:\n\n```mermaid\nsequenceDiagram\n    participant client\n    participant mitmproxy\n    participant upstream\n    participant inventory\n    client-\u003e\u003emitmproxy: Request\n    mitmproxy-\u003e\u003eupstream: Upstream Request\n    upstream-\u003e\u003emitmproxy: Response\n    mitmproxy-\u003e\u003einventory: Store URL / Hash\n    mitmproxy-\u003e\u003eclient: Response\n```\n\nDuring replay, operation looks like this:\n\n```mermaid\nsequenceDiagram\n    participant client\n    participant mitmproxy\n    participant inventory\n    client-\u003e\u003emitmproxy: Request\n    mitmproxy-\u003e\u003einventory: Lookup\n    inventory-\u003e\u003emitmproxy: Nix Store Path\n     mitmproxy-\u003e\u003eclient: Response\n```\n\n## Compatibility / Challenges\n\nThis package was originally built for and works out of the box with CMake. CMake respects the `HTTP_PROXY/HTTPS_PROXY` environment variables and by default ignores certificate errors. If you are using a different tool, you need to figure out how to configure the proxy server and how to tell the tool to accept the self-signed certificate of `mitmproxy`.\n\n`mitmproxy` will load responses full into memory, I have not yet found out if streaming from/to disk is possible. If this is being used to serve large files, expect RAM usage of at least the file size, possibly several times that.\n\nHTTP redirects are currently not properly handled. The resulting `proxy_content.nix` will contain the original request with a hash for an empty response and the redirected request with the actual hash separately. You can fix this issue by copying the final `sha256` to the entry that redirects to it. This will be properly addressed in a future update.\n\nYou can modify `proxy_content.nix` to deliver different files. To make builds more stable, it is recommended to replace requests to, e.g., the moving `main` branch of a project to a concrete commit hash. Otherwise, future builds might experience checksum failures. This can also be used as an effective patching mechanism but there is currently no support built in.\n\n## Open issues / Roadmap\n\n* [ ] Properly handle HTTP redirects: Undecided whether to replay the redirect or whether to deliver the resulting file immediately, the latter breaking in case the client modifies the request\n* [ ] Properly handle non-success HTTP status codes in general\n* [ ] Framework for patching of downloaded files\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpolygon%2Fnix-buildproxy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpolygon%2Fnix-buildproxy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpolygon%2Fnix-buildproxy/lists"}