{"id":13610483,"url":"https://github.com/berberman/nvfetcher","last_synced_at":"2025-05-15T22:11:53.531Z","repository":{"id":38018700,"uuid":"359679866","full_name":"berberman/nvfetcher","owner":"berberman","description":"Generate nix sources expr for the latest version of packages","archived":false,"fork":false,"pushed_at":"2024-11-25T04:41:30.000Z","size":3933,"stargazers_count":210,"open_issues_count":22,"forks_count":17,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-05-08T11:43:46.708Z","etag":null,"topics":["cli","flakes","haskell-library","nix","nvchecker"],"latest_commit_sha":null,"homepage":"https://nvfetcher.torus.icu","language":"Haskell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/berberman.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-04-20T04:07:58.000Z","updated_at":"2025-04-27T03:33:58.000Z","dependencies_parsed_at":"2024-04-08T02:29:29.014Z","dependency_job_id":"81596c50-798c-474b-9f36-814dcb390b0d","html_url":"https://github.com/berberman/nvfetcher","commit_stats":{"total_commits":168,"total_committers":15,"mean_commits":11.2,"dds":0.1071428571428571,"last_synced_commit":"bdb14eab6fe9cefc29efe01e60c3a3f616d6b62a"},"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/berberman%2Fnvfetcher","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/berberman%2Fnvfetcher/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/berberman%2Fnvfetcher/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/berberman%2Fnvfetcher/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/berberman","download_url":"https://codeload.github.com/berberman/nvfetcher/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254430331,"owners_count":22069909,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","flakes","haskell-library","nix","nvchecker"],"created_at":"2024-08-01T19:01:45.097Z","updated_at":"2025-05-15T22:11:53.480Z","avatar_url":"https://github.com/berberman.png","language":"Haskell","funding_links":[],"categories":["Haskell","nix","Development"],"sub_categories":[],"readme":"# nvfetcher\n\n[![Hackage](https://img.shields.io/hackage/v/nvfetcher.svg?logo=haskell)](https://hackage.haskell.org/package/nvfetcher)\n[![MIT license](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)\n[![nix](https://github.com/berberman/nvfetcher/actions/workflows/nix.yml/badge.svg)](https://github.com/berberman/nvfetcher/actions/workflows/nix.yml)\n\nnvfetcher is a tool to automate nix package updates. It's built on top of [shake](https://www.shakebuild.com/),\nintegrating [nvchecker](https://github.com/lilydjwg/nvchecker).\nnvfetcher cli program accepts a TOML file as config, which defines a set of package sources to run.\n\n## Overview\n\nFor example, feeding the following configuration to`nvfetcher`:\n\n```toml\n# nvfetcher.toml\n[feeluown]\nsrc.pypi = \"feeluown\"\nfetch.pypi = \"feeluown\"\n```\n\nit will create `_sources/generated.nix`:\n\n```nix\n{ fetchgit, fetchurl, fetchFromGitHub }:\n{\n  feeluown = {\n    pname = \"feeluown\";\n    version = \"3.8.2\";\n    src = fetchurl {\n      url = \"https://pypi.io/packages/source/f/feeluown/feeluown-3.8.2.tar.gz\";\n      sha256 = \"sha256-V2yzpkmjRkipZOvQGB2mYRhiiEly6QPrTOMJ7BmyWBQ=\";\n    };\n  };\n}\n```\n\nand `_sources/generated.json`:\n\n```json\n{\n  \"feeluown\": {\n    \"pinned\": false,\n    \"cargoLocks\": null,\n    \"name\": \"feeluown-core\",\n    \"version\": \"3.8.2\",\n    \"passthru\": null,\n    \"src\": {\n      \"url\": \"https://pypi.io/packages/source/f/feeluown/feeluown-3.8.2.tar.gz\",\n      \"name\": null,\n      \"type\": \"url\",\n      \"sha256\": \"sha256-V2yzpkmjRkipZOvQGB2mYRhiiEly6QPrTOMJ7BmyWBQ=\"\n    },\n    \"extract\": null,\n    \"rustGitDeps\": null\n  }\n}\n```\n\nWe tell nvfetcher how to get the latest version number of packages and how to fetch their sources given version numbers,\nand nvfetcher will help us keep their version and prefetched SHA256 sums up-to-date, producing ready-to-use nix expressions in `_sources/generated.nix`.\nNvfetcher reads `generated.json` to produce version change message, such as `feeluown: 3.8.1 → 3.8.2`.\nWe always check versions of packages during each run, but only do prefetch and further operations when needed.\n\n### Live examples\n\nHow to use the generated sources file? Here are several examples:\n\n- My [flakes repo](https://github.com/berberman/flakes)\n\n- Nick Cao's [flakes repo](https://gitlab.com/NickCao/flakes/-/tree/master/pkgs)\n\n## Installation\n\n`nvfetcher` package is available in [nixpkgs](https://github.com/NixOS/nixpkgs), so you can try it with:\n\n```\n$ nix-shell -p nvfetcher\n```\n\nThis repo also has flakes support:\n\n```\n$ nix run github:berberman/nvfetcher\n```\n\nTo use it as a Haskell library, the package is available on [Hackage](https://hackage.haskell.org/package/nvfetcher).\nIf you want to use the Haskell library from flakes, there is also a shell `ghcWithNvfetcher`:\n\n```\n$ nix develop github:berberman/nvfetcher#ghcWithNvfetcher\n$ runghc Main.hs\n```\n\nwhere you can define packages in `Main.hs`. See [Haskell library](#Haskell-library) for details.\n\n## Usage\n\nBasically, there are two ways to use nvfetcher, where the difference is how we provide package sources definitions to it.\n\n### CLI\n\nTo run nvfetcher as a CLI program, you'll need to provide package sources defined in TOML.\n\n```\nUsage: nvfetcher [--version] [--help] [-o|--build-dir DIR] [--commit-changes] \n                 [-l|--changelog FILE] [-j NUM] [-r|--retry NUM] [-t|--timing] \n                 [-v|--verbose] [-f|--filter REGEX] [-k|--keyfile FILE] \n                 [--keep-old] [--keep-going] [TARGET] [-c|--config FILE]\n\n  generate nix sources expr for the latest version of packages\n\nAvailable options:\n  --version                Show version\n  --help                   Show this help text\n  -o,--build-dir DIR       Directory that nvfetcher puts artifacts to\n                           (default: \"_sources\")\n  --commit-changes         `git commit` build dir with version changes as commit\n                           message\n  -l,--changelog FILE      Dump version changes to a file\n  -j NUM                   Number of threads (0: detected number of processors)\n                           (default: 0)\n  -r,--retry NUM           Times to retry of some rules (nvchecker, prefetch,\n                           nix-build, etc.) (default: 3)\n  -t,--timing              Show build time\n  -v,--verbose             Verbose mode\n  -f,--filter REGEX        Regex to filter packages to be updated\n  -k,--keyfile FILE        Nvchecker keyfile\n  --keep-old               Don't remove old files other than generated json and\n                           nix before build\n  --keep-going             Don't stop if some packages failed to be fetched\n  TARGET                   Three targets are available: 1.build 2.clean (remove\n                           all generated files) 3.purge (remove shake db)\n                           (default: build)\n  -c,--config FILE         Path to nvfetcher TOML config\n                           (default: \"nvfetcher.toml\")\n```\n\nEach _package_ corresponds to a TOML table, whose name is encoded as table key, with\ntwo required fields and three optional fields in each table.\nYou can find an example of the configuration file, see [`nvfetcher_example.toml`](nvfetcher_example.toml).\n\n### Keyfile\n\nYou can specify nvchecker keyfile via command line option.\nFor the format of this file, please refer to [nvchecker documentation](https://nvchecker.readthedocs.io/en/latest/usage.html#configuration-table).\n\n#### Nvchecker\n\nVersion source -- how do we track upstream version updates?\n\n- `src.github = owner/repo` - the latest github release\n- `src.github_tag = owner/repo` - the max github tag, usually used with list options (see below)\n- `src.pypi = pypi_name` - the latest pypi release\n- `src.git = git_url` (and an optional `src.branch = git_branch`) - **the latest commit** of a repo\n- `src.archpkg = archlinux_pkg_name` -- the latest version of an archlinux package\n- `src.aur = aur_pkg_name` -- the latest version of an aur package\n- `src.manual = v` -- a fixed version, which never updates\n- `src.repology = project:repo` -- the latest version from repology\n- `src.webpage = web_url` and `src.regex` -- a string in webpage that matches with regex\n- `src.httpheader = request_url` and `src.regex` -- a string in http header that matches with regex\n- `src.openvsx = publisher.ext_name` -- the latest version of a vscode extension from open vsx\n- `src.vsmarketplace = publisher.ext_name` -- the latest version of a vscode extension from vscode marketplace\n- `src.cmd = cmd` -- the version from a shell command (e.g. `echo Meow`)\n- `src.container = owner/name` - the latest tag of a container from the Docker registry\n\nOptional list options for some version sources (`src.github_tag`, `src.webpage`, and `src.httpheader` and `src.container`),\nsee the corresponding [nvchecker documentation](https://nvchecker.readthedocs.io/en/latest/usage.html#list-options) for details.\n\n- `src.include_regex`\n- `src.exclude_regex`\n- `src.sort_version_key`\n- `src.ignored`\n\nOptional global options for all kinds of version sources,\nsee the corresponding [nvchecker documentation](https://nvchecker.readthedocs.io/en/latest/usage.html#global-options) for details. You can tweak obtained version number using this option, e.g. stripping the prefix `v` or transforming the result by regex.\n\n- `src.prefix`\n- `src.from_pattern`\n- `src.to_pattern`\n\n#### Nix fetcher\n\nHow do we fetch the package source if we have the target version number?\n`$ver` is available in string, which will be set to the result of nvchecker.\n\n- `fetch.github = owner/repo`\n- `fetch.pypi = pypi_name`\n- `fetch.git = git_url`\n- `fetch.url = url`\n- `fetch.openvsx = publisher.ext_name`\n- `fetch.vsmarketplace = publisher.ext_name`\n- `fetch.tarball = tarball_url`\n- `fetch.docker = owner/name`\n\nOptional config for `nix-prefetch-url`, applies when the fetcher equals to `fetch.url`.\n`$ver` is available in string, just like for the fetch config.\n\n- `url.name = file_name`\n\nOptional config for `nix-prefetch-git`, applies when the fetcher equals to `fetch.github` or `fetch.git`.\n\n- `git.deepClone`\n- `git.fetchSubmodules`\n- `git.leaveDotGit`\n\nOptional config for `fetch.docker`/`dockerTools.pullImage`:\n\n- `docker.os`\n- `docker.arch`\n- `docker.finalImageName`\n- `docker.finalImageTag`\n- `docker.tlsVerify`\n\n#### Extract src\n\nOptional _extract src_ config, files are extracted into build directory, and then read by `readFile` in generated nix expr.\n\n- `extract = [ \"file_1\", \"file_2\", ...]` - file paths are relative to the source root\n\n#### Rust support\n\n`rustPlatform.buildRustPackage` now accepts an attribute [`cargoLock`](https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/rust.section.md#importing-a-cargolock-file) to vendor dependencies from `Cargo.lock`,\nso we can use this instead of TOFU `cargoSha256` for Rust packaging. `nvfetcher` supports automating this process,\nextracting the lock file to build and calculating `cargoLock.outputHashes`, as long as you set the config.\nThere can be many lock files in one source.\n\n- `cargo_locks = [ \"cargo_lock_path_1\", \"cargo_lock_path_2\", ...]` - relative to the source root\n\n#### Passthru\n\n_passthru_ config, an additional set of attrs to be generated.\n\n- `passthru = { k1 = \"v1\", k2 = \"v2\", ... }`\n\nNote: currently the values can only be strings\n\n#### Pinned\n\nIf a package is pinned, we call nvchecker to check the new version iff there's no existing version.\n\n- `pinned = true`\n\n#### Git commit date\n\nIf the version source of a package is `git`, nvfetcher can finds out the commit date of this revision,\nin the format of `%Y-%m-%d` by default. You can provide your own [`strftime`](https://strftime.org/) format:\n\n- `git.date_format = \"strftime_format\"`\n\n#### Force fetching\n\nAlways fetch the package, even if its version (nvchecker output) doesn't change.\nThis is useful when the file to be downloaded cannot not be determined by the url.\n\n- `fetch.force = true`\n\n\u003e Note: In such case, nvfetcher will not produce a version change log, since the version doesn't change at all.\n\u003e Only sha256 in generated files will be updated.\n\n### Haskell library\n\nnvfetcher itself is a Haskell library as well, whereas the CLI program is just a trivial wrapper of the library.\nYou can create a Haskell program depending on it directly, by using the `runNvFetcher` entry point.\nIn this case, we can define packages in Haskell language, getting rid of TOML constraints.\n\nYou can find an example of using nvfetcher in the library way, see [`Main_example.hs`](Main_example.hs).\n\n## Documentation\n\nFor details of the library, documentation of released versions is available on [Hackage](https://hackage.haskell.org/package/nvfetcher),\nand of master is on our [github pages](https://nvfetcher.berberman.space).\n\n## Contributing\n\nIssues and PRs are always welcome. **\\_(:з」∠)\\_**\n\nBuilding from source:\n\n```\n$ git clone https://github.com/berberman/nvfetcher\n$ nix develop\n$ cabal update\n$ cabal build\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fberberman%2Fnvfetcher","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fberberman%2Fnvfetcher","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fberberman%2Fnvfetcher/lists"}