{"id":16956374,"url":"https://github.com/bgamari/build-env","last_synced_at":"2025-03-22T13:31:59.214Z","repository":{"id":64611695,"uuid":"570589082","full_name":"bgamari/build-env","owner":"bgamari","description":null,"archived":false,"fork":false,"pushed_at":"2024-08-23T11:23:51.000Z","size":267,"stargazers_count":4,"open_issues_count":4,"forks_count":1,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-18T11:48:14.853Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Haskell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bgamari.png","metadata":{"files":{"readme":"readme.md","changelog":"changelog.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-11-25T14:56:44.000Z","updated_at":"2025-01-07T02:27:33.000Z","dependencies_parsed_at":"2024-10-28T13:20:26.417Z","dependency_job_id":"ac5127c4-ca1c-43f5-b35d-c55633544158","html_url":"https://github.com/bgamari/build-env","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bgamari%2Fbuild-env","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bgamari%2Fbuild-env/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bgamari%2Fbuild-env/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bgamari%2Fbuild-env/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bgamari","download_url":"https://codeload.github.com/bgamari/build-env/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244962937,"owners_count":20539247,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T22:14:46.589Z","updated_at":"2025-03-22T13:31:58.921Z","avatar_url":"https://github.com/bgamari.png","language":"Haskell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# build-env \u003ca href=\"https://hackage.haskell.org/package/build-env\" alt=\"Hackage\"\u003e\u003cimg src=\"https://img.shields.io/hackage/v/build-env.svg\" /\u003e\u003c/a\u003e\n\n**`build-env`** is a utility to build a set of Cabal packages (computed\nby `cabal-install`'s solver) into a free-standing package database.\n\nThis enables the compilation of Haskell packages (with their dependencies)\nin hermetic build environments.\n\n---\n\n### Contents\n\n- [Example](#example)\n- [Commands](#commands)\n  - [Parallelism](#parallelism)\n- [What does `build-env` do?](#what-does-build-env-do)\n- [Hermetic builds](#hermetic-builds)\n  - [Narrowing a build down (for debugging)](#narrowing-a-build-down-for-debugging)\n- [Bootstrapping](#bootstrapping)\n  - [Bootstrap arguments](#bootstrap-arguments)\n  - [Ninja](#ninja)\n- [Specifying packages](#specifying-packages)\n  - [Local packages](#local-packages)\n\n---\n\n## Example\n\n```\n$ build-env build lens -f sources -o install -j8\n$ ghci -package-db install/package.conf -package lens\nghci\u003e :ty Control.Lens.Lens\nControl.Lens.Lens\n  :: Control.Lens.Type.Lens s t a b\n     -\u003e Control.Lens.Reified.ReifiedLens s t a b\n```\n\nIn this example, the `build-env build` invocation:\n\n  - computes a build plan for `lens`,\n  - fetches `lens` and its dependencies into the `sources` directory,\n  - configures and builds all the packages, with up to `8` units building\n    concurrently, and registers the libraries into the package database at\n    `install/package.conf`.\n\nWe then tell `ghci` to use this package database, making `lens` available.\n\n## Commands\n\n`build-env` has three distinct modes of operation:\n\n  - `plan` computes a build plan, outputting a `plan.json` Cabal plan.\n  - `fetch` fetches sources.\n  - `build` executes a build plan.\n\nEach command subsumes the functionality of the previous ones in the above list.\nIn particular, in the previous example, the `build` command computed a build\nplan, fetched the sources, and built the plan. The same could have been achieved\nwith three separate invocations:\n\n```\n$ build-env plan lens -p lens-plan.json\n$ build-env fetch -p lens-plan.json -f sources\n$ build-env build -p lens-plan.json -f sources -o install -j8 --prefetched\n```\n\nBeing able to separate these steps affords us some extra flexibility, as\nsubsequent sections will explain.\n\n__Note__: it is preferable to pass a previously computed build plan when calling\n`build-env build --prefetched`: an invocation of the form `build-env build a b c --prefetched`\nwill compute a new build plan, and this build plan could be different from\nthe plan that was previously fetched, for example if you have run `cabal update`\nin the meantime.  \nAnother option is to pass `--index-state`, which pins down a specific Hackage\nindex state.\n\n### Parallelism\n\n`build-env` supports parallel builds in much the same way as `cabal-install` does:\n\n  - `-jN` controls how many packages are built concurrently.\n  - `--configure-arg --ghc-option=-jN` controls the parallelism afforded to\n    GHC invocations when building individual packages.\n  - `--jsem N` allows `build-env` and GHC to coordinate usage of CPU resources.  \n    This is usually the fastest, but it requires:\n      - GHC \u003e= 9.8,\n      - that GHC and `build-env` have been compiled against the same libc implementation\n    (see [GHC issue #25087](https://gitlab.haskell.org/ghc/ghc/-/issues/25087)).\n\n## What does `build-env` do?\n\n- **plan:** `build-env` calls `cabal build --dry-run` on a dummy\n  project with the dependencies and constraints that were asked for.  \n  This produces a Cabal plan in the form of a `plan.json`, which `build-env`\n  parses.  \n  **Required executables:** `cabal`, `ghc`.\n- **fetch:** `build-env` calls `cabal get` on each package in the build plan.  \n  **Required executables:** `cabal`.\n- **build:** `build-env` builds the dependencies by compiling their `Setup`\n  scripts and calling `Setup configure`, `Setup build`,\n  `Setup haddock` (optional), `Setup copy`, and, for libraries, `Setup register`\n  and `ghc-pkg register`.  \n  **Required executables:** `ghc`, `ghc-pkg`.\n\nNote that, when building packages, `build-env` passes the following information\nwhen running the `Setup` script:\n\n  - `with-compiler`, `prefix` and `destdir` options supplied by the user,\n  - `cid`, `datadir`, `datasubdir`, `bindir` and `builddir` options supplied by `build-env`,\n  - package flags and `dependency` arguments, obtained from the build plan,\n  - `\u003cpkg\u003e_datadir` environment variables supplied by `build-env`.\n\nThese options cannot be overridden. If you absolutely need to change some of\nthese, please file a bug on the issue tracker asking for this customisability.\n\n## Hermetic builds\n\nWhen working in a hermetic build environment, we might need to compute a build\nplan and fetch all the dependencies first, before doing an offline build.\n\nThis can be achieved by two separate `build-env` invocations.\n\nIn a local environment (with access to internet):\n\n```\n$ build-env fetch lens -f sources --output-plan plan.json\n```\n\nIn a build environment in which `build-env` has been provisioned:\n\n```\n$ build-env build --prefetched -p plan.json -f sources -o install -j8\n```\n\nIn this example:\n\n  - the `fetch` command computes a plan which is written to\n    `plan.json` (using the Cabal JSON plan format), and fetches all\n    the required sources, putting them into the `sources` directory;\n  - the `build` command reads in the build plan and performs the build.\n\n### Narrowing a build down (for debugging)\n\nWhen attempting to execute a large build plan, it can be useful to be able\nto refine the build down to a small set of problematic packages, for debugging\npurposes. For this, you can use `--only`, e.g.:\n\n```\n$ build-env build --prefetched -p plan.json -f sources -o install --only badPackage\n```\n\nInstead of building the full plan from `plan.json`, this will instead restrict\nto only building units from package `badPackage` and their transitive dependencies.  \nMultiple `--only` arguments combine additively, e.g. `--only pkg1 --only pkg2`\nwill build the units from the build plan that belong to `pkg1` or `pkg2`, and\ntransitive dependencies thereof.\n\nNote that `--only` only affects which packages are built; it does not change\nthe computation of the build plan nor which packages are fetched.\n\nYou can also resume a build where it left off by passing `--resume`; this will\navoid rebuilding any of the units that have already been registered in the\npackage database.\n\n## Bootstrapping\n\nIn the example from [§ Hermetic builds](#hermetic-builds), we ran\n`build-env build` to perform a build. However, this requires that the build\nenvironment provision `build-env`.\n\nTo avoid this requirement, it is also possible to ask `build-env` to output\na shell script containing the build steps to execute.\n\nIn a local environment (with access to internet):\n\n```\n$ build-env build lens -f sources -o install --script build_lens.sh\n```\n\nIn a build environment:\n\n```\n$ ./build_lens.sh\n```\n\nThe downside is that we lose any parallelism from this approach, as the\ngenerated build script is inherently sequential.\n\nOne particularly useful application is the bootstrapping of `build-env` itself,\nas this supplies a mechanism for the provisioning of `build-env` in build\nenvironments.\n\n### Bootstrap arguments\n\nIf your build environment provides additional variables that need to be passed\nto the build script, you can pass them to the shell script using variables.\n\nFor example, in the local environment, you can first obtain all the information\nneeded to return a build script:\n\n```\n$ build-env fetch lens -f sources --output-plan myPlan.json\n```\n\nNext, compute a build script containing references to variables:\n\n```\n$ build-env build -p myPlan.json -f sources -o install --prefetched --configure-arg \\$myArgs --script script.sh\n```\n\nThis will output a script which passes `$myArgs` to each `Setup configure`\ninvocation. In the build environment, you can then set the value of `$myArgs`\nbefore running the shell script `script.sh`. Note that the build script\n__does not__ insert additional quotation marks around `$myArgs`, which allows\npassing multiple arguments at once (this is crucial when one doesn't know\nthe number of arguments ahead of time).\n\nIf you want to set `--fetchdir`, `--prefix` or `--destdir` to variables,\nyou should pass the `--variables` flag. This will set these to the values\n`$SOURCES`, `$PREFIX` and `$DESTDIR` in the output shell script, respectively.\nThese should be set to absolute paths before running the script.  \nThe script will also use `$GHC` and `$GHCPKG` variables for the compiler.  \n\n### Ninja\n\n`build-env` can also generate [Ninja](https://ninja-build.org/) files,\nwhich support parallel execution (unlike simple shell scripts):\n\n```\n$ build-env build lens -f sources -o install --script build_lens.ninja --ninja\n$ ninja -f build_lens.ninja\n```\n\n## Specifying packages\n\nInstead of passing the required packages through the command line,\nit can be more convenient to ask `build-env` to read them from files:\n\n```\n$ build-env build --seeds SEEDS --freeze cabal.config -f sources -o install\n```\n\nThis will build a plan for the packages specified in the `SEEDS` file,\nwith versions of packages that aren't in the `SEEDS` file constrained as\nspecified by the `cabal.config` file.\n\nEach line in the seeds file should be one of the following:\n\n  - A Cabal unit, in the format `unit +flag1 -flag2 \u003e= 0.1 \u0026\u0026 \u003c 0.3`.\n\n    A unit can be of the form `pkgName`, `lib:pkgName`, `exe:pkgName`,\n    `pkgName:lib:compName`, ... as per `cabal` component syntax.\n\n    The unit name must be followed by a space.\n\n    Flags and constraints are optional.  \n    When both are present, flags must precede constraints.  \n    Constraints must use valid Cabal constraint syntax.\n\n  - An allow-newer specification such as:\n\n    ```\n    allow-newer: pkg1:pkg2,pkg3:base,*:ghc\n    ```\n\n    This uses Cabal `allow-newer` syntax.\n\n  - An empty line or comment (starting with `--`).\n\nThe `cabal.config` file should use valid `cabal.project` syntax, e.g.:\n\n```cabal\nconstraints: pkg1 == 0.1,\n             pkg2 \u003e= 3.0\n```\n\nOnly the constraints are processed; everything else from the `cabal.config`\nfile is ignored.\n\n### Local packages\n\nTo specify that a package should be found locally rather than fetched from\nHackage, use `--local`:\n\n```\nbuild-env build myPkg --local myPkg=../pkgs/myPkg\n```\n\nThe part to the right of the `=` sign corresponds to what you would write\nin a `cabal.project` file: the directory in which `myPkg.cabal` is located.\n\nIf this is a relative path, it will be interpreted relative to the working\ndirectory of the `build-env` invocation, which can be overriden by passing\n`--cwd \u003cdir\u003e` (works like the `-C` argument to `make`).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbgamari%2Fbuild-env","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbgamari%2Fbuild-env","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbgamari%2Fbuild-env/lists"}