{"id":20132494,"url":"https://github.com/aherrmann/2024-hiw-amazonka","last_synced_at":"2025-09-21T05:32:12.954Z","repository":{"id":255755749,"uuid":"848144318","full_name":"aherrmann/2024-hiw-amazonka","owner":"aherrmann","description":"Building Haskell with Buck2 - ICFP 2024 - Haskell Implementors Workshop (Haskell Symposium)","archived":false,"fork":false,"pushed_at":"2024-09-19T08:46:19.000Z","size":5810,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"2024-hiw","last_synced_at":"2025-04-03T21:51:10.459Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Nix","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aherrmann.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"open_collective":"bazel-rules-authors-sig"}},"created_at":"2024-08-27T08:04:22.000Z","updated_at":"2025-02-21T06:28:59.000Z","dependencies_parsed_at":null,"dependency_job_id":"24f0eafd-ba4a-4ec5-a5d5-33cfda11efe9","html_url":"https://github.com/aherrmann/2024-hiw-amazonka","commit_stats":null,"previous_names":["aherrmann/2024-hiw-amazonka"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/aherrmann/2024-hiw-amazonka","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aherrmann%2F2024-hiw-amazonka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aherrmann%2F2024-hiw-amazonka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aherrmann%2F2024-hiw-amazonka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aherrmann%2F2024-hiw-amazonka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aherrmann","download_url":"https://codeload.github.com/aherrmann/2024-hiw-amazonka/tar.gz/refs/heads/2024-hiw","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aherrmann%2F2024-hiw-amazonka/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":276195629,"owners_count":25601152,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-21T02:00:07.055Z","response_time":72,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-13T20:53:47.965Z","updated_at":"2025-09-21T05:32:12.533Z","avatar_url":"https://github.com/aherrmann.png","language":"Nix","funding_links":["https://opencollective.com/bazel-rules-authors-sig"],"categories":[],"sub_categories":[],"readme":"# Building Haskell with Buck2\n\nPresentation at the [Haskell Implementors Workshop (Haskell Symposium) at ICFP 2024][hiw-2024], ([slides]).\n\n[hiw-2024]: https://icfp24.sigplan.org/details/haskellsymp-2024-papers/12/-HIW-Building-Haskell-with-Buck2\n[slides]: https://aherrmann.github.io/2024-hiw-amazonka/\n\n## Buck2 build of Amazonka\n\nUses [Amazonka] as an example project to illustrate [Buck2] for [Haskell].\n\n[Amazonka]: https://github.com/brendanhay/amazonka\n[Buck2]: https://buck2.build/\n[Haskell]: https://www.haskell.org/\n\nUses an extension [funded by Mercury][mercury-prelude] of the upstream [Buck2 prelude by Meta][meta-prelude].\n\n[mercury-prelude]: https://github.com/MercuryTechnologies/buck2-prelude/\n[meta-prelude]: https://github.com/facebook/buck2-prelude\n\n## Setup\n\nMake sure to checkout all git submodules.\n\n```shell\n$ git submodule update --init --recursive\n```\n\nMake sure to have [Nix] installed.\n\n[Nix]: https://nixos.org/\n\nEnter the Nix shell with the following command or using [direnv].\n\n```shell\n$ nix develop .#buck2\n```\n\n[direnv]: https://direnv.net/\n\n## Usage\n\nBuild select Amazonka components:\n\n```shell\n$ buck2 build //amazonka/lib/amazonka //amazonka/lib/services/amazonka-s3\n```\n\nQuery for all available Amazonka components:\n\n```shell\n$ buck2 uquery //amazonka/...\n```\n\nBuild all of Amazonka at once (this will take a while):\n\n```shell\n$ buck2 build //amazonka/...\n```\n\nOnly build and import the Haskell dependencies from Nix:\n\n```shell\n$ buck2 build //haskell/...\n```\n\nUpdate the Haskell library imports after you made a change:\n\n```shell\n$ buck2 bxl haskell/toolchain.bxl:libs\n```\n\nRe-generate the Buck2 targets after you changed the Cabal files:\n\n```shell\n$ ./convert.hs\n```\n\nClean previous build outputs and start over:\n```shell\n$ buck2 clean\n```\n\nGenerate a Chrome trace of the latest build, you can view it with [Perfetto UI]:\n```shell\n$ buck2 debug chrome-trace --trace-path=trace.json\n```\n\n[Perfetto UI]: https://ui.perfetto.dev/\n\n# buck2-ghcHEAD\n\nThis setup is based on a Buck2 project template supporting both nix-based GHC env and custom.\n\n## Getting started\n\nThis build template supports two build toolchain. One is GHC compiler and libraries configured via Nix,\nand the other is those prepared externally (custom GHC). For the latter, the environment variables `GHC`, `GHC_PATH`,\n`GHC_PKG_DB` and `GHC_EXTRA_OPTS` are used for specifying the locations of external tools and package configuration.\n\nFor nix-based GHC mode, enter the shell by\n```\n$ nix develop .#buck2-nix\n```\n\nFor custom GHC mode, enter the shell by\n```\n$ nix develop .#buck2-ghcHEAD\n```\nand set the environmental variables. For example,\n```\n$ export GHC_PATH=/nix/store/943sxl4vcfpfg6xaagxvgwbgz9scl7lc-coreutils-9.3/bin:/nix/store/m488d5iwzn93bdk1j5gxl77k3zb8y285-cctools-binutils-darwin-wrapper-11.1.0-973.0.1/bin:/nix/store/n6s41h0vwcllawzpxbmhxkbla4lhj9va-cctools-llvm-11.1.0-973.0.1/bin\n$ export GHC=~/repo/srcc/ghcHEAD/_build/stage1/bin/ghc\n$ export GHC_PKG_DB=~/.local/state/cabal/store/ghc-9.11.20240625/package.db\n$ export GHC_EXTRA_OPTS=\"-L/nix/store/cmxms1ix9ki9n5x7wkgilrhbcq97b48z-gmp-with-cxx-6.3.0/lib\"\n$ export GHC_LD_LIBRARY_PATH=\"/nix/store/cmxms1ix9ki9n5x7wkgilrhbcq97b48z-gmp-with-cxx-6.3.0/lib\"\n```\nNote that one can specify multiple `package.db` locations by a colon-separated list in the `GHC_PKG_DB` variable.\n`GHC_EXTRA_OPTS` and `GHC_LD_LIBRARY_PATH` are also colon-separated lists.\n\n## Build\n\n```\n$ buck2 build //myproject:pkg-b[static]\n```\n\nTo build a specific target and show which output was generated for it\n```\n$ buck2 build //myproject:pkg-b --show-output\n```\n\nTo show compiler output during the build\n```\n$ buck2 build //myproject:pkg-b -v 2,stderr\n```\n\n## Documentation\n\nRead the documentation on the [Buck2 website][buck2] for detailed documentation.\nIn particular:\n- https://buck2.build/docs/users/cheat_sheet/\n- https://buck2.build/docs/users/faq/common_issues/\n- https://buck2.build/docs/concepts/key_concepts/\n\n[buck2]: https://buck2.build/\n\n## Haskell toolchain libraries\n\nLibraries provided by nix need to be added to the `ghcWithPackages` call in order to make them available for the compiler, and they need\nto be added to the `toolchain_libraries` list for buck2 to make them available as `//haskell:lib-name` targets.\n\nIn order to add a new toolchain library from nix:\n\n1. add it to the `deps` attribute of a `haskell_library` or `haskell_binary` target by referring to `//haskell:foo`\n\n    ```bzl\n    haskell_library(\n       name = \"mylib\",\n       deps = [ \"//haskell:foo\", ... ],\n    )\n    ```\n2. run `buck2 bxl haskell/toolchain.bxl:libs`\n3. commit the changes to `toolchains/libs.bzl` and `toolchains/nix/ghc-toolchain-libraries.nix`\n\n## Update buck2\n\nRun the following command and commit the changed files in `nix/overlays/buck2`:\n\n```\n$ nix run '.#buck2-update'\n```\n\n## Query\n\nSee the [Buck2 cheat sheet][buck2-cheat] for common cases and the [Buck2 query\ndocs][buck2-query] for further details.\n\n[buck2-cheat]: https://buck2.build/docs/users/cheat_sheet/\n[buck2-query]: https://buck2.build/docs/users/query/cquery/\n\nList the sub-targets available under a given target\n```\n$ buck2 audit subtargets //myproject:pkg-b\n```\n\nShow the attributes of a given target\n```\n$ buck2 uquery //myproject:pkg-b -A\n```\n\nList the providers of a given target in detail\n```\n$ buck2 audit providers //myproject:pkg-b\n```\n\nVisualize the dependency graph of a target\n```\n$ buck2 cquery 'deps(backend) ^ //myproject/...' --dot | xdot -\n```\n\nQuery the build actions that will be performed to build a given target\n```\n$ buck2 aquery \\\n    'kind(run, deps(//myproject:pkg-b))' \\\n    --output-attribute='cmd|category'\n```\n\n## Debug\n\nIf the build failed, understand which command failed\n```\n$ buck2 log what-failed\n```\n\nUnderstand what build commands were executed\n```\n$ buck2 log what-ran\n```\n\nPrint commands that ran including their output\n```\n$ buck2 log what-ran --show-std-err --format json | jq\n```\n\nReplay the Buck2 terminal output\n```\n$ buck2 log replay\n```\n\nDisplay the logs for the recent n-th build\n```\n$ buck2 log replay --recent 0\n```\n\nDisplay the logs for a specific build ID\n```\n$ buck2 log replay --trace-id ff7a9c37-7637-4c98-9c39-9ef51d5f824b\n```\n\nExtract exit code and output of all build actions\n```\n$ buck2 log show | jq -r '\n    .Event.data.SpanEnd.data.ActionExecution\n    | try .commands[]\n    | ( \"exit: \" + (.details.signed_exit_code|tostring)\n      + \"\\nstdout:\\n\" + .details.stdout\n      + \"\\nstderr:\\n\" + .details.stderr\n      )\n  '\n```\n\nExtract build commands from the log\n```\n$ buck2 log show | jq -r '\n    .Event.data.SpanStart.data.ExecutorStage.stage.Local.stage.Execute.command.argv\n  | try join(\" \")\n  '\n```\n\nSee [Buck2 Logging documentation][buck2-logging] for further information.\n\n[buck2-logging]: https://buck2.build/docs/users/build_observability/logging/\n\n## Profile and Optimize\n\nDisplay the critical path\n```\n$ buck2 log critical-path\n```\n\nGenerate a Chrome trace viewable in [Perfetto UI][perfetto] or similar.\n```\n$ buck2 debug chrome-trace --trace-path=OUT.json\n```\n\n[Perfetto UI supports SQL queries][perfetto-sql] that can be used to analyze\nthe performance of specific types of actions. However, one must be careful to\nseparate the actual execution time from any potential time spent in the queue.\n\nThe following query accumulates all the (dynamic) Haskell module compile\nactions that executed locally.\n```sql\nSELECT AVG(exec_slice.dur), SUM(exec_slice.dur)\nFROM slice as exec_slice\nINNER JOIN slice as parent_slice ON exec_slice.parent_id = parent_slice.id\nWHERE exec_slice.name = \"local_execute\" AND parent_slice.name LIKE \"%haskell_compile_shared%\"\n```\n\nSee [Buck2 Observability and Optimization][buck2-opt] for information on\nprofiling Buck2 and its rules themselves.\n\n[perfetto]: https://ui.perfetto.dev/\n[perfetto-sql]: https://perfetto.dev/docs/quickstart/trace-analysis\n[buck2-opt]: https://buck2.build/docs/rule_authors/optimization/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faherrmann%2F2024-hiw-amazonka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faherrmann%2F2024-hiw-amazonka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faherrmann%2F2024-hiw-amazonka/lists"}