{"id":19743378,"url":"https://github.com/jpenuchot/ctbench","last_synced_at":"2025-10-26T01:38:06.792Z","repository":{"id":42630628,"uuid":"363880788","full_name":"JPenuchot/ctbench","owner":"JPenuchot","description":"Compiler-assisted variable size benchmarking for the study of C++ metaprogram compile times.","archived":false,"fork":false,"pushed_at":"2024-09-26T15:18:18.000Z","size":737,"stargazers_count":25,"open_issues_count":8,"forks_count":7,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-10-20T22:27:57.214Z","etag":null,"topics":["benchmark","clang","compilation","data-analysis","data-visualization","gcc","metaprogramming"],"latest_commit_sha":null,"homepage":"https://jpenuchot.github.io/ctbench-docs/","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JPenuchot.png","metadata":{"files":{"readme":"readme.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-05-03T09:36:17.000Z","updated_at":"2025-01-10T11:19:53.000Z","dependencies_parsed_at":"2023-02-17T22:01:21.360Z","dependency_job_id":"68563980-862a-4616-8afa-be32b48275b5","html_url":"https://github.com/JPenuchot/ctbench","commit_stats":null,"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"purl":"pkg:github/JPenuchot/ctbench","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JPenuchot%2Fctbench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JPenuchot%2Fctbench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JPenuchot%2Fctbench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JPenuchot%2Fctbench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JPenuchot","download_url":"https://codeload.github.com/JPenuchot/ctbench/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JPenuchot%2Fctbench/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":281047794,"owners_count":26435124,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-25T02:00:06.499Z","response_time":81,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","clang","compilation","data-analysis","data-visualization","gcc","metaprogramming"],"created_at":"2024-11-12T01:36:52.531Z","updated_at":"2025-10-26T01:38:06.759Z","avatar_url":"https://github.com/JPenuchot.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"Compiler-assisted benchmarking for the study of C++ metaprogram compile times.\n\n- Github project: https://github.com/jpenuchot/ctbench\n- Online documentation: https://jpenuchot.github.io/ctbench-docs/\n- Discord server: https://discord.gg/NvJFFrdS7p\n\n[![DOI](https://joss.theoj.org/papers/10.21105/joss.05165/status.svg)](https://doi.org/10.21105/joss.05165)\n\nctbench allows you to declare and generate compile-time benchmark batches for\ngiven ranges, run them, aggregate and wrangle Clang profiling data,\nand plot them.\n\nThe project was made to fit the needs of **scientific data collection and\nanalysis**, thus it is not a one-shot profiler, but a set of tools that enable\n**reproductible data gathering** from user-defined, variably sized compile-time\nbenchmarks using Clang's time-trace feature to understand the impact of\nmetaprogramming techniques on compile time. On top of that, ctbench is also able\nto measure compiler execution time to support compilers that do not have\nbuilt-in profilers like GCC.\n\nIt has two main components: a C++ plotting toolset that can be used as a CLI\nprogram and as a library, and a CMake boilerplate library to generate benchmark\nand graph targets.\n\nThe CMake library contains all the boilerplate code to define benchmark targets\ncompatible with the C++ plotting toolset called `grapher`.\n\n[Rule of Cheese](https://github.com/jpenuchot/rule-of-cheese) can be used as an\nexample project for using ctbench.\n\n### Examples\n\nAs an example here are benchmark curves from the Poacher project. The benchmark\ncase sources are available [here](\nhttps://github.com/JPenuchot/poacher/tree/main/brainfog/benchmark/consecutive_loops).\n\n![](docs/images/ExecuteCompiler.svg)\n\n*Clang ExecuteCompiler time curve from\n[poacher](https://github.com/jpenuchot/poacher),\ngenerated by the `compare_by` plotter*\n\n![](docs/images/Total_Frontend.svg)\n\n*Clang Total Frontend time curve from\n[poacher](https://github.com/jpenuchot/poacher),\ngenerated by the `compare_by` plotter*\n\n\n## Using ctbench\n\n### Build prerequisites\n\nArchLinux and Ubuntu 23.04 are officially supported as tests are compiled and\nexecuted on both of these Linux distributions. Others including Fedora or any\nother Linux distro that provides CMake 3.25 or higher should be compatible.\n\n- Required ArchLinux packages: `boost boost-libs catch2 clang cmake curl fmt git\nllvm llvm-libs ninja nlohmann-json tar tbb unzip zip`\n\n- Required Ubuntu packages: `catch2 clang cmake curl git libboost-all-dev\nlibclang-dev libfmt-dev libllvm15 libtbb-dev libtbb12 llvm llvm-dev ninja-build\nnlohmann-json3-dev pkg-config tar unzip zip`\n\nThe Sciplot library is required too. It can be installed on ArchLinux using the\n[`sciplot-git` AUR package](https://aur.archlinux.org/packages/sciplot-git)\n(NB: the non-git package isn't up-to-date). Otherwise, [you can install it for\nyour whole system using CMake](\nhttps://sciplot.github.io/installation/#installation-using-cmake) or locally\nusing [vcpkg](https://vcpkg.io/en/getting-started.html):\n\n```sh\ngit clone https://github.com/Microsoft/vcpkg.git\n./vcpkg/bootstrap-vcpkg.sh\n./vcpkg/vcpkg install sciplot fmt\n\ncmake --preset release \\\n  -DCMAKE_TOOLCHAIN_FILE=vcpkg/scripts/buildsystems/vcpkg.cmake\n```\n\nNote: The `fmt` dependency is needed, as vcpkg breaks fmt's CMake integration if\nyou have it already installed.\n\n### Installing ctbench\n\n```sh\ngit clone https://github.com/jpenuchot/ctbench\ncd ctbench\ncmake --preset release\ncmake --build --preset release\nsudo cmake --build --preset release --target install\n```\n\nAn [AUR package](https://aur.archlinux.org/packages/ctbench-git) is available\nfor easier install and update.\n\n### Integrating ctbench in your project\n\nctbench can be integrated to a CMake project using `find_package`:\n\n```cmake\nfind_package(ctbench REQUIRED)\n```\n\nThe example project is provided as a reference project for ctbench integration\nand usage. For more details, an exhaustive [CMake API reference](\nhttps://jpenuchot.github.io/ctbench-docs/md_generated_docs_ctbench_api.html)\nis available.\n\n### Declaring a benchmark case target\n\nA benchmark case is represented by a C++ file. It will be \"instanciated\", ie.\ncompiled with `BENCHMARK_SIZE` defined to values in a range that you provide.\n\n`BENCHMARK_SIZE` is intended to be used by the preprocessor to generate a\nbenchmark instance of the desired size:\n\n```cpp\n#include \u003cboost/preprocessor/repetition/repeat.hpp\u003e\n\n// First we generate foo\u003cint\u003e().\n// foo\u003cint\u003e() uses C++20 requirements to dispatch function calls accross 16\n// of its instances, according to the value of its integer template parameter.\n\n#define FOO_MAX 16\n\n#define DECL(z, i, nope)                                                       \\\n  template \u003cint N\u003e                                                             \\\n  requires(N % FOO_MAX == i) constexpr int foo() { return N * i; }\n\nBOOST_PP_REPEAT(BENCHMARK_SIZE, DECL, FOO_MAX);\n#undef DECL\n\n// Now we generate the sum() function for instanciation\n\nint sum() {\n  int i;\n\n#define CALL(z, n, nop) i += foo\u003cn\u003e();\n  BOOST_PP_REPEAT(BENCHMARK_SIZE, CALL, i);\n#undef CALL\n  return i;\n}\n```\n\nBy default, only compiler execution time is measured.\nIf you want to generate plots using Clang's profiler data, add the following:\n\n```cmake\nadd_compile_options(-ftime-trace -ftime-trace-granularity=1)\n```\n\nNote that plotting profiler data takes more time\nand will generate a *lot* of plot files.\n\nThen you can declare a benchmark case target in CMake with the following:\n\n```cmake\nctbench_add_benchmark(function_selection.requires # Benchmark case name\n  function_selection-requires.cpp                 # Benchmark case file\n  1                                               # Range begin\n  32                                              # Range end\n  1                                               # Range step\n  10)                                             # Iterations per size\n```\n\n### Declaring a graph target\n\nOnce you have several benchmark cases, you can start writing a graph config.\n\nExample configs can be found [here](\nhttps://github.com/JPenuchot/ctbench/tree/main/grapher/configs), or by running\n`ctbench-grapher-utils --plotter=\u003cplotter\u003e --command=get-default-config`. A list\nof available plotters can be retrieved by running\n`ctbench-grapher-utils --help`.\n\n```json\n{\n  \"plotter\": \"compare_by\",\n  \"demangle\": true,\n  \"draw_average\": true,\n  \"draw_points\": true,\n  \"key_ptrs\": [\n    \"/name\",\n    \"/args/detail\"\n  ],\n  \"legend_title\": \"Timings\",\n  \"plot_file_extensions\": [\n    \".svg\",\n    \".png\"\n  ],\n  \"value_ptr\": \"/dur\",\n  \"width\": 1500,\n  \"height\": 500,\n  \"x_label\": \"Benchmark size factor\",\n  \"y_label\": \"Time (µs)\"\n}\n```\n\nThis configuration uses the `compare_by` plotter. It compares features targeted\nby the JSON pointers in `key_ptrs` across all benchmark cases. This is the\neasiest way to extract and compare as many relevant time-trace features at once.\n\nBack to CMake, you can now declare a graph target using this config to compare\nthe time spent in the compiler execution, the frontend, and the backend between\nthe benchmark cases you declared previously:\n\n```cmake\nctbench_add_graph(function_selection-feature_comparison-graph # Target name\n  ${CONFIGS}/feature_comparison.json                          # Config\n  function_selection.enable_if                                # First case\n  function_selection.enable_if_t                              # Second case\n  function_selection.if_constexpr                             # ...\n  function_selection.control\n  function_selection.requires)\n```\n\nFor each group descriptor, a graph will be generated with one curve\nper benchmark case. In this case, you would then get 3 graphs\n(`ExecuteCompiler`, `Frontend`, and `Backend`) each with 5 curves (`enable_if`,\n`enable_if_t`, `if_constexpr`, `control`, and `requires`).\n\n## Related work\n\n- [poacher](https://github.com/jpenuchot/poacher)\n- [Rule of Cheese](https://github.com/jpenuchot/rule-of-cheese)\n\n## References\n\n- [ctbench: compile time benchmarking for Clang](\n  https://www.youtube.com/watch?v=1RZY6skM0Rc) at [CPPP 2021](\n  https://cppp.fr/schedule2021/)\n- [A totally constexpr standard library - Paul Keir, Joel Falcou et al - Meeting C++ 2022](\n  https://www.youtube.com/watch?v=ekFPm7e__vI)\n- [Pyperf - Tune the system for benchmarks](\n  https://pyperf.readthedocs.io/en/latest/system.html)\n- [Metabench](https://github.com/ldionne/metabench)\n\n## Citing ctbench\n\n```bibtex\n@article{Penuchot2023,\n  doi = {10.21105/joss.05165},\n  url = {https://doi.org/10.21105/joss.05165},\n  year = {2023},\n  publisher = {The Open Journal},\n  volume = {8},\n  number = {88},\n  pages = {5165},\n  author = {Jules Penuchot and Joel Falcou},\n  title = {ctbench - compile-time benchmarking and analysis},\n  journal = {Journal of Open Source Software},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjpenuchot%2Fctbench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjpenuchot%2Fctbench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjpenuchot%2Fctbench/lists"}