{"id":15429732,"url":"https://github.com/ldionne/metabench","last_synced_at":"2025-08-20T08:31:52.189Z","repository":{"id":144918568,"uuid":"52907839","full_name":"ldionne/metabench","owner":"ldionne","description":"A simple framework for compile-time benchmarks","archived":false,"fork":false,"pushed_at":"2021-05-01T15:47:22.000Z","size":358065,"stargazers_count":183,"open_issues_count":12,"forks_count":17,"subscribers_count":17,"default_branch":"master","last_synced_at":"2025-04-07T12:22:43.639Z","etag":null,"topics":["benchmark","cmake","cpp","metaprogramming"],"latest_commit_sha":null,"homepage":"metaben.ch","language":"CMake","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ldionne.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-03-01T20:33:37.000Z","updated_at":"2025-03-31T11:51:14.000Z","dependencies_parsed_at":null,"dependency_job_id":"3ee46611-307c-4c77-a82f-93d33fae9c94","html_url":"https://github.com/ldionne/metabench","commit_stats":{"total_commits":372,"total_committers":10,"mean_commits":37.2,"dds":0.3360215053763441,"last_synced_commit":"3322ce7b54e426688fe60349e059d2832689e5fb"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ldionne/metabench","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ldionne%2Fmetabench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ldionne%2Fmetabench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ldionne%2Fmetabench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ldionne%2Fmetabench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ldionne","download_url":"https://codeload.github.com/ldionne/metabench/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ldionne%2Fmetabench/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271287619,"owners_count":24733424,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-20T02:00:09.606Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","cmake","cpp","metaprogramming"],"created_at":"2024-10-01T18:12:41.863Z","updated_at":"2025-08-20T08:31:51.851Z","avatar_url":"https://github.com/ldionne.png","language":"CMake","funding_links":[],"categories":["Modules"],"sub_categories":[],"readme":"## Metabench \u003ca target=\"_blank\" href=\"https://travis-ci.org/ldionne/metabench\"\u003e![Travis status][badge.Travis]\u003c/a\u003e \u003ca target=\"_blank\" href=\"https://ci.appveyor.com/project/ldionne/metabench\"\u003e![Appveyor status][badge.Appveyor]\u003c/a\u003e\n\u003e A simple framework for compile-time microbenchmarks\n\n\n### Overview\nMetabench is a single, self-contained CMake module making it easy to create\ncompile-time microbenchmarks. Compile-time benchmarks measure the performance\nof compiling a piece of code instead of measuring the performance of running\nit, as regular benchmarks do. The __micro__ part in **micro**benchmark means\nthat Metabench can be used to benchmark precise parts of a C++ file, such as\nthe instantiation of a single function. Writing benchmarks of this kind is\nvery useful for C++ programmers writing metaprogramming-heavy libraries, which\nare known to cause long compilation times. Metabench was designed to be very\nsimple to use, while still allowing fairly complex benchmarks to be written.\n\nMetabench is also a collection of compile-time microbenchmarks written using\nthe `metabench.cmake` module. The benchmarks measure the compile-time performance\nof various algorithms provided by different metaprogramming libraries. The\nbenchmarks are updated nightly with the latest version of each library, and\nthe results are published at http://metaben.ch.\n\n### Requirements\nMetabench requires [CMake][] 3.1 or higher and [Ruby][] 2.1 or higher.\nMetabench is known to work with CMake's _Unix Makefiles_ and _Ninja_\ngenerators.\n\n### Usage\nTo use Metabench, make sure you have the dependencies listed above and simply\ndrop the `metabench.cmake` file somewhere in your CMake search path for modules.\nThen, use `include(metabench)` to include the module in your CMake file, add\nindividual datasets to be benchmarked using `metabench_add_dataset`, and finally\nspecify which datasets should be put together into a chart via `metabench_add_chart`.\nFor example, a minimal CMake file using Metabench would look like:\n\n```CMake\n# Make sure Metabench can be found when writing include(metabench)\nlist(APPEND CMAKE_MODULE_PATH \"path/to/metabench/directory\")\n\n# Actually include the module\ninclude(metabench)\n\n# Add new datasets\nmetabench_add_dataset(dataset1 \"path/to/dataset1.cpp.erb\" \"[1, 5, 10]\")\nmetabench_add_dataset(dataset2 \"path/to/dataset2.cpp.erb\" \"(1...15)\")\nmetabench_add_dataset(dataset3 \"path/to/dataset3.cpp.erb\" \"(1...20).step(5)\")\n\n# Add a new chart\nmetabench_add_chart(chart DATASETS dataset1 dataset2 dataset3)\n```\n\nThis will create a target named `chart`, which, when run, will gather benchmark\ndata from each `dataset` and output JSON files for easy integration with other\ntools. A HTML file is generated for easy visualization of the datasets as a\n[NVD3][] chart. To understand what the `path/to/datasetN.cpp.erb` files are,\nread what follows.\n\n#### The principle\nBenchmarking the compilation time of a single `.cpp` file is rather useless,\nbecause one could simply run the compiler and time that single execution instead.\nWhat is really useful is to have a means of running variations of the same\n`.cpp` file automatically. For example, we might be interested in benchmarking\nthe compilation time for creating a `std::tuple` with many elements in it. To\ndo so, we could write the following test case:\n\n```c++\n#include \u003ctuple\u003e\n\nint main() {\n    auto tuple = std::make_tuple(1, 2, 3, 4, 5);\n}\n```\n\nWe would run the compiler and time the compilation, and then change the test\ncase by augmenting the number of elements in the tuple:\n\n```c++\n#include \u003ctuple\u003e\n\nint main() {\n    auto tuple = std::make_tuple(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);\n}\n```\n\nWe would measure the compilation time for this file, and repeat the process\nuntil satisfactory data has been gathered. This tedious task of generating\ndifferent (but obviously related) `.cpp` files and running the compiler to\ngather timings is what Metabench automates. It does this by taking a `.cpp.erb`\nfile written using the ERB template system, and generating a family of `.cpp`\nfiles from that template. It then compiles these `.cpp` files and gathers\nbenchmark data from these compilations.\n\nConcretely, you start by writing a `.cpp.erb` file (say `std_tuple.cpp.erb`)\nthat may contain ERB markup:\n\n```c++\n#include \u003ctuple\u003e\n\nint main() {\n    auto tuple = std::make_tuple(\u003c%= (1..n).to_a.join(', ') %\u003e);\n}\n```\n\nCode contained inside `\u003c%= ... %\u003e` is just normal Ruby code. When the file\nwill be rendered, the contents of `\u003c%= ... %\u003e` will be replaced with the\nresult of evaluating this Ruby code, which will look like:\n\n```c++\n#include \u003ctuple\u003e\n\nint main() {\n    auto tuple = std::make_tuple(1, 2, 3, ..., n);\n}\n```\n\nThe ERB markup language has many other features; we encourage readers to take\na look at the [Wikipedia page][ERB]. What happens is that Metabench will\ngenerate a `.cpp` file for different values of `n`, and will gather benchmark\ndata for each of these values. Now, this isn't the whole story. More often\nthan not, we're only interested in benchmarking part of a C++ file. Indeed,\nif we benchmark the whole file in our example above, we'll end up measuring\nthe time required to `#include` the `\u003ctuple\u003e` header in addition to the time\nrequired for creating the `std::tuple`. While this might be negligible in our\nexample, this situation arises in nontrivial examples, and would make the\nresulting data nearly worthless. Hence, we have to tell Metabench what part(s)\nof the file it should measure. This is done by guarding the relevant part(s)\nof the code with a preprocessor `#if`:\n\n```c++\n#include \u003ctuple\u003e\n\nint main() {\n#if defined(METABENCH)\n    auto tuple = std::make_tuple(\u003c%= (1..n).to_a.join(', ') %\u003e);\n#endif\n}\n```\n\nWhat Metabench will actually do is compile the file once with the macro defined\n(and hence with the content of the block), and once without it. It will then\nsubtract the time for compiling the file without the content of the block to\nthe time for compiling the whole file, which should represent a good\napproximation of the time for compiling what's inside the block.\n\nOn the C++ side of things, the `.cpp` file will be compiled (to benchmark it)\nas if it were located in the directory containing the `.cpp.erb` file, so that\nrelative include paths can be used. Furthermore, it will be compiled as if the\n`.cpp` file were part of a CMake executable added in the same directory as the\ncall to `metabench_add_dataset`. This way, any variable or property set in CMake\nwill also apply when benchmarking the file. In other words, Metabench tries to\ncreate the illusion that the code is actually compiled as if it were written\nin the `.cpp.erb` file.\n\nThis is it for the basic usage of the module! The `example/` directory contains\na fully working example of using Metabench to create benchmarks. For a more\ninvolved example, you can take a look at the benchmark suite in the `benchmark/`\ndirectory. Note that only the most basic usage of Metabench was covered here.\nTo know all the features provided by the module, you should read the reference\ndocumentation provided as comments inside the CMake module.\n\n#### A note on benchmark resolution\nLike any measurement tool, Metabench has a limited resolution. For example, when\nthe code being measured (inside the `#ifdef METABENCH`/`#endif` pair) takes only\na few milliseconds to compile, the timings reported by Metabench may be completely\ninside the noise. Typically, the resolution of timings taken by Metabench is\nsimilar to that of the `time` command. A good technique to make sure the results\nof a benchmark are not inside the noise is to reduce the _relative_ uncertainty\nof the measurement. This can be done by increasing the total compilation time\nof the measured block, by repeating the same thing (or a similar one) multiple\ntimes:\n\n```c++\n#include \u003ctuple\u003e\n\nint main() {\n#if defined(METABENCH)\n    auto tuple1 = std::make_tuple(\u003c%= (1..n).to_a.join(', ') %\u003e);\n    auto tuple2 = std::make_tuple(\u003c%= (1..n).to_a.join(', ') %\u003e);\n    auto tuple3 = std::make_tuple(\u003c%= (1..n).to_a.join(', ') %\u003e);\n    auto tuple4 = std::make_tuple(\u003c%= (1..n).to_a.join(', ') %\u003e);\n#endif\n}\n```\n\n### History\nMetabench was initially developed inside the [Boost.Hana][] library as a\nmean to benchmark compile-time algorithms. After seeing that a self-standing\nframework would be useful to the general C++ community, it was decided to\nextract it into its own project.\n\n### License\nPlease see [LICENSE.md](LICENSE.md).\n\n\n\u003c!-- Links --\u003e\n[badge.Appveyor]: https://ci.appveyor.com/api/projects/status/github/ldionne/metabench?svg=true\u0026branch=master\n[badge.Travis]: https://travis-ci.org/ldionne/metabench.svg?branch=master\n[Boost.Hana]: http://github.com/boostorg/hana\n[CMake]: http://www.cmake.org\n[ERB]: http://en.wikipedia.org/wiki/ERuby\n[Ruby]: https://www.ruby-lang.org/en/\n[NVD3]: http://nvd3.org/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fldionne%2Fmetabench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fldionne%2Fmetabench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fldionne%2Fmetabench/lists"}