https://github.com/aburdulescu/gbenchdiff

Diff results of two C++ google benchmarks
https://github.com/aburdulescu/gbenchdiff

command-line-tool golang google-benchmark

Last synced: about 2 months ago
JSON representation

Diff results of two C++ google benchmarks

• Host: GitHub
• URL: https://github.com/aburdulescu/gbenchdiff
• Owner: aburdulescu
• License: mit
• Created: 2021-06-01T09:24:59.000Z (over 3 years ago)
• Default Branch: main
• Last Pushed: 2023-02-11T21:28:50.000Z (almost 2 years ago)
• Last Synced: 2024-10-23T22:30:12.630Z (3 months ago)
• Topics: command-line-tool, golang, google-benchmark
• Language: Go
• Homepage: https://bandr.me/compare-cpp-benchmark-results/
• Size: 54.7 KB
• Stars: 1
• Watchers: 3
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# gbenchdiff

Diff results of two C++ google benchmarks

## Install

### From source using `go`

```

go install bandr.me/p/gbenchdiff@latest

```

### Using prebuilt binary

See [here](https://github.com/aburdulescu/gbenchdiff/releases/latest)

## Usage

```

Usage: gbenchdiff [options] old.json new.json

options:

  -filter string

        select only the benchmarks with names that match the given regex

  -no-ctx

        don't compare benchmark contexts

  -with-cpu

        compare also CPU time

For each benchmark in both files, the tool will:

- remove outliers with interquartile range rule

- perform significance test(Man-Whitney U-test)

- print % change in mean from the first to the second file

- print the p-value and sample sizes from a test of the two distributions of benchmark times

Small p-values indicate that the two distributions are significantly different.

If the test indicates that there was no significant change between the two

benchmarks (defined as p > 0.05), a single ~ will be displayed instead of

the percent change.

IMPORTANT:

Run the benchmark with the following flags:

    --benchmark_out=file.json

    --benchmark_repetitions(=10 should be enough in most cases)

```

For a example, see [example](./example) directory.

## Acknowledgements

Heavily inspired by [Go benchstat](https://github.com/golang/perf) tool

(the statistics code is copied from there).