https://github.com/willabides/benchdiff
https://github.com/willabides/benchdiff
benchmark benchstat golang
Last synced: 11 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/willabides/benchdiff
- Owner: WillAbides
- License: mit
- Created: 2020-12-27T21:08:28.000Z (over 5 years ago)
- Default Branch: main
- Last Pushed: 2023-05-16T16:03:27.000Z (about 3 years ago)
- Last Synced: 2025-05-17T19:08:37.294Z (about 1 year ago)
- Topics: benchmark, benchstat, golang
- Language: Go
- Homepage:
- Size: 138 KB
- Stars: 68
- Watchers: 4
- Forks: 2
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# benchdiff
[](https://pkg.go.dev/github.com/willabides/benchdiff)
[](https://github.com/WillAbides/benchdiff/actions?query=workflow%3Aci+branch%3Amain+event%3Apush)
Benchdiff runs go benchmarks on your current git worktree and a base ref then uses benchstat to show
the delta.
## What it does
Benchdiff automates comparing the results of the same go benchmarks on your worktree and a git reference.
It essentially combines multiple `git`, `go test -bench` and `benchstat` commands into one.
These are the basic steps `benchdiff` performs:
- Runs `go test -bench ...` on your current worktree and saves the results to a cache directory.
- Creates a new worktree at the base ref
- Runs `go test -bench ...` on the base worktree and saves the results to cache again.
- Runs `benchstat` to compare the base and head results from cache.
## GitHub Action
benchdiff can be run as a GitHub Action. https://github.com/WillAbides/benchdiff-action
## Usage
### `benchdiff --help` output
```
Usage: benchdiff
benchdiff runs go benchmarks on your current git worktree and a base ref then uses benchstat to show
the delta.
More documentation at https://github.com/willabides/benchdiff.
Flags:
-h, --help Show context-sensitive help.
--version Output the benchdiff version and exit.
--debug write verbose output to stderr
--base-ref="HEAD" The git ref to be used as a baseline.
--cooldown=100ms How long to pause for cooldown between head and base runs.
--force-base Rerun benchmarks on the base reference even if the output already exists.
--git-cmd="git" The executable to use for git commands.
--json Format output as JSON.
--on-degrade=0 Exit code when there is a statistically significant degradation in the
results.
--tolerance=10.0 The minimum percent change before a result is considered degraded.
benchmark command line
--bench="." Run only those benchmarks matching a regular expression. To run all
benchmarks, use '--bench .'.
--benchmark-args=args Override the default args to the go command. This may be a template.
See https://github.com/willabides/benchdiff for details."
--benchmark-cmd="go" The command to use for benchmarks.
--benchmem Memory allocation statistics for benchmarks.
--benchtime=STRING Run enough iterations of each benchmark to take t, specified as a
time.Duration (for example, --benchtime 1h30s). The default is 1 second
(1s). The special syntax Nx means to run the benchmark N times (for
example, -benchtime 100x).
--count=10 Run each benchmark n times. If --cpu is set, run n times for each
GOMAXPROCS value.'
--cpu=GOMAXPROCS,... Specify a list of GOMAXPROCS values for which the benchmarks should be
executed. The default is the current value of GOMAXPROCS.
--packages="./..." Run benchmarks in these packages.
--show-bench-cmdline Instead of running benchmarks, output the command that would be used and
exit.
--tags=STRING Set the -tags flag on the go test command
--warmup-count=INT Run benchmarks with -count=n as a warmup
--warmup-time=STRING When warmups are run, set -benchtime=n
benchstat options
--alpha=0.05 consider change significant if p < α
--benchstat-output="text" format for benchstat output (csv,html,markdown or text)
--delta-test="utest" significance test to apply to delta: utest, ttest, or none
--geomean print the geometric mean of each file
--norange suppress range columns (CSV and markdown only)
--reverse-sort reverse sort order
--sort="none" sort by order: delta, name, none
--split="pkg,goos,goarch" split benchmarks by labels
benchmark result cache
--cache-dir=STRING Override the default directory where benchmark output is kept.
--clear-cache Remove benchdiff files from the cache dir.
--show-cache-dir Output the cache dir and exit.
```
### `--benchmark-args`
The default should work for most cases.
`--benchmark-args` is everything after "go" in the `go test` command that runs the benchmarks. You can use either
pass in the raw values or use a go template.
Default value:
```
test {{ .Packages }} -run '^$'
{{- if .Bench }} -bench {{ .Bench }}{{end}}
{{- if .Count }} -count {{ .Count }}{{end}}
{{- if .Benchtime }} -benchtime {{ .Benchtime }}{{end}}
{{- if .CPU }} -cpu {{ .CPU }}{{ end }}
{{- if .Tags }} -tags "{{ .Tags }}"{{ end }}
{{- if .Benchmem }} -benchmem{{ end }}
```
## Install
### go get
`go get -u github.com/willabides/benchdiff/cmd/benchdiff`
### bindown
Add a [bindown](https://github.com/willabides/bindown) dependency:
```
$ bindown template-source add benchdiff https://raw.githubusercontent.com/WillAbides/benchdiff/main/bindown.yml
$ bindown dependency add benchdiff benchdiff#benchdiff
Please enter a value for required variable "version":
```