Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/bencherdev/bencher

🐰 Bencher - Continuous Benchmarking
https://github.com/bencherdev/bencher

benchmark benchmarking cd ci ci-cd code-quality continuous-benchmarking performance

Last synced: 3 months ago
JSON representation

🐰 Bencher - Continuous Benchmarking

Host: GitHub
URL: https://github.com/bencherdev/bencher
Owner: bencherdev
License: other
Created: 2020-07-25T09:54:21.000Z (over 4 years ago)
Default Branch: main
Last Pushed: 2024-10-21T00:48:15.000Z (3 months ago)
Last Synced: 2024-10-22T00:28:26.428Z (3 months ago)
Topics: benchmark, benchmarking, cd, ci, ci-cd, code-quality, continuous-benchmarking, performance
Language: MDX
Homepage: https://bencher.dev
Size: 21.3 MB
Stars: 533
Watchers: 3
Forks: 25
Open Issues: 126
Metadata Files:
- Readme: README.md
- Changelog: changelog.md
- License: LICENSE.md
- Security: SECURITY.md

Awesome Lists containing this project

awesome-rust - Bencher - A suite of continuous benchmarking tools designed to catch performance regressions in CI (Development tools / Profiling)
awesome-rust - Bencher - A suite of continuous benchmarking tools designed to catch performance regressions in CI (Development tools / Profiling)
awesome-scala - **bencher** - Continuous Benchmarking | ![GitHub stars](https://img.shields.io/github/stars/bencherdev/bencher) ![GitHub commit activity](https://img.shields.io/github/commit-activity/y/bencherdev/bencher) (Table of Contents / Testing)
fucking-awesome-rust - Bencher - A suite of continuous benchmarking tools designed to catch performance regressions in CI (Development tools / Profiling)
fucking-awesome-rust - Bencher - A suite of continuous benchmarking tools designed to catch performance regressions in CI (Development tools / Profiling)

README

        


  

    

  

  Bencher



[Bencher](https://bencher.dev) is a suite of [continuous benchmarking](https://bencher.dev/docs/explanation/continuous-benchmarking/) tools.

Have you ever had a performance regression impact your users?

Bencher could have prevented that from happening.

Bencher allows you to detect and prevent performance regressions _before_ they hit production.

- **Run**: Run your benchmarks locally or in CI using your favorite benchmarking tools. The `bencher` CLI simply wraps your existing benchmark harness and stores its results.

- **Track**: Track the results of your benchmarks over time. Monitor, query, and graph the results using the Bencher web console based on the source branch, testbed, benchmark, and measure.

- **Catch**: Catch performance regressions in CI. Bencher uses state of the art, customizable analytics to detect performance regressions before they make it to production.

For the same reasons that unit tests are run in CI to prevent feature regressions, benchmarks should be run in CI with Bencher to prevent performance regressions. Performance bugs are bugs!




Bencher consists of:

- `bencher` CLI

- Bencher API Server

- Bencher Console Web UI




Though Bencher is open source, there is also a hosted version available [Bencher Cloud](https://bencher.dev/).

The best place to start is the [Bencher Quick Start](https://bencher.dev/docs/tutorial/quick-start/) tutorial.






  

    

  



> 🐰 [Use the GitHub Action with your project](#github-actions)

## Documentation

- Tutorial

  - [Quick Start](https://bencher.dev/docs/tutorial/quick-start/)

  - [Docker Quick Start](https://bencher.dev/docs/tutorial/docker/)

- How To

  - [Install CLI](https://bencher.dev/docs/how-to/install-cli/)

  - [Track Benchmarks in CI](https://bencher.dev/docs/how-to/track-benchmarks/)

  - [GitHub Actions](https://bencher.dev/docs/how-to/github-actions/)

  - [GitLab CI/CD](https://bencher.dev/docs/how-to/gitlab-ci-cd/)

  - [Track Custom Benchmarks](https://bencher.dev/docs/how-to/track-custom-benchmarks/)

  - [Track File Size](https://bencher.dev/docs/how-to/track-file-size/)

  - [Self-Hosted GitHub App](https://bencher.dev/docs/how-to/github-app/)

- Explanation

  - [Benchmarking Overview](https://bencher.dev/docs/explanation/benchmarking/)

  - [`bencher run`](https://bencher.dev/docs/explanation/bencher-run/)

  - [Branch Selection](https://bencher.dev/docs/explanation/branch-selection/)

  - [Thresholds & Alerts](https://bencher.dev/docs/explanation/thresholds/)

  - [Benchmark Adapters](https://bencher.dev/docs/explanation/adapters/)

  - [Continuous Benchmarking](https://bencher.dev/docs/explanation/continuous-benchmarking/)

  - [Bencher Self-Hosted](https://bencher.dev/docs/explanation/bencher-self-hosted/)

  - [Talks](https://bencher.dev/docs/explanation/talks/)

- Reference

  - [REST API](https://bencher.dev/docs/api/)

  - [Architecture](https://bencher.dev/docs/reference/architecture/)

  - [Console Server Config](https://bencher.dev/docs/reference/console-config/)

  - [API Server Config](https://bencher.dev/docs/reference/server-config/)

  - [Bencher Metric Format](https://bencher.dev/docs/reference/bencher-metric-format/)

  - [Prior Art](https://bencher.dev/docs/reference/prior-art/)

  - [Roadmap](https://bencher.dev/docs/reference/roadmap/)

  - [Changelog](https://bencher.dev/docs/reference/changelog/)

🌐 Also available in:

- [简体中文](https://bencher.dev/zh/docs/)

- [Español](https://bencher.dev/es/docs/)

- [Português do Brasil](https://bencher.dev/pt/docs/)

- [Русский](https://bencher.dev/ru/docs/)

- [日本語](https://bencher.dev/ja/docs/)

- [Français](https://bencher.dev/fr/docs/)

- [Deutsch](https://bencher.dev/de/docs/)

- [한국어](https://bencher.dev/ko/docs/)

## Supported Benchmark Harnesses

- {...} JSON

  - [Custom benchmark harness support](https://bencher.dev/docs/explanation/adapters/#-json)

  - [File Size (Binary Size)](https://bencher.dev/docs/explanation/adapters/#%EF%B8%8F-file-size)

- #️⃣ C#

  - [BenchmarkDotNet](https://bencher.dev/docs/explanation/adapters/#%EF%B8%8F%E2%83%A3-c-dotnet)

- ➕ C++

  - [Catch2](https://bencher.dev/docs/explanation/adapters/#-c-catch2)

  - [Google Benchmark](https://bencher.dev/docs/explanation/adapters/#-c-google)

- 🕳 Go

  - [go test -bench](https://bencher.dev/docs/explanation/adapters/#-go-bench)

- ☕️ Java

  - [Java Microbenchmark Harness (JMH)](https://bencher.dev/docs/explanation/adapters/#%EF%B8%8F-java-jmh)

- 🕸 JavaScript

  - [Benchmark.js](https://bencher.dev/docs/explanation/adapters/#-javascript-benchmark)

  - [console.time/console.timeEnd](https://bencher.dev/docs/explanation/adapters/#-javascript-time)

- 🐍 Python

  - [airspeed velocity](https://bencher.dev/docs/explanation/adapters/#-python-asv)

  - [pytest-benchmark](https://bencher.dev/docs/explanation/adapters/#-python-pytest)

- ♦️ Ruby

  - [Benchmark](https://bencher.dev/docs/explanation/adapters/#%EF%B8%8F-ruby-benchmark)

- 🦀 Rust

  - [libtest bench](https://bencher.dev/docs/explanation/adapters/#-rust-bench)

  - [Criterion](https://bencher.dev/docs/explanation/adapters/#-rust-criterion)

  - [Iai](https://bencher.dev/docs/explanation/adapters/#-rust-iai)

  - [Iai-Callgrind](https://bencher.dev/docs/explanation/adapters/#-rust-iai-callgrind)

- ❯_ Shell

  - [Hyperfine](https://bencher.dev/docs/explanation/adapters/#_%EF%B8%8F-shell-hyperfine)

👉 For more details see the [explanation of benchmark harness adapters](https://bencher.dev/docs/explanation/adapters/).

## Showcase

  

    

      


        

          

        

      

      Microsoft CCF

    

    

      

        

          

        

      

      Rustls

    

    

      

        

          

        

      

      Diesel

    

  

  

    

      

        

          

        

      

      Hydra Database

    

    

      

        

          

        

      

      GreptimeDB

    

    

      

        

          

        

      

      Tailcall

    

  

  

    

      

        

          

        

      

      Ratatui

    

    

      

        

          

        

      

      Poolifier

    

    

      

        

          

        

      

      Wire

    

  

👉 Checkout [all public projects](https://bencher.dev/perf).

## GitHub Actions

Install the Bencher CLI using the [GitHub Action](https://github.com/marketplace/actions/bencher-cli),

and use it for [continuous benchmarking](https://bencher.dev/docs/explanation/continuous-benchmarking/) in your project.

```yaml

name: Continuous Benchmarking with Bencher

on:

  push:

    branches: main

jobs:

  benchmark_with_bencher:

    name: Benchmark with Bencher

    runs-on: ubuntu-latest

    env:

      BENCHER_PROJECT: my-project-slug

      BENCHER_API_TOKEN: ${{ secrets.BENCHER_API_TOKEN }}

    steps:

      - uses: actions/checkout@v4

      - uses: bencherdev/bencher@main

      - run: bencher run "bencher mock"

```

Supported Operating Systems:

- Linux (x86_64 & ARM64)

- MacOS (x86_64 & ARM64)

- Windows (x86_64 & ARM64)




👉 For more details see the [explanation of how to use GitHub Actions](https://bencher.dev/docs/how-to/github-actions/).

### Repository Secrets

Add `BENCHER_API_TOKEN` to you **Repository** secrets (ex: `Repo -> Settings -> Secrets and variables -> Actions -> New repository secret`). You can find your API tokens by running `bencher token list my-user-slug` or [view them in the Bencher Console](https://bencher.dev/console/users/tokens).

### Error on Alert

You can set the `bencher run` CLI subcommand to error

if [an Alert is generated](https://bencher.dev/docs/explanation/thresholds/) with the `--err` flag.

```bash

bencher run --err "bencher mock"

```

👉 For more details see the [explanation of `bencher run`](https://bencher.dev/docs/explanation/bencher-run/#--err).

### Comment on PRs

You can set the `bencher run` CLI subcommand to comment on a PR with the `--github-actions` argument.

```bash

bencher run --github-actions "${{ secrets.GITHUB_TOKEN }}" "bencher mock"

```

👉 For more details see the [explanation of `bencher run`](https://bencher.dev/docs/explanation/bencher-run/#--github-actions/).

### Example PR Comment




 Bencher Report
Branch254/mergeTestbedubuntu-latest

🚨 1 ALERT: Threshold Boundary Limit exceeded!
BenchmarkMeasure
UnitsViewBenchmark Result
(Result Δ%)Lower Boundary
(Limit %)Upper Boundary
(Limit %)Adapter::JsonLatency
nanoseconds (ns)📈 plot
🚨 alert
🚷 threshold3,445.60
(+1.52%)3,362.07
(102.48%)Click to view all benchmark results
BenchmarkLatencyBenchmark Results
nanoseconds (ns)
(Result Δ%)Upper Boundary
nanoseconds (ns)
(Limit %)Adapter::Json📈  view plot
🚨 view alert
🚷 view threshold3,445.60
(+1.52%)3,362.07
(102.48%)Adapter::Magic (JSON)📈 view plot
🚷 view threshold3,431.40
(+0.69%)3,596.95
(95.40%)Adapter::Magic (Rust)📈 view plot
🚷 view threshold22,095.00
(-0.83%)24,732.80
(89.33%)Adapter::Rust📈 view plot
🚷 view threshold2,305.70
(-2.76%)2,500.49
(92.21%)Adapter::RustBench📈 view plot
🚷 view threshold2,299.90
(-3.11%)2,503.41
(91.87%)🐰 View full continuous benchmarking report in Bencher


### Specify CLI Version

There is also an optional `version` argument to specify an exact version of the Bencher CLI to use.

Otherwise, it will default to using the latest CLI version.

```yaml

- uses: bencherdev/bencher@main

  with:

    version: 0.4.20

```

Specify an exact version if using Bencher _Self-Hosted_.

Do **not** specify an exact version if using Bencher _Cloud_ as there are still occasional breaking changes.

## Share Your Benchmarks

All public projects have their own [perf page](https://bencher.dev/perf). These results can easily be shared with an auto-updating perf image. Perfect for your README!



  

    

  



## Contributing

The easiest way to contribute is to open this repo as a [Dev Container](https://containers.dev) in [VSCode](https://code.visualstudio.com/download) by simply clicking one of the buttons below.

Everything you need will already be there!

Once set up, both the UI and API should be built, running, and seeded at [localhost:3000](http://localhost:3000) and [localhost:61016](http://localhost:61016) respectively.

To make any changes to the UI or API though, you will have to exit the startup process and restart the UI and API yourself.

#### 🐰 All pull requests should target the `devel` branch






  

    

  





  

    

  



There is also a [pre-built image from CI](https://github.com/orgs/bencherdev/packages/container/package/bencher-dev-container) available for each branch: `ghcr.io/bencherdev/bencher-dev-container`

## License

All content that resides under any directory or [feature](https://doc.rust-lang.org/cargo/reference/features.html) named "plus" is licensed under the [Bencher Plus License](license/LICENSE-PLUS).

All other content is licensed under the [Apache License, Version 2.0](license/LICENSE-APACHE) or [MIT License](license/LICENSE-MIT) at your discretion.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in Bencher by you, as defined in the Apache-2.0 license, shall be licensed as above, without any additional terms or conditions.