https://github.com/dapplion/benchmark

JS/TS benchmarking solution to track performance regressions in CI
https://github.com/dapplion/benchmark

benchmark performance

Last synced: about 19 hours ago
JSON representation

JS/TS benchmarking solution to track performance regressions in CI

• Host: GitHub
• URL: https://github.com/dapplion/benchmark
• Owner: dapplion
• License: apache-2.0
• Created: 2021-06-20T11:26:43.000Z (over 3 years ago)
• Default Branch: main
• Last Pushed: 2024-12-02T13:36:24.000Z (about 1 month ago)
• Last Synced: 2024-12-28T06:47:42.961Z (10 days ago)
• Topics: benchmark, performance
• Language: TypeScript
• Homepage:
• Size: 230 KB
• Stars: 1
• Watchers: 3
• Forks: 3
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Benchmark

Ensures that new code does not introduce performance regressions with CI. Tracks:

- Do PR against the base branch include a performance regression?

- Do new commits in the main branch include a performance regression?

This tooling provides both a easy to use runner for benchmarking and easy integrations to persist past benchmark data.

## Quick start

Create a test mocha test file but use `itBench` instead of `it`

```ts

import {itBench, setBenchOpts} from "../../src";

describe("Sum array benchmark", () => {

  itBench("sum array with reduce", () => {

    arr.reduce((total, curr) => total + curr, 0);

  });

});

```

Then run the CLI, compatible with all mocha options.

```

benchmark 'test/perf/**/*.perf.ts' --local

```

Inspect benchmark results in the terminal

```

  Sum array benchmark

    ✔ sum array with reduce                                               826.0701 ops/s    1.210551 ms/op   x0.993        578 runs   1.21 s

```

## How does it work?

This tool is a CLI wrapper around mocha, example usage:

```

benchmark 'test/perf/**/*.perf.ts' --s3

```

The above command will:

- Read benchmark history from the specified provider (AWS S3)

- Figure out the prev benchmark based on your option (defaults to latest commit in main branch)

- Run benchmark comparing with previous

  - Runs mocha programatically against the file globs

  - Collect benchmark data in-memory while streaming results with a familiar mocha reporter

  - Note: also runs any test that would regularly be run with mocha

- Add result to benchmark history and persist them to the specified provider (AWS S3)

- If in CI, post a PR or commit comment with an expandable summary of the benchmark results comparision

- If a performance regression was detected, exit 1

### Track performance in CI

Below is a suggested Github action to run this tool with s3 history provider:

```yaml

name: Benchmark

# Ensure a single benchmark is run at a time

concurrency: cd-benchmark-${{ github.ref }}

on:

  push:

    branches:

      - main

  pull_request:

    branches:

      - main

jobs:

  s3:

    if: always()

    runs-on: ubuntu-latest

    # Ensure both don't run at the same time

    needs:

      - local

      # - ga-cache

    steps:

      - uses: actions/checkout@v2

      - uses: actions/setup-node@v2-beta

      - run: yarn install --frozen-lockfile

      # Run benchmark with custom tooling and stores the output to a file

      - name: Run performance tests

        run: yarn benchmark --s3

        env:

          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

          # S3 credentials

          S3_ACCESS_KEY: ${{ secrets.S3_ACCESS_KEY }}

          S3_SECRET_KEY: ${{ secrets.S3_SECRET_KEY }}

          S3_REGION: ${{ secrets.S3_REGION }}

          S3_BUCKET: ${{ secrets.S3_BUCKET }}

          S3_ENDPOINT: ${{ secrets.S3_ENDPOINT }}

          # Key prefix to separate benchmark data from multiple repositories

          S3_KEY_PREFIX: ${{ github.repository }}/${{ runner.os }}

```

### Track performance locally

When working on optimizing a function you may want to know if your code is actually faster than the previous implementation.

To do that you can keep a benchmark history file locally and run the benchmark first against previous code

```

git checkout master

benchmark test/perf/func.perf.ts --local

```

- Runs benchmark without comparing with previous

- Writes single benchmark data to `./benchmark_data`

Then measure performance with the new code

```

git checkout fix1

benchmark test/perf/func.perf.ts --local

```

- Reads single benchmark data from `./benchmark_data`

- Run benchmark comparing with prev

- Does not write benchmark data

## Config

### `--defaultBranch`

Provide the default branch of this repository to prevent fetching from Github

- type: string

- default:

### `--persistBranches`

Choose what branches to persist benchmark data

- type: array

- default: default-branch

### `--benchmarksPerBranch`

Limit number of benchmarks persisted per branch

- type: number

- default: Infinity

### `--threshold`

Ratio of new average time per run vs previos time per run to consider a failure. Set to 'Infinity' to disable it.

- type: number

- default: 2

### `--compareBranch`

Compare new benchmark data against the latest available benchmark in this branch

- type: string

- default: default-branch

### `--compareCommit`

Compare new benchmark data against the benchmark data associated with a specific commit

- type: string

- default:

### `--prune`

When persisting history, delete benchmark data associated with commits that are no longer in the current git history

- type: boolean

- default:

### `--persist`

Force persisting benchmark data in history

- type: boolean

- default:

### `--noThrow`

Exit cleanly even if a preformance regression was found

- type: boolean

- default:

### `--historyLocal`, `--local`

Persist benchmark history locally. May specify just a boolean to use a default path, or provide a path

- type: string

- default: ./benchmark_data

### `--historyGaCache`, `--ga-cache`

Persist benchmark history in Github Actions cache. Requires Github authentication. May specify just a boolean to use a default cache key or provide a custom key

- type: string

- default: benchmark_data

### `--historyS3`, `--s3`

Persist benchmark history in an Amazon S3 bucket. Requires Github authentication

- type: string

- default:

## Roadmap

### Compare performance

```

bench compare --from  --to 

```

- Retrieves benchmark history

- Don't run benchmark

- Print comparision of from benchmark with to benchmark

### Add more providers

- Github actions cache: Doesn't work due to Github's limitations

- Commited in the repository (gh-pages, or data branch)