Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/tkf/benchmarkci.jl
https://github.com/tkf/benchmarkci.jl
benchmark continuous-integration github-actions julia
Last synced: about 1 month ago
JSON representation
- Host: GitHub
- URL: https://github.com/tkf/benchmarkci.jl
- Owner: tkf
- License: mit
- Created: 2020-01-17T23:59:53.000Z (almost 5 years ago)
- Default Branch: master
- Last Pushed: 2022-05-05T08:28:06.000Z (over 2 years ago)
- Last Synced: 2024-10-19T17:45:12.334Z (2 months ago)
- Topics: benchmark, continuous-integration, github-actions, julia
- Language: Julia
- Size: 1.57 MB
- Stars: 53
- Watchers: 7
- Forks: 6
- Open Issues: 13
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# BenchmarkCI.jl
[![Latest documentation][docs-dev-img]][docs-dev-url]
[![CI Status][ci-img]][ci-url]
[![codecov.io][codecov-img]][codecov-url]
BenchmarkCI.jl provides an easy way to run benchmark suite via GitHub
Actions. It only needs a minimal setup if there is a benchmark suite
declared by
[BenchmarkTools.jl](https://github.com/JuliaCI/BenchmarkTools.jl) /
[PkgBenchmark.jl](https://github.com/JuliaCI/PkgBenchmark.jl) API.
**Warning** This package is still experimental. Make sure to fix the
version number in your CI setup.
## Setup
BenchmarkCI.jl requires PkgBenchmark.jl to work. See
[Defining a benchmark suite · PkgBenchmark.jl](https://juliaci.github.io/PkgBenchmark.jl/stable/define_benchmarks/)
for more information. BenchmarkCI.jl also requires a Julia project
`benchmark/Project.toml` that is used for running the benchmark.
### Create a workflow file (required)
Create (say) `.github/workflows/benchmark.yml` with the following
configuration:
```yaml
name: Run benchmarks
on:
pull_request:
jobs:
Benchmark:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: julia-actions/setup-julia@latest
with:
version: 1
- uses: julia-actions/julia-buildpkg@latest
- name: Install dependencies
run: julia -e 'using Pkg; pkg"add PkgBenchmark [email protected]"'
- name: Run benchmarks
run: julia -e 'using BenchmarkCI; BenchmarkCI.judge()'
- name: Post results
run: julia -e 'using BenchmarkCI; BenchmarkCI.postjudge()'
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
```
If you don't want to benchmark your code for every push of every PR,
then you can conditionally trigger the jobs on a label:
```yaml
name: Run benchmarks
on:
pull_request:
types: [labeled, opened, synchronize, reopened]
# Only trigger the benchmark job when you add `run benchmark` label to the PR
jobs:
Benchmark:
runs-on: ubuntu-latest
if: contains(github.event.pull_request.labels.*.name, 'run benchmark')
steps:
- uses: actions/checkout@v2
- uses: julia-actions/setup-julia@latest
with:
version: 1
- uses: julia-actions/julia-buildpkg@latest
- name: Install dependencies
run: julia -e 'using Pkg; pkg"add PkgBenchmark [email protected]"'
- name: Run benchmarks
run: julia -e 'using BenchmarkCI; BenchmarkCI.judge()'
- name: Post results
run: julia -e 'using BenchmarkCI; BenchmarkCI.postjudge()'
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
```
### Setup with `benchmark/Manifest.toml`
If `benchmark/Manifest.toml` is checked into the repository,
`benchmark/Project.toml` must include parent project as well. Run
`dev ..` in `benchmark/` directory to add it:
```
shell> cd ~/.julia/dev/MyProject/
shell> cd benchmark/
(@v1.x) pkg> activate .
Activating environment at `~/.julia/dev/MyProject/benchmark/Project.toml`
(benchmark) pkg> dev ..
```
### Additional setup (recommended)
It is recommended to add following two lines in `.gitignore`:
```
/.benchmarkci
/benchmark/*.json
```
This is useful for running BenchmarkCI locally (see below).
### Printing benchmark result (optional)
Posting the benchmark result as a comment for every push for each PR
may be too noisy. In such case, using
`BenchmarkCI.displayjudgement()` instead of `BenchmarkCI.postjudge()`
may be useful.
```yaml
- name: Print judgement
run: julia -e 'using BenchmarkCI; BenchmarkCI.displayjudgement()'
```
### Store benchmark result in a Git branch (optional; _very_ experimental)
Alternatively, the benchmark result and report markdown can be pushed
to a git branch `benchmark-results`
([example](https://github.com/tkf/BenchmarkCI-data/tree/benchmark-results)).
```yaml
- name: Push results
run: julia -e "using BenchmarkCI; BenchmarkCI.pushresult()"
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
SSH_KEY: ${{ secrets.DOCUMENTER_KEY }}
```
This method can also be used in Travis CI. See
[this example `.travis.yml`](.travis.yml).
**WARNING**: Storage format may be changed across releases.
## Running BenchmarkCI interactively
```
shell> cd ~/.julia/dev/MyProject/
julia> using BenchmarkCI
julia> BenchmarkCI.judge()
...
julia> BenchmarkCI.displayjudgement()
...
```
[ci-img]: https://github.com/tkf/BenchmarkCI.jl/workflows/Run%20tests/badge.svg
[ci-url]: https://github.com/tkf/BenchmarkCI.jl/actions?query=workflow%3A%22Run+tests%22
[codecov-img]: http://codecov.io/github/tkf/BenchmarkCI.jl/coverage.svg?branch=master
[codecov-url]: http://codecov.io/github/tkf/BenchmarkCI.jl?branch=master
[docs-dev-img]: https://img.shields.io/badge/docs-dev-blue.svg
[docs-dev-url]: https://tkf.github.io/BenchmarkCI.jl/dev