https://github.com/fluxml/fluxmlbenchmarks.jl
A benchmarking suite for the FluxML org
https://github.com/fluxml/fluxmlbenchmarks.jl
Last synced: 10 months ago
JSON representation
A benchmarking suite for the FluxML org
- Host: GitHub
- URL: https://github.com/fluxml/fluxmlbenchmarks.jl
- Owner: FluxML
- License: mit
- Created: 2023-06-04T18:22:33.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2024-04-30T04:58:29.000Z (about 2 years ago)
- Last Synced: 2025-02-27T03:50:04.840Z (over 1 year ago)
- Language: Julia
- Size: 147 KB
- Stars: 6
- Watchers: 5
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# FluxMLBenchmarks
`FluxMLBenchmarks` is a benchmarking tool designed for FluxML community, which allows for the creation of different benchmarking environments by installing different sets of dependencies and comparing the results.
## Use cases
### 1. Command-Line Interface
#### i. Single Package
To observe whether there is a performance difference between two versions of **the same package**, `FluxMLBenchmarks` provides 2 arguments, `--baseline` and `--target`, to specify the 2 versions of **the same package**.
```shell
> BASELINE=
> TARGET=
> julia --project=benchmark benchmark/runbenchmarks.jl --pr --target=$TARGET --baseline=$BASELINE
```
For specification, `Dependency Representation` is similar to the `word` of [add - REPL command - Pkg.jl](https://pkgdocs.julialang.org/v1/repl/#package-commands), described as follows:
| format | example |
| :-: | :-: |
| `` | `Flux` |
| `@` | `NNlib@0.8.20` |
| `#` | `Zygote#master` or `Zygote#2f4937096ee1db4b5a67c1c31fe3ebeab1c96c8c` |
| `` | `https://github.com/FluxML/Optimisers.jl` |
| `#` | `https://github.com/FluxML/Functors.jl#master` |
e.g.
```shell
> BASELINE="https://github.com/FluxML/NNlib.jl#backports-0.8.21"
> TARGET="https://github.com/skyleaworlder/NNlib.jl#dummy-benchmark-test"
> julia --project=benchmark benchmark/runbenchmarks.jl --pr --target=$TARGET --baseline=$BASELINE
```
#### ii. Multiple Packages
The performance of a package need measured under the condition that other packages and tools remain constant. However, in the case of mutual influence between **multiple packages of different versions**, **2 sets of dependencies** need to be provided simultaneously. As for this scenario, you can use `--baseline` and `--target` as well:
```shell
> BASELINE=,,
> TARGET=,,
> julia --project=benchmark benchmark/runbenchmarks.jl --pr --target=$TARGET --baseline=$BASELINE
```
#### iii. Multiple Sets of Dependencies
Sometimes we need to run benchmarks for multiple sets of dependencies simultaneously. To meet this benchmarking requirements, you can use `--deps-list`:
```shell
> DEPS_LIST=
> julia --project=benchmark benchmark/runbenchmarks.jl --cli --deps-list=$DEPS_LIST
```
For specification, `Dependencies List` is a single string that simulates an array, with each element separated by a semicolon. Each element adheres to the format of `Dependency Representation`. However, **Unlike the previous output `result-baseline.json` and `result-target.json`, the output format for this feature is `result-1.json`, `result-2.json`, `result-n.json`...**
e.g.
```shell
> DEPS_LIST="NNlib,Flux;https://github.com/FluxML/NNlib.jl#backports-0.8.21,Flux;https://github.com/skyleaworlder/NNlib.jl#backports-0.8.21,Flux@0.13.12"
> julia --project=benchmark benchmark/runbenchmarks.jl --cli --deps-list=$DEPS_LIST
```
### 2. GitHub Pull Request
TODO
## Command Arguments
### 0. `--pr` / `--cli` / `--cache-setup` / `--merge-reports`
Each argument represents an operation this tool will perform. The corresponding relationship is:
* `--pr`: "benchmark/script/runbenchmarks-pr.jl" You can specify `--target` `--baseline` `--enable` `--disable`
* `--cli`: "benchmark/script/runbenchmarks-cli.jl" You can specify `--deps-list` `--enable` `--disable`
* (**Not recommended, used by GitHub Actions**) `--cache-setup`: "benchmark/script/cachesetup-cli.jl" You can specify `--target` `--baseline`
* (**Not recommended, used by GitHub Actions**) `--merge-reports`: "benchmark/script/mergereports-cli.jl" You can specify `--target` `--baseline` `--push-result` `--push-username` `--push-useremail` `--push-password`
### 1. `--target` / `--baseline` / `--deps-list`
See [Use cases - Single Package](#i-single-package) and [Use cases - Multiple Packages](#ii-multiple-packages).
### 2. `--enable` / `--disable`
Benchmarking always takes amount of time. In order to focus on the targets and reduce the time consumption of our benchmarking tool, the `--enable` and `--disable` options are used to specify **the parts to be included** and **the parts to be excluded** respectively.
```shell
> julia --project=benchmark benchmark/runbenchmarks.jl --cli \
> --enable= \
> --disable= \
> --deps-list=
```
For specification, `Enabled Parts` and `Disabled Parts` have the same format, which is a single string that simulates an array, with each element separated by a semicolon.
`--enable` is used to specify the files that should be included, and by default (`--enable` not specified) all files in the `benchmark/benchmark` are included. `--disable` is used to specify the files that should be excluded, and the default value is an empty string.
More precisely, the granularity of the element of `Enabled Parts` and `Disabled Parts` is currently at the file-level, and supports two levels of files, which means that now **our tool will recognize the name of each file in `benchmark/benchmark` and all the files under `benchmark/benchmark/**` before benchmarking**.
**Each top-level element in `Enabled Parts` and `Disable Parts` should be exactly the name of the file under `benchmark/benchmark`; each second-level element should be the name of the file under the dir that has the same name of top-level file**.
> I don't recommend using `--enable` and `--disable` at the same time. But if you do, `--disable` takes priority over `--enable`.
>
> e.g. if `--enable` is set "flux,nnlib" while `--disable` is set "nnlib", only benchmarks in "benchmark/benchmark/flux.jl" will be executed.
e.g.
```shell
> DEPS_LIST="https://github.com/FluxML/NNlib.jl#backports-0.8.21,Flux;https://github.com/skyleaworlder/NNlib.jl#dummy-benchmark-test,Flux@0.13.12"
> # Only Flux-MLP and all NNlib
> julia --project=benchmark benchmark/runbenchmarks.jl --cli --enable="flux(mlp);nnlib" --deps-list=$DEPS_LIST
> # All benchmarks except Flux, NNlib-gemm and NNlib-activations
> julia --project=benchmark benchmark/runbenchmarks.jl --cli --disable="flux;nnlib(gemm,activations)" --deps-list=$DEPS_LIST
> # Only Flux
> julia --project=benchmark benchmark/runbenchmarks.jl --cli --enable="flux;nnlib" --disable="nnlib" --deps-list=$DEPS_LIST
```
### 3. `--fetch-result` / `--push-result` / `--push-username` / `--push-useremail` / `--push-password`
These arguments are only used in `--merge-reports`, not recommended.