https://github.com/optuna/kurobako
A black-box optimization benchmark tool
https://github.com/optuna/kurobako
black-box-benchmarking black-box-optimization rust
Last synced: 8 months ago
JSON representation
A black-box optimization benchmark tool
- Host: GitHub
- URL: https://github.com/optuna/kurobako
- Owner: optuna
- License: mit
- Created: 2019-03-12T12:56:25.000Z (about 7 years ago)
- Default Branch: master
- Last Pushed: 2023-06-21T22:41:28.000Z (almost 3 years ago)
- Last Synced: 2024-05-01T11:35:05.519Z (about 2 years ago)
- Topics: black-box-benchmarking, black-box-optimization, rust
- Language: Rust
- Homepage:
- Size: 816 KB
- Stars: 80
- Watchers: 8
- Forks: 9
- Open Issues: 10
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
kurobako
=========
[](https://crates.io/crates/kurobako)
[](https://docs.rs/kurobako)
[](https://github.com/optuna/kurobako/actions)
[](https://coveralls.io/github/optuna/kurobako?branch=master)
[](LICENSE)
[](https://hub.docker.com/r/sile/kurobako)
`kurobako` is a command-line tool to benchmark black-box optimization algorithms.
Features:
- It has the capability to support wide range of optimization problems:
- various search sapce:
- Continuous numerical, discrete numerical and categorical
- Uniform distribution and log uniform distribution
- Conditional
- Constrainted problems
- Multi-objective problems
- Generating a markdown report and PNG plots from benchmarking results
- Easy to add user-defined optimization problems and solvers
- Simulating a concurrent environment in which an optimization process is executed by multiple workers simultaneously
- Reproducible
Installation
------------
### Precompiled binaries
Precompiled binaries for Linux are available in the [releases] page.
```console
$ curl -L https://github.com/optuna/kurobako/releases/download/${VERSION}/kurobako-${VERSION}.linux-amd64 -o kurobako
$ chmod +x kurobako
$ ./kurobako -h
```
[releases]: https://github.com/optuna/kurobako/releases
### Using Cargo
If you have already installed [Cargo][cargo], you can install `kurobako` by executing the following command:
```console
$ cargo install kurobako
```
[cargo]: https://doc.rust-lang.org/cargo/
### Dependencies
If you want to use `kurobako plot` command, you need to have installed `gnuplot` package on your environment.
Usage Example
--------------
```console
// Define solver.
$ kurobako solver random | tee solver.json
{"random":{}}
// Define problem.
$ curl -OL http://ml4aad.org/wp-content/uploads/2019/01/fcnet_tabular_benchmarks.tar.gz
$ tar xf fcnet_tabular_benchmarks.tar.gz && cd fcnet_tabular_benchmarks/
$ kurobako problem hpobench fcnet_protein_structure_data.hdf5 | tee problem.json
{"hpobench":{"dataset":"fcnet_protein_structure_data.hdf5"}}
// Run benchmark.
$ kurobako studies --solvers $(cat solver.json) --problems $(cat problem.json) | kurobako run > result.json
(ALL) [00:00:01] [STUDIES 10/10 100%] [ETA 0s] done
// Report the benchmark result.
$ cat result.json | kurobako report
...abbrev...
// Plot the benchmark result.
$ cat result.json | kurobako plot curve
(PLOT) [00:00:01] [1/1 100%] [ETA 0s] done (dir="images/curve/")
```
Build-in Solvers and Problems
-----------------------------
Solvers:
- Random Search
- [NSGA-II](https://ieeexplore.ieee.org/document/996017)
- [ASHA](https://arxiv.org/abs/1810.05934)
- [Optuna](https://github.com/optuna/optuna)
Problems:
- [NASBench](https://github.com/automl/nas_benchmarks) ([detail](https://github.com/optuna/kurobako/wiki/NASBench))
- [HPOBench](https://github.com/automl/nas_benchmarks)
- [sigopt/evalset](https://github.com/sigopt/evalset)
- [Two-objective ZDT functions](http://repository.ias.ac.in/9404/1/306.pdf)
Where does the name come from?
-----------------------------------
"kurobako" is a Japanese translation of "black box".
References
----------
- [Introduction to Kurobako: A Benchmark Tool for Hyperparameter Optimization Algorithms](https://medium.com/optuna/kurobako-a2e3f7b760c7)