https://github.com/clojure/test.benchmark
Benchmark and Regression Suite for Clojure
https://github.com/clojure/test.benchmark
Last synced: about 2 months ago
JSON representation
Benchmark and Regression Suite for Clojure
- Host: GitHub
- URL: https://github.com/clojure/test.benchmark
- Owner: clojure
- Created: 2010-10-26T11:08:09.000Z (about 15 years ago)
- Default Branch: master
- Last Pushed: 2024-07-15T18:02:15.000Z (over 1 year ago)
- Last Synced: 2025-01-30T06:51:16.203Z (11 months ago)
- Language: Roff
- Homepage:
- Size: 184 KB
- Stars: 37
- Watchers: 27
- Forks: 10
- Open Issues: 0
-
Metadata Files:
- Readme: README.markdown
- Contributing: CONTRIBUTING.md
Awesome Lists containing this project
README
Benchmark (and regression) suite for Clojure Copyright (c) Rich Hickey.
License and CA same as Clojure. See epl-v10.html at the root of the project for details.
## Getting Started
### Requirements
* JDK with java/javac on path - http://www.oracle.com/technetwork/java/javase/downloads/index.html
* maven - http://maven.apache.org/guides/getting-started/index.html
### Add the following to your ~/.m2/settings.xml file
clojure-dev
clojure-dev
false
clojars
http://clojars.org/repo/
...
...
### Building
```bash
git clone git@github.com:clojure/test.benchmark.git
cd test.benchmark
mvn test
```
### Misc
```bash
# starts a swank server
mvn clojure:swank
```
### Running a benchmark (e.g. alioth.thread-ring)
```bash
script/run alioth.thread-ring 1000
```
### Timing a benchmark (e.g. alioth.fasta-redux)
```bash
time script/run alioth.fasta-redux 25000000 > /dev/null
real 0m2.389s
user 0m3.329s
sys 0m0.115s
```
In these results, alioth uses a measurement similar to the `real` timing.
### Running a baseline test
Baseline tests execute a java-equivalent benchmark and compare execution times to the respective clojure benchmarks.
The specifications for the baseline tests are in https://github.com/clojure/test.benchmark/blob/master/src/baseline/clojure/baseline/spec.clj
```bash
# this will pull down the non-CA repo into src/baseline and build
# Currently some Java programs require JDK 1.7 for Fork/Join library.
mvn compile -Pbaseline
# afterwards you can execute baseline tests as follows
# default, run all baseline tests 3x, sample and report
script/run baseline.exec
# run all baseline tests 10x, sample and report
script/run baseline.exec 10
# run mandelbrot and thread-ring baseline tests once and report
script/run baseline.exec 1 mandelbrot thread-ring
({:failures :none,
:id "mandelbrot",
:metrics {:runtime {:baseline 9451.773698, :target 10316.261803}},
:tollerances {:runtime 0.25},
:sample-size 1}
{:failures :none,
:id "thread-ring",
:metrics {:runtime {:baseline 4308.804918, :target 309.521524}},
:tollerances {:runtime -9.0},
:sample-size 1})
```
TODO: hook baseline test execution into maven test cycle and generate a jenkins-consumable report
## Contributing
There are a number of completed benchmarks, but there is a lot more left to do.
Andy Fingerhut has a suit of benchmarks tailored for Clojure 1.2 and need updated or redone for Clojure 1.3 (https://github.com/jafingerhut/clojure-benchmarks).
A good general approach is to examine the fastest alioth implementations, usually Java or C, and write a Clojure port.
Fast small programs in any language are also worthy of examination. These benchmarks should demonstrate how to write idiomatic high-performance Clojure.
Long term goal is a performance regression test suit ...
Alioth Benchmark site is here: http://shootout.alioth.debian.org/
Work on the benchmarks is tracked here: http://dev.clojure.org/jira/browse/TBENCH