https://github.com/sgraf812/ghc-benchmark

Benchmarking harness for GHC diffs
https://github.com/sgraf812/ghc-benchmark

Last synced: about 2 months ago
JSON representation

Benchmarking harness for GHC diffs

• Host: GitHub
• URL: https://github.com/sgraf812/ghc-benchmark
• Owner: sgraf812
• Created: 2018-04-03T19:41:23.000Z (almost 7 years ago)
• Default Branch: master
• Last Pushed: 2023-10-02T09:18:47.000Z (over 1 year ago)
• Last Synced: 2024-10-13T10:42:16.605Z (3 months ago)
• Language: Shell
• Homepage:
• Size: 186 KB
• Stars: 2
• Watchers: 4
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# `ghc-benchmark`

This is a harness I stole from Joachim Breitner (@nomeata) for reproducible, CI-like `NoFib` runs.

# Setup

1. Make sure you have Nix installed

2. Checkout the appropriate base commit in `ghc/`

3. Add your diff to `diffs/` (leave the `base.diff` untouched, unless you don't want to compare to the baseline commit). This may come from Phabricator, which is the result of `arc diff`, or just do `git diff > faster.diff` in your local working copy. The file name must match `$DIFF.diff`, where `$DIFF` is used to identify your patch in generated files and in the eventual output.

# Benchmark

Run `./benchmark.sh`.

This will benchmark **all** `diffs/` anew on every run. It will use cachegrind to count instructions and only run each benchmark once, so measured Runtimes will be garbage (they would be on Windows anyway due to VM overhead). This is also the default behavior for https://perf.haskell.org/ghc.

If you just want to re-run selected diffs because you amended some changes, pass them as arguments: `./docker-benchmark.sh $DIFF1 $DIFF2`.

You can find buildlogs (`buildlog-$DIFF-.log`) and logs from the `NoFib` run (`$DIFF-.log`) in `logs/`. You probably want to feed these to `nofib-analyse`.

# `gen-tables.sh`

To automate the invocation of `nofib-analyse`, try `./gen-tables.sh`. This will (re-)generate select reports by taking the most recent log file of every diff it can find. To generate reports that only mention a subset of diffs, do `gen-tables.sh $DIFF1 $DIFF2`. The `base` diff will always be used as baseline.

You can see how various the `.tex` and `.txt` reports are built by invoking `nofib-analyse` in `gen-tables.sh`. The analysed output is passed to the `fixup.pl` script, which takes care of weeding out insignificant changes for readability. Play around with the threshold parameters to find a setting you like.