Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/kostya/benchmarks
Some benchmarks of different languages
https://github.com/kostya/benchmarks
benchmarks languages
Last synced: 20 days ago
JSON representation
Some benchmarks of different languages
- Host: GitHub
- URL: https://github.com/kostya/benchmarks
- Owner: kostya
- License: mit
- Created: 2014-10-13T16:08:23.000Z (over 10 years ago)
- Default Branch: master
- Last Pushed: 2024-11-21T03:10:16.000Z (3 months ago)
- Last Synced: 2024-12-05T04:24:34.744Z (3 months ago)
- Topics: benchmarks, languages
- Language: Makefile
- Homepage:
- Size: 1.8 MB
- Stars: 2,824
- Watchers: 98
- Forks: 255
- Open Issues: 52
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-list - benchmarks
- awesome-rust-list - kostya/benchmarks
- awesome-rust-list - kostya/benchmarks
README
# Table of Content
* [Overview](#overview)
* [Measurements](#measurements)
* [Test Cases](#test-cases)
* [Brainfuck](#brainfuck)
* [bench.b](#benchb)
* [mandel.b](#mandelb)
* [Base64](#base64)
* [Json](#json)
* [Matmul](#matmul)
* [Primes](#primes)
* [Tests Execution](#tests-execution)
* [Environment](#environment)
* [Using Docker](#using-docker)
* [Manual Execution](#manual-execution)
* [Prerequisites](#prerequisites)
* [Contribution](#contribution)
* [Makefile guide](#makefile-guide)
* [Binary executables](#binary-executables)
* [Compiled artifacts](#compiled-artifacts)
* [Scripting language](#scripting-language)
* [README update](#readme-update)
* [Docker image update](#docker-image-update)
# Overview
The benchmarks follow the criteria:
- They are written as the average software developer would write them, i.e.
- The algorithms are implemented as cited in public sources;
- The libraries are used as described in the tutorials, documentation and examples;
- The used data structures are idiomatic.
- The used algorithms are similar between the languages (as the reference implementations), variants are acceptable if the reference implementation exists.
- All final binaries are releases (optimized for performance if possible) as debug performance may vary too much depending on the compiler.
My other benchmarks: [jit-benchmarks](https://github.com/kostya/jit-benchmarks), [crystal-benchmarks-game](https://github.com/kostya/crystal-benchmarks-game)
## Measurements
The measured values are:
- time spent for the benchmark execution (loading required data and code self-testing are not measured);
- memory consumption of the benchmark process, reported as `base` + `increase`, where `base` is the RSS before the benchmark and `increase` is the peak increase of the RSS during the benchmark;
- energy consumption of the CPU package during the benchmark: PP0 (cores) + PP1 (uncores like GPU) + DRAM. Currently, only Intel CPU are supported via the powercap interface.
All values are presented as: `median`±`median absolute deviation`.
UPDATE: 2024-11-13
# Test Cases
## Brainfuck
Testing brainfuck implementations using two code samples (bench.b and mandel.b).
Supports two mode:
- Verbose (default). Prints the output immediately.
- Quiet (if QUIET environment variable is set). Accumulates the output using Fletcher-16 checksum, and prints it out after the benchmark.
[Brainfuck](brainfuck)
### bench.b
| Language | Time, s | Memory, MiB | Energy, J |
| :--------------------- | -----------------------: | ------------------------------------------------: | -------------------------: |
| Scala (Staged) | 0.425±0.017 | 208.26±02.66 + 36.81±04.81 | 27.57±01.62 |
| Racket (Staged) | 0.885±0.000 | 105.98±00.09 + 0.00±00.00 | 34.27±00.03 |
| Rust | 1.012±0.000 | 2.00±00.00 + 0.00±00.00 | 42.99±00.30 |
| C++/g++ | 1.095±0.002 | 3.38±00.00 + 0.00±00.00 | 45.08±00.23 |
| C/gcc | 1.111±0.001 | 1.50±00.00 + 0.00±00.00 | 46.73±00.25 |
| D/gdc | 1.118±0.002 | 6.75±00.00 + 0.00±00.00 | 48.63±00.23 |
| C/clang | 1.138±0.001 | 1.38±00.00 + 0.00±00.00 | 48.14±00.11 |
| C++/clang++ | 1.166±0.000 | 2.88±00.00 + 0.00±00.00 | 49.24±01.17 |
| D/ldc2 | 1.167±0.001 | 3.12±00.06 + 0.00±00.00 | 49.12±00.20 |
| Nim/gcc | 1.168±0.001 | 1.94±00.06 + 0.00±00.00 | 48.20±00.22 |
| Java | 1.188±0.000 | 42.25±00.11 + 0.81±00.06 | 48.12±00.33 |
| V/gcc | 1.198±0.001 | 2.12±00.00 + 0.00±00.00 | 47.85±00.12 |
| Vala/gcc | 1.210±0.001 | 5.38±00.06 + 0.00±00.00 | 50.36±00.52 |
| Kotlin/JVM | 1.227±0.002 | 44.92±00.09 + 0.88±00.12 | 51.69±00.13 |
| Vala/clang | 1.234±0.000 | 5.31±00.06 + 0.00±00.00 | 52.04±00.89 |
| Go | 1.247±0.001 | 3.62±00.06 + 0.00±00.00 | 51.44±00.38 |
| Zig | 1.266±0.003 | 1.25±00.00 + 0.00±00.00 | 53.30±01.05 |
| Go/gccgo | 1.288±0.000 | 23.88±00.12 + 0.00±00.00 | 54.12±00.51 |
| C#/.NET Core | 1.367±0.001 | 32.38±00.06 + 0.12±00.00 | 58.25±00.89 |
| Julia | 1.551±0.014 | 248.75±00.06 + 0.25±00.00 | 66.53±00.84 |
| Nim/clang | 1.582±0.000 | 2.12±00.00 + 0.00±00.00 | 64.79±00.92 |
| F#/.NET Core | 1.593±0.002 | 37.49±00.06 + 0.38±00.12 | 68.98±01.23 |
| Crystal | 1.635±0.004 | 3.44±00.06 + 0.00±00.00 | 68.71±00.54 |
| OCaml | 1.657±0.003 | 3.25±00.00 + 2.62±00.00 | 76.47±00.71 |
| Chez Scheme | 1.748±0.002 | 25.12±00.00 + 3.38±00.00 | 73.58±00.24 |
| Racket | 1.794±0.027 | 93.84±00.05 + 22.75±00.19 | 73.46±01.48 |
| V/clang | 1.891±0.019 | 2.06±00.06 + 0.00±00.00 | 82.57±01.03 |
| C#/Mono | 2.034±0.001 | 25.84±00.12 + 0.00±00.00 | 86.11±00.25 |
| MLton | 2.077±0.011 | 1.75±00.00 + 0.38±00.00 | 86.06±01.09 |
| Scala | 2.777±0.005 | 62.17±00.16 + 145.05±00.32 | 122.07±00.64 |
| Node.js | 3.150±0.005 | 46.78±00.00 + 6.12±00.00 | 131.99±01.28 |
| Haskell (MArray) | 3.236±0.003 | 4.12±00.00 + 4.12±00.00 | 134.34±00.29 |
| D/dmd | 3.328±0.001 | 3.50±00.00 + 0.00±00.00 | 125.22±00.34 |
| Haskell (FP) | 3.786±0.003 | 4.25±00.06 + 4.00±00.00 | 160.86±00.26 |
| Ruby/truffleruby | 4.669±0.152 | 201.88±00.69 + 814.31±49.81 | 225.04±10.28 |
| Swift | 5.593±0.008 | 17.38±00.00 + 0.00±00.00 | 210.94±02.31 |
| Lua/luajit | 5.687±0.024 | 2.62±00.00 + 0.00±00.00 | 234.94±01.49 |
| Ruby/truffleruby (JVM) | 5.884±0.028 | 390.44±02.98 + 468.57±35.67 | 289.91±02.92 |
| Python/pypy | 9.575±0.073 | 60.50±00.12 + 29.45±00.01 | 426.59±05.95 |
| Idris | 15.559±0.040 | 20.63±00.06 + 8.38±00.00 | 700.59±04.61 |
| Elixir | 20.530±0.047 | 71.27±00.45 + 0.00±00.00 | 816.30±01.74 |
| Ruby (--jit) | 31.953±0.050 | 21.47±00.07 + 4.88±00.00 | 1320.18±03.40 |
| PHP | 34.067±0.088 | 18.88±00.00 + 0.00±00.00 | 1442.70±16.86 |
| Lua | 37.027±0.060 | 2.62±00.00 + 0.00±00.00 | 1523.51±15.61 |
| Python | 54.076±0.669 | 11.88±00.00 + 0.00±00.00 | 2418.75±26.40 |
| Ruby | 71.836±0.205 | 11.31±00.06 + 0.00±00.00 | 3123.03±27.59 |
| Ruby/jruby | 79.903±1.524 | 203.90±02.51 + 222.73±32.77 | 3573.69±72.79 |
| Tcl (FP) | 193.372±0.720 | 4.62±00.12 + 0.00±00.00 | 8541.35±121.54 |
| Perl | 223.395±1.420 | 7.06±00.06 + 0.12±00.00 | 9755.30±45.94 |
| Tcl (OOP) | 382.403±3.408 | 4.62±00.12 + 0.12±00.00 | 16926.54±148.14 |
### mandel.b
[Mandel in Brainfuck](brainfuck/mandel.b)
| Language | Time, s | Memory, MiB | Energy, J |
| :--------------------- | ----------------------: | ------------------------------------------------: | -----------------------: |
| Scala (Staged) | 7.751±0.157 | 211.17±04.45 + 132.95±12.75 | 465.58±11.30 |
| C++/g++ | 10.076±0.018 | 3.50±00.00 + 0.62±00.00 | 416.16±01.64 |
| C#/.NET Core | 11.991±0.035 | 32.56±00.06 + 1.38±00.00 | 481.39±02.69 |
| Java | 12.441±0.041 | 42.18±00.16 + 2.18±00.07 | 500.84±03.37 |
| C/gcc | 12.597±0.005 | 1.62±00.00 + 0.25±00.00 | 509.44±01.23 |
| C++/clang++ | 12.910±0.015 | 3.00±00.00 + 0.62±00.00 | 534.77±00.89 |
| Kotlin/JVM | 13.168±0.070 | 44.94±00.06 + 1.94±00.27 | 551.88±02.60 |
| C/clang | 13.205±0.013 | 1.62±00.00 + 0.25±00.00 | 581.89±04.42 |
| Zig | 13.314±0.028 | 1.38±00.06 + 1.00±00.00 | 558.65±04.76 |
| F#/.NET Core | 13.335±0.024 | 37.61±00.06 + 2.12±00.12 | 534.94±02.52 |
| Go | 14.113±0.014 | 3.62±00.06 + 0.00±00.00 | 568.61±02.43 |
| Rust | 14.242±0.020 | 1.88±00.00 + 0.25±00.00 | 569.99±01.08 |
| D/ldc2 | 14.272±0.019 | 3.25±00.06 + 0.75±00.00 | 584.52±03.64 |
| V/gcc | 14.333±0.038 | 2.12±00.06 + 1.00±00.00 | 560.92±03.16 |
| Vala/gcc | 14.436±0.031 | 5.38±00.00 + 0.62±00.00 | 572.24±04.78 |
| Racket (Staged) | 14.758±0.159 | 105.70±00.14 + 71.38±01.87 | 584.52±06.50 |
| Vala/clang | 14.781±0.010 | 5.38±00.12 + 0.62±00.00 | 593.05±01.04 |
| Crystal | 14.942±0.019 | 3.50±00.00 + 0.25±00.00 | 631.98±02.50 |
| Nim/gcc | 15.017±0.014 | 2.06±00.06 + 1.50±00.00 | 608.83±01.49 |
| D/gdc | 15.433±0.016 | 6.75±00.00 + 0.88±00.00 | 642.43±03.76 |
| Scala | 16.477±0.112 | 62.29±00.09 + 144.88±00.19 | 717.49±03.80 |
| Nim/clang | 17.695±0.052 | 2.38±00.12 + 1.50±00.00 | 719.34±09.74 |
| Swift | 18.038±0.044 | 17.62±00.12 + 0.12±00.00 | 740.60±03.26 |
| V/clang | 18.356±0.084 | 2.12±00.06 + 1.00±00.00 | 806.70±06.57 |
| Go/gccgo | 18.686±0.314 | 24.06±00.06 + 0.00±00.00 | 773.76±12.67 |
| OCaml | 24.514±0.018 | 4.38±00.00 + 3.12±00.00 | 1178.94±06.82 |
| Julia | 30.494±0.068 | 248.88±00.00 + 0.25±00.00 | 1291.74±10.50 |
| C#/Mono | 31.075±0.053 | 25.81±00.15 + 0.75±00.00 | 1320.05±03.33 |
| Chez Scheme | 31.201±0.191 | 25.45±00.00 + 3.12±00.00 | 1372.33±16.13 |
| Lua/luajit | 31.900±0.061 | 2.50±00.06 + 0.25±00.00 | 1318.65±02.87 |
| MLton | 33.684±0.058 | 1.75±00.00 + 4.27±00.00 | 1527.54±10.09 |
| Node.js | 34.372±0.081 | 46.80±00.00 + 8.62±00.12 | 1380.19±03.68 |
| Haskell (MArray) | 35.178±0.039 | 4.00±00.06 + 5.12±00.00 | 1431.66±08.02 |
| Racket | 35.789±0.831 | 93.83±00.07 + 23.62±00.12 | 1591.23±45.98 |
| D/dmd | 37.954±0.004 | 3.38±00.06 + 0.88±00.00 | 1390.20±05.03 |
| Python/pypy | 41.239±0.111 | 60.38±00.12 + 30.13±00.05 | 1834.39±16.72 |
| Ruby/truffleruby | 48.778±0.439 | 201.31±00.94 + 734.44±18.06 | 2375.16±22.12 |
| Ruby/truffleruby (JVM) | 50.823±2.734 | 393.21±05.77 + 405.25±32.51 | 2255.45±41.59 |
| Idris | 66.743±0.120 | 21.82±00.00 + 9.25±00.00 | 2977.76±12.58 |
| Haskell (FP) | 78.765±0.269 | 4.00±00.00 + 76.00±00.00 | 3323.00±06.05 |
## Base64
Testing base64 encoding/decoding of the large blob into the newly allocated buffers.
[Base64](base64)
| Language | Time, s | Memory, MiB | Energy, J |
| :------------------------ | ----------------------: | ------------------------------------------------: | ----------------------: |
| C/clang (aklomp) | 0.095±0.001 | 2.12±00.00 + 0.00±00.00 | 4.50±00.05 |
| C/gcc (aklomp) | 0.097±0.000 | 2.12±00.00 + 0.00±00.00 | 4.56±00.02 |
| PHP | 0.105±0.000 | 19.25±00.00 + 0.00±00.00 | 4.93±00.03 |
| Go (base64x) | 0.275±0.002 | 6.62±00.06 + 0.00±00.00 | 13.00±00.06 |
| Node.js | 0.663±0.004 | 45.19±00.02 + 40.96±00.16 | 29.74±00.50 |
| Zig | 0.703±0.000 | 1.75±00.00 + 0.12±00.00 | 26.41±00.10 |
| Rust | 0.875±0.000 | 2.38±00.00 + 0.12±00.00 | 36.15±00.19 |
| C/clang | 0.918±0.000 | 2.00±00.00 + 0.12±00.00 | 33.80±00.13 |
| Nim/clang | 0.944±0.002 | 2.81±00.06 + 4.94±00.06 | 37.54±00.31 |
| Crystal | 1.039±0.001 | 3.88±00.00 + 1.00±00.00 | 42.82±00.44 |
| C/gcc | 1.106±0.001 | 1.88±00.00 + 0.12±00.00 | 40.27±00.33 |
| D/ldc2 | 1.179±0.001 | 3.75±00.00 + 3.38±00.00 | 48.73±00.37 |
| Nim/gcc | 1.347±0.001 | 2.62±00.00 + 4.62±00.00 | 54.54±00.35 |
| Java | 1.433±0.002 | 43.05±00.10 + 215.12±03.94 | 59.06±00.69 |
| Ruby (--jit) | 1.503±0.002 | 15.09±00.06 + 72.88±00.38 | 58.96±00.26 |
| Scala | 1.531±0.002 | 59.09±00.11 + 265.94±05.44 | 65.30±00.39 |
| Kotlin/JVM | 1.567±0.003 | 45.92±00.12 + 263.50±05.75 | 67.15±00.22 |
| V/gcc | 1.630±0.001 | 2.62±00.00 + 2379.12±01.50 | 59.57±00.39 |
| Vala/gcc | 1.645±0.001 | 5.75±00.00 + 0.12±00.00 | 63.19±00.43 |
| Vala/clang | 1.645±0.001 | 5.81±00.06 + 0.12±00.00 | 63.50±01.04 |
| Go | 1.659±0.005 | 4.25±00.00 + 0.00±00.00 | 68.14±00.65 |
| C++/clang++ (libcrypto) | 1.705±0.003 | 6.06±00.06 + 0.11±00.00 | 68.13±00.60 |
| C++/g++ (libcrypto) | 1.706±0.004 | 6.50±00.00 + 0.12±00.00 | 68.03±00.51 |
| Ruby | 1.738±0.002 | 11.75±00.06 + 39.74±00.39 | 68.37±00.46 |
| Perl (MIME::Base64) | 2.000±0.003 | 14.94±00.06 + 0.00±00.00 | 80.98±00.50 |
| C#/.NET Core | 2.494±0.039 | 33.85±00.03 + 17.63±01.91 | 97.82±01.66 |
| F#/.NET Core | 2.580±0.027 | 38.48±00.01 + 19.97±01.83 | 98.40±01.39 |
| Tcl | 2.753±0.005 | 5.12±00.00 + 0.12±00.00 | 114.23±00.59 |
| Python | 2.872±0.001 | 11.62±00.00 + 0.12±00.00 | 112.57±00.90 |
| Julia | 2.983±0.002 | 263.48±00.11 + 28.98±00.08 | 120.37±00.91 |
| Go/gccgo | 3.020±0.005 | 25.00±00.12 + 0.00±00.00 | 142.55±00.34 |
| V/clang | 3.248±0.001 | 2.75±00.00 + 2388.56±00.50 | 119.99±00.66 |
| Python/pypy | 3.249±0.009 | 60.25±00.12 + 31.31±00.05 | 143.38±01.06 |
| Ruby/truffleruby (JVM) | 3.443±0.007 | 389.16±05.72 + 294.50±17.44 | 173.84±01.07 |
| Racket | 3.903±0.006 | 93.95±00.11 + 25.88±00.12 | 155.48±00.43 |
| D/gdc | 3.920±0.001 | 7.06±00.06 + 3.75±00.06 | 162.11±01.78 |
| D/dmd | 4.020±0.002 | 3.62±00.00 + 3.38±00.00 | 164.62±00.95 |
| C#/Mono | 4.800±0.010 | 26.58±00.20 + 18.86±00.03 | 198.99±01.14 |
| Ruby/jruby | 6.210±0.070 | 196.29±01.99 + 144.32±10.97 | 273.42±04.78 |
| Ruby/truffleruby | 8.259±0.038 | 200.88±00.50 + 542.00±02.56 | 399.78±03.55 |
| Perl (MIME::Base64::Perl) | 10.394±0.088 | 16.25±00.06 + 0.21±00.03 | 457.56±03.07 |
## Json
Testing parsing and simple calculating of values from a big JSON file.
Few notes:
- gason mutates input strings;
- simdjson requires input strings with batch of trailing zeros: a special zero padding for SIMD instructions;
- DAW JSON Link "NoCheck" skips some JSON structure correctness checks;
- DAW JSON Link, gason, default (not "Precise") RapidJSON, and D implementations except Mir-based
have some inaccuracies in number parsing:
- [DAW JSON Link's number parsing issue](https://github.com/beached/daw_json_link/issues/226)
- [gason's number parsing issue](https://github.com/vivkin/gason/issues/35)
- [D stdlib number parsing issue](https://issues.dlang.org/show_bug.cgi?id=20967)
[Json](json)
| Language | Time, s | Memory, MiB | Energy, J |
| :---------------------------------- | ----------------------: | --------------------------------------------------: | ----------------------: |
| C++/clang++ (simdjson On-Demand) | 0.061±0.000 | 112.88±00.00 + 59.75±00.00 | 2.55±00.01 |
| C++/g++ (simdjson On-Demand) | 0.061±0.000 | 113.50±00.00 + 59.75±00.00 | 2.56±00.01 |
| C++/clang++ (DAW JSON Link NoCheck) | 0.071±0.000 | 112.71±00.00 + 0.00±00.00 | 2.95±00.03 |
| C++/g++ (DAW JSON Link NoCheck) | 0.083±0.000 | 113.21±00.00 + 0.00±00.00 | 3.39±00.02 |
| C++/clang++ (DAW JSON Link) | 0.092±0.000 | 112.77±00.06 + 0.00±00.00 | 3.84±00.03 |
| C++/g++ (DAW JSON Link) | 0.092±0.000 | 113.33±00.01 + 0.00±00.00 | 3.78±00.04 |
| C++/clang++ (simdjson DOM) | 0.103±0.001 | 112.81±00.06 + 174.75±00.81 | 4.67±00.06 |
| C++/g++ (simdjson DOM) | 0.105±0.001 | 113.38±00.00 + 172.62±00.31 | 4.82±00.05 |
| Rust (Serde Custom) | 0.107±0.000 | 111.62±00.00 + 0.00±00.00 | 4.54±00.03 |
| Rust (Serde Typed) | 0.112±0.000 | 111.62±00.00 + 12.12±00.00 | 4.73±00.07 |
| C++/g++ (gason) | 0.127±0.000 | 113.21±00.00 + 96.75±00.00 | 5.11±00.06 |
| D/ldc2 (Mir Asdf DOM) | 0.132±0.000 | 112.88±00.00 + 61.25±00.00 | 5.41±00.05 |
| C++/clang++ (gason) | 0.134±0.000 | 112.70±00.06 + 96.75±00.00 | 5.32±00.03 |
| C++/g++ (RapidJSON) | 0.144±0.002 | 113.34±00.06 + 125.54±01.73 | 6.15±00.12 |
| Scala (jsoniter-scala) | 0.153±0.001 | 279.53±00.11 + 26.48±00.73 | 8.43±00.17 |
| Go (rjson custom) | 0.194±0.000 | 113.94±00.06 + 0.00±00.00 | 7.52±00.02 |
| C++/clang++ (RapidJSON) | 0.195±0.001 | 112.83±00.00 + 128.62±00.00 | 8.20±00.03 |
| C++/g++ (RapidJSON Precise) | 0.202±0.001 | 113.34±00.00 + 128.75±00.00 | 8.61±00.08 |
| Go (Sonic) | 0.204±0.002 | 123.62±00.06 + 0.00±00.00 | 8.87±00.07 |
| D/ldc2 (Mir Amazon's Ion DOM) | 0.220±0.001 | 112.88±00.00 + 80.75±00.00 | 9.32±00.03 |
| Go (rjson) | 0.221±0.000 | 114.00±00.00 + 0.00±00.00 | 8.58±00.04 |
| Zig | 0.225±0.000 | 111.12±00.00 + 39.25±00.00 | 9.76±00.13 |
| Go (goccy/go-json) | 0.260±0.001 | 114.44±00.06 + 0.00±00.00 | 10.41±00.16 |
| C++/clang++ (RapidJSON Precise) | 0.262±0.001 | 112.71±00.00 + 128.75±00.00 | 11.06±00.14 |
| C++/g++ (RapidJSON SAX) | 0.351±0.001 | 113.21±00.00 + 0.00±00.00 | 15.53±00.10 |
| C/clang (yajl) | 0.362±0.001 | 111.31±00.06 + 0.00±00.00 | 15.71±00.27 |
| C/gcc (yajl) | 0.365±0.000 | 111.31±00.06 + 0.00±00.00 | 15.77±00.10 |
| C++/g++ (Boost.JSON) | 0.365±0.001 | 113.60±00.00 + 307.75±00.00 | 15.48±00.06 |
| C++/clang++ (Boost.JSON) | 0.375±0.002 | 112.96±00.12 + 307.75±00.00 | 16.06±00.10 |
| Nim/clang (jsony) | 0.382±0.001 | 112.06±00.06 + 154.12±00.00 | 16.10±00.29 |
| Nim/gcc (jsony) | 0.408±0.001 | 111.62±00.00 + 156.19±01.75 | 17.33±00.14 |
| C++/clang++ (RapidJSON SAX) | 0.417±0.001 | 195.08±00.00 + 0.00±00.00 | 17.70±00.04 |
| C++/g++ (RapidJSON SAX Precise) | 0.427±0.000 | 113.34±00.00 + 0.12±00.00 | 19.06±00.18 |
| Node.js | 0.437±0.001 | 155.54±00.06 + 201.31±03.75 | 21.57±00.11 |
| C++/clang++ (RapidJSON SAX Precise) | 0.514±0.000 | 195.08±00.00 + 0.12±00.00 | 22.66±00.11 |
| Go (jsoniter) | 0.542±0.001 | 114.25±00.12 + 0.00±00.00 | 22.00±00.31 |
| C#/.NET Core (System.Text.Json) | 0.549±0.002 | 490.00±00.08 + 140.62±00.19 | 24.45±00.29 |
| Rust (Serde Untyped) | 0.558±0.001 | 111.75±00.00 + 839.88±00.00 | 23.69±00.30 |
| Java (DSL-JSON) | 0.571±0.004 | 267.72±00.12 + 288.95±01.92 | 30.06±00.56 |
| Python/pypy | 0.614±0.000 | 280.79±00.01 + 125.21±00.00 | 26.83±00.15 |
| V/clang | 0.634±0.001 | 111.75±00.12 + 496.00±00.00 | 26.49±00.14 |
| V/gcc | 0.634±0.000 | 111.81±00.06 + 496.00±00.00 | 26.77±00.27 |
| Crystal (Pull) | 0.634±0.002 | 113.44±00.06 + 18.12±00.00 | 27.95±00.31 |
| Crystal (Schema) | 0.647±0.001 | 113.38±00.00 + 50.75±00.00 | 28.21±00.09 |
| Nim/gcc (Packedjson) | 0.673±0.001 | 112.00±00.00 + 294.19±00.06 | 28.78±00.44 |
| Nim/clang (Packedjson) | 0.675±0.001 | 112.38±00.12 + 294.25±00.00 | 29.45±00.42 |
| CPython (UltraJSON) | 0.720±0.002 | 123.90±00.00 + 476.31±00.88 | 28.65±00.08 |
| Perl (Cpanel::JSON::XS) | 0.772±0.006 | 125.44±00.06 + 402.88±00.00 | 32.07±00.29 |
| PHP | 0.805±0.002 | 128.44±00.06 + 517.88±00.00 | 34.83±00.19 |
| Go | 0.825±0.001 | 114.00±00.00 + 0.00±00.00 | 34.28±00.40 |
| Python | 0.895±0.004 | 121.73±00.06 + 325.88±00.00 | 37.83±00.56 |
| Crystal | 0.947±0.008 | 113.50±00.00 + 392.12±00.00 | 41.02±00.80 |
| Nim/gcc | 1.023±0.002 | 112.06±00.06 + 1001.25±00.00 | 42.77±00.18 |
| C#/.NET Core | 1.064±0.004 | 496.26±00.18 + 273.00±00.12 | 50.88±00.49 |
| Nim/clang | 1.091±0.002 | 112.38±00.00 + 999.00±00.00 | 45.69±00.46 |
| Clojure | 1.141±0.022 | 411.01±04.96 + 545.75±15.06 | 60.70±00.83 |
| C++/g++ (json-c) | 1.203±0.002 | 113.35±00.00 + 1215.88±00.00 | 50.53±00.81 |
| C++/clang++ (Nlohmann) | 1.205±0.001 | 112.89±00.06 + 359.88±00.00 | 50.78±00.19 |
| C++/clang++ (json-c) | 1.205±0.005 | 112.91±00.06 + 1215.88±00.00 | 50.10±00.29 |
| Go/gccgo | 1.227±0.002 | 139.38±00.06 + 0.00±00.00 | 50.43±00.12 |
| C++/g++ (Nlohmann) | 1.274±0.002 | 113.40±00.06 + 447.88±00.00 | 53.66±00.57 |
| Ruby (--jit) | 1.330±0.017 | 127.03±00.43 + 213.50±00.62 | 56.45±00.76 |
| Ruby | 1.360±0.004 | 121.25±00.00 + 212.75±00.00 | 57.49±00.53 |
| F#/.NET Core (System.Text.Json) | 1.498±0.006 | 498.96±00.06 + 231.38±04.12 | 68.65±00.71 |
| D/ldc2 | 1.742±0.002 | 113.25±00.00 + 708.38±00.06 | 73.21±00.37 |
| Ruby (YAJL) | 1.745±0.005 | 121.38±00.12 + 219.06±00.06 | 73.96±00.69 |
| C#/Mono | 1.789±0.010 | 253.37±00.07 + 31.54±00.03 | 77.28±00.84 |
| C3 | 1.962±0.004 | 111.88±00.00 + 795.88±00.00 | 81.93±00.35 |
| Haskell | 2.008±0.005 | 115.88±00.00 + 723.56±00.19 | 86.20±00.70 |
| C++/g++ (Boost.PropertyTree) | 2.515±0.009 | 113.40±00.06 + 1439.88±00.00 | 107.37±01.05 |
| Rust (jq) | 2.597±0.006 | 113.50±00.06 + 903.47±01.19 | 108.44±00.97 |
| C++/clang++ (Boost.PropertyTree) | 2.625±0.007 | 195.14±00.06 + 1232.62±00.00 | 112.22±00.99 |
| Odin | 2.824±0.003 | 111.44±00.06 + 20.00±00.00 | 117.93±00.75 |
| Ruby/jruby | 2.869±0.026 | 469.56±03.09 + 890.16±23.07 | 146.96±02.03 |
| Vala/gcc | 3.089±0.009 | 115.31±00.06 + 980.00±00.00 | 131.83±00.56 |
| Vala/clang | 3.094±0.006 | 115.38±00.00 + 980.00±00.00 | 132.11±00.59 |
| D/dmd | 3.101±0.003 | 113.38±00.00 + 708.56±00.06 | 133.40±01.11 |
| D/gdc | 3.493±0.022 | 116.62±00.06 + 708.75±00.00 | 148.35±00.78 |
| Racket | 3.819±0.025 | 320.69±00.42 + 225.56±00.12 | 159.99±01.22 |
| Perl (JSON::Tiny) | 9.297±0.067 | 126.00±00.00 + 528.69±00.01 | 409.51±01.80 |
| Ruby/truffleruby (JVM) | 10.154±0.186 | 483.81±04.05 + 2243.44±141.72 | 637.52±13.44 |
| Ruby/truffleruby | 10.192±0.089 | 398.88±02.12 + 1932.31±17.38 | 586.73±04.37 |
## Matmul
Testing allocating and multiplying matrices.
[Matmul](matmul)
| Language | Time, s | Memory, MiB | Energy, J |
| :---------------------- | -----------------------: | -------------------------------------------------: | -------------------------: |
| D/ldc2 (lubeck) | 0.042±0.001 | 16.19±02.25 + 48.19±02.31 | 4.17±00.04 |
| Python (NumPy) | 0.066±0.001 | 39.59±02.13 + 52.95±02.12 | 5.61±00.04 |
| Nim/gcc (Arraymancer) | 0.073±0.002 | 10.69±03.00 + 53.19±03.06 | 5.81±00.10 |
| Java (ND4J) | 0.082±0.001 | 117.87±01.02 + 92.62±00.00 | 6.38±00.11 |
| Rust (ndarray) | 0.091±0.001 | 2.50±00.00 + 68.48±00.00 | 6.18±00.06 |
| Nim/clang (Arraymancer) | 0.112±0.027 | 20.44±02.25 + 44.06±02.25 | 7.99±01.36 |
| Julia (threads: 2) | 0.119±0.000 | 263.91±00.08 + 57.19±00.06 | 6.62±00.02 |
| C++/g++ (Eigen) | 0.147±0.000 | 4.21±00.00 + 85.54±00.00 | 7.23±00.09 |
| C++/clang++ (Eigen) | 0.147±0.000 | 4.75±00.00 + 85.42±00.00 | 7.31±00.03 |
| Julia (threads: 1) | 0.170±0.000 | 263.98±00.07 + 56.62±00.00 | 7.95±00.04 |
| V/clang (VSL + CBLAS) | 0.274±0.003 | 14.31±02.00 + 44.88±01.88 | 19.26±00.20 |
| V/clang (VSL) | 0.274±0.004 | 14.38±02.75 + 44.69±02.69 | 18.94±00.23 |
| V/gcc (VSL + CBLAS) | 0.469±0.006 | 12.94±02.44 + 46.31±02.38 | 35.29±00.27 |
| V/gcc (VSL) | 0.484±0.002 | 13.25±02.44 + 45.94±02.25 | 32.84±00.15 |
| Julia (no BLAS) | 1.110±0.018 | 263.50±00.00 + 51.88±00.00 | 48.27±00.89 |
| D/gdc | 1.504±0.000 | 7.12±00.06 + 4.00±00.00 | 56.83±00.29 |
| D/ldc2 | 1.723±0.002 | 3.62±00.06 + 70.38±00.00 | 64.10±00.41 |
| D/dmd | 1.881±0.002 | 3.44±00.06 + 70.38±00.00 | 71.95±00.51 |
| C/gcc | 3.034±0.000 | 2.00±00.00 + 68.38±00.00 | 112.68±00.46 |
| V/gcc | 3.036±0.000 | 2.50±00.00 + 68.75±00.00 | 113.43±00.28 |
| Vala/clang | 3.061±0.001 | 5.66±00.03 + 68.38±00.00 | 107.97±01.66 |
| Rust | 3.064±0.001 | 2.25±00.00 + 68.50±00.00 | 107.36±01.14 |
| V/clang | 3.064±0.001 | 2.88±00.00 + 68.75±00.00 | 106.71±00.68 |
| C/clang | 3.066±0.000 | 2.00±00.00 + 68.38±00.00 | 105.48±00.31 |
| Zig | 3.068±0.002 | 1.75±00.00 + 68.62±00.00 | 109.29±00.23 |
| Nim/gcc | 3.094±0.002 | 2.50±00.00 + 57.88±00.00 | 115.51±00.14 |
| Java | 3.107±0.006 | 43.27±00.10 + 78.61±00.08 | 123.58±00.40 |
| Swift | 3.117±0.001 | 8.31±00.06 + 68.62±00.00 | 112.98±00.66 |
| Nim/clang | 3.121±0.001 | 2.88±00.12 + 57.88±00.00 | 108.74±00.94 |
| Vala/gcc | 3.130±0.001 | 5.53±00.03 + 68.50±00.00 | 115.26±00.63 |
| Go | 3.153±0.000 | 4.00±00.12 + 0.00±00.00 | 114.91±00.69 |
| Crystal | 3.154±0.001 | 4.00±00.06 + 59.50±00.00 | 113.86±01.39 |
| Go/gccgo | 3.160±0.000 | 24.38±00.06 + 0.00±00.00 | 112.72±00.89 |
| Kotlin/JVM | 3.197±0.002 | 43.74±00.09 + 78.32±00.17 | 124.82±00.65 |
| Node.js | 3.201±0.001 | 54.88±00.13 + 70.75±00.12 | 125.28±00.24 |
| Python/pypy | 3.263±0.002 | 61.00±00.00 + 68.75±00.12 | 136.35±00.22 |
| Scala | 3.295±0.004 | 61.33±00.10 + 150.06±00.06 | 121.09±00.77 |
| C#/.NET Core | 4.890±0.001 | 34.75±00.07 + 68.88±00.00 | 199.16±01.76 |
| C#/Mono | 7.394±0.002 | 26.13±00.15 + 69.62±00.00 | 307.51±01.02 |
| Ruby/truffleruby | 23.778±0.432 | 323.12±01.06 + 495.12±08.81 | 822.87±11.67 |
| Ruby/truffleruby (JVM) | 24.266±0.702 | 460.01±13.73 + 344.48±85.17 | 849.21±19.35 |
| Python | 131.064±0.887 | 11.94±00.06 + 68.62±00.00 | 6080.98±50.15 |
| Perl | 149.885±1.491 | 8.75±00.12 + 379.62±00.00 | 6523.98±63.31 |
| Ruby (--jit) | 153.769±0.077 | 21.87±00.03 + 68.44±00.06 | 6836.84±17.55 |
| Tcl | 205.038±0.931 | 7.50±00.00 + 400.25±00.00 | 9334.51±36.78 |
| Ruby | 213.180±0.222 | 11.62±00.00 + 69.25±00.00 | 9606.45±45.45 |
| Ruby/jruby | 378.246±8.095 | 269.65±12.79 + 1169.97±99.95 | 15729.25±399.84 |
## Primes
Testing:
- generating primes using the optimized [sieve of Atkin](https://www.geeksforgeeks.org/sieve-of-atkin/);
- prefix search for their decimal numbers using Trie data structure.
Notes:
- All languages but V and Python use unordered hashmaps (V and Python don't provide those out of box, and
their hashmaps use keys in the insertion order);
- The results are always sorted (could be unstable or stable though).
[Primes](primes)
| Language | Time, s | Memory, MiB | Energy, J |
| :--------------------- | ---------------------: | ------------------------------------------------: | ----------------------: |
| Zig | 0.060±0.000 | 1.50±00.00 + 47.88±00.25 | 2.52±00.02 |
| C++/g++ | 0.064±0.001 | 3.62±00.12 + 85.86±00.56 | 2.50±00.02 |
| Go | 0.072±0.001 | 3.56±00.06 + 0.00±00.00 | 3.13±00.06 |
| C++/clang++ | 0.075±0.000 | 3.12±00.12 + 64.66±00.12 | 2.85±00.03 |
| V/clang | 0.100±0.000 | 2.25±00.00 + 200.38±00.94 | 4.15±00.03 |
| Rust | 0.103±0.000 | 2.00±00.00 + 72.98±00.00 | 4.05±00.06 |
| V/gcc | 0.105±0.001 | 2.12±00.00 + 215.81±02.50 | 4.36±00.05 |
| Java | 0.136±0.003 | 41.95±00.12 + 126.09±05.43 | 7.73±00.17 |
| Crystal | 0.142±0.000 | 3.44±00.06 + 89.06±00.62 | 5.77±00.06 |
| Scala | 0.193±0.002 | 63.27±00.10 + 147.50±04.91 | 11.42±00.09 |
| Node.js | 0.195±0.000 | 44.05±00.00 + 147.14±00.26 | 10.22±00.04 |
| Nim/gcc | 0.290±0.002 | 1.75±00.00 + 596.00±04.81 | 11.18±00.16 |
| Nim/clang | 0.291±0.001 | 2.12±00.00 + 585.56±03.38 | 11.34±00.10 |
| Lua/luajit | 0.305±0.001 | 2.56±00.06 + 157.53±00.65 | 12.31±00.03 |
| Julia | 0.419±0.002 | 263.62±00.00 + 216.46±02.14 | 15.98±00.08 |
| Python/pypy | 0.636±0.003 | 60.00±00.12 + 248.82±00.07 | 26.09±00.26 |
| Racket | 0.751±0.001 | 111.73±00.41 + 246.77±00.46 | 29.66±00.18 |
| Ruby/truffleruby | 0.895±0.009 | 201.19±02.19 + 649.06±40.88 | 60.60±00.54 |
| Ruby (--jit) | 1.214±0.003 | 23.14±00.06 + 163.74±00.49 | 49.45±00.50 |
| Lua | 1.218±0.004 | 2.50±00.00 + 283.91±00.62 | 49.71±00.55 |
| Ruby/truffleruby (JVM) | 1.305±0.052 | 393.06±03.76 + 462.28±33.59 | 84.79±03.35 |
| Ruby | 1.964±0.007 | 11.38±00.06 + 172.75±00.31 | 82.09±00.37 |
| Ruby/jruby | 2.245±0.083 | 200.22±01.74 + 552.07±44.14 | 120.12±05.85 |
| Python | 2.275±0.007 | 11.62±00.00 + 172.88±00.69 | 98.93±01.08 |
# Tests Execution
## Environment
CPU: Intel(R) Xeon(R) E-2324G
Base Docker image: Debian GNU/Linux trixie/sid
| Language | Version |
| ---------------- | ------------------------------- |
| .NET Core | 8.0.403 |
| C#/.NET Core | 4.11.0-3.24468.6 (b4e5d1dd) |
| C#/Mono | 6.12.0.200 |
| C3 | 0.6.4 |
| Chez Scheme | 10.0.0 |
| Clojure | "1.12.0" |
| Crystal | 1.14.0 |
| D/dmd | v2.109.1 |
| D/gdc | 14.2.0 |
| D/ldc2 | 1.39.0 |
| Elixir | 1.14.0 |
| F#/.NET Core | 12.8.401.0 for F# 8.0 |
| Go | go1.23.2 |
| Go/gccgo | 14.2.0 |
| Haskell | 9.8.2 |
| Idris 2 | 0.6.0 |
| Java | 23.0.1 |
| Julia | v"1.11.1" |
| Kotlin | 2.0.21 |
| Lua | 5.4.6 |
| Lua/luajit | 2.1.1723681758 |
| MLton | 20210117 |
| Nim | 2.2.0 |
| Node.js | v23.1.0 |
| OCaml | 5.2.0 |
| Odin | dev-2024-11-nightly |
| PHP | 8.2.24 |
| Perl | v5.40.0 |
| Python | 3.12.6 |
| Python/pypy | 7.3.17-final0 for Python 3.10.14 |
| Racket | "8.14" |
| Ruby | 3.3.5p100 |
| Ruby/jruby | 9.4.8.0 |
| Ruby/truffleruby | 24.1.1 |
| Rust | 1.82.0 |
| Scala | 3.5.2 |
| Swift | 6.0.2 |
| Tcl | 8.6 |
| V | 0.4.8 2ab1523 |
| Vala | 0.56.17 |
| Zig | 0.13.0 |
| clang/clang++ | 19.1.2 (1) |
| gcc/g++ | 14.2.0 |
## Using Docker
Build the image:
$ docker build docker/ -t benchmarks
Run the image:
$ docker run -it --rm -v $(pwd):/src benchmarks
where `` is:
- `versions` (print installed language versions);
- `shell` (start the shell);
- `brainfuck bench` (build and run Brainfuck bench.b benchmarks);
- `brainfuck mandel` (build and run Brainfuck mandel.b benchmarks);
- `base64` (build and run Base64 benchmarks);
- `json` (build and run Json benchmarks);
- `matmul` (build and run Matmul benchmarks);
- `primes` (build and run Primes benchmarks);
Please note that the actual measurements provided in the project are taken semi-manually (via `shell`) as the full update takes days and could have occassional issues in Docker.
There is a `./run.sh` that could be used to simplify Docker usage:
- `./run.sh build` (build the image);
- `./run.sh make versions` (run the image with the `versions` command);
- `sudo ./run.sh shell` (run the image with the `shell' command, sudo is required to read energy levels).
## Manual Execution
Makefiles contain recipes for building and executing tests with the
proper dependencies. Please use `make run` (and `make run2` where applicable).
The measurements are taken using `analyze.rb` script:
$ cd
$ ../analyze.rb make run
$ ../analyze.rb make run[]
Please note that the measurements could take hours. It uses 10 iterations
by default, but it could be changed using ATTEMPTS environment variable:
$ ATTEMPTS=1 ../analyze.rb make run
### Prerequisites
Please use [Dockerfile](docker/Dockerfile) as a reference regarding which
packages and tools are required.
For all (optional):
- [Powercap](https://github.com/powercap/powercap) for reading energy
counters in Linux (Debian package `powercap-utils`).
For Python:
- [NumPy](https://numpy.org/) for matmul tests
(Debian package `python3-numpy`).
- [UltraJSON](https://pypi.org/project/ujson/) for JSON tests
(Debian package `python3-ujson`).
For C++:
- [Boost](https://www.boost.org/) for JSON tests
(Debian package `libboost-dev`).
- [JSON-C](https://github.com/json-c/json-c) for JSON tests
(Debian package `libjson-c-dev`).
For Rust:
- [libjq](https://stedolan.github.io/jq/) for jq test
(Debian packages `libjq-dev`, `libonig-dev` and environment variable
`JQ_LIB_DIR=/usr/lib/x86_64-linux-gnu/`).
For Java, Scala:
- [Coursier](https://get-coursier.io/) for downloading Maven artifacts.
For Haskell:
- [network](http://hackage.haskell.org/package/network) for
TCP connectivity between the tests and the test runner.
- [raw-strings-qq](http://hackage.haskell.org/package/raw-strings-qq) for
raw string literals used in tests.
For Perl:
- [cpanminus](https://metacpan.org/pod/App::cpanminus) for installing
modules from CPAN (Debian package `cpanminus`).
For Vala:
- [JSON-GLib](https://wiki.gnome.org/Projects/JsonGlib) for JSON tests
(Debian package `libjson-glib-dev`).
# Contribution
Please follow the criteria specified in the [overview](#overview). Besides
that please ensure that the communication protocol between a test and the
test runner is satisfied:
- The test runner listens on localhost:9001;
- All messages are sent using TCP sockets closed immediately after the
message has been sent;
- There are two messages sent from a test (it establishes the measurement
boundary):
1. The beginning message having the format *name of the test*/t*process ID*
(the process ID is used to measure the memory consumption). Please note that
the name of the test couldn't use Tab character as it's a delimiter;
2. The end message with any content (mostly it's "stop" for consistency).
- The test runner could be unavailable (if the test is launched as is) and
the test should gracefully handle it.
## Makefile guide
### Binary executables
If the test is compiled into a single binary, then two sections of
the `Makefile` require changes:
- append a new target (the final binary location) into `executables`
variable;
- append the proper target rule.
### Compiled artifacts
If the test is compiled, but can't be executed directly as a binary, then
three sections of the `Makefile` require changes:
- append a new target (the final artifact location) into `artifacts`
variable;
- append the proper target rule to compile the test;
- append `run[]` rule to run the test.
### Scripting language
If the test doesn't require compilation, then two sections of the `Makefile`
requires changes:
- append `run[]` into `all_runners` variable;
- append `run[]` rule to run the test.
## README update
TOC is regenerated using [git-markdown-toc](https://github.com/ildar-shaimordanov/git-markdown-toc):
```
./run.sh toc
```