Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/composewell/streaming-benchmarks
Benchmarks to compare Haskell streaming library performance
https://github.com/composewell/streaming-benchmarks
benchmarks haskell performance streaming
Last synced: 3 days ago
JSON representation
Benchmarks to compare Haskell streaming library performance
- Host: GitHub
- URL: https://github.com/composewell/streaming-benchmarks
- Owner: composewell
- License: other
- Created: 2017-07-26T17:38:39.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2024-11-17T04:15:15.000Z (about 1 month ago)
- Last Synced: 2024-12-16T20:08:25.219Z (10 days ago)
- Topics: benchmarks, haskell, performance, streaming
- Language: Haskell
- Homepage:
- Size: 1.28 MB
- Stars: 102
- Watchers: 14
- Forks: 9
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- Changelog: Changelog.md
- License: LICENSE
Awesome Lists containing this project
README
# Streaming Benchmarks
[](https://hackage.haskell.org/package/streaming-benchmarks)
[](https://gitter.im/composewell/streamly)
[](https://travis-ci.org/composewell/streaming-benchmarks)
[](https://ci.appveyor.com/project/harendra-kumar/streaming-benchmarks)
This package provides micro-benchmarks to measure and compare the
performance of various streaming implementations in Haskell.
We have taken due to care to make sure that we are
benchmarking correctly and fairly. See [the notes on correct
benchmarking](docs/benchmarking-notes.md).
DISCLAIMER: This package is a result of benchmarking effort done during the
development of [streamly](https://github.com/composewell/streamly) by the
authors of [streamly](https://github.com/composewell/streamly).
## Benchmarks
The benchmark names are obvious, some of them are described below. Single
operation benchmarks:
| Name | Description |
| -------------- | ----------------------------------------------------------- |
| `drain` | Just discards all the elements in the stream |
| `drop-all` | drops all element using the ``drop`` operation |
| `last` | extract the last element of the stream |
| `fold` | sum all the numbers in the stream |
| `map` | increments each number in the stream by 1 |
| `take-all` | Use ``take`` to retain all the elements in the stream |
| `filter-even` | Keep even numbers, discard odd |
| `scan` | scan the stream using ``+`` operation |
| `mapM` | transform the stream using a monadic action |
| `zip` | combines corresponding elements of the two streams together |
Composite operation benchmarks:
| Name | Description |
| -------------- | ----------------------------------------------------------- |
| `map x 4` | perform `map` operation 4 times |
| `take-map` | `take` followed by a `map` |
For more details on how each benchmark is implemented see
[this benchmark file](https://github.com/composewell/streaming-benchmarks/blob/master/Benchmarks/Streamly.hs).
Each benchmark is run in a separate process to avoid any effects of GC
interference and sharing across benchmarks.
## Benchmark Results
Below we present some results comparing
[streamly](https://github.com/composewell/streamly) with other streaming
implementations.
Due care has been taken to keep the comparisons fair. We have optimized
each library's code to the best of our knowledge, please point out if
you find any measurement issues.
## Running Benchmarks
You can run individual benchmarks using `cabal bench `. You may
need to specify a build flag to include a particular streaming library
e.g. `--flag pipes` to benchmark the `pipes` library. Please consult the cabal
file to find the exact flag names.
### Generating Comparison Reports
You can generate the comparison reports presented in this page yourself.
To do so you need to run the benchmarks using the reporting tool, first build
the reporting tool using the following command:
```
$ cd bench-runner
$ cabal install --project-file cabal.project.user --installdir ../bin
```
If you want to create a report for benchmarks showing a 10% or greater
improvement with Streamly over Lists, use:
```
$ bin/bench-runner --package-name streaming-benchmarks --package-version 0.4.0 --compare --diff-cutoff-percent 10 --diff-style absolute --targets "StreamlyPure List"
```
After running once, you can add `--no-measure` option to use the same benchmark
measurements for different reports. For example, use:
* `--diff-cutoff-percent -10` to know where lists are better than streamly
* `--diff-style multiples` to generate ratios instead of absolute values
### Streamly vs Haskell Lists
Streamly, when used with `Identity` monad, is almost the same as Haskell lists
(in the `base` package).
[See this](https://github.com/composewell/streamly/blob/master/docs/streamly-vs-lists.md)
for more details.
The following table compares the timing of several operations for
[streamly](https://github.com/composewell/streamly)
with lists using a one million element stream. For brevity only
those operations where the performance of the two packages differ by
more than 10% are shown in the table below. The last column shows how
many times slower list is compared to streamly.
| Benchmark | streamly(μs) | list(μs) | list/streamly |
| ------------------- | ------------ | ---------- | ------------- |
| drop-map x 4 | 375.09 | 76925.32 | 205.08 |
| filter-drop x 4 | 382.03 | 54848.54 | 143.57 |
| drop-scan x 4 | 795.81 | 76716.79 | 96.40 |
| filter-scan x 4 | 795.60 | 44559.15 | 56.01 |
| scan-map x 4 | 1192.19 | 48838.22 | 40.97 |
| take-map x 4 | 1500.99 | 60126.58 | 40.06 |
| filter-take x 4 | 1502.01 | 48766.87 | 32.47 |
| take-drop x 4 | 1499.62 | 41720.03 | 27.82 |
| take-scan x 4 | 1874.94 | 51283.30 | 27.35 |
| drop-one x 4 | 375.33 | 8993.87 | 23.96 |
| dropWhile-false x 4 | 374.61 | 8957.79 | 23.91 |
| dropWhile-false | 374.83 | 8670.05 | 23.13 |
| drop-one | 390.77 | 8681.85 | 22.22 |
| dropWhile-true | 571.60 | 12237.48 | 21.41 |
| drop-all | 562.94 | 8262.38 | 14.68 |
| take-all | 624.83 | 564.34 | 1/1.11 |
| scan x 4 | 795.83 | 385.85 | 1/2.06 |
| appendR[10000] | 360.75 | 126.95 | 1/2.84 |
| concatMap | 34957.71 | 1124.85 | 1/31.08 |
* streamly-0.8.0, base-4.14.1.0, ghc-8.10.4, Linux
To generate these reports run `bench-runner` with:
* `--targets "StreamlyPure List"`
### Streamly vs Streaming
The following table compares the timing of several operations
for [streamly](https://github.com/composewell/streamly) with
[streaming](https://hackage.haskell.org/package/streaming) using a
million element stream.
| Benchmark | streamly(μs) | streaming(μs) | streaming/streamly |
| ------------------- | ------------ | ------------- | ------------------ |
| appendR[10000] | 326.56 | 1301176.69 | 3984.54 |
| mapM x 4 | 374.42 | 223591.08 | 597.17 |
| filter-map x 4 | 381.07 | 194903.88 | 511.47 |
| filter-scan x 4 | 795.66 | 233527.90 | 293.50 |
| filter-all-in x 4 | 375.40 | 102629.64 | 273.38 |
| filter-drop x 4 | 387.15 | 99096.98 | 255.96 |
| map x 4 | 386.49 | 94944.87 | 245.66 |
| drop-map x 4 | 375.62 | 89669.37 | 238.73 |
| scan x 4 | 797.00 | 166332.40 | 208.70 |
| scan-map x 4 | 1194.30 | 238804.48 | 199.95 |
| filter-even x 4 | 396.37 | 77865.47 | 196.45 |
| drop-scan x 4 | 796.98 | 156063.52 | 195.82 |
| takeWhile-true x 4 | 562.49 | 90183.53 | 160.33 |
| scan | 375.24 | 47520.57 | 126.64 |
| filter-take x 4 | 1498.55 | 189635.34 | 126.55 |
| mapM | 388.10 | 46689.61 | 120.30 |
| take-map x 4 | 1500.71 | 178954.50 | 119.25 |
| zip | 656.65 | 66689.73 | 101.56 |
| take-scan x 4 | 2380.35 | 241675.75 | 101.53 |
| filter-all-in | 375.97 | 33590.14 | 89.34 |
| map | 375.02 | 33081.13 | 88.21 |
| filter-even | 393.26 | 30458.46 | 77.45 |
| filter-all-out | 382.87 | 26826.21 | 70.07 |
| take-all x 4 | 1499.71 | 101332.53 | 67.57 |
| take-drop x 4 | 1498.53 | 98281.99 | 65.59 |
| takeWhile-true | 562.62 | 31863.25 | 56.63 |
| foldl' | 388.22 | 18503.15 | 47.66 |
| drop-all | 562.08 | 25200.32 | 44.83 |
| take-all | 768.65 | 33247.97 | 43.26 |
| dropWhile-true | 564.87 | 24431.50 | 43.25 |
| last | 385.53 | 15240.85 | 39.53 |
| dropWhile-false | 374.83 | 14566.70 | 38.86 |
| drop-one | 374.80 | 14565.01 | 38.86 |
| drop-one x 4 | 375.88 | 14448.67 | 38.44 |
| dropWhile-false x 4 | 390.12 | 14619.42 | 37.47 |
| drain | 375.06 | 13702.29 | 36.53 |
| toList | 117708.83 | 201444.81 | 1.71 |
* streamly-0.8.0, streaming-0.2.3.0, ghc-8.10.4, Linux
To generate these reports run `bench-runner` with:
* `--targets "Streamly Streaming" --cabal-build-options "--flag streaming"`
### Streamly vs Pipes
The following table compares the timing of several operations
for [streamly](https://github.com/composewell/streamly) with
[pipes](https://hackage.haskell.org/package/pipes) using a
million element stream.
| Benchmark | streamly(μs) | pipes(μs) | pipes/streamly |
| ------------------- | ------------ | --------- | -------------- |
| appendR[10000] | 327.90 | 901135.92 | 2748.21 |
| mapM x 4 | 375.20 | 407184.39 | 1085.23 |
| filter-map x 4 | 381.52 | 366759.70 | 961.31 |
| drop-map x 4 | 375.48 | 281296.82 | 749.16 |
| filter-all-in x 4 | 375.60 | 222331.68 | 591.93 |
| filter-drop x 4 | 387.44 | 222830.71 | 575.14 |
| drop-scan x 4 | 797.23 | 336737.89 | 422.39 |
| filter-even x 4 | 389.87 | 152688.91 | 391.64 |
| filter-scan x 4 | 797.38 | 309733.91 | 388.44 |
| drop-one x 4 | 375.48 | 139851.13 | 372.46 |
| map x 4 | 386.56 | 136289.32 | 352.57 |
| dropWhile-false x 4 | 390.72 | 137395.44 | 351.65 |
| scan-map x 4 | 1194.38 | 381286.88 | 319.23 |
| takeWhile-true x 4 | 562.86 | 165143.23 | 293.40 |
| scan x 4 | 796.68 | 222986.17 | 279.90 |
| mapM | 388.19 | 95576.97 | 246.21 |
| filter-all-in | 375.21 | 71297.42 | 190.02 |
| take-map x 4 | 1502.76 | 275887.24 | 183.59 |
| scan | 374.81 | 65549.13 | 174.89 |
| take-drop x 4 | 1503.43 | 256448.45 | 170.58 |
| filter-even | 390.29 | 66183.72 | 169.57 |
| filter-all-out | 376.99 | 59074.54 | 156.70 |
| drop-one | 375.19 | 58395.24 | 155.64 |
| dropWhile-false | 375.35 | 58223.03 | 155.12 |
| map | 375.05 | 57736.43 | 153.94 |
| filter-take x 4 | 1503.00 | 227925.71 | 151.65 |
| take-scan x 4 | 2455.91 | 354284.33 | 144.26 |
| zip | 657.07 | 86011.93 | 130.90 |
| takeWhile-true | 564.14 | 61390.21 | 108.82 |
| take-all x 4 | 1502.32 | 139730.70 | 93.01 |
| dropWhile-true | 564.03 | 49227.19 | 87.28 |
| drop-all | 562.05 | 46505.37 | 82.74 |
| take-all | 824.09 | 60511.34 | 73.43 |
| drain | 375.29 | 26390.59 | 70.32 |
| foldl' | 397.34 | 19064.05 | 47.98 |
| last | 387.11 | 17364.44 | 44.86 |
| toList | 117257.09 | 207405.94 | 1.77 |
* streamly-0.8.0, pipes-4.3.16, ghc-8.10.4, Linux
To generate these reports run `bench-runner` with:
* `--targets "Streamly Pipes" --cabal-build-options "--flag pipes"`
### Streamly vs Conduit
The following table compares the timing of several operations
for [streamly](https://github.com/composewell/streamly) with
[conduit](https://hackage.haskell.org/package/conduit) using a
million element stream.
| Benchmark | streamly(μs) | conduit(μs) | conduit/streamly |
| ------------------- | ------------ | ----------- | ---------------- |
| mapM x 4 | 375.46 | 297002.31 | 791.04 |
| filter-map x 4 | 380.79 | 267543.81 | 702.60 |
| drop-map x 4 | 375.66 | 232307.84 | 618.39 |
| filter-drop x 4 | 386.05 | 235029.15 | 608.81 |
| filter-scan x 4 | 796.56 | 306556.67 | 384.85 |
| drop-scan x 4 | 797.19 | 300789.06 | 377.31 |
| zip | 657.29 | 210069.05 | 319.60 |
| filter-all-in x 4 | 375.24 | 118506.68 | 315.82 |
| scan-map x 4 | 1194.67 | 360671.18 | 301.90 |
| map x 4 | 387.00 | 113497.14 | 293.27 |
| drop-one x 4 | 375.49 | 101842.95 | 271.23 |
| dropWhile-false x 4 | 389.44 | 102051.22 | 262.04 |
| scan x 4 | 796.72 | 190479.35 | 239.08 |
| takeWhile-true x 4 | 564.58 | 114459.57 | 202.73 |
| filter-even x 4 | 391.76 | 72369.30 | 184.73 |
| filter-take x 4 | 1502.04 | 267921.27 | 178.37 |
| take-map x 4 | 1502.88 | 238875.95 | 158.95 |
| take-drop x 4 | 1500.34 | 232606.19 | 155.04 |
| take-scan x 4 | 2443.83 | 309738.86 | 126.74 |
| mapM | 389.15 | 41897.48 | 107.66 |
| scan | 375.40 | 38137.85 | 101.59 |
| take-all x 4 | 1502.32 | 110682.74 | 73.67 |
| filter-all-in | 375.31 | 26024.21 | 69.34 |
| dropWhile-false | 375.10 | 25307.13 | 67.47 |
| map | 375.18 | 23088.09 | 61.54 |
| drop-one | 375.43 | 22020.65 | 58.65 |
| filter-even | 392.28 | 21504.28 | 54.82 |
| takeWhile-true | 562.79 | 29012.68 | 51.55 |
| filter-all-out | 378.76 | 15736.05 | 41.55 |
| drop-all | 562.89 | 19916.48 | 35.38 |
| foldl' | 388.88 | 12499.03 | 32.14 |
| dropWhile-true | 564.43 | 17983.35 | 31.86 |
| take-all | 784.67 | 24425.36 | 31.13 |
| last | 385.75 | 10974.84 | 28.45 |
| drain | 375.18 | 4272.15 | 11.39 |
| appendR[10000] | 326.93 | 1207.88 | 3.69 |
| toList | 116441.26 | 199138.09 | 1.71 |
* streamly-0.8.0, conduit-1.3.4.1, ghc-8.10.4, Linux
To generate these reports run `bench-runner` with:
* `--targets "Streamly Conduit" --cabal-build-options "--flag conduit"`
## Stack and heap utilization
`bench-runner` also generates a `maxrss` comparison report, displaying the
maximum resident memory for each benchmark.
## Comparing other libraries
This package supports many streaming libraries, bytestring, text, vector etc.
You can run the benchmarks directly or use `bench-runner` to create comparison
reports. Check out the targets in the cabal file.
## Adding New Libraries
It is trivial to add a new package. This is how
[a benchmark file](https://github.com/composewell/streaming-benchmarks/blob/master/Benchmarks/Streamly.hs)
for a streaming package looks like. Pull requests are welcome, we will be happy
to help, [just join the gitter chat](https://gitter.im/composewell/streamly)
and ask!