https://github.com/easbar/jetty_jmh_benchmark
https://github.com/easbar/jetty_jmh_benchmark
Last synced: 2 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/easbar/jetty_jmh_benchmark
- Owner: easbar
- Created: 2022-05-06T14:44:31.000Z (about 3 years ago)
- Default Branch: master
- Last Pushed: 2022-05-09T07:23:52.000Z (about 3 years ago)
- Last Synced: 2025-01-24T11:44:32.395Z (4 months ago)
- Language: Java
- Size: 9.77 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Jetty JMH Benchmark
```shell
mvn clean package
# run full benchmark
java -jar target/jetty-jmh-benchmark.jar
# run faster benchmark (probably less precise but quicker results)
java -jar target/jetty-jmh-benchmark.jar -f 2 -i 3 -wi 3 -r 3 -w 2
# use different array size(s)
java -jar target/jetty-jmh-benchmark.jar -p size=1000000,2000000
```
# What this benchmark does
Basically we start a jetty server instance with two servlets. One servlet returns a constant number and the other does
some work (it sums up all integers of an array). We then run queries against the two servlets and measure the response
time. We also run a benchmark that measures the time it takes to run the work method (taking the array sum)
in isolation.
* measureBaseline - baseline that does nothing at all
* measureSumArray - sums random integers that were written to an int[] array, the array length is given by the size
parameter
* measureBaselineHttp - runs queries against a servlet that returns a constant number
* measureSumArrayHttp - runs queries against a servlet that does the same work as measureSumArray
# Results
On some machines everything seems to make sense:
```
Benchmark (size) Mode Cnt Score Error Units
JettyBenchmark.measureBaseline 10000000 avgt 25 ≈ 10⁻⁶ ms/op
JettyBenchmark.measureBaseline 20000000 avgt 25 ≈ 10⁻⁶ ms/op
JettyBenchmark.measureBaseline 40000000 avgt 25 ≈ 10⁻⁶ ms/op
JettyBenchmark.measureBaselineHttp 10000000 avgt 25 0.052 ± 0.001 ms/op
JettyBenchmark.measureBaselineHttp 20000000 avgt 25 0.052 ± 0.001 ms/op
JettyBenchmark.measureBaselineHttp 40000000 avgt 25 0.051 ± 0.001 ms/op
JettyBenchmark.measureSumArray 10000000 avgt 25 9.015 ± 0.012 ms/op
JettyBenchmark.measureSumArray 20000000 avgt 25 18.099 ± 0.027 ms/op
JettyBenchmark.measureSumArray 40000000 avgt 25 36.262 ± 0.059 ms/op
JettyBenchmark.measureSumArrayHttp 10000000 avgt 25 10.401 ± 0.120 ms/op
JettyBenchmark.measureSumArrayHttp 20000000 avgt 25 18.701 ± 0.108 ms/op
JettyBenchmark.measureSumArrayHttp 40000000 avgt 25 36.843 ± 0.047 ms/op
```
measureBaseline takes almost no time at all measureBaselineHttp is quite fast (less than 0.1ms server response time)
measureSumArray is proportional to the array size measureSumArrayHttp does not take much longer than measureSumArray,
which can be expected because the server overhead is small according to measureBaselineHttp
... However, on some other machines things aren't as clear, like this one:
```
Benchmark (size) Mode Cnt Score Error Units
JettyBenchmark.measureBaseline 10000000 avgt 25 ≈ 10⁻⁶ ms/op
JettyBenchmark.measureBaseline 20000000 avgt 25 ≈ 10⁻⁶ ms/op
JettyBenchmark.measureBaseline 40000000 avgt 25 ≈ 10⁻⁶ ms/op
JettyBenchmark.measureBaselineHttp 10000000 avgt 25 0.064 ± 0.006 ms/op
JettyBenchmark.measureBaselineHttp 20000000 avgt 25 0.067 ± 0.006 ms/op
JettyBenchmark.measureBaselineHttp 40000000 avgt 25 0.069 ± 0.004 ms/op
JettyBenchmark.measureSumArray 10000000 avgt 25 6.460 ± 0.086 ms/op
JettyBenchmark.measureSumArray 20000000 avgt 25 12.975 ± 0.072 ms/op
JettyBenchmark.measureSumArray 40000000 avgt 25 25.975 ± 0.166 ms/op
JettyBenchmark.measureSumArrayHttp 10000000 avgt 25 11.287 ± 0.319 ms/op
JettyBenchmark.measureSumArrayHttp 20000000 avgt 25 18.906 ± 0.191 ms/op
JettyBenchmark.measureSumArrayHttp 40000000 avgt 25 31.976 ± 0.046 ms/op
```
measureBaseline, measureBaselineHttp and measureSumArray all seem fine, but measureSumArrayHttp seems to add around 6ms
additional query time! Where is this coming from?
On some other machines it is even a lot worse, like this one:
```
Benchmark (size) Mode Cnt Score Error Units
JettyBenchmark.measureBaseline 10000000 avgt 25 ? 10?? ms/op
JettyBenchmark.measureBaseline 20000000 avgt 25 ? 10?? ms/op
JettyBenchmark.measureBaseline 40000000 avgt 25 ? 10?? ms/op
JettyBenchmark.measureBaselineHttp 10000000 avgt 25 0.044 ? 0.003 ms/op
JettyBenchmark.measureBaselineHttp 20000000 avgt 25 0.043 ? 0.002 ms/op
JettyBenchmark.measureBaselineHttp 40000000 avgt 25 0.045 ? 0.003 ms/op
JettyBenchmark.measureSumArray 10000000 avgt 25 6.442 ? 0.114 ms/op
JettyBenchmark.measureSumArray 20000000 avgt 25 12.948 ? 0.165 ms/op
JettyBenchmark.measureSumArray 40000000 avgt 25 25.829 ? 0.419 ms/op
JettyBenchmark.measureSumArrayHttp 10000000 avgt 25 13.675 ? 0.521 ms/op
JettyBenchmark.measureSumArrayHttp 20000000 avgt 25 27.514 ? 0.176 ms/op
JettyBenchmark.measureSumArrayHttp 40000000 avgt 25 54.949 ? 0.247 ms/op
```
measureSumArrayHttp is not just slower by a few ms, but the difference even depends on the array length:
Depending on the array size it takes around 7ms/15ms/29ms longer than measureSumArray, so the additional response time
seems to increase proportionally to the array size! What is going on here?
# What we know so far
* All Intel CPUs we tried so far seem to show the first case (no issues) (for example we tried: i7-8700, i7-8550U,
i9-9900K)
* The problematic second and third cases occur for AMD CPUs (for example we tried: AMD Ryzen 9 5950X, AMD EPYC 7502P)
* For the problematic AMD CPUs it seems like we get the second case using Debian 10 and the third case using Debian 11
* There is at least one AMD CPU that shows the first case (AMD Ryzen 7 PRO 5850U) using Ubuntu 21.10
* Using different JDK versions (8, 11, 17) does not seem to change anything
* We see the same problems using Tomcat and Jetty, but not using Vert.x