{"id":16841490,"url":"https://github.com/easbar/jetty_jmh_benchmark","last_synced_at":"2025-03-18T05:16:05.742Z","repository":{"id":136305766,"uuid":"489390404","full_name":"easbar/jetty_jmh_benchmark","owner":"easbar","description":null,"archived":false,"fork":false,"pushed_at":"2022-05-09T07:23:52.000Z","size":10,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-24T11:44:32.395Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/easbar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-05-06T14:44:31.000Z","updated_at":"2022-05-06T14:45:42.000Z","dependencies_parsed_at":null,"dependency_job_id":"ad0450f6-c856-4413-a2e7-e7baae329c13","html_url":"https://github.com/easbar/jetty_jmh_benchmark","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/easbar%2Fjetty_jmh_benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/easbar%2Fjetty_jmh_benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/easbar%2Fjetty_jmh_benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/easbar%2Fjetty_jmh_benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/easbar","download_url":"https://codeload.github.com/easbar/jetty_jmh_benchmark/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244160051,"owners_count":20408021,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T12:42:05.046Z","updated_at":"2025-03-18T05:16:05.734Z","avatar_url":"https://github.com/easbar.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Jetty JMH Benchmark\n\n```shell\nmvn clean package\n# run full benchmark\njava -jar target/jetty-jmh-benchmark.jar\n# run faster benchmark (probably less precise but quicker results)\njava -jar target/jetty-jmh-benchmark.jar -f 2 -i 3 -wi 3 -r 3 -w 2\n# use different array size(s)\njava -jar target/jetty-jmh-benchmark.jar -p size=1000000,2000000\n```\n\n# What this benchmark does\n\nBasically we start a jetty server instance with two servlets. One servlet returns a constant number and the other does\nsome work (it sums up all integers of an array). We then run queries against the two servlets and measure the response\ntime. We also run a benchmark that measures the time it takes to run the work method (taking the array sum)\nin isolation.\n\n* measureBaseline - baseline that does nothing at all\n* measureSumArray - sums random integers that were written to an int[] array, the array length is given by the size\n  parameter\n* measureBaselineHttp - runs queries against a servlet that returns a constant number\n* measureSumArrayHttp - runs queries against a servlet that does the same work as measureSumArray\n\n# Results\n\nOn some machines everything seems to make sense:\n\n```\nBenchmark                             (size)  Mode  Cnt   Score     Error  Units\nJettyBenchmark.measureBaseline      10000000  avgt   25  ≈ 10⁻⁶            ms/op\nJettyBenchmark.measureBaseline      20000000  avgt   25  ≈ 10⁻⁶            ms/op\nJettyBenchmark.measureBaseline      40000000  avgt   25  ≈ 10⁻⁶            ms/op\nJettyBenchmark.measureBaselineHttp  10000000  avgt   25   0.052 ±   0.001  ms/op\nJettyBenchmark.measureBaselineHttp  20000000  avgt   25   0.052 ±   0.001  ms/op\nJettyBenchmark.measureBaselineHttp  40000000  avgt   25   0.051 ±   0.001  ms/op\nJettyBenchmark.measureSumArray      10000000  avgt   25   9.015 ±   0.012  ms/op\nJettyBenchmark.measureSumArray      20000000  avgt   25  18.099 ±   0.027  ms/op\nJettyBenchmark.measureSumArray      40000000  avgt   25  36.262 ±   0.059  ms/op\nJettyBenchmark.measureSumArrayHttp  10000000  avgt   25  10.401 ±   0.120  ms/op\nJettyBenchmark.measureSumArrayHttp  20000000  avgt   25  18.701 ±   0.108  ms/op\nJettyBenchmark.measureSumArrayHttp  40000000  avgt   25  36.843 ±   0.047  ms/op\n```\n\nmeasureBaseline takes almost no time at all measureBaselineHttp is quite fast (less than 0.1ms server response time)\nmeasureSumArray is proportional to the array size measureSumArrayHttp does not take much longer than measureSumArray,\nwhich can be expected because the server overhead is small according to measureBaselineHttp\n\n... However, on some other machines things aren't as clear, like this one:\n\n```\nBenchmark                             (size)  Mode  Cnt   Score    Error  Units\nJettyBenchmark.measureBaseline      10000000  avgt   25  ≈ 10⁻⁶           ms/op\nJettyBenchmark.measureBaseline      20000000  avgt   25  ≈ 10⁻⁶           ms/op\nJettyBenchmark.measureBaseline      40000000  avgt   25  ≈ 10⁻⁶           ms/op\nJettyBenchmark.measureBaselineHttp  10000000  avgt   25   0.064 ±  0.006  ms/op\nJettyBenchmark.measureBaselineHttp  20000000  avgt   25   0.067 ±  0.006  ms/op\nJettyBenchmark.measureBaselineHttp  40000000  avgt   25   0.069 ±  0.004  ms/op\nJettyBenchmark.measureSumArray      10000000  avgt   25   6.460 ±  0.086  ms/op\nJettyBenchmark.measureSumArray      20000000  avgt   25  12.975 ±  0.072  ms/op\nJettyBenchmark.measureSumArray      40000000  avgt   25  25.975 ±  0.166  ms/op\nJettyBenchmark.measureSumArrayHttp  10000000  avgt   25  11.287 ±  0.319  ms/op\nJettyBenchmark.measureSumArrayHttp  20000000  avgt   25  18.906 ±  0.191  ms/op\nJettyBenchmark.measureSumArrayHttp  40000000  avgt   25  31.976 ±  0.046  ms/op\n```\n\nmeasureBaseline, measureBaselineHttp and measureSumArray all seem fine, but measureSumArrayHttp seems to add around 6ms\nadditional query time! Where is this coming from?\n\nOn some other machines it is even a lot worse, like this one:\n\n```\nBenchmark                             (size)  Mode  Cnt   Score    Error  Units\nJettyBenchmark.measureBaseline      10000000  avgt   25  ? 10??           ms/op\nJettyBenchmark.measureBaseline      20000000  avgt   25  ? 10??           ms/op\nJettyBenchmark.measureBaseline      40000000  avgt   25  ? 10??           ms/op\nJettyBenchmark.measureBaselineHttp  10000000  avgt   25   0.044 ?  0.003  ms/op\nJettyBenchmark.measureBaselineHttp  20000000  avgt   25   0.043 ?  0.002  ms/op\nJettyBenchmark.measureBaselineHttp  40000000  avgt   25   0.045 ?  0.003  ms/op\nJettyBenchmark.measureSumArray      10000000  avgt   25   6.442 ?  0.114  ms/op\nJettyBenchmark.measureSumArray      20000000  avgt   25  12.948 ?  0.165  ms/op\nJettyBenchmark.measureSumArray      40000000  avgt   25  25.829 ?  0.419  ms/op\nJettyBenchmark.measureSumArrayHttp  10000000  avgt   25  13.675 ?  0.521  ms/op\nJettyBenchmark.measureSumArrayHttp  20000000  avgt   25  27.514 ?  0.176  ms/op\nJettyBenchmark.measureSumArrayHttp  40000000  avgt   25  54.949 ?  0.247  ms/op\n```\n\nmeasureSumArrayHttp is not just slower by a few ms, but the difference even depends on the array length:\nDepending on the array size it takes around 7ms/15ms/29ms longer than measureSumArray, so the additional response time\nseems to increase proportionally to the array size! What is going on here?\n\n# What we know so far\n\n* All Intel CPUs we tried so far seem to show the first case (no issues) (for example we tried: i7-8700, i7-8550U,\n  i9-9900K)\n* The problematic second and third cases occur for AMD CPUs (for example we tried: AMD Ryzen 9 5950X, AMD EPYC 7502P)\n* For the problematic AMD CPUs it seems like we get the second case using Debian 10 and the third case using Debian 11\n* There is at least one AMD CPU that shows the first case (AMD Ryzen 7 PRO 5850U) using Ubuntu 21.10\n* Using different JDK versions (8, 11, 17) does not seem to change anything\n* We see the same problems using Tomcat and Jetty, but not using Vert.x","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feasbar%2Fjetty_jmh_benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feasbar%2Fjetty_jmh_benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feasbar%2Fjetty_jmh_benchmark/lists"}