{"id":16620321,"url":"https://github.com/chrisgleissner/loom-webflux-benchmarks","last_synced_at":"2026-06-04T17:31:16.271Z","repository":{"id":229220347,"uuid":"776155779","full_name":"chrisgleissner/loom-webflux-benchmarks","owner":"chrisgleissner","description":"Benchmarks of Spring Boot REST service comparing Java 21 Virtual Threads (Project Loom) with WebFlux (Project Reactor). ","archived":false,"fork":false,"pushed_at":"2026-06-02T04:39:55.000Z","size":1173181,"stargazers_count":47,"open_issues_count":4,"forks_count":4,"subscribers_count":2,"default_branch":"main","last_synced_at":"2026-06-02T06:23:43.784Z","etag":null,"topics":["benchmark","http","java","java21","jpa","matplotlib","microservice","performance-optimization","performance-testing","postgresql","projectloom","python3","rest","scalability","spring-boot-cache","springboot3","ubuntu2404","virtualthreads","webflux"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chrisgleissner.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-03-22T19:46:52.000Z","updated_at":"2026-06-02T04:39:59.000Z","dependencies_parsed_at":"2025-09-29T16:22:41.117Z","dependency_job_id":null,"html_url":"https://github.com/chrisgleissner/loom-webflux-benchmarks","commit_stats":null,"previous_names":["chrisgleissner/loom-webflux","chrisgleissner/loom-webflux-benchmarks"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/chrisgleissner/loom-webflux-benchmarks","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrisgleissner%2Floom-webflux-benchmarks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrisgleissner%2Floom-webflux-benchmarks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrisgleissner%2Floom-webflux-benchmarks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrisgleissner%2Floom-webflux-benchmarks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chrisgleissner","download_url":"https://codeload.github.com/chrisgleissner/loom-webflux-benchmarks/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrisgleissner%2Floom-webflux-benchmarks/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33916319,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-04T02:00:06.755Z","response_time":64,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","http","java","java21","jpa","matplotlib","microservice","performance-optimization","performance-testing","postgresql","projectloom","python3","rest","scalability","spring-boot-cache","springboot3","ubuntu2404","virtualthreads","webflux"],"created_at":"2024-10-12T02:43:59.939Z","updated_at":"2026-06-04T17:31:16.254Z","avatar_url":"https://github.com/chrisgleissner.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Benchmark of Java Virtual Threads vs WebFlux\n\n[![build](https://github.com/chrisgleissner/loom-webflux-benchmarks/actions/workflows/build.yaml/badge.svg)](https://github.com/chrisgleissner/loom-webflux-benchmarks/actions)\n[![benchmark](https://github.com/chrisgleissner/loom-webflux-benchmarks/actions/workflows/benchmark.yaml/badge.svg)](https://github.com/chrisgleissner/loom-webflux-benchmarks/actions/workflows/benchmark.yaml)\n[![soaktest](https://github.com/chrisgleissner/loom-webflux-benchmarks/actions/workflows/soaktest.yaml/badge.svg)](https://github.com/chrisgleissner/loom-webflux-benchmarks/actions/workflows/soaktest.yaml)\n[![Coverage Status](https://coveralls.io/repos/github/chrisgleissner/loom-webflux-benchmarks/badge.svg)](https://coveralls.io/github/chrisgleissner/loom-webflux-benchmarks)\n\nThis Java 21 project benchmarks a simple [Spring Boot 3.4](https://spring.io/projects/spring-boot) microservice using\nconfigurable scenarios, comparing Java Virtual Threads (introduced by [Project Loom, JEP 444](https://openjdk.org/jeps/444)) using Tomcat and Netty\nwith [Spring WebFlux](https://docs.spring.io/spring-framework/reference/web/webflux.html) (relying on [Project Reactor](https://projectreactor.io/)) using Netty.\n\nAll benchmark results below come from a dedicated bare metal test environment. The benchmark is also scheduled to run monthly on GitHub-hosted runners, using [all combinations](./results/ci/ci.md) of (Ubuntu 22.04, Ubuntu 24.04) and (Java 21, Java 23).\n\n## Background\n\nBoth Spring WebFlux and Virtual Threads are alternative technologies to create Java microservices that support a high\nnumber of concurrent users, mapping all incoming requests to very few shared operating system threads. This reduces the\nresource overhead incurred by dedicating a single operating system thread to each user.\n\nSpring WebFlux was first introduced in September 2017. Virtual Threads were first introduced as preview feature with\nJava 19 and were fully rolled out with Java 21 in September 2023.\n\n## TL;DR\n\n\u003e [!NOTE]\n\u003e In a nutshell, the benchmark results are:\n\u003e\n\u003e **Virtual Threads on Netty** (using [blocking code](https://docs.oracle.com/javase/8/docs/api/java/lang/Thread.html#sleep-long-)) showed very similar and often superior performance characteristics (latency percentiles, requests per second,\n\u003e system load) compared with **WebFlux on Netty** (using non-blocking code and relying on [Mono](https://projectreactor.io/docs/core/release/api/reactor/core/publisher/Mono.html)\n\u003e and [Flux](https://projectreactor.io/docs/core/release/api/reactor/core/publisher/Flux.html) from Project Reactor):\n\u003e - Virtual Threads on Netty was the [benchmark winner](#Netty-based-Approaches) for ca. 40% more combinations of metrics and benchmark scenarios than Project Reactor on Netty.\n\u003e - For all high user count scenarios, it had the lowest latency as well as the largest number of requests for the entirety of each benchmark run.\n\u003e - In many cases (e.g. [60k-vus-smooth-spike-get-post-movies](#60k-vus-smooth-spike-get-post-movies)), the 90th and 99th percentile latencies (P90 and P99)\n    were considerably lower for Virtual Threads on Netty when compared with WebFlux on Netty.\n\u003e - For both approaches, we could scale up to the same number of virtual users (and thus TCP connections) before\n    exhausting the CPU and running into time-outs due to rejected TCP connection requests.\n\u003e\n\u003e **Virtual Threads on Tomcat** are not recommended for high load:\n\u003e - We saw considerably higher resource use compared with the two Netty-based approaches.\n\u003e - There were many time-out errors as visualized by red dots in the charts, even when the CPU use was far below 100%. In contrast, none the Netty-based scenarios experienced any errors, even with a CPU use of 100%.\n\n## Benchmark Winners\n\nBelow are top-performing approaches across all scenarios and metrics, visualizing the contents of [results/scenarios-default/results.csv](results/scenarios-default/results.csv):\n\n- Each cell shows the metric values of best approach (on top) and runner-up.\n    - What \"best\" is depends on the metric: A lower value is better for all metrics except for metrics starting with `requests_ok`, `requests_per_second`, or `sockets`.\n    - Approaches which encountered request errors are ranked below those approaches with only successful requests. Such \"failed\" approaches have all their metric values printed in red and suffixed with `E`.\n    - An overall ranking based on the win count of each approach is shown in the legend: `(1)` indicates the overall best approach, `(2)` the runner-up, and so on. This overall ranking is also shown next to each metric value.\n- Cells are colored based on the winning approach. The darker the color, the bigger the lead of the winning approach over the runner-up. If cells are white or faded, there's no clear winner as the top two approaches performed similarly.\n- For [detailed charts](#Charts) on each approach and test scenario combination, have a look at the second half of this document.\n- All measurements below were performed on the dedicated, non-virtualized test environment described under [Results](#Results). Additional [monthly measurements](./results/ci/ci.md) are performed on virtualized GitHub-hosted Runners.\n\n### All Approaches\n\nThis chart compares Project Loom (on both Tomcat and Netty) with Project Reactor (on Netty).\n\n![All Results](results/scenarios-default/results.png)\n\n### Netty-based Approaches\n\nThis chart is based on same benchmark as before, but only considers Netty-based approaches.\n\n![Netty Results](results/scenarios-default/results-netty.png)\n\n## Benchmark Features\n\n* Fully automated and CLI-driven by running a single command: `benchmark.sh`.\n* Different test scenario files, each containing one or more scenarios. Example: `src/main/resources/scenarios/scenario.csv`.\n* Operating system thread re-use by waiting and by performing transitive HTTP calls of configurable call depth.\n* Interacts with realistic JSON APIs.\n* Creates single PNG image via [Matplotlib](https://matplotlib.org/) for each combination of scenario and approach which contains:\n    * Raw latencies and P50/90/99 percentiles, as well as any errors.\n    * System metrics for CPU, RAM, sockets, and network throughput.\n    * JVM metrics such as heap usage, garbage collections (GCs), and platform thread count.\n* Creates summary PNG image of all scenarios which shows best approaches.\n\n## Benchmark Design\n\nThe benchmark is driven by [k6](https://k6.io/docs/) which repeatedly issues HTTP requests to a service listening at http://localhost:8080/\n\nThe service exposes multiple REST endpoints. The implementation of each has the same 3 stages:\n\n1. **HTTP Call**: If `$delayCallDepth \u003e 0`, call `GET /$approach/epoch-millis` recursively `$delayCallDepth` times to mimic calls to upstream service(s).\n    - By default, all approaches use `Spring Boot`'s [WebFlux WebClient](https://docs.spring.io/spring-framework/reference/integration/rest-clients.html#rest-webclient)\n      based on Netty.\n    - The scenarios in [scenarios-clients.csv](src/main/resources/scenarios/scenarios-clients) compare the `WebClient` with Spring Boot's\n      [RestClient](https://docs.spring.io/spring-framework/reference/integration/rest-clients.html#rest-restclient) using various client implementations. For details see the [Multi-Client Scenarios](#multi-client-scenarios) chapter.\n2. **Wait**: If `$delayCallDepth = 0`, wait `$delayInMillis` (default: `100`) to mimic the delay incurred by a network call, filesystem access, or similar.\n    - Whilst the request waits, its operating system thread can be reused by another request.\n    - The imperative approaches (`platform-tomcat`, `loom-tomcat`, and `loom-netty`) use blocking wait whilst the reactive approach (`webflux-netty`) uses non-blocking wait.\n3. **Calculate and Return Response** specific to REST endpoint.\n\n### Sample Flow\n\nGet all [movies](#movies) using `loom-netty` approach, an HTTP call depth of `1` and a delay of `100` milliseconds:\n\n```mermaid\nsequenceDiagram\n    participant k6s\n    participant service\n    k6s-\u003e\u003e+service: GET /loom-netty/movies?delayCallDepth=1\u0026delayMillis=100\n    service-\u003e\u003e+service: GET /loom-netty/epoch-millis?delayCallDepth=0\u0026delayMillis=100\n    service-\u003e\u003eservice: Wait 100 milliseconds\n    service--\u003e\u003e-service: Return current epoch millis\n    service-\u003e\u003eservice: Find movies\n    service--\u003e\u003e-k6s: Return movies\n```\n\n### REST APIs\n\nThe microservice under test exposes several RESTful APIs. In the following descriptions, `$approach` is the approach\nunder test and can be one of `loom-tomcat`, `loom-netty`, and `webflux-netty`.\n\nAll REST APIs support the following query parameters:\n\n- `delayCallDepth`: Depth of recursive HTTP call stack to `$approach/epoch-millis` endpoint prior to server-side delay; see [Scenario Columns](#Columns) for more details.\n- `delayInMillis`: Server-side delay in milliseconds; see [Scenario Columns](#Columns) for more details.\n\n#### epoch-millis\n\nThe [TimeController](src/main/java/uk/gleissner/loomwebflux/time/TimeController.java) returns the milliseconds since the\nepoch, i.e. 1 Jan 1970:\n\n- This is one of the simplest possible APIs to provide a best-case performance scenario.\n- Supported requests:\n    - `GET /$approach/epoch-millis`\n\n#### movies\n\nThe [MovieController](src/main/java/uk/gleissner/loomwebflux/movie/MovieController.java) gets and saves movies which are\nstored in an [H2](https://h2database.com/) in-memory DB via [Spring Data JPA](https://spring.io/projects/spring-data-jpa), fronted by a [Caffeine](https://github.com/ben-manes/caffeine)-backed [Spring Boot cache](https://docs.spring.io/spring-boot/reference/io/caching.html):\n\n- This is a realistic JSON API as exposed by a typical microservice.\n- Several hard-coded movies by three directors are provided.\n\nDB Considerations:\n\n- By default, writes are not saved since the code under test is identical for all approaches and would thus only\n  contribute to CPU use. However, this can be controlled with the Spring Boot property `loom-webflux.repo-read-only`\n  in `src/main/resources/application.yaml`.\n- The H2 DB was chosen for the same reason. To swap it for PostgreSQL, specify `postgres` in the `serverProfiles` column of the scenario CSV file. See [scenarios-postgres.csv](./src/main/resources/scenarios/scenarios-postgres.csv)\n  and [PostgreSQL results](results/scenarios-postgres/results-netty.png).\n\nSupported requests:\n\n- `GET /$approach/movies?directorLastName={director}`:\n    - Returns movies by the specified director.\n    - Supported `{director}` values and their respective response body size in bytes, based on the default movies:\n        - `Allen`: 1597 bytes (unindented)\n        - `Hitchcock`: 1579 bytes (unindented)\n        - `Kubrick`: 1198 bytes (unindented)\n- `POST /$approach/movies`:\n    - Saves one or more movies.\n    - The [sample movies](src/main/resources/scenarios/movies.json) saved during the load tests measure 7288 bytes (indented).\n\n## Requirements\n\n### Software\n\n* Unix-based OS; tested with Ubuntu 22.04 and 24.04\n* Java 21 or above\n* [k6](https://k6.io/docs/) and Python 3 with [Matplotlib](https://matplotlib.org/) to drive load and measure latency\n* [sar/sadf](https://linux.die.net/man/1/sar) to measure system resource use\n* Python 3 and [Matplotlib](https://matplotlib.org/) to convert latency and system CSV measurements into a PNG image\n\n### Hardware\n\nThe hardware requirements depend on the scenarios you intend to run. If you run all scenarios (e.g. via `./benchmarks.sh`), then the following is recommended:\n\n* CPU: Intel i5 12600K or similar\n* RAM: 32 GiB\n\nIf you only run the default scenarios configured in `src/main/resources/scenarios/scenarios-default.csv`, the following is sufficient:\n\n* CPU: Intel 6700K or similar\n* RAM: 16 GiB\n\n## Setup\n\nThe following instructions assume you are using a Debian-based Linux such as Ubuntu 22.04 or 24.04.\n\n### Java 21\n\nYou'll need Java 21 or above:\n\n```shell\nsudo apt install openjdk-21-jdk\n```\n\n### k6\n\n[k6](https://k6.io/docs/) is used to load the service:\n\n```shell\nsudo gpg -k\nsudo gpg --no-default-keyring --keyring /usr/share/keyrings/k6-archive-keyring.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D69\necho \"deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main\" | sudo tee /etc/apt/sources.list.d/k6.list\nsudo apt-get update\nsudo apt-get install k6\n```\n\n### Python 3, matplotlib, sar and sadf\n\nPython 3 and `matplotlib` are used to convert the CSV output of `k6` and `sar`/`sadf` to a single PNG chart. The `sar`\nand `sadf` tools come as part of `sysstat` and are used to measure resource use. To install them run:\n\n```shell\nsudo apt update \u0026\u0026 sudo apt install -y python3 python3-matplotlib sysstat\n```\n\n### Linux Optimizations\n\nThe following adjustments optimize Linux for HTTP load tests.\n\n#### Increase Open File Limit\n\nEnsure your system can handle a large number of concurrent connections:\n\n```shell\nprintf '* soft nofile 1048576\\n* hard nofile 1048576\\n' | sudo tee -a /etc/security/limits.conf \n```\n\n#### Increase Port Range and Allow Fast Connection Reuse\n\nIncrease the port range for outgoing TCP connections and allow quick connection reuse:\n\n```shell\nprintf 'net.ipv4.ip_local_port_range=1024 65535\\nnet.ipv4.tcp_tw_reuse = 1\\n' | sudo tee -a /etc/sysctl.conf \u0026\u0026 sudo sysctl -p\n```\n\n#### Activate Changes\n\nLog out and back in.\n\n## Execution\n\n### benchmark.sh\n\nRun a benchmark for each combination of approaches and scenarios defined in a scenario CSV file. Results are stored in\n`build/results/`:\n\n```shell\n./benchmark.sh \n```\n\nUsage as per `benchmark.sh -h`:\n\n```\nUsage: benchmark.sh [OPTION]... [SCENARIO_FILE]\nRuns benchmarks configured by a scenario file.\n\nSCENARIO_FILE:     Scenario configuration CSV file in src/main/resources/scenarios/. Default: scenarios-default.csv\n\nOPTION:\n  -a \u003capproaches\u003e  Comma-separated list of approaches to test. Default: loom-tomcat, loom-netty, webflux-netty\n                   Supported approaches: platform-tomcat, loom-tomcat, loom-netty, webflux-netty\n  -C               Keep CSV files used to create chart. Default: false\n  -h               Print this help\n```\n\n### benchmarks.sh\n\nThis is a wrapper over `benchmark.sh` and supports multiple scenario files:\n\n```shell\n./benchmarks.sh \n```\n\nUsage as per `benchmarks.sh -h`:\n\n```\nUsage: benchmarks.sh [OPTION]... [SCENARIO_FILE]...\nWrapper over benchmark.sh that supports multiple scenario files and optionally suspends the system on completion.\n\nSCENARIO_FILE:           Zero or more space-separated scenario configuration CSV files in src/main/resources/scenarios/.\n                         Default: Default: scenarios-default.csv scenarios-clients.csv scenarios-deep-call-stack.csv scenarios-postgres.csv scenarios-sharp-spikes.csv scenarios-soaktest.csv\n\nOPTION:\n  -d, --dry-run          Print what would be done without actually performing it.\n  -k, --kill-java        Kill all Java processes after each benchmark. Default: false\n  -o, --options \"\u003copts\u003e\" Pass additional options to the benchmark.sh script. Run \"./benchmark.sh -h\" for supported options.\n  -s, --suspend          Suspend the system upon completion of the script. Default: false\n  -h, --help             Show this help message and exit.\n```\n\nPlease note that the default configured scenarios may take several hours to complete.\n\n### Approaches\n\n- **platform-tomcat**: Platform threads using [Tomcat](https://tomcat.apache.org/) server\n- **loom-tomcat**: Virtual Threads using [Tomcat](https://tomcat.apache.org/) server\n- **loom-netty**: Virtual Threads on [Netty](https://netty.io/) server\n- **webflux-netty**: WebFlux on Netty server\n\nAll approaches use the same Spring Boot 3.2 version.\n\n### Scenarios\n\n#### Default Scenarios\n\nThese scenarios cover a mixture of load patterns between 5k and 20k users.\n\n- Config: [./src/main/resources/scenarios/scenarios-default.csv](./src/main/resources/scenarios/scenarios-default.csv)\n- Results: [./results/scenarios-default/results.md](./results/scenarios-default/results.md)\n\n#### High-Load Scenarios\n\nThese are steady-state scenarios for 40k users and ramp-up/down scenarios for 60k users.\n\n- Config: [./src/main/resources/scenarios/scenarios-high-load.csv](./src/main/resources/scenarios/scenarios-high-load.csv)\n- Results: [./results/scenarios-high-load/results.md](./results/scenarios-high-load/results.md)\n\n#### Multi-Client Scenarios\n\nThese scenarios compare both Spring Boot [RestClient](https://docs.spring.io/spring-framework/reference/integration/rest-clients.html#rest-restclient) and\n[WebClient](https://docs.spring.io/spring-framework/reference/integration/rest-clients.html#rest-webclient) implementations with each other.\n\n- Config: [./src/main/resources/scenarios/scenarios-clients.csv](./src/main/resources/scenarios/scenarios-clients.csv)\n- Results: [./results/scenarios-clients/results.md](./results/scenarios-clients/results.md)\n\nAll scenarios except for those tested with a `webflux-netty` approach use the `WebClient` or `RestClient` implementation specified in the scenario name. However,\nthe `webflux-netty` approach always uses a fully reactive approach and therefore always uses the non-blocking `WebClient`.\n\nThe following clients are compared:\n\n- Spring Boot `RestClient` based on:\n    - [JDK HttpClient](https://docs.oracle.com/en/java/javase/11/docs/api/java.net.http/java/net/http/HttpClient.html)\n    - [Apache Commons HttpClient 5](https://hc.apache.org/httpcomponents-client-5.4.x/current/httpclient5/apidocs/)\n    - [Netty](https://projectreactor.io/docs/netty/1.1.21/api/reactor/netty/http/client/HttpClient.html)\n- Spring Boot `WebClient` based on:\n    - [Netty](https://projectreactor.io/docs/netty/1.1.21/api/reactor/netty/http/client/HttpClient.html)\n\n#### Other Scenarios\n\n- [deep-call-stack](./results/scenarios-deep-call-stack/results.md): High delay call depths\n- [postgres](./results/scenarios-postgres/results.md): Use PostgreSQL (started via Docker) instead of H2\n- [sharp-spikes](./results/scenarios-sharp-spikes/results.md): Intermittent sharp load spikes from 0 to 10/20/30k users\n- [soaktest](./results/scenarios-soaktest/results.md): Slow ramp-up to 10k users over 15 minutes, followed by ramp-down\n\n### Steps\n\nThe benchmark run for each `$scenario` consists of the following phases and steps:\n\n#### Before Benchmark\n\n* Build and start the Spring Boot service with a specific `$approach` as Spring Boot profile, using the config in `src/main/resources/application.yaml` and overridden by `src/main/resources/application-$approach.yaml` if defined.\n\n#### Benchmark\n\n* Run the benchmark as configured by the `$scenario`.\n* For each `$resultType` (i.e. `latency`, `system`, or `jvm`), create a CSV file at `build/results/$scenario/$approach-$resultType.csv`.\n\n#### After Benchmark\n\n* Convert CSV files into `build/results/$scenario/$approach.png`\n* Delete the CSV files unless the `-C` CLI option was specified.\n* Stop the service.\n\n## Config\n\n### Common\n\n- The `build.gradle` file configures the heap space to 2 GiB.\n- The `src/main/resources/application.yaml` file enables HTTP/2.\n- Time-out is 60s for both client and server.\n\n### Scenario-specific\n\nEach line in [src/main/resources/scenarios/scenarios-default.csv](src/main/resources/scenarios/scenarios-default.csv) configures a test scenario which is performed first for Java\nVirtual Threads, then for WebFlux.\n\n#### Example\n\n| scenario                         | k6Config                               | serverProfiles | delayCallDepth | delayInMillis | connections | requestsPerSecond | warmupDurationInSeconds | testDurationInSeconds |\n|----------------------------------|----------------------------------------|----------------|----------------|---------------|-------------|-------------------|-------------------------|-----------------------|\n| 5k-vus-and-rps-get-time          | get-time.js                            |                | 0              | 100           | 5000        | 5000              | 10                      | 300                   |\n| 20k-vus-smooth-spike-get-movies] | k6-20k-vus-smooth-spike-get-movies].js | postgres       | 0              | 100           | 20000       |                   | 0                       | 300                   |\n\n#### Columns\n\n1. `scenario`: Name of scenario. Is printed on top of each diagram.\n2. `k6Config`: Name of the [K6 Config File](https://k6.io/docs/using-k6/http-requests/) which is assumed to be in\n   the `config` folder\n3. `serverProfiles`: Pipe-delimited Spring profiles which are also used to start and stop Docker containers. For example, specifying the value `postgres|no-cache` has these effects:\n    - The Spring Boot profiles `postgres,no-cache` are added to the default Spring Boot profile of `$approach`.\n    - The files `src/main/docker/docker-compose-postgres.yaml` and `src/main/docker/docker-compose-no-cache.yaml` (if existent)\n      are used to start/stop Docker containers before/after each scenario run.\n3. `delayCallDepth`: Depth of recursive HTTP call stack to `$approach/epoch-millis` endpoint prior to server-side delay.\n    - Mimics calls to upstream services which allow for reuse of the current platform thread.\n    - For example, a value of `0` means that the service waits for `$delayInMillis` milliseconds immediately upon receiving a request.\n    - Otherwise, it calls the `$approach/epoch-millis` with `${delayCallDepth - 1}`.\n    - This results in a recursive HTTP-request-based descent into the service, creating a call stack of depth `$delayCallDepth`.\n3. `delayInMillis`: Server-side delay of each request, in milliseconds. Mimics a delay such as invoking a DB which allow for reuse of the current platform thread.\n4. `connections`: Number of TCP connections, i.e. virtual users.\n5. `requestsPerSecond`: Number of requests per second across all connections. Left empty for scenarios where the number\n   of requests per second is organically derived based on the number of connections, the request latency, and any\n   explicit client-side delays.\n6. `warmUpDurationInSeconds`: Duration of the warm-up iteration before the actual test. Warm-up is skipped if `0`.\n7. `testDurationInSeconds`: Duration of the test iteration.\n\n## Test Environment\n\n- Unless noted otherwise, all tests were conducted on this test environment.\n- **Preparation**: The system was rebooted before each test and quieted down as much as possible. The baseline total CPU\n  use before test start was 0.3%.\n- **Co-location**: Test driver (k6) and server under test (Spring Boot microservice) were co-located on the same\n  physical machine. The aim of this benchmark is not to achieve maximum absolute performance, but rather to compare\n  different server-side approaches with each other. Considering that the test driver and the load it produced was\n  identical for the combination of server-side approach and scenario, this co-location should not affect the validity of\n  the test results.\n\n### Hardware\n\n- CPU: [Intel Core i5-14600K](https://www.intel.com/content/www/us/en/products/sku/236799/intel-core-i5-processor-14600k-24m-cache-up-to-5-30-ghz/specifications.html) with 5.3GHz, 12 cores, 20 threads, and a base/turbo power of both 180W\n- Motherboard: [Asus ProArt Z690-Creator WIFI](https://www.asus.com/uk/motherboards-components/motherboards/proart/proart-z690-creator-wifi/)\n- RAM: 64 GiB DDR5 (2 x Corsair Vengeance 32 GiB 5600 MT/s CL40)\n- Network: Loopback interface\n- Virtualization: None; bare metal desktop\n\n### Software\n\n- OS: Ubuntu 24.04.2 LTS\n- Kernel: 6.11.0-19-generic\n- Java: Amazon Corretto JDK 21.0.6.7.1\n- Spring Boot 3.4.3\n\n\u003e [!NOTE]\n\u003e The actual software versions used by a benchmark are automatically determined and shown at the beginning of each `results.md` file. If there are differences to the above, then the values in `results.md` are correct.\n\n## Results\n\nThis chapter shows the results of each default scenario, sorted by ascending scenario load. The results below can also be found in [./results/scenarios-default/results.md](./results/scenarios-default/results.md).\n\n### Errors\n\nAny lines in the client-side or error-side log files which contain the term `error` (case-insensitive) are preserved. You can find them in error log files, located in the results folder alongside the generated PNG files.\n\nAny failed requests appear both in the latency chart as red dots, as well as in the RPS chart as part of a continuous orange\nline. Additionally, they leave a trace in the `$approach-latency.csv` file, if preserved by running the benchmark with the `-C` option:\n\n- A very small latency below 3ms indicates that the client failed to establish a TCP connection. Example from the latency CSV file: `1715728866471,0.000000,0,dial: i/o timeout,1211`. Such requests are not considered when reporting minimum latency since this could obscure the minimum latency of\n  successful requests.\n- A very large latency above 60s indicates a server-side time-out. Example from the latency CSV file: `1715728861008,60001.327066,0,request timeout,1050`.\n\n### 5k-vus-and-rps-get-time\n\nThis scenario aims to maintain a steady number of 5k virtual users (VUs, i.e. TCP connections) as well as 5k requests\nper second (RPS) across all users for 3 minutes:\n\n- Each user issues a request and then waits. This wait between consecutive requests is controlled by k6 in order to\n  achieve the desired number of RPS.\n- The server-side delay is 100ms.\n- The server returns the current millis since the epoch.\n\n#### Virtual Threads (Tomcat)\n\n![Loom](results/scenarios-default/5k-vus-and-rps-get-time/loom-tomcat.png)\n\n#### Virtual Threads (Netty)\n\n![WebFlux](results/scenarios-default/5k-vus-and-rps-get-time/loom-netty.png)\n\n#### WebFlux (Netty)\n\n![WebFlux](results/scenarios-default/5k-vus-and-rps-get-time/webflux-netty.png)\n\n### 5k-vus-and-rps-get-movies\n\nLike the previous scenario, but the response body contains a JSON of movies.\n\nFor further details, please see the [movies](#movies) section.\n\n#### Virtual Threads (Tomcat)\n\n![Loom](results/scenarios-default/5k-vus-and-rps-get-movies/loom-tomcat.png)\n\n#### Virtual Threads (Netty)\n\n![Loom](results/scenarios-default/5k-vus-and-rps-get-movies/loom-netty.png)\n\n#### WebFlux (Netty)\n\n![WebFlux](results/scenarios-default/5k-vus-and-rps-get-movies/webflux-netty.png)\n\n### 10k-vus-and-rps-get-movies\n\nLike the previous scenario, but 10 virtual users and requests per second.\n\n#### Virtual Threads (Tomcat)\n\n![Loom](results/scenarios-default/10k-vus-and-rps-get-movies/loom-tomcat.png)\n\n#### Virtual Threads (Netty)\n\n![Loom](results/scenarios-default/10k-vus-and-rps-get-movies/loom-netty.png)\n\n#### WebFlux (Netty)\n\n![WebFlux](results/scenarios-default/10k-vus-and-rps-get-movies/webflux-netty.png)\n\n### 10k-vus-and-rps-get-movies-call-depth-1\n\nLike the previous scenario, but mimics a request to an upstream service.\n\n- On receiving an incoming HTTP request, the service calls itself via HTTP.\n- This secondary request then waits 100 milliseconds.\n\n#### Virtual Threads (Tomcat)\n\n![Loom](results/scenarios-default/10k-vus-and-rps-get-movies-call-depth-1/loom-tomcat.png)\n\n#### Virtual Threads (Netty)\n\n![Loom](results/scenarios-default/10k-vus-and-rps-get-movies-call-depth-1/loom-netty.png)\n\n#### WebFlux (Netty)\n\n![WebFlux](results/scenarios-default/10k-vus-and-rps-get-movies-call-depth-1/webflux-netty.png)\n\n### 20k-vus-stepped-spike-get-movies\n\nThis scenario ramps up virtual users (and thus TCP connections) from 0 to 20k in multiple steps, then back down:\n\n- Each step has a short [riser](https://en.wikipedia.org/wiki/Stair_riser) time when users are increased,\n  followed by a longer [tread](https://en.wikipedia.org/wiki/Stair_tread) time when users are held\n  constant.\n- Each user issues a request, waits for the response, and then waits for a random delay between 1s and 3s. This reduces\n  the load and better mimics real user interactions with a service, assuming the service calls are driven by user\n  interactions with a website that relies on the service under test.\n- The server-side delay before returning a response is 100ms.\n\n#### Virtual Threads (Tomcat)\n\n![Loom](results/scenarios-default/20k-vus-stepped-spike-get-movies/loom-tomcat.png)\n\n#### Virtual Threads (Netty)\n\n![Loom](results/scenarios-default/20k-vus-stepped-spike-get-movies/loom-netty.png)\n\n#### WebFlux (Netty)\n\n![WebFlux](results/scenarios-default/20k-vus-stepped-spike-get-movies/webflux-netty.png)\n\n### 20k-vus-smooth-spike-get-movies\n\nLike the previous scenario, but linear ramp-up and down.\n\n#### Virtual Threads (Tomcat)\n\n![Loom](results/scenarios-default/20k-vus-smooth-spike-get-movies/loom-tomcat.png)\n\n#### Virtual Threads (Netty)\n\n![Loom](results/scenarios-default/20k-vus-smooth-spike-get-movies/loom-netty.png)\n\n#### WebFlux (Netty)\n\n![WebFlux](results/scenarios-default/20k-vus-smooth-spike-get-movies/webflux-netty.png)\n\n### 20k-vus-smooth-spike-get-post-movies\n\nLike the previous scenario, but instead of just getting movies, we are now additionally saving them:\n\n- 75% of requests are GET requests which are split into three groups, each requesting movies by a different director.\n- 25% of requests are POST requests.\n\nFor further details, please see the [movies](#movies) section.\n\n#### Virtual Threads (Tomcat)\n\n![Loom](results/scenarios-default/20k-vus-smooth-spike-get-post-movies/loom-tomcat.png)\n\n#### Virtual Threads (Netty)\n\n![Loom](results/scenarios-default/20k-vus-smooth-spike-get-post-movies/loom-netty.png)\n\n#### WebFlux (Netty)\n\n![WebFlux](results/scenarios-default/20k-vus-smooth-spike-get-post-movies/webflux-netty.png)\n\n### 20k-vus-smooth-spike-get-post-movies-call-depth-1\n\nLike the previous scenario, but mimics call to upstream service as explained in [10k-vus-and-rps-get-movies-call-depth-1](#10k-vus-and-rps-get-movies-call-depth-1).\n\n\u003e [!NOTE]\n\u003e For `loom-netty` and `webflux-netty`, this scenario was CPU-contended on the test environment upon reaching ca. 5,000 RPS.\n\u003e Whilst causing no errors, it drastically increased latencies.\n\n#### Virtual Threads (Tomcat)\n\n![Loom](results/scenarios-default/20k-vus-smooth-spike-get-post-movies-call-depth-1/loom-tomcat.png)\n\n#### Virtual Threads (Netty)\n\n![Loom](results/scenarios-default/20k-vus-smooth-spike-get-post-movies-call-depth-1/loom-netty.png)\n\n#### WebFlux (Netty)\n\n![WebFlux](results/scenarios-default/20k-vus-smooth-spike-get-post-movies-call-depth-1/webflux-netty.png)\n\n## High Load Results\n\nThis chapter highlights interesting results based on `scenarios-high-load.csv` which scales up to 60k users. For full\nresults, see [results/scenarios-high-load/results.md](results/scenarios-high-load/results.md).\n\n### Summary\n\n![Summary](results/scenarios-high-load/results.png)\n\n### 60k-vus-smooth-spike-get-post-movies\n\nLike [20k-vus-smooth-spike-get-post-movies](#20k-vus-smooth-spike-get-post-movies), but scaling up to 60k users.\n\n#### Virtual Threads (Netty)\n\n![Loom](results/scenarios-high-load/60k-vus-smooth-spike-get-post-movies/loom-netty.png)\n\n#### WebFlux (Netty)\n\n![WebFlux](results/scenarios-high-load/60k-vus-smooth-spike-get-post-movies/webflux-netty.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchrisgleissner%2Floom-webflux-benchmarks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchrisgleissner%2Floom-webflux-benchmarks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchrisgleissner%2Floom-webflux-benchmarks/lists"}