{"id":15990600,"url":"https://github.com/gaplo917/generic-http-benchmark","last_synced_at":"2025-05-07T06:41:13.727Z","repository":{"id":77489562,"uuid":"543960804","full_name":"gaplo917/generic-http-benchmark","owner":"gaplo917","description":"Generic HTTP benchmark using Gatling. Check out the results 👇","archived":false,"fork":false,"pushed_at":"2023-11-23T06:17:56.000Z","size":1198,"stargazers_count":4,"open_issues_count":0,"forks_count":3,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-31T07:22:00.090Z","etag":null,"topics":["benchmark","java","jvm","kotin-coroutines","kotlin","reactor","virtual-threads"],"latest_commit_sha":null,"homepage":"https://benchmarks.gaplo.tech/index.html#","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gaplo917.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2022-10-01T09:05:56.000Z","updated_at":"2023-03-29T08:03:16.000Z","dependencies_parsed_at":"2023-12-09T16:47:30.306Z","dependency_job_id":null,"html_url":"https://github.com/gaplo917/generic-http-benchmark","commit_stats":{"total_commits":67,"total_committers":4,"mean_commits":16.75,"dds":0.07462686567164178,"last_synced_commit":"57bcbe94410059e4ca3ad88f1d6874f43c628e78"},"previous_names":["gaplo917/generic-http-benchmark","gaplo917/kotlin-coroutine-benchmark"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gaplo917%2Fgeneric-http-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gaplo917%2Fgeneric-http-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gaplo917%2Fgeneric-http-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gaplo917%2Fgeneric-http-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gaplo917","download_url":"https://codeload.github.com/gaplo917/generic-http-benchmark/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252830514,"owners_count":21810769,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","java","jvm","kotin-coroutines","kotlin","reactor","virtual-threads"],"created_at":"2024-10-08T05:22:37.717Z","updated_at":"2025-05-07T06:41:13.708Z","avatar_url":"https://github.com/gaplo917.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Generic HTTP benchmark\n\nCheck out the interesting result at [https://benchmarks.gaplo.tech](https://benchmarks.gaplo.tech).\n\nContributions are\nwelcome. [Rust and C# implementations were contributed by others!](https://github.com/gaplo917/generic-http-benchmark/pulls?q=is%3Apr+is%3Aclosed)\nIf you are interested to add your favourite framework in this project, take a look on their PRs for reference.\n\n| Goal                                         | Description                                                                                                                             |\n|----------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------|\n| ✅ End-to-End HTTP benchmark                  | Utilizing [Gatling](https://gatling.io/) to generate real-world HTTP load and output fruitful performance report                        |\n| ✅ Fully Containerized                        | Single `docker compose` dependency is enough to get started. Say good bye to countless installations of project dependencies.           |\n| ✅ Scalable Design                            | Adding a new benchmark target only needs a Dockerfile and a simple configuration.                                                       |\n| ✅ Dev-Friendly Design                        | Run the benchmark in **your local** machine instead of trusting the report online. Easy to make code changes to run to learn.           |\n| ✅ Zero Code Configuration                    | Kotlin implementation of utilizing Gatling to work like `ab` or `wrk` CLI tools                                                         |\n| ✅ High Concurrency                           | Thanks for the containerization. Tested 20k+ concurrent users with no problem. No more file description or TCP connection limit issues. |\n| ✅ Continuous Concurrency Ramp Up and Sustain | Discover the application performance across continuous time series charts with configuration ramp up and sustain duration.              |\n| ✅ Gatling Reports Automation                 | Automatically parse Gatling output and upload to https://benchmarks.gaplo.tech CDN                                                      |\n| 🔧 Self-host action runner                   | Create a self-hosted action runner base image on cloud and provision faster to run benchmarks                                           |\n| 🔧 Provisioning Automation                   | Use GitHub Actions to automatically provision a self-hosted action runner to run benchmarks on                                          |\n| 🔧 Report Summary                            | Aggregate all-in-one Gatling simulation log to visualize avg. request/s, response time                                                  |\n| 🔧 More features complaint with `ab`         | Support to define http methods, cookies, headers, payload, http connections, custom cert...etc                                          |\n| ...more                                      |                                                                                                                                         |\n\n| Gatling Scenario    | Goal                                                                                                                                                                       |\n|---------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| ✅ Two dependent I/O | Compare various implementations of non-blocking IO or blocking IO  within its HTTP framework. (i.e. Spring w/ Kotlin Coroutine, Reactive Stream, and Java virtual threads) |\n| 🔧 Two parallel I/O | Compare various implementations of non-blocking or blocking IO implementation                                                                                              |\n| ...                 |                                                                                                                                                                            |\n\n## Getting Started (end-to-end HTTP benchmark)\n\nAll benchmark targets are designed to run in Docker container and use [Gatling](https://gatling.io/)\nto run the load test and get the benchmark of end-to-end result.\n\n![cover](cover.png)\n\n### 1. Define benchmark scenarios in `./config/*.env`\n\nIn `./config/*.env`, there are key configurations to control gatling's concurrency.\n![duration](duration-explained.png)\n\n### 2. Start up the benchmark\n\nFirst, you need to build the docker image for each application you want to test.\n\nYou might need at least 7 vCPU and 16GB Memory resources for the whole docker engine.\n\nRun a single benchmark with the follow commands, it will build automatically if there is no available images.\n\n```bash\n# In case you need to change architecture to build cross platform\nDOCKER_DEFAULT_PLATFORM=linux/arm64/v8 # Mac M1/M2 CPU\nDOCKER_DEFAULT_PLATFORM=linux/amd64 # Intel / AMD CPU\n\n# Run single benchmark (i.e. ktor)\nENV_FILE=./config/ktor.env\ndocker compose --env-file $ENV_FILE build \u0026\u0026 \\\ndocker compose --env-file $ENV_FILE up -d benchmark-target \u0026\u0026 \\\ndocker compose --env-file $ENV_FILE up gatling-runner \u0026\u0026 \\\ndocker compose --env-file $ENV_FILE down\n```\n\nOR run all benchmarks through the scripts\n\n```bash\n# Run all benchmarks, config available in `./config/`\nsh all-gatling-benchmarks.sh\n```\n\n### 3. During the test, Grab a ☕️ or 🍵 and wait.\n\n- `gatling-runner` output all the logs to `./logs/`\n\n- gatling reports available in `./reports/`\n\n### 4. Study the Gatling reports\n\nCheckout the benchmark report in `./reports/`. Hope you would have a\ngreat understanding of your applications!\n\n### (Optional) Grafana Dashboard to view application metrics\n\n1. Un-comment the `prometheus` and `grafana` services in `docker-compose.yaml`\n2. Make sure the application support prometheus, and add job and endpoints in `prometheus.yml`\n\n```bash\nENV_FILE=./config/ktor.env\ndocker compose --env-file $ENV_FILE up -d benchmark-target prometheus grafana \u0026\u0026\n```\n\nGo to http://localhost:3000 to configure the grafana dashboard.\n\n## Benchmark Environment\n\n| Configuration  | Description                       |\n|----------------|-----------------------------------|\n| Platform       | x86 Intel                         |\n| Cloud Provider | Digital Ocean                     |\n| CPU            | 8 vCPU (Dedicated, CPU-Optimized) |\n| RAM            | 16GB Memory                       |\n| Disk           | 100GB SSD                         |\n|                |                                   |\n\nResult: https://benchmarks.gaplo.tech/index.html\n\n## Develop Gatling Kotlin Project\n\nAfter making Gatling code changes, make sure you run to build the docker image.\n\n```bash\ndocker compose --env-file $ENV_FILE build gatling-runner\n```\n\n## Contribution\n\n1. Add new web framework `frameworks/xxx` implementation under that is functionally equivalent to other implementations.\n2. Add benchmark configuration in `./config/xxx.env`\n\n## Troubleshooting\n\n```bash\n# Remove orphans containers\ndocker compose --env-file ./config/ktor.env down --remove-orphans\n```\n\n## History\n\nThis project is originally designed for testing the performance of Kotlin Coroutines, Reactor,\nand cutting edge Java 19 Virtual Threads on __Spring Boot__ for my application of KotlinConf 2023 talk (Not being\nselected by KotlinConf, [moved in here](https://github.com/gaplo917/coroutine-reactor-virtualthread-microbenchmark)).\n\nMeanwhile, I realized if my work go the extra miles, it can be further extended to benchmark generic frameworks. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgaplo917%2Fgeneric-http-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgaplo917%2Fgeneric-http-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgaplo917%2Fgeneric-http-benchmark/lists"}