{"id":25009213,"url":"https://github.com/barista-benchmarks/barista","last_synced_at":"2026-04-10T17:05:29.991Z","repository":{"id":275751567,"uuid":"914439611","full_name":"barista-benchmarks/barista","owner":"barista-benchmarks","description":"Barista is an open-source Microservice Benchmark Suite for the JVM platform. It comprises applications written in a variety of popular frameworks and supports both JIT and AOT compilations.","archived":false,"fork":false,"pushed_at":"2025-02-04T11:35:18.000Z","size":11795,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-04T12:26:04.264Z","etag":null,"topics":["benchmark-suite","helidon","java","jvm","ktor","micronaut","microservices","playframework","quarkus","spring","vertx"],"latest_commit_sha":null,"homepage":"","language":"CSS","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/barista-benchmarks.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-09T15:47:23.000Z","updated_at":"2025-02-04T11:35:22.000Z","dependencies_parsed_at":"2025-02-04T12:37:27.424Z","dependency_job_id":null,"html_url":"https://github.com/barista-benchmarks/barista","commit_stats":null,"previous_names":["barista-benchmarks/barista"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/barista-benchmarks%2Fbarista","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/barista-benchmarks%2Fbarista/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/barista-benchmarks%2Fbarista/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/barista-benchmarks%2Fbarista/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/barista-benchmarks","download_url":"https://codeload.github.com/barista-benchmarks/barista/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246266252,"owners_count":20749754,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark-suite","helidon","java","jvm","ktor","micronaut","microservices","playframework","quarkus","spring","vertx"],"created_at":"2025-02-05T04:26:18.155Z","updated_at":"2025-12-30T23:17:48.617Z","avatar_url":"https://github.com/barista-benchmarks.png","language":"CSS","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Barista Suite\n\nWelcome!\n\nBarista Suite is an open-source benchmark suite dedicated to measuring the performance of Java microservices.\n\n\nJust like a good barista knows what the client needs, this benchmark suite provides key metrics for evaluating microservices. \nWhen measuring the performance there are many more metrics than peak throughput, for microservices. \nSome microservices may have higher throughput, while some may use less resources and have better latency. \nHaving large selection of metrics can be helpful in understanding differences between microservice builds and frameworks. Currently supported metrics are:\n\n- Throughput\n- Latency\n- Startup time\n- First response times\n- Warmup throughput\n- Memory usage(RSS and VMS)\n- CPU usage\n\n## Outline\n\n- [Quick start](#quick-start)\n- [Dependencies](#dependencies)\n- [Execution modes](#execution-modes)\n- [Barista applications](#barista-applications)\n- [Building the Barista apps](#building-the-barista-apps)\n- [Benchmarking](#benchmarking)\n- [Supported harness options](#supported-harness-options)\n- [Pipeline](#pipeline)\n- [Examples](#example-configurations)\n- [Contributing](#contributing)\n\n## Dependencies\n\n### Python dependencies (only for non-Linux platforms)\n\nBarista runs with no python dependency on Linux operating systems. For MacOS and Windows, the harness relies on [psutil](https://pypi.org/project/psutil/). Here are the setup instructions:\n\nFirst of all create or change to your preferred virtual environment.\nTo create a virtual environment run:\n```console\npython -m venv venv\n```\n\nThen switch to the created environment run the command in the same folder:\n```console\nsource venv/bin/activate\n```\n\nEfforts have been made to minimize the number of dependencies to ease usage. However, `psutil` is still required for non-Linux platforms. Barista uses this package for process termination and the collection of process resource usage information.\n\nIf you're running Barista on any platform other than Linux, please install `psutil` by running:\n```console\npip install psutil\n```\n\n### wrk and wrk2\n\n[wrk](https://github.com/wg/wrk) tool is used for throughput measurements and warming up microservices with JIT compilers. \nFor detailed latency measures we use [wrk2](https://github.com/giltene/wrk2).\n\nYou can get the tools source code and build it yourself from here:\n- [wrk](https://github.com/wg/wrk)\n- [wrk2](https://github.com/giltene/wrk2)\n\nOr from default linux packages.\n\nOnce you have tested (and built) the tools add them to your `PATH` environment variable:\n\n```bash\nexport PATH=\"$PATH:/your/path/to/wrk\"\nexport PATH=\"$PATH:/your/path/to/wrk2\"\n```\n\nAlso make sure to rename the executable in wrk2 folder from `wrk` to `wrk2` this is necessary for benchmark to differentiate the two.\n\nYou can simply do that by going to the `wrk2` directory and executing:\n```console\nmv wrk wrk2\n```\n\n## Quick Start\n\nYou can inform yourself on the details of the benchmark suite in the upcoming sections, this section will just give you instructions for default use-cases.\n\nIt is necessary to first build the Barista applications that we wish to benchmark.\n```console\n./build\n```\nThis command will build all the Barista apps, if you want to speed up the process you can build just a specific one. Take a look at the [Building the Barista apps](#building-the-barista-apps) section for information.\n\nTo quickly test the suite, you can execute the following command:\n```console\n./barista micronaut-hello-world\n```\nWhich benchmarks the `micronaut-hello-world` app with default configurations.\n\nTo run the same benchmark in native mode, run the following:\n```console\n./barista micronaut-hello-world --mode native\n```\n\n## Barista applications\n\nThe Barista benchmark suite comprises a broad and varied selection of applications, covering an array of widely-used microservice frameworks.\n\nThis is the list of Barista applications:\n- ### vanilla-hello-world\n    Minimal microservice implemented using the standard library package [com.sun.net.httpserver](https://docs.oracle.com/en/java/javase/17/docs/api/jdk.httpserver/com/sun/net/httpserver/package-summary.html)\n- ### micronaut-hello-world\n    Minimal microservice implemented in the [Micronaut](https://micronaut.io/) framework\n- ### micronaut-shopcart\n    A fairly complex demo application modelling a web shop implemented in the [Micronaut](https://micronaut.io/) framework\n- ### micronaut-similarity\n    A text similarity application implemented in the [Micronaut](https://micronaut.io/) framework\n- ### quarkus-hello-world\n    Minimal microservice implemented in the [Quarkus](https://quarkus.io/) framework\n- ### quarkus-tika\n    A PDF/ODT document processing application implemented in the [Quarkus](https://quarkus.io/) framework\n- ### spring-hello-world\n    Minimal microservice implemented in the [Spring](https://spring.io/) framework\n- ### spring-petclinic\n    A fairly complex demo application modelling a pet clinic implemented in the [Spring](https://spring.io/) framework\n- ### helidon-hello-world\n    Minimal microservice implemented in the [Helidon](https://helidon.io/) framework\n- ### vertx-hello-world\n    Minimal microservice implemented in the [Vert.x](https://vertx.io/docs/) framework\n- ### ktor-hello-world\n    Minimal microservice written in Kotlin in the [Ktor](https://ktor.io/) framework\n- ### play-scala-hello-world\n    Minimal microservice written in Scala in the [Play](https://www.playframework.com/) framework\n- ### dropwizard-hello-world\n    Sample application using the [Dropwizard](https://www.dropwizard.io/) framework on which only a hello endpoint is stressed\n\nYou can find the implementations in the `benchmarks` subdirectory.\n\n## Execution modes\n\nBarista supports benchmarking in two modes: JVM and native.\n- In the JVM mode the application is executed on a Java Virtual Machine.\n- In the native mode the application is natively executed. This mode is compatible with any natively executable file, but is primarily focused on executing GraalVM native images, including the integrated workflow for generating Native Image Bundles (nibs) and then building application images using the bundles. You can learn more about native images [here](https://www.graalvm.org/latest/reference-manual/native-image/).\n\n## Building the Barista apps\n\nThere is a top-level Barista build script which you can use to build a selection (or all) of the apps. This script builds the application jar and nib (Native Image Bundle) files. The application jar is used for benchmarking in the JVM mode, while the nib is used for benchmarking in the native mode.\nSpecify the name(s) of the applications you wish to build, adding the `--skip-nib-generation` option if you don't need to generate nibs. If you don't specify a single application, all of them will be built.\n\nBuild all applications, generating both jars and nibs:\n```console\n./build\n```\n\nExample command for building a selection of applications, requesting only the generation of jars, skipping the generation of nibs:\n```console\n./build micronaut-hello-world quarkus-tika --skip-nib-generation\n```\n\nThe interface of the Barista build script:\n```\nusage: build [-h] [-s] [-m MAVEN_OPTIONS] [-d] [bench-name ...]\n\npositional arguments:\n  bench-name            name(s) of the benchmark(s) to be built, all are built if unspecified\n\noptions:\n  -h, --help            show this help message and exit\n  -s, --skip-nib-generation\n                        skip building nibs (Native Image Bundles), only build the jars\n  -m MAVEN_OPTIONS, --maven-options MAVEN_OPTIONS\n                        additional options to pass to mvn when building maven projects\n  -d, --debug           show debug logs\n```\n\nAdditionally, each Barista application contains a `build.sh` script at its directory level. The top-level Barista build script invokes these app-level build scripts for its selection.\n```\nBuilds the project jar and then uses GraalVM to generate a nib file (Native Image Bundle)\n\nusage: build.sh [--help] [--skip-nib-generation] [--get-jar] [--get-nib]\n\noptions:\n    --help                  shows this help message and exits\n    --skip-nib-generation   skips building the application nib (Native Image Bundle) file, only builds the jar\n    --get-jar               prints the path of the built jar without building anything. The path will be printed in the pattern of 'application jar file path is: \u003cpath\u003e\\n'\n    --get-nib               prints the path of the built nib (Native Image Bundle) file without building anything. The path will be printed in the pattern of 'application nib file path is: \u003cpath\u003e\\n'\n```\n\n## Benchmarking\n\nHere is an example where several of the options are provided:\n```console\n./barista micronaut-hello-world --mode native --warmup-duration=1 --throughput-duration=1 --latency-duration=1 --cmd-app-prefix=\"time -v\" --vm-options=\"-XX:+PrintGC\" --native-image-build-options=\"-O0\"\n```\nThis command benchmarks the `micronaut-hello-world` app in the `native` execution mode, with a 1s warmup, 1s throughput measurement, 1s latency measurement, and values from the default configuration file (`workloads/default.barista.json`) for the rest of the harness options. It prefixes the app command with `time -v`, propagating `-XX:+PrintGC` to the virtual machine, and propagating `-O0` to the native image build process.\n\nExplore the supported harness options in the following section.\n\n## Supported harness options\n\nAll of the configuration options in the command line are optional and will overwrite any configuration options coming from the configuration file. \nIf no configuration file is provided with `--config path/to/your/config.json` flag the default config is taken from `\u003cbench-dir\u003e/workloads/default.barista.json`.\n\n**NOTE**: A special `BENCHMARK_HOME` environment variable points to the directory of each benchmark project source and can be used by all configuration options responsible for forming the command line starting the microservice like `app_args` and `vm_options`.\n\nFor more detailed configuration options, you can run:\n```console\n./barista --help\n```\n to print out all of the available options.\n\n```console\nusage: barista [-h] [-j JAVA_HOME] [-m {jvm,native}] [-c CONFIG] [-x APP_EXECUTABLE] [-e ENDPOINT] [-o OUTPUT] [-t THREADS] [-k CONNECTIONS] [-s LUA_SCRIPT] [--resource-usage-polling-interval RESOURCE_USAGE_POLLING_INTERVAL] [--skip-prepare] [--skip-cleanup] [-d] [-p CMD_APP_PREFIX] [-v VM_OPTIONS] [-a APP_ARGS] [-b NATIVE_IMAGE_BUILD_OPTIONS] [--startup-iteration-count STARTUP_ITERATION_COUNT]\n               [--startup-timeout STARTUP_TIMEOUT] [--warmup-iteration-count WARMUP_ITERATION_COUNT] [--warmup-duration WARMUP_DURATION] [--warmup-threads WARMUP_THREADS] [--warmup-connections WARMUP_CONNECTIONS] [--warmup-lua-script WARMUP_LUA_SCRIPT] [--throughput-iteration-count THROUGHPUT_ITERATION_COUNT] [--throughput-duration THROUGHPUT_DURATION] [--throughput-threads THROUGHPUT_THREADS]\n               [--throughput-connections THROUGHPUT_CONNECTIONS] [--throughput-lua-script THROUGHPUT_LUA_SCRIPT] [--latency-iteration-count LATENCY_ITERATION_COUNT] [--latency-duration LATENCY_DURATION] [--latency-threads LATENCY_THREADS] [--latency-connections LATENCY_CONNECTIONS] [--latency-search-strategy LATENCY_SEARCH_STRATEGY] [--latency-rate LATENCY_RATE] [--latency-percentages LATENCY_PERCENTAGES]\n               [--latency-min-step-percent LATENCY_MIN_STEP_PERCENT] [--latency-sla LATENCY_SLA] [--latency-lua-script LATENCY_LUA_SCRIPT]\n               {helidon-hello-world,ktor-hello-world,micronaut-hello-world,micronaut-shopcart,micronaut-similarity,play-scala-hello-world,quarkus-hello-world,quarkus-tika,spring-hello-world,spring-petclinic,vanilla-hello-world,vertx-hello-world}\n\npositional arguments:\n  {helidon-hello-world,ktor-hello-world,micronaut-hello-world,micronaut-shopcart,micronaut-similarity,play-scala-hello-world,quarkus-hello-world,quarkus-tika,spring-hello-world,spring-petclinic,vanilla-hello-world,vertx-hello-world}\n                        Name of the benchmark\n\noptions:\n  -h, --help            show this help message and exit\n  -j JAVA_HOME, --java-home JAVA_HOME\n                        Path to the JVM distribution to be used. If not provided, the JAVA_HOME environment variable is used\n  -m {jvm,native}, --mode {jvm,native}\n                        Execution mode of the app\n  -c CONFIG, --config CONFIG\n                        Path to the configuration JSON file to be used for load testing, can be either absolute or relative to the \u003cbench-dir\u003e/workloads directory. Defaults to 'default.barista.json'\n  -x APP_EXECUTABLE, --app-executable APP_EXECUTABLE\n                        Path to the application executable. If this is not set, the application executable is retrieved (built, in the case of native execution) from the benchmark directory\n  -e ENDPOINT, --endpoint ENDPOINT\n                        Endpoint of the application which will be loaded\n  -o OUTPUT, --output OUTPUT\n                        Path to the directory in which a timestamped directory will be created. Barista stores all its output in this timestamped directory. Defaults to the current working directory.\n  -t THREADS, --threads THREADS\n                        Number of threads to use during the warmup, throughput and latency load-testing phases. This option can be overwritten for each of the mentioned phases. During each phase, the number of threads is propagated to wrk/wrk2. The number of threads is also propagated to the Lua script that the wrk/wrk2 tool executes, enabling the writing of scripts that are aware of the number of threads executing them.\n  -k CONNECTIONS, --connections CONNECTIONS\n                        Connections to keep open during the warmup, throughput and latency load-testing phases. This option can be overwritten for each of the mentioned phases. During each phase, the number of connections is propagated to wrk/wrk2.\n  -s LUA_SCRIPT, --lua-script LUA_SCRIPT\n                        Lua script to be executed by wrk/wrk2 for general benchmarking purposes\n  --resource-usage-polling-interval RESOURCE_USAGE_POLLING_INTERVAL\n                        Time interval in seconds between two subsequent resource usage polls. Determines how often resource usage metrics, such as rss (Resident Set Size), vms (Virtual Memory Size), and CPU utilization, are collected. If set to 0 resource usage polling is disabled. Defaults to 0.02s (20ms)\n  --skip-prepare        Explicitly skip the prepare step of the benchmark, even if a prepare script is present in the benchmark directory\n  --skip-cleanup        Explicitly skip the cleanup step of the benchmark, even if a cleanup script is present in the benchmark directory\n  -d, --debug           Show debug logs\n  -p CMD_APP_PREFIX, --cmd-app-prefix CMD_APP_PREFIX\n                        Command to be prefixed to the application startup command\n  -v VM_OPTIONS, --vm-options VM_OPTIONS\n                        Options to be propagated to the virtual machine (JVM in jvm execution mode, native-image in native execution mode)\n  -a APP_ARGS, --app-args APP_ARGS\n                        Arguments to be propagated to the application\n  -b NATIVE_IMAGE_BUILD_OPTIONS, --native-image-build-options NATIVE_IMAGE_BUILD_OPTIONS\n                        Options to be propagated to the native-image build command (used only in native execution mode when no '--app-executable' option is provided)\n  --startup-iteration-count STARTUP_ITERATION_COUNT\n                        Number of requests to make and record the response time of, immediately after starting the application\n  --startup-timeout STARTUP_TIMEOUT\n                        Period of time without receiving a response from the app after which it is deemed unresponsive and the benchmark is stopped. If set to 0 the app will never be deemed unresponsive. Defaults to 60\n  --warmup-iteration-count WARMUP_ITERATION_COUNT\n                        Number of iterations that should be performed before testing the application\n  --warmup-duration WARMUP_DURATION\n                        Single iteration warmup time duration in seconds. How long should the application be stressed before testing\n  --warmup-threads WARMUP_THREADS\n                        Number of threads to use for warmup, overrides the '--threads' option specifically for warmup iterations\n  --warmup-connections WARMUP_CONNECTIONS\n                        Connections to keep open during warmup, overrides the '--connections' option specifically for warmup iterations\n  --warmup-lua-script WARMUP_LUA_SCRIPT\n                        Lua script to be executed by wrk during warmup, overrides the '--lua-script' option specifically for warmup iterations\n  --throughput-iteration-count THROUGHPUT_ITERATION_COUNT\n                        Number of iterations that will be performed to measure throughput\n  --throughput-duration THROUGHPUT_DURATION\n                        Duration in seconds of a single iteration of throughput measurement\n  --throughput-threads THROUGHPUT_THREADS\n                        Number of threads to use for throughput measurements, overrides the '--threads' option specifically for throughput iterations\n  --throughput-connections THROUGHPUT_CONNECTIONS\n                        Connections to keep open during throughput measurements, overrides the '--connections' option specifically for throughput iterations\n  --throughput-lua-script THROUGHPUT_LUA_SCRIPT\n                        Lua script to be executed by wrk during throughput measurements, overrides the '--lua-script' option specifically for throughput iterations\n  --latency-iteration-count LATENCY_ITERATION_COUNT\n                        Number of iterations that will be performed to measure latency\n  --latency-duration LATENCY_DURATION\n                        Time in seconds of how long should single iteration of latency measurment take\n  --latency-threads LATENCY_THREADS\n                        Number of threads to use for latency measurements, overrides the '--threads' option specifically for latency iterations\n  --latency-connections LATENCY_CONNECTIONS\n                        Connections to keep open during latency measurements, overrides the '--connections' option specifically for latency iterations\n  --latency-search-strategy LATENCY_SEARCH_STRATEGY\n                        Strategy to be used when searching for the optimal throughput\n  --latency-rate LATENCY_RATE\n                        Constant throughput (in ops/sec) applied to measure latency\n  --latency-percentages LATENCY_PERCENTAGES\n                        Fraction of throughput recorded in throughput measurements to be used in latency measurements\n  --latency-min-step-percent LATENCY_MIN_STEP_PERCENT\n                        Accuracy with which to perform the latency search\n  --latency-sla LATENCY_SLA\n                        Latency Service Level Agreement entry\n  --latency-lua-script LATENCY_LUA_SCRIPT\n                        Lua script to be executed by wrk2 during latency measurements, overrides the '--lua-script' option specifically for latency iterations\n```\n### Lua scripts\n\nThe wrk and wrk2 load testers support scripts for generating custom loads. Lua is supported the programming language for scripting functionality in wrk and wrk2.\nLua scripts are split across multiple layers. The top level Lua script resides under the `\"load_testing\"` key in the json file named `\"lua_script\"` or `--lua-script` \nflag. \nThis option propagates the given lua script to the all the measurement steps that are:\n\n- Warmup\n- Throughput\n- Latency\n\nHowever, to overwrite the script in one of these fields you can add `\"lua_script\":\"path/to/your/lua/script.lua\"` key and value pair in the JSON under the specific \nload testing step. \nOr you can add a command line flag:\n\n- Warmup: `--warmup-lua-script`\n- Throughput: `--throughput-lua-script`\n- Latency: `--latency-lua-script`\n\n\n## Pipeline\n\nA quick overview of the whole flow can be seen here:\n\n\u003cimg src=\"images/Barista-Pipeline.png\" title=\"Barista Pipeline\" alt=\"Barista Pipeline\" style=\"max-height: 1200px\" /\u003e\n\nThe main phases are as follows. *NOTE*: they can all be skipped by providing `0` as the number of iterations for the configuration.\n\n### Startup Phase\n\nThe startup phase occurs immediately after the application's cold start. During this phase the harness records the response time to `--startup-iteration-count` requests sent to the application.\n\n### Warmup Phase\n\nThe warmup phase is dedicated for JVM applications with JIT compiler to ensure that they reach the peak throughput before actual measurements begin.\nCurrently there is no automation when does the warmup period \"end\"(Warmup is tricky and may never end), so it is up to a user to define a good number of iterations \nand time duration for their application so they would ensure it has warmed up.\n\n### Throughput Phase\n\nFirst time measurement phase after warmup. Records requests per second with `wrk` to return the average seen throughput for each of the iterations defined.\n\n### Latency Phase\n\nThe suite uses `wrk2` to measure latency under constant load in a distinct step than the throughput measurement. The reason is that it is not desirable to report latency while measuring throughput because latency won't be comparable across experiments since a different throughput will directly impact latency (more throughput means more memory pressure, hence higher tail latency).\n\n`wrk2` reports the full latency distribution and this is captured in separate file for further analysis if need be. However, the harness extracts the major percentiles (p50, p75, p99, p99.99, p99.999, p100) for convenience and stores them in a CSV. \n\n#### Fixed rates and percentages\n\nThe latencies can be measured in fixed rates and percentages. \nFixed rate is required to be provided in order for `wrk2` tool for preventing [Coordinated Omission](https://www.artillery.io/blog/load-testing-workload-models#uh-oh-coordinated-omission) problem. \nFurthermore, some microservices have already expected load and user count they want to test.\nThe fixed `\"rates\"` allow benchmark users to define an expected traffic and then observe the latencies microservice will experience.\nOn the other hand, if a throughput of a microservice is unknown `\"percentages\"` can be used to determine throughput at a given overall load of an application.\nFor example one could test how well does their application respond to requests under a 50%(`0.5`) or 100%(`1.0`) load.\n\n\n\nThe search strategy that should be defined under `latency_measurement` is `\"FIXED\"`.\nThe fixed `\"rates\"` must be given in an array `[1000,2000,3000]`.\nThe fixed `\"percentages\"` must be given in an array `[0.1, 0.05]`.\nThe latency measurements will perform all the measurements for the given number of `\"iterations\"` for each of the `\"rates\"` and `\"percentages\"`\n\n### Concurrent Reader\n\nThroughout the duration of all the load-testing phases, a separate thread called `ConcurrentReader` collects the application resource usage metrics as well as the application's output.\n\n#### Resource usage\n\nThe resource usage metrics are collected by periodically, with a period specified with the `--resource-usage-polling-interval` option, polling the OS for metrics such as:\n    - rss (Resident Set Size)\n    - vms (Virtual Memory Size)\n    - CPU usage\n\nAfter the load-testing is completed, the gathered samples are used to compile P-values for a range of percentiles.\n\n#### Application standard output\n\nThe standard output is collected and the self-reported startup time is extracted. However, this metric is far less reliable than the one recorded during startup, as it is up to the application/framework to decide what is reported.\nThe output of the application is finally dumped into a file called `app-dump.txt` in the output directory.\n\n## Maximizing throughput with given SLAs\n\nSometimes services require to meet specific Service Level Agreements or SLAs. \nThey can be throughput, latency, memory usage and many more. \nBarista Suite has an option where it maximises the throughput given a latency SLA.\n\nThe SLA can be configured by providing the major percentile restrictions under `\"latency_measurement\"`:\n```json\n\"SLA\":[\n    [\"p99\", 5]\n    [\"p75\", 4]\n    [\"p50\", 2]\n]\n```\nIt is an array of pair arrays for each of the percentile the search should target. \nEach pair is a  `[percentile, latency_in_milliseconds]`\n\n\nThe algorithm will then perform search with `\"min_step_percent\"` accuracy, that dictates how close, relevant to the average throughput seen, should the algorithm get for the final value.\n\nBoth [Binary Search](#binary-search) and [AIMD](#additive-increase-multiplicative-decrease-aimd) algorithms require that the `\"throughput\"` load test must have at least 1 iteration for algorithms to determine a starting point. \nThe throughput(s) seen in the throughput phase will be averaged to perform search.\n\nOnce the value is found by one of the algorithms, the latency will be measured at a given rate for the given number of `\"iterations\"` each lasting for `\"iteration_time_seconds\"`.\n\n### Binary Search\nThe binary search splits the range between 0 and maximum average throughput seen in `\"throughput\"` measurements into 1/`\"min_step_percent\"` pieces. Then performs measurements to determine the optimal rate.\n\nSearch strategy should be defined under `\"latency_measurement\"` as `\"BINARY_SEARCH\"`.\n\n\n\n### Additive Increase/ Multiplicative Decrease (AIMD)\n\nThe [AIMD](https://en.wikipedia.org/wiki/Additive_increase/multiplicative_decrease) search strategy will try to perform search over range throughput values.\n\nThe algorithm continuosly doubles the rate from the user given percentage rate `\"min_step_percent\"` until the rate exceeds the maximum average seen throughput. Then it splits the range again from last successful request and continues to do so for fixed number of times.\n\nSearch strategy should be defined under `\"latency_measurement\"` as `\"AIMD\"`.\n\n\n## Example Configurations\n\n### Native image\n\n```json\n{\n    \"mode\": \"NATIVE\",\n    \"app_executable\": \"benchmarks/spring-hello-world/target/spring-hello-world\",\n    \"cmd_app_prefix\" : [\"taskset\", \"-c\", \"0-1\"],\n    \"vm_options\": [\"-Xmx512M\", \"-Dserver.port=8090\"],\n    \"endpoint\": \"http://127.0.0.1:8090/hello\",\n    \"output_dir\": \"my-spring-logs/\",\n    \"load_testing\":{\n        \"warmup\":{\n            \"iterations\": 0,\n            \"iteration_time_seconds\": 30\n        },\n        \"throughput\":{\n            \"iterations\": 6,\n            \"iteration_time_seconds\": 30\n        },\n        \"latency_measurement\":{\n            \"iterations\": 1,\n            \"iteration_time_seconds\": 120,\n            \"search_strategy\": \"FIXED\",\n            \"rates\" : [500],\n            \"percentages\": [0.1],\n        }\n    }\n}\n```\n- This json configuration will run the native image given in the `\"app_executable\"` value.\n- The `\"cmd_app_prefix\"` in this example allows us to limit the cpu cores used with `taskset` unix command.\n- `\"vm_options\"` will be appended to the end of the command to limit the memory and change server port\n- `\"endpoint\"` flag will be the endpoint wrk, wrk2 and python will ping for determining metrics.\n- `\"output\"` dir will be the folder where all the result files will be created. NOTE: the files will be created in a folder with a timestamp + hash so they will not write over each other.\n- `\"load_testing\"` defines all the load tests that will be performed across the benchmark\n- `\"warmup\"` in this case warmup has 0 iterations and will be skipped as it is not required for Native Image (AOT) applications\n- `\"throughput\"` will perfor 6 measurements of 30 seconds each with `wrk` to determine average rates\n- `\"latency_measurement\"` will then finally perform 1 iteration of each of the given `\"FIXED\"` strategy in `\"search_strategy\"` each lasting 120 seconds. In this case there will be:\n    - 1 iteration of 120 seconds at a rate of 500 ops/s(requests/second)\n    - 1 iteration of 120 seoncds at a 10% rate of average throughput seen in the `\"throughput\"` tests.\n\n### Binary Search\n\n```json\n{\n    \"mode\": \"JVM\",\n    \"app_executable\": \"benchmarks/spring-hello-world/target/spring-hello-world-3.0.5.jar\",\n    \"cmd_app_prefix\" : [\"taskset\", \"-c\", \"0-7\"],\n    \"vm_options\": [\"-Xmx512M\", \"-Dserver.port=8090\"],\n    \"endpoint\": \"http://127.0.0.1:8090/hello\",\n    \"output_dir\": \"logs-lat/\",\n    \"load_testing\":{\n        \"startup\":{\n            \"iterations\": 5\n        },\n        \"warmup\":{\n            \"iterations\": 2,\n            \"iteration_time_seconds\": 30\n        },\n        \"throughput\":{\n            \"iterations\": 6,\n            \"iteration_time_seconds\": 30\n        },\n        \"latency_measurement\":{\n            \"iterations\": 1,\n            \"iteration_time_seconds\": 150,\n            \"min_step_percent\": 0.01,\n            \"search_strategy\": \"BINARY_SEARCH\",\n            \"SLA\":[\n                [\"p99\",5]\n                [\"p50\",2]\n            ]\n        }\n    }\n}\n```\n\n- This json configuration will run the jar file given in the `\"app_executable\"` value.\n- The `\"cmd_app_prefix\"` in this example allows us to limit the cpu cores used with `taskset` unix command.\n- `\"vm_options\"` will be appended to the end of the command to limit the memory and change server port\n- `\"endpoint\"` flag will be the endpoint wrk, wrk2 and python will ping for determining metrics.\n- `\"output\"` dir will be the folder where all the result files will be created. NOTE: the files will be created in a folder with a timestamp + hash so they will not write over each other.\n- `\"load_testing\"` defines all the load tests that will be performed across the benchmark\n- `\"startup\"` will measure the response time to 5 requests made immediately after the app startup\n- `\"warmup\"` there will be 2 meaeasurements of 30 seconds each with `wrk` to warmup up the JIT application\n- `\"throughput\"` will perfor 6 measurements of 30 seconds each with `wrk` to determine average rates\n- `\"latency_measurement\"` will perform a `\"search_strategy\"` of `\"BINARY_SEARCH\"` to match the `\"SLA\"` pairs given. In this example the rate range would be split into 1 / `\"min_step_percent\"` = 100 rates over which the binary search would be done. Afterwards when the rate that meets the `\"SLA\"` requirement is found there will be performed:\n    - 1 iteration of 120 seconds at a rate of 150 seonds on the final rate found with `\"BINARY_SEARCH\"`\n\n## Contributing\n\nPlease see the [contribution guide](CONTRIBUTING.md) for a description of the contribution process.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbarista-benchmarks%2Fbarista","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbarista-benchmarks%2Fbarista","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbarista-benchmarks%2Fbarista/lists"}