{"id":29482616,"url":"https://github.com/ConduitIO/benchi","last_synced_at":"2025-07-15T02:02:07.480Z","repository":{"id":277555780,"uuid":"932300713","full_name":"ConduitIO/benchi","owner":"ConduitIO","description":"Benchmark any tool from the CLI","archived":false,"fork":false,"pushed_at":"2025-06-17T05:56:04.000Z","size":15214,"stargazers_count":156,"open_issues_count":3,"forks_count":7,"subscribers_count":10,"default_branch":"main","last_synced_at":"2025-06-22T20:46:19.227Z","etag":null,"topics":["benchmark","benchmark-framework"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ConduitIO.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-02-13T17:34:55.000Z","updated_at":"2025-06-18T05:50:32.000Z","dependencies_parsed_at":"2025-03-10T11:21:12.983Z","dependency_job_id":"9a50c82c-af92-46a9-bd23-f48a9d3bc9d7","html_url":"https://github.com/ConduitIO/benchi","commit_stats":null,"previous_names":["conduitio/benchi"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/ConduitIO/benchi","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ConduitIO%2Fbenchi","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ConduitIO%2Fbenchi/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ConduitIO%2Fbenchi/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ConduitIO%2Fbenchi/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ConduitIO","download_url":"https://codeload.github.com/ConduitIO/benchi/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ConduitIO%2Fbenchi/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265386079,"owners_count":23756747,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","benchmark-framework"],"created_at":"2025-07-15T02:01:17.577Z","updated_at":"2025-07-15T02:02:07.471Z","avatar_url":"https://github.com/ConduitIO.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# Benchi\n\nBenchi is a minimal benchmarking framework designed to help you measure the\nperformance of your applications and infrastructure. It leverages Docker to\ncreate isolated environments for running benchmarks and collecting metrics.\n\nIt was developed to simplify the process of setting up and running benchmarks\nfor [Conduit](https://github.com/ConduitIO/conduit).\n\n![demo](https://github.com/user-attachments/assets/334d854d-0466-489c-bff6-95f4471b457f)\n\n## Features\n\n- **Docker Integration**: Define and manage your benchmarking environments using\n  Docker Compose.\n- **Metrics Collection**: Collect and export metrics in CSV format for further\n  analysis.\n- **Custom Hooks**: Define custom hooks to run commands at various stages of the\n  benchmark.\n- **Progress Monitoring**: Real-time monitoring of container statuses and\n  metrics during the benchmark run.\n\n## Installation\n\nTo install Benchi, download\nthe [latest release](https://github.com/ConduitIO/benchi/releases/latest) or\ninstall it using Go:\n\n```sh\ngo install github.com/conduitio/benchi/cmd/benchi@latest\n```\n\nAlternatively, you can install just the Benchi binary on a Linux OS with:\n\n```shell\ncurl https://raw.githubusercontent.com/ConduitIO/benchi/main/install.sh | sh\n```\n\n## Usage\n\n### Running Benchmarks\n\nRun benchi and point `-config` to a benchmark [configuration file](#configuration).\nThe repository includes an [example benchmark](./example), which can be run\nusing the following command:\n\n```sh\nbenchi -config ./example/bench-kafka-kafka/bench.yml\n```\n\n### Results\n\nRunning the benchmark will store the results in a folder named after the current\ndate and time inside of `results` (e.g. `results/20060102_150405`). You can\nadjust the output folder using the `-out` flag.\n\nThe output folder will contain two files:\n\n- `benchi.log`: Log file containing the full output of benchi.\n- `aggregated-results.csv`: Aggregated metric results from all collectors and\n  all tests. The results are aggregated using a\n  [trimmed mean](https://en.wikipedia.org/wiki/Truncated_mean), where the top\n  and bottom 5% of the results are removed. Benchi also disregards any 0 values\n  from the start and end of the test, to accomodate for warm-up and cool-down\n  periods.\n\nThe output folder will also contain one folder per benchmark run (i.e. per test\nand tool combination). Each benchmark run folder will contain:\n\n- `infra_NAME.log`: Log file containing the output of the infrastructure docker\n  containers, split per infra service.\n- `tool_NAME.log`: Log file containing the output of the tool docker containers.\n- `COLLECTOR.csv`: Raw metrics collected using the corresponding\n  [metrics collector](#collectors).\n\n### Command-Line Flags\n\n- `-config`: Path to the benchmark config file (required).\n- `-out`: Path to the output folder (default: `./results/${now}`*).\n- `-tool`: Filter tool to be tested (can be provided multiple times).\n- `-tests`: Filter test to run (can be provided multiple times).\n\n\\* `${now}` is replaced with the current time formatted as `YYYYMMDD_HHMMSS`.\n\n### Docker network\n\nBenchi creates a Docker network named `benchi` to connect the infrastructure\nservices and tools. This network is created automatically and removed after the\nbenchmark run. Please make sure to connect your services to this network to\nensure they can communicate with each other.\n\nExample Docker Compose configuration:\n\n```yaml\nservices:\n  my-service:\n    networks:\n      - benchi\n\nnetworks:\n  benchi:\n    external: true\n```\n\n### Environment variables\n\nBenchi runs all Docker Compose commands using the same environment variables as\nthe current shell. This means that you can use environment variables to pass\nvalues to your services.\n\nFor instance, having the following Docker Compose configuration:\n\n```yaml\nservices:\n  my-service:\n    environment:\n      - MY_ENV_VAR=${MY_ENV_VAR}\n```\n\nYou can inject the environment variable by running Benchi as follows:\n\n```sh\nMY_ENV_VAR=my-value benchi -config ./my-benchmark.yml\n```\n\n### Configuration\n\nBenchi uses a YAML configuration file to define the benchmark in combination\nwith Docker Compose configurations.\n\nBelow is an example configuration:\n\n```yaml\ninfrastructure:\n  database:\n    compose: \"./compose-database.yml\"\n  cache:\n    compose: \"./compose-cache.yml\"\n\ntools:\n  my-app:\n    compose: \"./compose-my-app.yml\"\n\nmetrics:\n  prometheus:\n    collector: \"prometheus\"\n    settings:\n      url: \"http://localhost:9090/metrics\"\n      queries:\n        - name: \"http_requests_rate\"\n          query: \"rate(request_count{endpoint=hello}[2s])\"\n          unit: \"req/s\"\n          interval: \"1s\"\n\ntests:\n  - name: Endpoint Load\n    duration: 2m\n    steps:\n      pre-infrastructure:\n      post-infrastructure:\n        - name: Setup Database\n          container: database\n          run: /scripts/setup-database.sh\n      pre-tools:\n      post-tools:\n      pre-test:\n      during:\n        - name: Run Load Test\n          container: my-app\n          run: /scripts/run-load-test.sh\n      post-test:\n      pre-cleanup:\n        - name: Cleanup\n          container: my-app\n          run: /scripts/cleanup.sh\n      post-cleanup:\n```\n\n#### `infrastructure`\n\nThe `infrastructure` section defines the Docker Compose configurations for the\ninfrastructure services required for the benchmark. Each service is identified\nby a custom name, used in logging and to correlate overridden configurations\nspecified in a test (see [tests](#tests)). The path to the docker compose file\nis relative to the location of the benchmark configuration file.\n\nExample:\n\n```yaml\ninfrastructure:\n  name-of-infrastructure-service:\n    compose: \"./path/to/compose-file.yml\"\n```\n\n#### `tools`\n\nThe `tools` section defines the Docker Compose configurations for the tools\nbeing benchmarked. Each tool is identified by a custom name, used in logging and\nto correlate overridden configurations specified in a test (see [tests](#tests).\nThe path to the docker compose file is relative to the location of the benchmark\nconfiguration file.\n\nExample:\n\n```yaml\ntools:\n  name-of-tool:\n    compose: \"./path/to/compose-file.yml\"\n```\n\n#### `metrics`\n\nThe `metrics` section defines the metric collectors running during the\nbenchmark. Each metric collector has a custom name used for logging. The\n`collector` field specifies the type of metric collector to use. The `settings`\nfield contains the configuration for the chosen collector.\n\nExample:\n\n```yaml\nmetrics:\n  name-of-metric-collector:\n    collector: \"conduit\"\n    settings:\n      url: \"http://localhost:8080/metrics\"\n```\n\n\u003e [!NOTE]  \n\u003e Metrics collectors run in the benchi process, which runs outside of docker\n\u003e on the host machine. Ensure that the metric collector can access the\n\u003e endpoints of the services being benchmarked by exposing the necessary ports\n\u003e in the Docker Compose configurations.\n\nSee [collectors](#collectors) for available collectors and their configurations.\n\n#### `tests`\n\nThe `tests` section defines the benchmarks to run. Each test has a custom name\nused for logging. The `duration` field specifies the duration of the test. The\n`steps` field contains the commands to run at various stages of the benchmark.\n\nThe `steps` field contains the following stages:\n\n- `pre-infrastructure`: Commands to run before starting the infrastructure\n  services.\n- `post-infrastructure`: Commands to run after starting the infrastructure\n  services.\n- `pre-tools`: Commands to run before starting the tools.\n- `post-tools`: Commands to run after starting the tools.\n- `pre-test`: Commands to run before starting the test.\n- `during`: Commands to run during the test.\n- `post-test`: Commands to run after the test.\n- `pre-cleanup`: Commands to run before cleaning up the test.\n- `post-cleanup`: Commands to run after cleaning up the test.\n\n\u003e [!NOTE]\n\u003e Steps are generally executed sequentially and in the order specified in the\n\u003e configuration. However, the `during` step is an exception, as all commands\n\u003e under this step are executed concurrently and will run for the duration of the\n\u003e test.\n\nEach hook can run its commands either in an existing container or in a temporary\ncontainer created from a specified image. The `container` field specifies the\nname of the container to run the commands in. The `image` field specifies the\nimage to use for the temporary container. If neither `container` nor `image` is\nspecified, the commands will run in a temporary container using the\n`alpine:latest` image. Note that running a custom script in a container requires\nthe container to contain the necessary script (hint: mount the script as a\nvolume).\n\nYou can optionally configure the `tools` field to run a hook only for certain\ntools. If the field is not present or empty, the hook is applied for all tools.\n\nExample:\n\n```yaml\ntests:\n  - name: My Test\n    duration: 2m\n    steps:\n      pre-infrastructure:\n      post-infrastructure:\n        # This script will run after the infrastructure services have started.\n        # It will run in the `database` container (presumably started by the\n        # infrastructure service).\n        - name: Setup Database\n          container: database\n          run: /scripts/setup-database.sh\n      pre-tools:\n      post-tools:\n      pre-test:\n      during:\n        # This script will run for the duration of the test (2 minutes). It will\n        # run in the `my-app` container (presumably started by the tool). The\n        # script will receive a SIGINT signal when the test duration is over.\n        - name: Run Load Test\n          container: my-app\n          tools:\n            - my-app # Only run this hook when benchmarking my-app\n          run: /scripts/run-load-test.sh\n      post-test:\n      pre-cleanup:\n        # This script will run before infrastructure and tools containers are\n        # stopped and removed. It will run in a temporary container created\n        # from the `busybox:latest` image and connected to the `benchi` network.\n        - name: Cleanup\n          image: \"busybox:latest\"\n          run: |\n            echo \"Cleaning up...\"\n            sleep 5\n      post-cleanup:\n```\n\nYou can also include custom `infrastructure` and `tools` configurations to\noverride the default configurations specified in the `infrastructure` and\n`tools` sections. Note that the global configurations will still be applied, the\nadditional configurations are merged with the global configurations (see\n[merging compose files](https://docs.docker.com/compose/how-tos/multiple-compose-files/merge/)).\nThis can be useful to inject custom configurations for a specific test.\n\n\u003e [!IMPORTANT]\n\u003e Since the provided configurations are merged with the global configurations,\n\u003e any paths specified in the custom docker compose configurations should be\n\u003e relative to the location of the global docker compose configuration. See\n\u003e [merging rules](https://docs.docker.com/compose/how-tos/multiple-compose-files/merge/#merging-rules).\n\nExample:\n\n```yaml\ntests:\n  - name: My Test\n    duration: 2m\n    infrastructure:\n      name-of-infrastructure-service:\n        compose: \"./compose-file-infra.override.yml\"\n    tools:\n      name-of-tool:\n        compose: \"./compose-file-tool.override.yml\"\n```\n\n## Collectors\n\nCollectors are used to collect metrics from various sources during the benchmark\nrun. The collected metrics are exported in CSV format for further analysis.\n\nCollectors are configured in the [`metrics` section](#metrics) of the benchmark\nconfiguration file.\n\nSupported collectors:\n\n- [Conduit](#conduit)\n- [Docker](#docker)\n- [Kafka](#kafka)\n- [Prometheus](#prometheus)\n\n### Conduit\n\nThe Conduit metrics collector tracks the throughput for each configured\npipeline in [Conduit](https://github.com/conduitio/conduit).\n\nSettings:\n\n- `url`: URL of the Conduit metrics endpoint (needs to be reachable from the\n  benchi process).\n- `pipelines`: Array of pipelines to track.\n\n```yaml\nmetrics:\n  my-conduit-collector:\n    collector: \"conduit\"\n    settings:\n      url: \"http://localhost:8080/metrics\"\n      pipelines:\n        - \"pipeline1\"\n        - \"pipeline2\"\n```\n\nMetrics:\n\n- `msg-rate-per-second[PIPELINE]`: Messages per second per pipeline.\n- `msg-megabytes-in-per-second[PIPELINE]`: Incoming megabytes per second per\n  pipeline (measured as records read by the source connector).\n- `msg-megabytes-out-per-second[PIPELINE]`: Outgoing megabytes per second per\n  pipeline (measured as records written by the destination connector).\n\n### Docker\n\nThe Docker metrics collector tracks the container resource usage during the\nbenchmark run.\n\nSettings:\n\n- `containers`: Array of containers to track.\n\n```yaml\nmetrics:\n  my-docker-collector:\n    collector: \"docker\"\n    settings:\n      containers:\n        - \"my-app\"\n        - \"database\"\n        - \"cache\"\n```\n\nMetrics:\n\n- `cpu-percentage[CONTAINER]`: CPU usage in percent at a specific point in time\n  per container.\n- `memory-usage[CONTAINER]`: Memory usage in megabytes at a specific point in\n  time per container.\n\n### Kafka\n\nThe Kafka metrics collector tracks the throughput for each configured topic in\n[Apache Kafka](https://kafka.apache.org/).\n\nThe collector expects Kafka to expose a Prometheus metrics endpoint via the\n[Prometheus JMX exporter](https://prometheus.github.io/jmx_exporter/). To see\nhow to configure it, check out the\n[example configuration](./example/infra/compose-kafka.yml).\n\nSettings:\n\n- `url`: URL of the Kafka metrics endpoint (needs to be reachable from the\n  benchi process).\n- `topics`: Array of topics to track.\n\n```yaml\nmetrics:\n  my-kafka-collector:\n    collector: \"kafka\"\n    settings:\n      url: \"http://localhost:7071/metrics\"\n      topics:\n        - \"topic1\"\n        - \"topic2\"\n```\n\nMetrics:\n\n- `msg-rate-in-per-second[TOPIC]`: Incoming messages per second per topic.\n- `msg-megabytes-in-per-second[TOPIC]`: Incoming megabytes per second per topic.\n- `msg-megabytes-out-per-second[TOPIC]`: Outgoing megabytes per second per topic.\n\n### Prometheus\n\nThe Prometheus metrics collector continuously scrapes a metrics endpoint, stores\nthe metrics in memory and queries them using\n[PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/).\n\nIt is expected that the query returns a matrix with a single series. The query\nshould be a\n[ranged query](https://prometheus.io/docs/prometheus/latest/querying/basics/#range-vector-selectors),\nwhich will be evaluated between the start of the test and the end of the test.\n\nSettings:\n\n- `url`: URL of the Prometheus metrics endpoint.\n- `queries`: Array of queries to run.\n   - `name`: Name of the query.\n   - `query`: [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) query.\n   - `unit`: Unit of the query (optional, only for displaying in the CLI).\n   - `interval`: Resolution of the ranged query.\n\n```yaml\nmetrics:\n  my-prometheus-collector:\n    collector: \"prometheus\"\n    settings:\n      url: \"http://localhost:8080/metrics\"\n      queries:\n        - name: \"http_request_success_rate\"\n          query: \"rate(request_count{endpoint=hello,status=200}[2s])\"\n          unit: \"req/s\"\n          interval: \"1s\"\n        - name: \"http_request_fail_rate\"\n          query: \"rate(request_count{endpoint=hello,status!=200}[2s])\"\n          unit: \"req/s\"\n          interval: \"1s\"\n```\n\nMetrics are user defined using the `queries` field. The `name` field is used as\nthe metric name in the CSV output.\n\n## Troubleshooting\n\n- Benchi fails with the error `Cannot connect to the Docker daemon at\n  unix:///var/run/docker.sock. Is the docker daemon running?`.\n\nBenchi is communicating with Docker using the default Docker socket. If you are\nusing Docker Desktop, you can enable it under Settings -\u003e Advanced and check the\nbox for \"Allow the default Docker socket to be used\".\n\n## License\n\nBenchi is licensed under the Apache License, Version 2.0. See the\n[`LICENSE`](./LICENSE.md) file for more details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FConduitIO%2Fbenchi","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FConduitIO%2Fbenchi","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FConduitIO%2Fbenchi/lists"}