{"id":50653910,"url":"https://github.com/cloudamqp/lavinmq-benchmark","last_synced_at":"2026-06-07T22:36:03.981Z","repository":{"id":326056778,"uuid":"935313086","full_name":"cloudamqp/lavinmq-benchmark","owner":"cloudamqp","description":"Run benchmark on LavinMQ with Terraform infrastructure setup","archived":false,"fork":false,"pushed_at":"2026-05-25T12:03:53.000Z","size":727,"stargazers_count":3,"open_issues_count":11,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2026-05-25T14:08:19.086Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cloudamqp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-02-19T08:43:36.000Z","updated_at":"2026-05-25T12:03:56.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/cloudamqp/lavinmq-benchmark","commit_stats":null,"previous_names":["cloudamqp/lavinmq-benchmark"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/cloudamqp/lavinmq-benchmark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudamqp%2Flavinmq-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudamqp%2Flavinmq-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudamqp%2Flavinmq-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudamqp%2Flavinmq-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cloudamqp","download_url":"https://codeload.github.com/cloudamqp/lavinmq-benchmark/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudamqp%2Flavinmq-benchmark/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34041088,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-07T02:00:07.652Z","response_time":124,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-07T22:36:02.526Z","updated_at":"2026-06-07T22:36:03.972Z","avatar_url":"https://github.com/cloudamqp.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LavinMQ Benchmark Tool\n\nInfrastructure-as-code benchmarking tool for [LavinMQ](https://lavinmq.com), using Terraform to\nprovision AWS resources and [`lavinmqperf`](https://lavinmq.com/documentation/lavinmqperf) to run\nperformance tests.\n\n## Prerequisites\n\n- Terraform \u003e= 1.3.0 — [installation guide](https://developer.hashicorp.com/terraform/install)\n- AWS credentials with permission to create EC2 and VPC resources\n- An SSH key pair for instance access\n\n## Scenarios\n\n### [`lavinmq_throughput`](scenarios/aws/lavinmq_throughput/)\n\nProvisions a broker and one or more load generator instances, then runs a single user-supplied\n`lavinmqperf` command. Useful for ad-hoc throughput or latency measurements where you control the\nexact test parameters.\n\n### [`multiple_run_throughput`](scenarios/aws/multiple_run_throughput/)\n\nRuns sequential AMQP throughput tests across a configurable set of message sizes, repeated N times.\nProduces a CSV file of raw per-run results and a JSON config file on the load generator. Aggregated\n(median) summaries are written to `results/v{version}/throughput.md` by `scripts/aggregate_results.py`.\n\n### [`multiple_run_latency`](scenarios/aws/multiple_run_latency/)\n\nRuns sequential AMQP latency tests across a configurable set of message sizes and rate limits,\nrepeated N times. Produces a CSV file of raw per-run results and a JSON config file on the load\ngenerator. Aggregated (median) summaries are written to `results/v{version}/latency_P95.md` and\n`latency_P99.md` by `scripts/aggregate_results.py`.\n\n### [`mqtt_throughput`](scenarios/aws/mqtt_throughput/)\n\nRuns sequential MQTT throughput tests across a configurable set of message sizes, repeated N times.\nInstalls `emqtt-bench` and `mqttloader` on the load generator. Produces a CSV file of raw per-run\nresults and a JSON config file on the load generator. Aggregated (median) summaries are written to\n`results/v{version}/mqtt_throughput.md` by `scripts/aggregate_results.py`.\n\n## Configuration\n\nVariables can be supplied in three ways:\n\n**`terraform.auto.tfvars`** — loaded automatically by Terraform:\n\n```shell\nterraform apply\n```\n\n**`terraform.tfvars`** — loaded explicitly:\n\n```shell\nterraform apply -var-file=\"terraform.tfvars\"\n```\n\n**`.env` file via [dotenv](https://github.com/bkeepers/dotenv)** — environment variables prefixed\nwith `TF_VAR_`:\n\n```shell\ndotenv terraform apply\n```\n\nUse the templates in `modules/providers/aws/variables_template/` as a starting point:\n\n- `terraform_tfvars.txt` — for `.tfvars` files\n- `env.txt` — for `.env` files\n\nAWS credentials must be set as environment variables regardless of the method used:\n\n```shell\nexport AWS_ACCESS_KEY=***\nexport AWS_SECRET_KEY=***\n```\n\n## CI / Parallel Benchmarks\n\nThe [Benchmark](../../actions/workflows/benchmark.yml) GitHub Actions workflow provisions\ninfrastructure, runs benchmarks across all supported instance types in parallel, aggregates the\nresults into markdown summary files, and opens a pull request against `main` with the results\ncommitted to a `results/v{version}` branch.\n\n### Triggering a run\n\n**Via the GitHub UI:** go to **Actions → Benchmark → Run workflow**, fill in the version and pick a scenario.\n\n**Via the GitHub CLI:**\n\n```shell\n# Run all benchmarks (AMQP throughput, AMQP latency, MQTT throughput) for all instance types\ngh workflow run benchmark.yml \\\n  -f lavinmq_version=2.7.0 \\\n  -f scenarios=all\n\n# Latency only\ngh workflow run benchmark.yml \\\n  -f lavinmq_version=2.7.0 \\\n  -f scenarios=latency\n\n# AMQP throughput only\ngh workflow run benchmark.yml \\\n  -f lavinmq_version=2.7.0 \\\n  -f scenarios=throughput\n\n# MQTT throughput only\ngh workflow run benchmark.yml \\\n  -f lavinmq_version=2.7.0 \\\n  -f scenarios=mqtt-throughput\n\n# Run from a specific branch (e.g. when testing workflow changes)\ngh workflow run benchmark.yml \\\n  -r my-branch \\\n  -f lavinmq_version=2.7.0 \\\n  -f scenarios=all\n```\n\nResults are committed to `results/v{version}/` and a pull request is created (or updated if one\nalready exists for that version).\n\n### Re-running specific broker instance types\n\nAll three workflows accept an optional `brokers` input — a comma-separated list of broker instance\ntypes to run. When omitted or empty, all instance types are benchmarked. This is useful for\nre-running a single instance type that failed without triggering the full suite.\n\n```shell\n# Re-run latency benchmark for a single instance type\ngh workflow run benchmark.yml \\\n  -f lavinmq_version=2.7.0 \\\n  -f scenarios=latency \\\n  -f brokers=\"c8g.large\"\n\n# Re-run latency benchmark for multiple specific instance types\ngh workflow run benchmark.yml \\\n  -f lavinmq_version=2.7.0 \\\n  -f scenarios=latency \\\n  -f brokers=\"c8g.large,t4g.medium\"\n```\n\nThe individual `benchmark-latency.yml`, `benchmark-throughput.yml`, and `benchmark-mqtt-throughput.yml`\nworkflows can also be triggered directly in the same way if you want to skip the aggregate/PR step.\n\n## Publishing results\n\nRaw results are stored and available in this repository. These are also available at https://lavinmq.com/benchmark\nwith graphs and a UI to easier compare the numbers.\n\nAggregated JSONs under `results/` are published to `benchmark.lavinmq.com` and consumed by the\ncharts page in lavinmq-website.\n\nTo preview a local copy, start jekyll in lavinmq-website and run the following here:\n\n```shell\npython3 scripts/build_data.py        # regenerate throughput/latency/mqtt_throughput JSONs\npython3 scripts/serve.py             # serve results/ on http://127.0.0.1:8081 with CORS\n```\n\n## Logging\n\nEnable detailed Terraform provider logs by setting:\n\n```shell\nexport TF_LOG_PROVIDER=DEBUG\n```\n\nAccepted levels: `INFO`, `DEBUG`, `WARN`, `ERROR`, `TRACE`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloudamqp%2Flavinmq-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcloudamqp%2Flavinmq-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloudamqp%2Flavinmq-benchmark/lists"}