{"id":27203566,"url":"https://github.com/cometbft/cometbft-load-test","last_synced_at":"2025-04-09T22:40:15.494Z","repository":{"id":213317277,"uuid":"733465318","full_name":"cometbft/cometbft-load-test","owner":"cometbft","description":"Load testing framework for CometBFT-based networks","archived":false,"fork":false,"pushed_at":"2024-11-26T04:02:38.000Z","size":813,"stargazers_count":4,"open_issues_count":1,"forks_count":6,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-11-26T05:18:50.811Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cometbft.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-19T11:45:46.000Z","updated_at":"2024-11-26T04:02:41.000Z","dependencies_parsed_at":"2024-01-04T09:45:24.813Z","dependency_job_id":"09894574-7fcf-41dc-9f97-f4176ff6ed77","html_url":"https://github.com/cometbft/cometbft-load-test","commit_stats":null,"previous_names":["cometbft/cometbft-load-test"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cometbft%2Fcometbft-load-test","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cometbft%2Fcometbft-load-test/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cometbft%2Fcometbft-load-test/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cometbft%2Fcometbft-load-test/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cometbft","download_url":"https://codeload.github.com/cometbft/cometbft-load-test/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248124840,"owners_count":21051757,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-09T22:40:14.746Z","updated_at":"2025-04-09T22:40:15.479Z","avatar_url":"https://github.com/cometbft.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# cometbft-load-test\n\n`cometbft-load-test` is a **framework** for load testing\n[CometBFT](https://cometbft.com/)-based networks, forked from\n[tm-load-test](https://github.com/informalsystems/tm-load-test).\n\nThe structure/format of transactions sent to a CometBFT-based network are\nspecific to the ABCI application running on that network, so while\n`cometbft-load-test` comes with built-in support for the `kvstore` ABCI\napplication (as an example), you have to build your own clients for your own\napps.\n\n## Requirements\n\n`cometbft-load-test` is currently tested using Go v1.22 and CometBFT v0.38.\n\n## Usage\n\n### Step 1: Create your project\n\nYou have to create your own load testing tool for your own ABCI application by\nimporting the `cometbft-load-test` package into a new project.\n\n```bash\nmkdir -p /your/project/\ncd /your/project\ngo mod init github.com/you/my-load-tester\n```\n\n### Step 2: Create your transaction generator\n\nCreate a client that generates transactions for your ABCI app. For an example,\nyou can look at the [kvstore client code](./pkg/loadtest/client_kvstore.go). Put\nthis in `./pkg/myabciapp/client.go`\n\n```go\npackage myabciapp\n\nimport \"github.com/cometbft/cometbft-load-test/pkg/loadtest\"\n\n// MyABCIAppClientFactory creates instances of MyABCIAppClient\ntype MyABCIAppClientFactory struct {}\n\n// MyABCIAppClientFactory implements loadtest.ClientFactory\nvar _ loadtest.ClientFactory = (*MyABCIAppClientFactory)(nil)\n\n// MyABCIAppClient is responsible for generating transactions. Only one client\n// will be created per connection to the remote CometBFT RPC endpoint, and\n// each client will be responsible for maintaining its own state in a\n// thread-safe manner.\ntype MyABCIAppClient struct {}\n\n// MyABCIAppClient implements loadtest.Client\nvar _ loadtest.Client = (*MyABCIAppClient)(nil)\n\nfunc (f *MyABCIAppClientFactory) ValidateConfig(cfg loadtest.Config) error {\n    // Do any checks here that you need to ensure that the load test\n    // configuration is compatible with your client.\n    return nil\n}\n\nfunc (f *MyABCIAppClientFactory) NewClient(cfg loadtest.Config) (loadtest.Client, error) {\n    return \u0026MyABCIAppClient{}, nil\n}\n\n// GenerateTx must return the raw bytes that make up the transaction for your\n// ABCI app. The conversion to base64 will automatically be handled by the\n// loadtest package, so don't worry about that. Only return an error here if you\n// want to completely fail the entire load test operation.\nfunc (c *MyABCIAppClient) GenerateTx() ([]byte, error) {\n    return []byte(\"this is my transaction\"), nil\n}\n```\n\n### Step 3: Create your CLI\n\nCreate your own CLI in `./cmd/my-load-tester/main.go`:\n\n```go\npackage main\n\nimport (\n    \"github.com/cometbft/cometbft-load-test/pkg/loadtest\"\n    \"github.com/you/my-load-tester/pkg/myabciapp\"\n)\n\nfunc main() {\n    if err := loadtest.RegisterClientFactory(\"my-abci-app-name\", \u0026myabciapp.MyABCIAppClientFactory{}); err != nil {\n        panic(err)\n    }\n    // The loadtest.Run method will handle CLI argument parsing, errors,\n    // configuration, instantiating the load test and/or coordinator/worker\n    // operations, etc. All it needs is to know which client factory to use for\n    // its load testing.\n    loadtest.Run(\u0026loadtest.CLIConfig{\n        AppName:              \"my-load-tester\",\n        AppShortDesc:         \"Load testing application for My ABCI App\",\n        AppLongDesc:          \"Some long description on how to use the tool\",\n        DefaultClientFactory: \"my-abci-app-name\",\n    })\n}\n```\n\nFor an example of very simple integration testing, you could do something\nsimilar to what's covered in\n[integration\\_test.go](./pkg/loadtest/integration_test.go).\n\n### Step 4: Build your CLI\n\nThen build the executable:\n\n```bash\ngo build -o ./build/my-load-tester ./cmd/my-load-tester/main.go\n```\n\n## Running your load testing tool\n\nA `cometbft-load-test`-based load testing application can be executed in one of\ntwo modes: **standalone**, or **coordinator/worker**.\n\nNB: In all of the following examples, replace `cometbft-load-test` with the name\nof your load testing application you have built (e.g. `my-load-tester`).\n\n### Standalone Mode\n\nIn standalone mode, `cometbft-load-test` simply broadcasts transactions to a\nsingle endpoint from a single binary:\n\n```bash\ncometbft-load-test -c 1 -T 10 -r 1000 -s 250 \\\n    --broadcast-tx-method async \\\n    --endpoints ws://cmt-endpoint1.somewhere.com:26657/websocket,ws://cmt-endpoint2.somewhere.com:26657/websocket\n```\n\nTo see a description of what all of the parameters mean, simply run:\n\n```bash\ncometbft-load-test --help\n```\n\n### Coordinator/Worker Mode\n\nIn coordinator/worker mode, which is best used for large-scale, distributed load\ntesting, `cometbft-load-test` allows you to have multiple worker machines\nconnect to a single coordinator to obtain their configuration and coordinate\ntheir operation.\n\nThe coordinator acts as a simple WebSockets host, and the workers are WebSockets\nclients.\n\nOn the coordinator machine:\n\n```bash\n# Run cometbft-load-test with similar parameters to the standalone mode, but now\n# specifying the number of workers to expect (--expect-workers) and the host:port\n# to which to bind (--bind) and listen for incoming worker requests.\ncometbft-load-test \\\n    coordinator \\\n    --expect-workers 2 \\\n    --bind localhost:26670 \\\n    -c 1 -T 10 -r 1000 -s 250 \\\n    --broadcast-tx-method async \\\n    --endpoints ws://cmt-endpoint1.somewhere.com:26657/websocket,ws://cmt-endpoint2.somewhere.com:26657/websocket\n```\n\nOn each worker machine:\n\n```bash\n# Just tell the worker where to find the coordinator - it will figure out the rest.\ncometbft-load-test worker --coordinator localhost:26680\n```\n\nFor more help, see the command line parameters' descriptions:\n\n```bash\ncometbft-load-test coordinator --help\ncometbft-load-test worker --help\n```\n\n### Endpoint Selection Strategies\n\nAn endpoint selection strategy can now be given to `cometbft-load-test` as a\nparameter (`--endpoint-select-method`) to control the way in which endpoints are\nselected for load testing. There are several options:\n\n1. `supplied` (the default) - only use the supplied endpoints (via the\n   `--endpoints` parameter) to submit transactions.\n2. `discovered` - only use endpoints discovered through the supplied endpoints\n   (by way of crawling the CometBFT peers' network info), but do not use any of\n   the supplied endpoints.\n3. `any` - use both the supplied and discovered endpoints to perform load\n   testing.\n\n**NOTE**: These selection strategies only apply if, and only if, the\n`--expect-peers` parameter is supplied and is non-zero. The default behaviour if\n`--expect-peers` is not supplied is effectively the `supplied` endpoint\nselection strategy.\n\n### Minimum Peer Connectivity\n\n`cometbft-load-test` can wait for a minimum level of P2P connectivity before\nstarting the load testing. By using the `--min-peer-connectivity` command line\nswitch, along with `--expect-peers`, one can restrict this.\n\nWhat this does under the hood is that it checks how many peers are in each\nqueried peer's address book, and for all reachable peers it checks what the\nminimum address book size is. Once the minimum address book size reaches the\nconfigured value, the load testing can begin.\n\n## Monitoring\n\n`cometbft-load-test` exposes a number of Prometheus metrics when in\ncoordinator/worker mode, but only from the coordinator's web server at the\n`/metrics` endpoint. So if you bind your coordinator node to `localhost:26670`,\nyou should be able to get these metrics from:\n\n```bash\ncurl http://localhost:26670/metrics\n```\n\nThe following kinds of metrics are made available here:\n\n* Total number of transactions recorded from the coordinator's perspective\n  (across all workers)\n* Total number of transactions sent by each worker\n* The status of the coordinator node, which is a gauge that indicates one of the\n  following codes:\n  * 0 = Coordinator starting\n  * 1 = Coordinator waiting for all peers to connect\n  * 2 = Coordinator waiting for all workers to connect\n  * 3 = Load test underway\n  * 4 = Coordinator and/or one or more worker(s) failed\n  * 5 = All workers completed load testing successfully\n* The status of each worker node, which is also a gauge that indicates one of\n  the following codes:\n  * 0 = Worker connected\n  * 1 = Worker accepted\n  * 2 = Worker rejected\n  * 3 = Load testing underway\n  * 4 = Worker failed\n  * 5 = Worker completed load testing successfully\n* Standard Prometheus-provided metrics about the garbage collector in\n  `cometbft-load-test`\n* The ID of the load test currently underway (defaults to 0), set by way of the\n  `--load-test-id` flag on the coordinator\n\n## Aggregate Statistics\n\nYou can write simple aggregate statistics to a CSV file once testing completes\nby specifying the `--stats-output` flag:\n\n```bash\n# In standalone mode\ncometbft-load-test -c 1 -T 10 -r 1000 -s 250 \\\n    --broadcast-tx-method async \\\n    --endpoints ws://cmt-endpoint1.somewhere.com:26657/websocket,ws://cmt-endpoint2.somewhere.com:26657/websocket \\\n    --stats-output /path/to/save/stats.csv\n\n# From the coordinator in coordinator/worker mode\ncometbft-load-test \\\n    coordinator \\\n    --expect-workers 2 \\\n    --bind localhost:26670 \\\n    -c 1 -T 10 -r 1000 -s 250 \\\n    --broadcast-tx-method async \\\n    --endpoints ws://cmt-endpoint1.somewhere.com:26657/websocket,ws://cmt-endpoint2.somewhere.com:26657/websocket \\\n    --stats-output /path/to/save/stats.csv\n```\n\nThe output CSV file has the following format at present:\n\n```csv\nParameter,Value,Units\ntotal_time,10.002,seconds\ntotal_txs,9000,count\navg_tx_rate,899.818398,transactions per second\n```\n\n## Development\n\nTo run the linter and the tests:\n\n```bash\nmake lint\nmake test\n```\n\n### Integration Testing\n\nIntegration testing requires Docker to be installed locally.\n\n```bash\nmake integration-test\n```\n\nThis integration test:\n\n1. Sets up a 4-validator, fully connected CometBFT-based network on a\n   192.168.0.0/16 subnet (the same kind of testnet as the CometBFT localnet).\n2. Executes integration tests against the network in series (it's important that\n   integration tests be executed in series so as to not overlap with one\n   another).\n3. Tears down the 4-validator network, reporting code coverage.\n\n## License\n\nCopyright 2023 CometBFT team and contributors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcometbft%2Fcometbft-load-test","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcometbft%2Fcometbft-load-test","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcometbft%2Fcometbft-load-test/lists"}