{"id":27203564,"url":"https://github.com/cometbft/qa-infra","last_synced_at":"2025-04-09T22:40:14.088Z","repository":{"id":74820078,"uuid":"592401850","full_name":"cometbft/qa-infra","owner":"cometbft","description":"QA infrastructure for CometBFT","archived":false,"fork":false,"pushed_at":"2024-04-12T08:02:37.000Z","size":340,"stargazers_count":4,"open_issues_count":7,"forks_count":2,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-04-12T15:23:44.257Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jinja","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cometbft.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-01-23T16:51:44.000Z","updated_at":"2024-03-22T09:27:39.000Z","dependencies_parsed_at":null,"dependency_job_id":"d99b16f0-a51f-451b-a3cb-ccc2b2d17f43","html_url":"https://github.com/cometbft/qa-infra","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cometbft%2Fqa-infra","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cometbft%2Fqa-infra/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cometbft%2Fqa-infra/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cometbft%2Fqa-infra/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cometbft","download_url":"https://codeload.github.com/cometbft/qa-infra/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248124840,"owners_count":21051757,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-09T22:40:13.538Z","updated_at":"2025-04-09T22:40:14.074Z","avatar_url":"https://github.com/cometbft.png","language":"Jinja","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CometBFT Test Networks\n\nThis repo contains [Ansible] and [Terraform] scripts for spinning up CometBFT test networks on Digital Ocean (DO).\n\n## Requirements\n\n- [Digital Ocean CLI][doctl]\n- [Terraform CLI][Terraform]\n- [Ansible CLI][Ansible]\n- Go\n\n## Instructions\n\n### Setup\n\nAfter you have all the prerequisites installed:\n\n1. Set up your [personal access token for DO](https://docs.digitalocean.com/reference/api/create-personal-access-token/)\n\n    ```bash\n    doctl auth init\n    ```\n\n    If you have executed this and the following steps before, you may be able to skip to step 5.\n    And if your token expired, you may need to force the use of the one you just generated here by using `doctl auth init -t \u003cnew token\u003e` instead.\n\n    ```bash\n    doctl auth init -t dop_v1_0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef\n    ```\n\n2. Get the fingerprint of the SSH key you want to be associated with the root user on the created VMs\n\n    ```bash\n    doctl compute ssh-key list\n    ```\n\n3. Set up your Digital Ocean credentials as Terraform variables. Be sure to write them to `./tf/terraform.tfvars` as this file is ignored in `.gitignore`.\n\n    ```bash\n    cat \u003c\u003cEOF \u003e ./tf/terraform.tfvars\n    do_token = \"dop_v1_0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef\"\n    ssh_keys = [\"ab:cd:ef:01:23:45:67:89:ab:cd:ef:01:23:45:67:89\"]\n    EOF\n    ```\n\n4. Initialize Terraform (only needed once)\n\n    ```bash\n    make terraform-init\n    ```\n\n### Start the network\n\nAfter you have set up the infrastructure:\n\n5. Set up the test you will run in the `experiment.mk` file:\n    1. Set the path to your manifest file in the variable `MANIFEST`.\n    2. Set the commit hash of CometBFT that you to install in the nodes in the variable `VERSION_TAG`.\n    3. If you want to deploy a subset of the validators with a different version of CometBFT, set\n       the variable `VERSION2_TAG` to the commit hash you want to install in that subset. Then set\n       the proportion of nodes that will run `VERSION_TAG` and `VERSION2_TAG` in the variables\n       `VERSION_WEIGHT` and `VERSION2_WEIGHT` respectively.\n    4. If necessary, set the variables `DO_INSTANCE_TAGNAME` and `DO_VPC_SUBNET` to customized\n    values to prevent collisions with other QA runs, including possible other users of the\n    DigitalOcean project who might be running these scripts. If the subnet is allocated in the\n    private IP address range 172.16.0.0/12, as it is in the unmodified file, a good choice should be\n    in the range 172.16.16.0/20 - 172.31.240.0/20. You may also need to rename the DO project\n    `cmt-testnet` in the `tf/project.tf` file to a unique name.\n\n6. Create the VMs for the validators and Prometheus as specified in the manifest file.\n    Be sure to use your actual DO token and SSH key fingerprints for the `do_token` and `do_ssh_keys` variables.\n\n    ```bash\n    make terraform-apply\n    ```\n    \n    After creating the DO droplets, this command will generate two files with information about the\n    IP addresses of the nodes: an Ansible inventory file `./ansible/hosts`, and\n    `./ansible/testnet/infrastructure-data.json` for E2E's `runner` tool.\n\n    Note that installing packages defined in `tf/user-data.txt` may take more time than expected.\n    It's possible that the installation process has not yet finished even when DO says that droplets\n    have been created successfully.\n\n7. Generate the testnet configuration\n\n    ```bash\n    make configgen\n    ```\n\n8. Install all necessary software on the created VMs using Ansible\n\n    ```bash\n    make ansible-install\n    ```\n\n9. Initialize the Prometheus instance\n\n    ```bash\n    make prometheus-init\n    ```\n\n10. Start the test application on all of the validators\n\n    ```bash\n    make start-network\n    ```\n\n### Execute the load test\n\nInitialize the load-runner node, if not it's yet running:\n```bash\nmake loadrunners-init\n```\n\nThe following command will start sending load until Ctrl-C is sent, so consider running this in its own terminal:\n```bash\nmake runload\n```\n\n### Stop the network and retrieve data\n\n11. Once the execution is over, stop the network:\n\n    ```bash\n    make stop-network\n    ```\n\n12. Retrieve the data produced during the execution.\n    You can either use the following command to retrieve both the prometheus and the blockstore databases together\n\n    ```bash\n    make retrieve-data\n    ```\n\n    To retrieve them independently use the following for prometheus, which will retrieve the data from all nodes.\n\n    ```bash\n    make retrieve-prometheus-data\n    ```\n\n    For blockstore, use the following. Here, notice that the target node from which the data is retrieved can be changed via the environment variable `RETRIEVE_TARGET_HOST`.\n      - `\"any\"` (default) - retrieve from one random validator from the inventory.\n      - `\"all\"` - retrieve from all nodes (very slow!);\n      - set it to the exact name of a validator to retrieve from that particular validator.\n\n    ```bash\n    make retrieve-blockstore\n    ```\n\n## Additional Commands\n\n### Restart the network\n\nIf you need to restart the running experiment, run the following command:\n\n```sh\nmake restart\n```\n\nThis command will delete all of the prometheus data, and re-initialize the nodes\non the network. The nodes will restart with the same configuration files and\nIDs that they previously used, but all of their data will be deleted and reset.\n\n### Destroy the network\n\nDo not forget to destroy the experiment to stop charging.\n\n```sh\nmake terraform-destroy\n```\n\n#### Keep the Prometheus node running\n\nYou may want to keep running some nodes to retrieve data from them and destroy the others.\nThe following commands will destroy all nodes except the Prometheus node and the last validator.\n```\ncd tf \u0026\u0026 terraform state rm digitalocean_droplet.testnet-prometheus digitalocean_droplet.testnet-node[199]\nmake terraform-destroy\n```\n\n## Metrics\n\nOnce you've completed setting up the network, take a look at your\n`ansible/hosts` file for the IP address of the Prometheus server, then navigate\nto that address on port 9090 in your web browser in order to query collected\nmetrics and view their associated graphs.\n\n[Ansible]: https://docs.ansible.com/ansible/latest/index.html\n[Terraform]: https://www.terraform.io/docs\n[doctl]: https://docs.digitalocean.com/reference/doctl/how-to/install/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcometbft%2Fqa-infra","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcometbft%2Fqa-infra","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcometbft%2Fqa-infra/lists"}