{"id":50385036,"url":"https://github.com/stackhpc/terraform-kayobe-multinode","last_synced_at":"2026-05-30T14:30:54.010Z","repository":{"id":52955640,"uuid":"521174203","full_name":"stackhpc/terraform-kayobe-multinode","owner":"stackhpc","description":null,"archived":false,"fork":false,"pushed_at":"2026-04-27T10:46:18.000Z","size":470,"stargazers_count":2,"open_issues_count":8,"forks_count":3,"subscribers_count":8,"default_branch":"main","last_synced_at":"2026-04-27T12:26:09.196Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stackhpc.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2022-08-04T07:51:56.000Z","updated_at":"2026-02-02T09:37:55.000Z","dependencies_parsed_at":"2023-12-19T20:17:49.151Z","dependency_job_id":"f5dd62c8-71a4-4cdd-8bec-989297e919a4","html_url":"https://github.com/stackhpc/terraform-kayobe-multinode","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/stackhpc/terraform-kayobe-multinode","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackhpc%2Fterraform-kayobe-multinode","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackhpc%2Fterraform-kayobe-multinode/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackhpc%2Fterraform-kayobe-multinode/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackhpc%2Fterraform-kayobe-multinode/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stackhpc","download_url":"https://codeload.github.com/stackhpc/terraform-kayobe-multinode/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackhpc%2Fterraform-kayobe-multinode/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33696681,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-30T02:00:06.278Z","response_time":92,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-30T14:30:53.917Z","updated_at":"2026-05-30T14:30:54.001Z","avatar_url":"https://github.com/stackhpc.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"==========================\nTerraform Kayobe Multinode\n==========================\n\nThis Terraform configuration deploys a requested amount of instances on an OpenStack cloud, to be\nused as a Multinode Kayobe test environment. This includes:\n\n* 1x Ansible control host\n* 1x seed host\n* controller hosts\n* compute hosts\n* Ceph storage hosts\n* Optional Wazuh manager host\n\nThe high-level workflow to deploy a cluster is as follows:\n\n* Prerequisites\n* Configure Terraform and Ansible\n* Deploy infrastructure on OpenStack using Terraform\n* Configure Ansible control host using Ansible\n* Deploy multi-node OpenStack using Kayobe\n\nThis configuration is typically used with the `ci-multinode` environment in the\n`StackHPC Kayobe Configuration\n\u003chttps://stackhpc-kayobe-config.readthedocs.io/en/stackhpc-2024.1/contributor/environments/ci-multinode.html\u003e`__\nrepository.\n\nWhat's in the box?\n==================\n\nThis repository contains various items.\n\nScripts\n-------\n\n* ``scripts/deploy.sh`` - end-to-end cluster deployment and testing.\n* ``scripts/tear-down.sh`` - tear down test cluster infrastructure.\n\nTerraform\n---------\n\nTerraform configuration deploys test cluster infrastructure on an OpenStack\ncloud. It provides outputs that can be used to populate Kayobe Configuration\nwith the details of the test infrastructure.\n\nAnsible\n-------\n\nAnsible playbooks in the ``ansible/`` directory are provided to prepare and use\nthe Ansible control host.\n\n#. ``configure-hosts.yml`` - sequentially executes 3 other playbooks:\n\n   #. ``wait-control-host.yml`` - Waits for the Ansible control host to become\n      accessible and ready for deployment. Tag: ``wait``\n   #. ``grow-control-host.yml`` - Applies LVM configuration to the control host\n      to ensure it has enough space to continue with the rest of the\n      deployment. Tag: ``lvm``\n   #. ``deploy-openstack-config.yml`` - Prepares the Ansible control host as a\n      Kayobe control host, cloning the Kayobe configuration and installing\n      virtual environments. Tag: ``deploy``\n\n   These playbooks are tagged so that they can be invoked or skipped using\n   `tags` or `--skip-tags` as required.\n\n#. ``deploy-openstack.yml`` - runs the ``multinode.sh deploy_full`` command in\n   a `tmux` session on the Ansible control host. The session is logged to\n   ``~/tmux.kayobe\\:0.log`` on the Ansible control host.  Use ``less -r\n   ~/tmux.kayobe\\:0.log`` to view the logs in their original colourful glory.\n\n#. ``fetch-logs.yml`` - fetches logs, diagnostics and tests results from the\n   cluster to the host runnin the playbook..\n\nConfiguration variables for these playbooks are in\n``ansible/vars/defaults.yml``.\n\nmultinode.sh\n------------\n\nThe ``multinode.sh`` script is installed in ``/usr/local/bin/`` on the Ansible\ncontrol host. It provides high-level automation of various aspects of OpenStack\ndeployment, operations and testing. It accepts a single argument which is the\ncommand to perform. The supported commands may be listed by running it without\narguments.\n\nPrerequisites\n=============\n\nThese instructions show how to use this Terraform configuration manually. They\nassume you are running an Ubuntu host that will be used to run Terraform. The\nmachine should have access to the API of the OpenStack cloud that will host the\ninfrastructure, and network access to the Ansible control host once it has been\ndeployed. This may be achieved by direct SSH access, a floating IP on the\nAnsible control host, or using an SSH bastion.\n\nThe OpenStack cloud should have sufficient capacity to deploy the\ninfrastructure, and a suitable image registered in Glance. Ideally the image\nshould be one of the overcloud host images defined in StackHPC Kayobe\nconfiguration and available in `Ark \u003chttps://ark.stackhpc.com\u003e`__.\n\nInstall Terraform:\n\n.. code-block:: console\n\n   wget -qO - terraform.gpg https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/terraform-archive-keyring.gpg\n   sudo echo \"deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/terraform-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main\" | sudo tee /etc/apt/sources.list.d/terraform.list\n   sudo apt update\n   sudo apt install git terraform\n\nClone and initialise this Terraform config repository:\n\n.. code-block:: console\n\n   git clone https://github.com/stackhpc/terraform-kayobe-multinode\n   cd terraform-kayobe-multinode\n\nInitialise Terraform:\n\n.. code-block:: console\n\n   terraform init\n\nGenerate an SSH keypair. Note that `ED25519 keys are not currently supported by RHEL\n\u003chttps://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html-single/securing_networks/index#making-openssh-more-secure_assembly_using-secure-communications-between-two-systems-with-openssh\u003e`__\nwhen using the FIPS security standard (as enabled by the CIS benchmark hardening\nscripts in kayobe-config). The public key will be registered in OpenStack as a\nkeypair and authorised by the instances deployed by Terraform. The private and\npublic keys will be transferred to the Ansible control host to allow it to\nconnect to the other hosts. Note that password-protected keys are not currently\nsupported.\n\n.. code-block:: console\n\n   ssh-keygen -t rsa -f id_rsa -N ''\n\nCreate an OpenStack clouds.yml file with your credentials to access an\nOpenStack cloud. Alternatively, download and source an openrc file from Horizon.\n\n.. code-block:: console\n\n   cat \u003c\u003c EOF \u003e clouds.yml\n   ---\n   clouds:\n     openstack:\n       auth:\n         auth_url: https://api.sms-lab.cloud:5000\n         username: \u003cusername\u003e\n         project_name: \u003cproject\u003e\n         domain_name: default\n       interface: public\n   EOF\n\nExport environment variables to use the correct cloud and provide a password (you shouldn't do this if you have sourced the openrc file):\n\n.. code-block:: console\n\n   export OS_CLOUD=openstack\n   export OS_CLOUD_CONFIG_FILE=\u003cpath\u003e/\u003cto\u003e/clouds.yml\n\nOr you can source the provided `init.sh` script which shall initialise terraform and export two variables.\n`OS_CLOUD` is a variable which is used by Terraform and must match an entry within `clouds.yml` (Not needed if you have sourced the openrc file).\n`OS_PASSWORD` is the password used to authenticate when signing into OpenStack.\n\n.. code-block:: console\n   source ./init.sh\n\n   Initializing the backend...\n\n   Initializing provider plugins...\n   - Reusing previous version of terraform-provider-openstack/openstack from the dependency lock file\n   - Reusing previous version of hashicorp/local from the dependency lock file\n   - Using previously-installed terraform-provider-openstack/openstack v1.48.0\n   - Using previously-installed hashicorp/local v2.2.3\n\n   Terraform has been successfully initialized!\n\n   You may now begin working with Terraform. Try running \"terraform plan\" to see\n   any changes that are required for your infrastructure. All Terraform commands\n   should now work.\n\n   If you ever set or change modules or backend configuration for Terraform,\n   rerun this command to reinitialize your working directory. If you forget, other\n   commands will detect it and remind you to do so if necessary.\n   OpenStack Cloud Name: openstack\n   Password:\n\nYou must ensure that you have `Ansible installed \u003chttps://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html\u003e`_ on your local machine.\n\n.. code-block:: console\n\n   pip install --user ansible\n\nInstall the Ansible galaxy requirements.\n\n.. code-block:: console\n\n   ansible-galaxy install -r ansible/requirements.yml\n\nIf the deployed instances are behind an SSH bastion you must ensure that your SSH config is setup appropriately with a proxy jump.\n\n.. code-block::\n\n   Host lab-bastion\n      HostName BastionIPAddr\n      User username\n      IdentityFile ~/.ssh/key\n\n   Host 10.*\n      ProxyJump=lab-bastion\n      ForwardAgent no\n      IdentityFile ~/.ssh/key\n      UserKnownHostsFile /dev/null\n      StrictHostKeyChecking no\n      IdentitiesOnly yes\n\nConfigure Terraform variables\n=============================\n\nPopulate Terraform variables in `terraform.tfvars`. Examples are provided in\nfiles named `*.tfvars.example`. The available variables are defined in\n`variables.tf` along with their type, description, and optional default.\n\nYou will need to set the `multinode_keypair`, `prefix`, and `ssh_public_key`.\nBy default, Rocky Linux 9 will be used but Ubuntu Jammy is also supported by\nchanging `multinode_image` to `overcloud-ubuntu-jammy-\u003crelease\u003e-\u003cdatetime\u003e` and\n`ssh_user` to `ubuntu`.\n\nThe `multinode_flavor` will change the flavor used for controller and compute\nnodes. Both virtual machines and baremetal are supported, but the `*_disk_size`\nvariables must be set to 0 when using baremetal host. This will stop a block\ndevice being allocated. When any baremetal hosts are deployed, the\n`multinode_vm_network` and `multinode_vm_subnet` should also be changed to\na VLAN network and associated subnet.\n\nIf `deploy_wazuh` is set to true, an infrastructure VM will be created that\nhosts the Wazuh manager. The Wazuh deployment playbooks will also be triggered\nautomatically to deploy Wazuh agents to the overcloud hosts.\n\nIf `add_ansible_control_fip` is set to `true`, a floating IP will be created\nand attached to the Ansible control host. In that case\n`ansible_control_fip_pool` should be set to the name of the pool (network) from\nwhich to allocate the floating IP, and the floating IP will be used for SSH\naccess to the control host.\n\nConfigure Ansible variables\n===========================\n\nReview the vars defined within `ansible/vars/defaults.yml`. In here you can customise the version of kayobe, kayobe-config or openstack-config.\nMake sure to define `ssh_key_path` to point to the location of the SSH key in use by the nodes and also `vxlan_vni` which should be unique value between 1 to 100,000.\nVNI should be much smaller than the officially supported limit of 16,777,215 as we encounter errors when attempting to bring interfaces up that use a high VNI.\nYou must set `vault_password_path`; this should be set to the path to a file containing the Ansible vault password.\n\nDeployment: The fast(er) way\n============================\n\nThe `scripts/deploy.sh` script provides a fully automated deployment method\nthat can be used to perform all steps from infrastructure deployment through to\nTempest testing without user interaction. Any errors encountered will be\nreported and halt the deployment.\n\nThis script makes use of the `ansible/deploy-openstack.yml` Ansible playbook\nthat runs the `multinode.sh deploy_full` command in a `tmux` session on the Ansible\ncontrol host. The session is logged to `~/tmux.kayobe\\:0.log` on the Ansible\ncontrol host. Use `less -r ~/tmux.kayobe\\:0.log` to view the logs in their\noriginal colourful glory.\n\nNote that this approach requires all Terraform, Ansible, Kayobe and OpenStack\nconfiguration to be provided in advance. For Kayobe and OpenStack\nconfiguration, this may be achieved by providing suitable branches for the\nkayobe-config and openstack-config repositories and referencing them in\n`ansible/vars/defaults.yml`.\n\nTo tear down the cluster immediately after a successful deployment, combine\nwith the `scripts/tear-down.sh` script.\n\n.. code-block:: console\n\n   ./scripts/deploy.sh \u0026\u0026 ./scripts/tear-down.sh -a -k\n\nNote that this will not tear down the cluster if deployment fails, allowing for\ndebugging the issue and/or retrying.\n\nDeployment: The slow(er) way\n============================\n\nThis section describes a more hands-on, interactive deployment method. It may\nbe useful to gain a better understanding of how the deployment works, modify\nthe deployment process in some way, or iterate on configuration.\n\nTerraform Deploy infrastructure using Terraform\n-----------------------------------------------\n\nGenerate a plan:\n\n.. code-block:: console\n\n   terraform plan\n\nApply the changes:\n\n.. code-block:: console\n\n   terraform apply -auto-approve\n\nYou should have requested a number of resources to be spawned on Openstack.\n\nConfigure Ansible control host\n------------------------------\n\nRun the configure-hosts.yml playbook to configure the Ansible control host.\n\n.. code-block:: console\n\n   ansible-playbook -i ansible/inventory.yml ansible/configure-hosts.yml\n\nDeploy OpenStack\n----------------\n\nOnce the Ansible control host has been configured with a Kayobe/OpenStack configuration you can then begin the process of deploying OpenStack.\nThis can be achieved by either manually running the various commands to configure the hosts and deploy the services or automated by using the ``multinode.sh deploy_full`` command.\n``multinode.sh`` should be available within ``/usr/local/bin/`` on your Ansible control host provided you ran `deploy-openstack-config.yml` earlier.\nThis script will go through the process of performing the following tasks:\n\n   * kayobe control host bootstrap\n   * kayobe seed host configure\n   * kayobe overcloud host configure\n   * cephadm deployment\n   * HashiCorp Vault deployment \u0026 certificate generation\n   * kayobe overcloud service deploy\n   * OpenStack configuration\n   * Tempest testing\n\nTempest test results will be written to ``~/tempest-artifacts``.\n\nIf you choose to opt for the automated method you must first SSH into your Ansible control host.\n\n.. code-block:: console\n\n   ssh $(terraform output -raw ssh_user)@$(terraform output -raw ansible_control_access_ip_v4)\n\nStart a `tmux` session to avoid halting the deployment if you are disconnected.\n\n.. code-block:: console\n\n   tmux\n\nRun the `multinode.sh` script.\n\n.. code-block:: console\n\n   multinode.sh deploy_full\n\nAccessing OpenStack\n===================\n\nAfter a successful deployment of OpenStack you make access the OpenStack API and Horizon by proxying your connection via the seed node, as it has an interface on the public network (192.168.39.X).\nUsing software such as sshuttle will allow for easy access.\n\n.. code-block:: console\n\n   sshuttle -r $(terraform output -raw ssh_user)@$(terraform output -raw seed_access_ip_v4) 192.168.39.0/24\n\nYou may also use sshuttle to proxy DNS via the multinode environment. Useful if you are working with Designate.\nImportant to note this will proxy all DNS requests from your machine to the first controller within the multinode environment.\n\n.. code-block:: console\n\n   sshuttle -r $(terraform output -raw ssh_user)@$(terraform output -raw seed_access_ip_v4) 192.168.39.0/24 --dns --to-ns 192.168.39.4\n\nTear Down\n=========\n\nAfter you are finished with the multinode environment please destroy the nodes to free up resources for others.\nThis can acomplished by using the provided `scripts/tear-down.sh` which will destroy your controllers, compute, seed and storage nodes whilst leaving your Ansible control host and keypair intact.\n\nIf you would like to delete your Ansible control host then you can pass the `-a` flag however if you would also like to remove your keypair then pass `-a -k`\n\nIssues \u0026 Fixes\n==============\n\nSometimes a compute instance fails to be provisioned by Terraform or fails on boot for any reason.\nIf this happens the solution is to mark the resource as tainted and perform terraform apply again which shall destroy and rebuild the failed instance.\n\n.. code-block:: console\n\n   terraform taint 'openstack_compute_instance_v2.controller[2]'\n   terraform apply\n\nAlso sometimes the provider may fail to notice that some resources are functioning as expected due to timeouts or other network issues.\nIf you can confirm via Horizon or via SSH that the resource is functioning as expected you may untaint the resource preventing Terraform from destroying on subsequent terraform apply.\n\n.. code-block:: console\n\n   terraform untaint 'openstack_compute_instance_v2.controller[2]'\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstackhpc%2Fterraform-kayobe-multinode","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstackhpc%2Fterraform-kayobe-multinode","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstackhpc%2Fterraform-kayobe-multinode/lists"}