{"id":22451975,"url":"https://github.com/odennav/on-premise-vsphere-devops","last_synced_at":"2025-03-27T12:28:24.838Z","repository":{"id":234909482,"uuid":"786609477","full_name":"odennav/on-premise-vsphere-devops","owner":"odennav","description":"Automate deployment of a load balanced application on vSphere datacenter with Ansible, Vault, Packer, Terraform and Jenkins.","archived":false,"fork":false,"pushed_at":"2024-04-26T20:40:20.000Z","size":17073,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-04-26T21:35:02.859Z","etag":null,"topics":["ansible","datastore","devops","docker","esxi","jenkins","nginx-proxy","nginx-server","on-prem","on-premise","packer","terraform","vault","vcenter-server","vmfs","vmware","vsphere"],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/odennav.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-15T00:29:56.000Z","updated_at":"2024-04-26T21:35:04.339Z","dependencies_parsed_at":"2024-04-26T21:45:13.597Z","dependency_job_id":null,"html_url":"https://github.com/odennav/on-premise-vsphere-devops","commit_stats":null,"previous_names":["odennav/on-premise-devops-vsphere","odennav/on-premise-vsphere-devops"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/odennav%2Fon-premise-vsphere-devops","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/odennav%2Fon-premise-vsphere-devops/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/odennav%2Fon-premise-vsphere-devops/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/odennav%2Fon-premise-vsphere-devops/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/odennav","download_url":"https://codeload.github.com/odennav/on-premise-vsphere-devops/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245844027,"owners_count":20681622,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ansible","datastore","devops","docker","esxi","jenkins","nginx-proxy","nginx-server","on-prem","on-premise","packer","terraform","vault","vcenter-server","vmfs","vmware","vsphere"],"created_at":"2024-12-06T06:09:25.445Z","updated_at":"2025-03-27T12:28:24.803Z","avatar_url":"https://github.com/odennav.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# On-Prem DevOps with VMware vSphere\n\nDeploy a load-balanced application using VMware vSphere, Docker, Gogs, Ansible, Vault, Packer, Terraform, Jenkins and Nginx.\n\n![](https://github.com/odennav/on-prem-devops-vsphere/blob/main/docs/pipeline.png)\n\n## Prerequisites\n  \n  - Deploy self hosted vSphere datacenter and compute cluster with 2 ESXi hosts.\n  - Enable vSAN, DRS and SDRS.\n  - Deploy vCenter server appliance on ESXi host.\n  - Provision a Build machine on next ESXi host with Ubuntu 20.04 server.\n  - Git bash or linux terminal on local machine.\n  - Assume IPv4 address of Build machine VM is `192.168.149.8` and it's hostname is `build-machine`\n  - Check this VMware datacenter lab [guide](https://github.com/odennav/vmware-sddc-private-cloud)\n  \n\n# Getting Started\n\n  Two Pipelines will be implemented:\n  - Manual Pipeline\n  - Automated Pipeline\n\n\n## Manual Pipeline\n\n  This workflow involves the following steps:\n  - Docker Installation\n  - Gogs(source control) Installation and Configuration\n  - Vault Installation and Configuration\n  - Packer Installation and Configuration\n  - Terraform Installation and VM Deployment \n  - Ansible Installation and Machine Configuration\n\n\n-----\n\n1.  **DOCKER INSTALLATION**\n\n    To install Docker Engine for the first time on build-machine, we'll set up the Docker repository.\n    Afterward, we can install and update Docker from the repository.\n    \n    Add Docker's official GPG key:\n    ```bash   \n    sudo apt-get update\n    sudo apt-get install ca-certificates curl\n    sudo install -m 0755 -d /etc/apt/keyrings\n    sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc\n    sudo chmod a+r /etc/apt/keyrings/docker.asc\n    ```\n    \n    Add the repository to Apt sources:\n    \n    ```bash\n    echo \\\n    \"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \\\n    $(. /etc/os-release \u0026\u0026 echo \"$VERSION_CODENAME\") stable\" | \\\n    sudo tee /etc/apt/sources.list.d/docker.list \u003e /dev/null\n    sudo apt-get update\n    ```\n\n    To install the latest version:\n    \n    ```bash\n    sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin.\n    ```\n    \n    Verify that the Docker Engine installation is successful by running the `hello-world` image.\n    ```bash\n    sudo docker run hello-world\n    ```\n\n    This command downloads a test image and runs it in a container. When the container runs, it prints a confirmation message and exits.\n\n    You have now successfully installed and started Docker Engine.\n\n\n-----\n\n2.  **GOGS INSTALLATION and CONFIGURATION**\n    \n    Build a simple, stable and extensible self-hosted Git service.\n\n    Pull image from Docker Hub.\n    ```bash\n    docker pull gogs/gogs\n    ```\n    \n    Create local directory for volume.\n    ```bash    \n    mkdir -p /opt/gogs\n    ```\n\n    Use `docker run` for the first time.\n    ```bash\n    docker run --name gogs --restart always -p 10022:22 -p 3880:3000 -v /opt/gogs:/data gogs/gogs\n    ```\n\n    It is important to map the SSH service from the container to the host and set the appropriate SSH Port and URI settings when setting up Gogs for the first time. \n\n    To access and clone Git repositories with the above configuration you would use: \n    ```bash\n    git clone ssh://git@192.168.149.8:10022/odennav/on-prem-devops-vsphere.git\n    ```\n\n    Files will be store in local path of build-machine instance,  /opt/gogs in my case.\n    \n\n    For first-time run installation, install gogs with mysqllite3 \n\n    Initialize local repository and create README\n    ```bash\n    git init\n    touch README.md\n    echo \"Hello Gogs!\" \u003e README.md\n    ```\n\n    ```bash\n    git config --global user.email \"odennav@odennav.com\"\n    git config --global user.name \"odennav\"\n    git config --global credentials.helper store\n    ```\n   \n    Add all changes to staging area and commit\n    ```bash\n    git add .\n    git commit -m \"first commit\"\n    ```\n\n    Connect local repo with remote repository\n    ```bash\n    git remote add origin https://192.168.149.8:3880/odennav/on-prem-devops-vsphere.git\n    ```\n\n    Push commits to remote repository\n    ```bash\n    git push -u origin master\n    ```\n\n    Set tracking information for this branch\n    ```bash\n    git branch --set-upstream-to=origin/master master\n    ```\n\n-----\n\n3. **VAULT INSTALLATION and CONFIGURATION**\n\n   Install jq to format the JSON output for vault\n   ```bash\n   sudo apt-get install jq -y\n   ```\n\n   Secure, store, and tightly control access to tokens, passwords, certificates, and encryption keys in modern computing.\n   \n    Create the vault to generate unseal and root tokens. \n    Unseal the vault. \n      - 5 Unseal keys are created upon a vault initialization.\n      -\t3 Unseal keys are required to unseal the vault.\t\n      \n    Install the HashiCorp GPG key, verify the key's fingerprint, and install Vault.\n    \n    Update the package manager and install GPG and wget.\n    \n    ```bash\n    sudo apt update \u0026\u0026 sudo apt install gpg wget\n    ```\n\n    Download the keyring\n    \n    ```bash\n    wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg\n    ```\n\n    Verify the keyring\n    ```bash\n    gpg --no-default-keyring --keyring /usr/share/keyrings/hashicorp-archive-keyring.gpg --fingerprint\n    ```\n\n    Add the HashiCorp repository\n    ```bash\n    echo \"deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main\" | \\\n    sudo tee /etc/apt/sources.list.d/hashicorp.list\n    ```\n\n    Install Vault\n    ```bash\n    sudo apt update \u0026\u0026 sudo apt install vault\n    ```\n\n    Verify the installation\n    ```bash\n    vault --version\n    ```\n\n\n    **Configure Vault**\n\n    Create directory path for vault\n    ```bash\n    mkdir -p /opt/vault/data\n    ```\n  \n    Edit configuration file\n    ```bash\n    sudo nano /etc/vault.d/vault.hcl\n    ```\n\n    ```yaml\n    ui = true\n    api_addr = \"http://0.0.0.0:8200\"\n    log_level = \"INFO\"\n\n    storage \"file\" {\n      path = \"/opt/vault/data\"\n    }\n\n    listener \"tcp\" {\n      address = \"0.0.0.0:8200\"\n      tls_disable = true\n    }\n    ```\n\n    TLS disabled since we're just communicating with `build-machine`. Use TLS encryption for production environment.\n\n    \n    Enable vault service\n    ```bash\n    sudo systemctl enable vault\n    sudo systemctl start vault\n    ```\n    \n    Confirm vault service is running\n    ```bash\n    sudo systemctl status vault\n    ```\n    \n    Set environment variable for vault address\n    This will configure the Vault client to talk to the dev server.\n    ```bash\n    export VAULT_ADDR=\"http://192.168.149.8:8200\"\n    ```\n    \n    Generate Unseal keys and Root token\n    ```bash\n    vault operator init\n    ```    \n\n    Verify the server is running\n    ```bash\n    vault status\n    ```\n\n    Start Unseal process with Unseal Key 1\n    \n    ```bash\n    vault operator unseal\n    ```\n    Implement this three times with other Unseal keys to increment Unseal progress by 1 until `Sealed` state is `false`.\n\n    \n    **Vault policy requirements**\n\n     it is recommended that root tokens are only used for just enough initial setup or in emergencies.\n    \n     As a best practice, use tokens with appropriate set of policies based on your role in the organization.   \n\n     **Write a Vault Policy**\n\n     As an admin user, we must be able to:\n     - Read system health check\n     - Create and manage ACL policies broadly across Vault\n     - Enable and manage authentication methods broadly across Vault\n     - Manage the Key-Value secrets engine enabled at secret/ path\n\n     Define the admin policy in the file named `admin-policy.hcl`\n  \n     ```bash  \n     tee admin-policy.hcl \u003c\u003cEOF\n     \n     # Read system health check\n     path \"sys/health\"\n     {\n       capabilities = [\"read\", \"sudo\"]\n     }\n\n     # Create and manage ACL policies broadly across Vault\n\n     # List existing policies\n     path \"sys/policies/acl\"\n     {\n       capabilities = [\"list\"]\n     }\n\n     # Create and manage ACL policies\n     path \"sys/policies/acl/*\"\n     {\n       capabilities = [\"create\", \"read\", \"update\", \"delete\", \"list\", \"sudo\"]\n     }\n\n     # Enable and manage authentication methods broadly across Vault\n\n     # Manage auth methods broadly across Vault\n     path \"auth/*\"\n     {\n       capabilities = [\"create\", \"read\", \"update\", \"delete\", \"list\", \"sudo\"]\n     }\n\n     # Create, update, and delete auth methods\n     path \"sys/auth/*\"\n     {\n       capabilities = [\"create\", \"update\", \"delete\", \"sudo\"]\n     }\n\n     # List auth methods\n     path \"sys/auth\"\n     {\n       capabilities = [\"read\"]\n     }\n\n     # Enable and manage the key/value secrets engine at `secrets/` path\n\n     # List, create, update, delete and patch key/value secrets\n     path \"secrets/*\"\n     {\n       capabilities = [\"create\", \"read\", \"update\", \"delete\",\"patch\", \"list\", \"sudo\"]\n     }\n\n     # Manage secrets engines\n     path \"sys/mounts/*\"\n     {\n       capabilities = [\"create\", \"read\", \"update\", \"delete\", \"list\", \"sudo\"]\n     }\n\n     # List existing secrets engines.\n     path \"sys/mounts\"\n     {\n       capabilities = [\"read\"]\n     }\n     EOF\n     ```\n\n     **Create admin policy**\n     \n     Create a policy named admin with the policy defined in admin-policy.hcl\n     \n     ```bash\n     vault policy write admin admin-policy.hcl\n     ```\n\n     **Display New Policy**\n   \n     List all policies\n     ```bash\n     vault policy list\n     ```\n  \n     Read the `admin` policy.\n     Displays the paths and capabilities defined for this policy.\n     ```bash\n     vault policy read admin\n     ```\n\n     **Create new token**\n     \n     Create a token with the admin policy attached and store the token in the variable `ADMIN_TOKEN`\n\n     ```bash\n     ADMIN_TOKEN=$(vault token create -format=json -policy=\"admin\" | jq -r \".auth.client_token\")\n     ```\n\n     Display the `ADMIN_TOKEN`\n     ```bash\n     echo $ADMIN_TOKEN\n     ```\n\n     The admin policy defines capabilities for the paths.\n     Retrieve the capabilities of this token for the `secrets/` path.\n     ```bash\n     vault token capabilities $ADMIN_TOKEN secrets/*\n     ```\n\n     Set the `VAULT_TOKEN` environment variable to interact with Vault.\n     Setting this environment variable is a way to provide the token to Vault via CLI\n     ```bash\n     export VAULT_TOKEN=\"\u003cADMIN_TOKEN\u003e\"\n     ```    \n     \n    Append environment variables to `.profile`\n    \n    Ensure they're automatically set up and available in every new shell session.\n    \n    Fill in your ADMIN_TOKEN\n    ```bash\n    cat \u003c\u003c EOF | sudo tee -a ~/.profile\n    export VAULT_ADDR=\"http://192.168.149.8:8200\"\n    export VAULT_TOKEN=\"\u003cADMIN_TOKEN\u003e\"\n    \n    EOF\n    ```\n\n    **Create KV secrets engine**\n    \n    Enable the key/value secrets engine v1 at secrets/.\n    ```bash\n    vault secrets enable -path=\"secrets\" -description=\"Secret engine for Vsphere Connection\" kv\n    ```\n\n    List enabled secrets engines\n    ```bash\n    vault secrets list \n    ```\n\n    **Save multiple key-value pairs**\n    \n    Create a file named vsphere.json that defines `cluster`, `datacenter`, `esx_datastore`, `esx_host`, `server`, `username` and  `password` fields.\n    \n    ```bash\n    tee vsphere.json \u003c\u003cEOF\n    {\n      \"username\": \"administrator@vsphere.local\",\n      \"password\": \"**********\"\n      \"server\": \"vcenter-II\",\n      \"datacenter\": \"odennav-labs\",\n      \"cluster\": \"odennav-labs-cluster\",\n      \"esx_host\": \"ESXi-2\",\n      \"esx_datastore\": \"datastore-2\"\n    }\n    EOF\n    ```\n\n    **Create new secrets**\n    \n    Create a secret at path secrets/vmware with keys and values defined in `vsphere.json`.    \n    ```bash\n    vault kv put secrets/vmware @vsphere.json\n    ```\n    \n    **Disable Vault command history**\n\n    The option above ensures that the contents of the secret do not appear in the shell history. \n    The secret path would still be accessible through the shell history.\n\n    We can configure our shell to avoid logging any `vault` commands to your history.\n\n    In Bash profile, set the history to ignore all commands that start with `vault`.\n    \n    ```bash\n    cat \u003c\u003c EOF | sudo tee -a ~/.profile\n    export HISTIGNORE=\"\u0026:vault*\"\n\n    EOF\n    ```\n\n-----   \n    \n  **Web UI option to create Secrets engine**\n   \n  Access Vault Web UI at `http://192.168.149.8:8200/ui`\n  Use Token method and input `Root token value` to login.\n  \n  - Enable secrets `new engine`, click the `KV` radio button and specify mount `Path`\n  - Click `Enable Engine`\n  - Click `Create secret` and set `Path` for this secret.\n  - Enter key and value as `Secret Data`. You can add multiple key/value pairs.\n  - Click `Save`\n\n-----\n\n*Please ensure vault remains unsealed until build project is completed*\n\n\n\n4.  **PACKER INSTALLATION and CONFIGURATION**\n\n    Packer is a modular tool built by Hashicorp to create raw VM images and templates.\n    It's much more scalable than using a specific hypervisor tool.\n    \n    We'll be using Packer to create our VM images.\n\n    **Packer Installation**   \n   \n     Add the HashiCorp GPG key.\n     ```bash\n     curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -\n     ```\n\n     Add the official HashiCorp Linux repository\n     ```bash\n     sudo apt-add-repository \"deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main\"\n     ```\n\n     Update and install\n     ```bash\n     sudo apt-get update \u0026\u0026 sudo apt-get install packer\n     ```\n\n     Verify installation\n     ```bash\n     packer --version\n     ```\n     \n     **Image Build with Packer Template**\n     \n     With Packer installed, it is time to build our first image.\n     \n     A Packer template is a configuration file that defines the image you want to build and how to build it. \n\n     View the HCL block in `ubuntu20.pkr.hcl` template in `packer/ubuntu20` directory.\n \n     ```yaml\n     local \"vcenter_username\" {\n         exprssion = vault(*/secrets/data/vmware\", \"username\")\n         sensitive = true\n     }\n\n     local \"vcenter_password\" {\n         exprssion = vault(*/secrets/data/vmware\", \"password\")\n         sensitive = true\n     }\n\n     local \"vcenter_server\" {\n         exprssion = vault(*/secrets/data/vmware\", \"server\")\n         sensitive = true\n     }\n\n     local \"vcenter_datacenter\" {\n         exprssion = vault(*/secrets/data/vmware\", \"datacenter\")\n         sensitive = true\n     }\n\n     local \"vcenter_cluster\" {\n         exprssion = vault(*/secrets/data/vmware\", \"vcenter_cluster\")\n         sensitive = true\n     }\n\n     local \"esx_host\" {\n         exprssion = vault(*/secrets/data/vmware\", \"esx_host\")\n         sensitive = true\n     }\n\n     local \"esx_datastore\" {\n         exprssion = vault(*/secrets/data/vmware\", \"esx_datastore\")\n         sensitive = true\n     }\n\n     locals {\n         buildtime = formatdate(\"YYYY-MM-DD hh:mm ZZZ\", timestamp())\n     }\n\n     source \"vsphere-iso\" \"ubuntu20\" {\n         username = local.vcenter_username\n         password = local.vcenter_password         \n         vcenter_server = local.vcenter_server\n         datacenter = local.vcenter_datacenter\n         cluster = local.vcenter_cluster\n         host = local.esx_host\n         folder = \"Templates\"\n         datastore = local.esx_datastore\n         insecure_connection = \"true\"\n\n \n         remove_cdrom = true\n         convert_to_template = true\n         guest_os_type = \"ubuntu64Guest\"\n         notes = \"Built by Packer on ${local.buildtime}\"\n\n         vm_name = \"packer_ubuntu20\"\n         CPUs = \"1\"\n         RAM = \"8192\"\n         disk_controller_type = [\"pvscsi\"]\n         firmware = \"bios\"\n\n         storage {\n             disk_size = \"40960\"\n             disk_thin_provisioned = true\n         }\n\n\n         network_adapters {\n             network = \"VM Network\"\n             network_card = \"vmxnet3\"\n         }\n\n         iso_paths = [\n             “[$(local.esx_datastore)] ubuntu-20.04.6-live-server-amd64”\n         ]\n         iso_checksum = “none”\n\n         boot_order = \"disk,cdrom\"\n         boot_wait = “5s” \n         boot_command = [\n             \"\u003cesc\u003e\u003cesc\u003e\u003cesc\u003e\",\n             \"\u003center\u003e\u003cwait\u003e\",\n             \"/casper/vmlinuz \",\n             \"root=/dev/sr0 \",\n             \"initrd=/casper/initdrd \",\n             \"autoinstall \",\n             \"ds=nocloud-net;s=http://192.168.149.8:8600/\",\n             \"\u003center\u003e\n         ]\n         ip_wait_timeout = \"20m\"\n         ssh_password = \"ubuntu\"\n         ssh_username = \"ubuntu\"\n         ssh_timeout = \"20m\"\n         ssh_handshake_attempts = \"100\"\n         communicator = \"ssh\"\n\n         shutdown_command = \"sudo -S -E shutdown -P now\"\n         shutdown_timeout = \"15m\"\n\n         http_port_min = 8600\n         http_port_max = 8600\n         http_directory = \"./artifacts\"\n     }\n\n\n     build {\n         sources = [\"source.vsphere-iso.ubuntu20\"]\n\n         provisioner \"shell\" {\n             inline [\n                 \"echo Running updates\",\n                 \"sudo apt-get update\",\n                 \"sudo apt-get -y install open-vm-tools\",\n                 \"sudo touch /etc/cloud/cloud-init.disabled\", # Fixing issues with preboot DHCP\n                 \"sudo apt-get -y purge cloud-init\",\n                 \"sudo sed -i \\\"s/D /tmp 1777/#D /tmp 1777/\\\" /usr/lib/tmpfiles.d/tmp.conf\",\n                 \"sudo sed -i \\\"s/After=/After=dbus.service /\\\" /lib/systemd/system/open-vm-tools.service\",\n                 \"sudo rm -rf /etc/machine-id\", # next four lines fix same ip address being assigned in vmware\n                 \"sudo rm -rf /var/lib/dbus/machine-id\",\n                 \"sudo touch /etc/machine-id\",\n                 \"sudo ln -s /etc/machine-id /var/lib/dbus/machine-id\"\n             ]    \n         }     \n     }     \n     ```\n\n     Note the cloud config boot file, `packer/artifacts/user-data` and the autoinstall configuration in it.\n\n \n     Initialize your Packer configuration\n     ```bash\n     cd ~/on-prem-devops-vsphere/packer/ubuntu20/\n     packer init ubuntu20.pkr.hcl\n     ```\n  \n     Ensure template has consistent format\n     ```bash\n     packer fmt ubuntu20.pkr.hcl\n     ```\n\n     Ensure your configuration is syntactically valid and internally consistent \n     ```bash\n     packer validate ubuntu20.pkr.hcl\n     ```\n\n     Build image\n     ```bash\n     packer build ubuntu20.pkr.hcl\n     ```\n      \n     View `packer_ubuntu20` VM template created in vcenter vsphere web client.\n\n-----\n \n5. **TERRAFORM INSTALLATION and VM DEPLOYMENT**\n\n\n   **Install Terraform**\n\n   Ensure that your system is up to date and you have installed the gnupg, software-properties-common, and curl packages installed.\n   \n   We'll use these packages to verify HashiCorp's GPG signature and install HashiCorp's Debian package repository.\n   ```bash\n   sudo apt-get update \u0026\u0026 sudo apt-get install -y gnupg software-properties-common\n   ```\n\n   Install the HashiCorp GPG key.\n   ```bash\n   wget -O- https://apt.releases.hashicorp.com/gpg | \\\n   gpg --dearmor | \\\n   sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg \u003e /dev/null\n   ```\n   \n   Verify the key's fingerprint. The `gpg` command will report the key fingerprint.\n   ```bash\n   gpg --no-default-keyring \\\n   --keyring /usr/share/keyrings/hashicorp-archive-keyring.gpg \\\n   --fingerprint\n   ```\n\n   Add the official HashiCorp repository to your system. \n   ```bash\n   echo \"deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] \\\n   https://apt.releases.hashicorp.com $(lsb_release -cs) main\" | \\\n   sudo tee /etc/apt/sources.list.d/hashicorp.list\n   ```\n\n   Download the package information from HashiCorp.\n   ```bash\n   sudo apt update\n   ```\n\n   Install Terraform from the new repository.\n   ```bash\n   sudo apt-get install terraform\n   ```\n\n   Verify that the installation\n   ```bash\n   terraform version\n   ```\n\n \n   **Provision VM Instances on VMware vSphere**\n   \n   Initialize the configuration directory `on-prem-devops-vsphere/terraform/` and install the `vSphere` providers defined in the configuration.\n   ```bash\n   cd on-prem-devops-vsphere/terraform/\n   terraform init\n   ```\n\n   Format your configuration\n   ```bash\n   terraform fmt\n   ```\n\n   Validate your configuration\n   ```bash\n   terraform validate\n   ```\n\n   Create an execution plan that describes the changes terraform will make to the infrastructure\n   ```bash\n   terraform plan\n   ```\n\n   Apply the configuration and provision the VMs\n   ```bash\n   terraform apply\n   ```\n \n   Note, the ansible inventory is built dynamically by terraform using the resource `\"local_file\" \"ansible_inventory\"` in the `myapp.tf` file.  \n   \n   \n   Here is the `.tpl` terraform template code for the logical groupings.\n   ```text\n   [lb]\n   %{ for ip in lb_ip ~}\n   ${ip} domain=odennav.com\n   %{ endfor ~}\n\n   [ws]\n   %{ for ip in ws_ip ~}\n   ${ip} domain=odennav.com\n   %{ endfor ~}\n   ```\n\n\n-----\n\n6. **ANSIBLE INSTALLATION and MACHINE CONFIGURATION**\n   \n   **Install Ansible**\n   \n   Configure the PPA on your system and install Ansible\n   ```bash\n   sudo apt update\n   sudo apt install software-properties-common\n   sudo add-apt-repository --yes --update ppa:ansible/ansible\n   sudo apt install ansible\n   ```\n\n   **Run ansible playbooks**\n   \n   Bootstrap each web server \n   Default user `ubuntu` used as `remote_user` in `on-prem-devops-vsphere/ansible/ansible.cfg` is not created yet\n   ```bash\n   ansible-playbook -u ubuntu add_user.yaml\n   ```\n\n   Remove Ubuntu default user from each web server\n   ```bash\n   ansible-playbook remove_ubuntu.yaml\n   ```\n\n   **Load-balanced application deployment with ansible**\n   \n   Deploy and configure a load-balancer and multiple servers on vSphere VMs in datacenter cluster.\n\n   Install Nginx on vSphere Virtual Machines and configure as web servers\n   ```bash\n   ansible-playbook install_nginx_ws.yaml\n   ```\n\n   Install Nginx on vSphere Virtual Machine and configure as load balancer\n   ```bash\n   ansible-playbook install_nginx_lb.yaml\n   ```\n\n   **Verify web server and load balancer installation**\n   \n   Make `http` requests to each webserver and the load balancer using `curl`\n   \n\n-----\n\n##  Automated Pipeline\n\n1.  **Jenkins Setup**\n\n    Jenkins is a self-contained, open source automation server which can be used to automate all sorts of tasks related to building, testing, and delivering or deploying software.\n\n    Our Jenkins Workflow:\n    - Define when to run job\n    - Download latest source code updates\n    - Run specified set of commands\n    - Create output artifacts(VM templates)\n    - Save console output for future debugging \u0026 troubleshooting\n\n    Generate ssh key-pair\n    ```bash\n    ssh-keygen -t rsa -b 4096 \n    ```\n\n    Create local jenkins directories for volume mapping\n    ```bash\n    sudo mkdir -p /opt/jenkins/bin\n    sudo mkdir -p /opt/jenkins/jenkins-docker-certs\n    ```\n    \n    Locate executable files of packer command\n    ```bash\n    which packer\n    ```\n\n    Locate executable files of terraform command\n    ```bash\n    which terraform\n    ```\n\n    Copy binaries to jenkins directory volume\n    ```bash\n    sudo cp -rf /usr/bin/packer /opt/jenkins/bin\n    sudo cp -rf /usr/bin/terraform /opt/jenkins/bin\n    ```\n \n    Confirm jenkins user permissions\n    ```bash\n    sudo chown -R odennav:odennav /opt/jenkins\n    ```\n\n    Run jenkins container\n    ```bash\n    sudo docker run \\\n      -d --name jenkins \\\n      --restart always \\\n      -v /opt/jenkins/jenkins-docker-certs:/certs/client \\\n      -v /opt/jenkins:/var/jenkins_home \\\n      -p 8080:8080 \\\n      -p 5000:5000 \\\n      -p 8600:8600 \\\n      jenkins/jenkins:lts-jdk17 \n    ```\n\n2.  **Unlock Jenkins**\n    \n    When you first access a new Jenkins instance, you are asked to unlock it using an automatically-generated password.\n\n    Browse to `http://192.168.149.8:8080` and wait until the Unlock Jenkins page appears.\n    \n    Discover `Administrator password` to unlock jenkins\n    Copy and paste password into setup wizard\n    ```bash\n    sudo cat /opt/jenkins/secrets/initialAdminPassword \n    ```\n\n   \n    After unlocking jenkins, click one of the options: `Install suggested plugins` to install the recommended set of plugins based on most common use cases.\n    \n3.  **Create the First Administrator User**\n    \n    Finally, after customizing Jenkins with plugins, Jenkins asks you to create your first administrator user.\n    Specify details for your administrator user, then save and continue setup.\n \n    When the `Jenkins is ready` page appears, click `Start using Jenkins`\n    \n    Notes:\n    - This page may indicate Jenkins is almost ready! instead and if so, click Restart.\n    - If the page does not automatically refresh after a minute, use your web browser to refresh the page manually.\n\n\n4.  **Add Gogs Plugin to Jenkins**\n    \n    We'll need to extend jenkins functionality with Gogs plugin.\n    \n    - At Jenkins Dashboard appears, Go to **Manage Jenkins** \u003e **Manage Plugins**\n    - Select **Available** tab and search for `gogs`\n    - Select box button of available gogs plugin\n    - Click on `Download now and install after restart` and select box button `Restart jenkins when installation is complete and no jobs are running`\n    - Wait while jenkins restarts.\n    \n    \n5.  **Configure Credentials**\n    \n    Next step is to create vault credentials in Jenkins.\n    \n    Login to Jenkins UI\n    Go to **Manage Jenkins** \u003e **Manage Credentials**\n    \n    Select `Jenkins` store\n    Under **System** tab, select `Global credentials(unrestricted)`\n    \n    Check left tab and click on `Add Credentials`, then choose the following:\n    - Kind: *secret text*\n    - Secret: \n    - ID: *vault_token*\n    \n    Get your ADMIN_TOKEN to fill in `Secret` field\n    ```bash\n    echo $ADMIN_TOKEN\n    ```\n\n    \n6.  **Add Private Key as Gogs Credential to Jenkins**\n    \n    Go to **Manage Jenkins** \u003e **Manage Credentials**\n    \n    Select `Jenkins` store\n    Under **System** tab, select `Global credentials(unrestricted)`\n    \n    Check left tab and click on `Add Credentials`, then choose the following:\n    - Kind: *SSH Username with private key*\n    - ID: *id_rsa*\n    - Username: *odennav*\n\n    Under `Private Key`, select `Enter directly` radio button\n    Copy your private key from here:\n    ```bash\n    cat ~/on-prem-devops-vsphere/keys/id_rsa\n    ```\n    Then paste in `Key` field and click `OK`\n    \n   \n    \n7.  **Add Public SSH Key to Gogs Server**\n    \n    We'll add our public key to Gogs to ensure ssh authentication with Jenkins.\n\n    Go to gogs settings page at `http://192.168.149.8:3880/user/settings`  \n    \n    Select `SSH Keys` tab and click on `Add Key` on `Manage SSH Keys` tab\n    \n    Enter `Key Name` as `id_rsa`\n\n    ```bash\n    cat ~/on-prem-devops-vsphere/keys/id_rsa.pub | tr -d '\\n'\n    ```\n    Copy your public key, paste into `Content` field and click `Add Key`\n\n\n\n8.  **Automate CI/CD Pipeline**\n \n    Go to Jenkins Dashboard and select `New Item` on left tab.\n\n    Enter an item name, select `Freestyle project` and click `OK`\n    \n    When `General` setup page appears, scroll down to `Source Code Management` under `Gogs Webhook` tab and select radio button for `Git`\n    \n    Fill the `Repository URL` field with:\n    ```bash\n    ssh://git@192.168.149.8:2222/odennav/on-prem-devops-vsphere.git\n    ```\n    Select `odennav` credentials for SSH keys\n     \n    Scroll down to `Build Environment` section under the `Build Triggers` tab\n    \n    Select box button `Use secret text(s) or file(s)`\n    \n    Add Binding for `Secret text`, enter variable name as `vault_token` and select credentials previously created `vault_token`\n\n\n    Sroll down to `Build Environment` section under the `Build Environment` tab\n    \n    Click on `Add build step` drop down and select `Execute shell`\n    \n    Fill in the following into `Command` field:\n    \n    ```bash\n    export VAULT_ADDR=http://192.168.149.8:8200\"\n    export VAULT_TOKEN=$vault_token\n    export PACKER_LOG=1\n    cd \"./packer/\"\n    /var/jenkins_home/bin/packer init ubuntu20.pkr.hcl\n    /var/jenkins_home/bin/packer build -force ubuntu20.pkr.hcl\n    export TF_LOG=INFO\n    cd \"./packer/terraform\"\n    /var/jenkins_home/bin/terraform init\n    /var/jenkins_home/bin/terraform validate\n    /var/jenkins_home/bin/terraform plan\n    /var/jenkins_home/bin/terraform apply -auto-approve\n    cd \"./packer/ansible\"\n    /var/jenkins_home/bin/ansible ansible-playbook -u ubuntu add_user.yaml\n    /var/jenkins_home/bin/ansible ansible-playbook remove_ubuntu.yaml\n    /var/jenkins_home/bin/ansible ansible-playbook install_nginx_ws.yaml\n    /var/jenkins_home/bin/ansible ansible-playbook install_nginx_lb.yaml\n    ```\n    Click `Save`\n\n    The Jenkins dashboard will show your project build just created.\n    \n    Select `Build Now` at left tab and the build job will show up under `Build History` section\n    \n    Click on this job, select `Console Output` at left tab and view the build job in real time.\n\n\n    When build job is completed, check the vSphere datacenter and confirm the following are deployed:\n    - VM template \n    - web01\n    - web02\n    - web03\n    - lb01\n\n   **Browse website deployed on web servers through load balancer**\n\n   View the Level website on your browser.\n\n   ![](https://github.com/odennav/on-prem-devops-vsphere/blob/main/docs/2095-level.jpg)\n\n-----\nEnjoy!\n   \n    \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fodennav%2Fon-premise-vsphere-devops","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fodennav%2Fon-premise-vsphere-devops","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fodennav%2Fon-premise-vsphere-devops/lists"}