Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/odennav/on-premise-vsphere-devops
Automate deployment of a load balanced application on vSphere datacenter with Ansible, Vault, Packer, Terraform and Jenkins.
https://github.com/odennav/on-premise-vsphere-devops
ansible datastore devops docker esxi jenkins nginx-proxy nginx-server on-prem on-premise packer terraform vault vcenter-server vmfs vmware vsphere
Last synced: 26 days ago
JSON representation
Automate deployment of a load balanced application on vSphere datacenter with Ansible, Vault, Packer, Terraform and Jenkins.
- Host: GitHub
- URL: https://github.com/odennav/on-premise-vsphere-devops
- Owner: odennav
- License: apache-2.0
- Created: 2024-04-15T00:29:56.000Z (9 months ago)
- Default Branch: main
- Last Pushed: 2024-04-26T20:40:20.000Z (8 months ago)
- Last Synced: 2024-04-26T21:35:02.859Z (8 months ago)
- Topics: ansible, datastore, devops, docker, esxi, jenkins, nginx-proxy, nginx-server, on-prem, on-premise, packer, terraform, vault, vcenter-server, vmfs, vmware, vsphere
- Language: HCL
- Homepage:
- Size: 16.3 MB
- Stars: 1
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# On-Prem DevOps with VMware vSphere
Deploy a load-balanced application using VMware vSphere, Docker, Gogs, Ansible, Vault, Packer, Terraform, Jenkins and Nginx.
## Prerequisites
- Deploy self hosted vSphere datacenter and compute cluster with 2 ESXi hosts.
- Enable vSAN, DRS and SDRS.
- Deploy vCenter server appliance on ESXi host.
- Provision a Build machine on next ESXi host with Ubuntu 20.04 server.
- Git bash or linux terminal on local machine.
- Assume IPv4 address of Build machine VM is `192.168.149.8` and it's hostname is `build-machine`
- Check this VMware datacenter lab [guide](https://github.com/odennav/vmware-sddc-private-cloud)
# Getting Started
Two Pipelines will be implemented:
- Manual Pipeline
- Automated Pipeline
## Manual Pipeline
This workflow involves the following steps:
- Docker Installation
- Gogs(source control) Installation and Configuration
- Vault Installation and Configuration
- Packer Installation and Configuration
- Terraform Installation and VM Deployment
- Ansible Installation and Machine Configuration
-----
1. **DOCKER INSTALLATION**
To install Docker Engine for the first time on build-machine, we'll set up the Docker repository.
Afterward, we can install and update Docker from the repository.
Add Docker's official GPG key:
```bash
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
```
Add the repository to Apt sources:
```bash
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
```
To install the latest version:
```bash
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin.
```
Verify that the Docker Engine installation is successful by running the `hello-world` image.
```bash
sudo docker run hello-world
```
This command downloads a test image and runs it in a container. When the container runs, it prints a confirmation message and exits.
You have now successfully installed and started Docker Engine.
-----
2. **GOGS INSTALLATION and CONFIGURATION**
Build a simple, stable and extensible self-hosted Git service.
Pull image from Docker Hub.
```bash
docker pull gogs/gogs
```
Create local directory for volume.
```bash
mkdir -p /opt/gogs
```
Use `docker run` for the first time.
```bash
docker run --name gogs --restart always -p 10022:22 -p 3880:3000 -v /opt/gogs:/data gogs/gogs
```
It is important to map the SSH service from the container to the host and set the appropriate SSH Port and URI settings when setting up Gogs for the first time.
To access and clone Git repositories with the above configuration you would use:
```bash
git clone ssh://[email protected]:10022/odennav/on-prem-devops-vsphere.git
```
Files will be store in local path of build-machine instance, /opt/gogs in my case.
For first-time run installation, install gogs with mysqllite3
Initialize local repository and create README
```bash
git init
touch README.md
echo "Hello Gogs!" > README.md
```
```bash
git config --global user.email "[email protected]"
git config --global user.name "odennav"
git config --global credentials.helper store
```
Add all changes to staging area and commit
```bash
git add .
git commit -m "first commit"
```
Connect local repo with remote repository
```bash
git remote add origin https://192.168.149.8:3880/odennav/on-prem-devops-vsphere.git
```
Push commits to remote repository
```bash
git push -u origin master
```
Set tracking information for this branch
```bash
git branch --set-upstream-to=origin/master master
```
-----
3. **VAULT INSTALLATION and CONFIGURATION**
Install jq to format the JSON output for vault
```bash
sudo apt-get install jq -y
```
Secure, store, and tightly control access to tokens, passwords, certificates, and encryption keys in modern computing.
Create the vault to generate unseal and root tokens.
Unseal the vault.
- 5 Unseal keys are created upon a vault initialization.
- 3 Unseal keys are required to unseal the vault.
Install the HashiCorp GPG key, verify the key's fingerprint, and install Vault.
Update the package manager and install GPG and wget.
```bash
sudo apt update && sudo apt install gpg wget
```
Download the keyring
```bash
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
```
Verify the keyring
```bash
gpg --no-default-keyring --keyring /usr/share/keyrings/hashicorp-archive-keyring.gpg --fingerprint
```
Add the HashiCorp repository
```bash
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | \
sudo tee /etc/apt/sources.list.d/hashicorp.list
```
Install Vault
```bash
sudo apt update && sudo apt install vault
```
Verify the installation
```bash
vault --version
```
**Configure Vault**
Create directory path for vault
```bash
mkdir -p /opt/vault/data
```
Edit configuration file
```bash
sudo nano /etc/vault.d/vault.hcl
```
```yaml
ui = true
api_addr = "http://0.0.0.0:8200"
log_level = "INFO"
storage "file" {
path = "/opt/vault/data"
}
listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = true
}
```
TLS disabled since we're just communicating with `build-machine`. Use TLS encryption for production environment.
Enable vault service
```bash
sudo systemctl enable vault
sudo systemctl start vault
```
Confirm vault service is running
```bash
sudo systemctl status vault
```
Set environment variable for vault address
This will configure the Vault client to talk to the dev server.
```bash
export VAULT_ADDR="http://192.168.149.8:8200"
```
Generate Unseal keys and Root token
```bash
vault operator init
```
Verify the server is running
```bash
vault status
```
Start Unseal process with Unseal Key 1
```bash
vault operator unseal
```
Implement this three times with other Unseal keys to increment Unseal progress by 1 until `Sealed` state is `false`.
**Vault policy requirements**
it is recommended that root tokens are only used for just enough initial setup or in emergencies.
As a best practice, use tokens with appropriate set of policies based on your role in the organization.
**Write a Vault Policy**
As an admin user, we must be able to:
- Read system health check
- Create and manage ACL policies broadly across Vault
- Enable and manage authentication methods broadly across Vault
- Manage the Key-Value secrets engine enabled at secret/ path
Define the admin policy in the file named `admin-policy.hcl`
```bash
tee admin-policy.hcl <",
"",
"/casper/vmlinuz ",
"root=/dev/sr0 ",
"initrd=/casper/initdrd ",
"autoinstall ",
"ds=nocloud-net;s=http://192.168.149.8:8600/",
"
]
ip_wait_timeout = "20m"
ssh_password = "ubuntu"
ssh_username = "ubuntu"
ssh_timeout = "20m"
ssh_handshake_attempts = "100"
communicator = "ssh"
shutdown_command = "sudo -S -E shutdown -P now"
shutdown_timeout = "15m"
http_port_min = 8600
http_port_max = 8600
http_directory = "./artifacts"
}
build {
sources = ["source.vsphere-iso.ubuntu20"]
provisioner "shell" {
inline [
"echo Running updates",
"sudo apt-get update",
"sudo apt-get -y install open-vm-tools",
"sudo touch /etc/cloud/cloud-init.disabled", # Fixing issues with preboot DHCP
"sudo apt-get -y purge cloud-init",
"sudo sed -i \"s/D /tmp 1777/#D /tmp 1777/\" /usr/lib/tmpfiles.d/tmp.conf",
"sudo sed -i \"s/After=/After=dbus.service /\" /lib/systemd/system/open-vm-tools.service",
"sudo rm -rf /etc/machine-id", # next four lines fix same ip address being assigned in vmware
"sudo rm -rf /var/lib/dbus/machine-id",
"sudo touch /etc/machine-id",
"sudo ln -s /etc/machine-id /var/lib/dbus/machine-id"
]
}
}
```
Note the cloud config boot file, `packer/artifacts/user-data` and the autoinstall configuration in it.
Initialize your Packer configuration
```bash
cd ~/on-prem-devops-vsphere/packer/ubuntu20/
packer init ubuntu20.pkr.hcl
```
Ensure template has consistent format
```bash
packer fmt ubuntu20.pkr.hcl
```
Ensure your configuration is syntactically valid and internally consistent
```bash
packer validate ubuntu20.pkr.hcl
```
Build image
```bash
packer build ubuntu20.pkr.hcl
```
View `packer_ubuntu20` VM template created in vcenter vsphere web client.
-----
5. **TERRAFORM INSTALLATION and VM DEPLOYMENT**
**Install Terraform**
Ensure that your system is up to date and you have installed the gnupg, software-properties-common, and curl packages installed.
We'll use these packages to verify HashiCorp's GPG signature and install HashiCorp's Debian package repository.
```bash
sudo apt-get update && sudo apt-get install -y gnupg software-properties-common
```
Install the HashiCorp GPG key.
```bash
wget -O- https://apt.releases.hashicorp.com/gpg | \
gpg --dearmor | \
sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg > /dev/null
```
Verify the key's fingerprint. The `gpg` command will report the key fingerprint.
```bash
gpg --no-default-keyring \
--keyring /usr/share/keyrings/hashicorp-archive-keyring.gpg \
--fingerprint
```
Add the official HashiCorp repository to your system.
```bash
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] \
https://apt.releases.hashicorp.com $(lsb_release -cs) main" | \
sudo tee /etc/apt/sources.list.d/hashicorp.list
```
Download the package information from HashiCorp.
```bash
sudo apt update
```
Install Terraform from the new repository.
```bash
sudo apt-get install terraform
```
Verify that the installation
```bash
terraform version
```
**Provision VM Instances on VMware vSphere**
Initialize the configuration directory `on-prem-devops-vsphere/terraform/` and install the `vSphere` providers defined in the configuration.
```bash
cd on-prem-devops-vsphere/terraform/
terraform init
```
Format your configuration
```bash
terraform fmt
```
Validate your configuration
```bash
terraform validate
```
Create an execution plan that describes the changes terraform will make to the infrastructure
```bash
terraform plan
```
Apply the configuration and provision the VMs
```bash
terraform apply
```
Note, the ansible inventory is built dynamically by terraform using the resource `"local_file" "ansible_inventory"` in the `myapp.tf` file.
Here is the `.tpl` terraform template code for the logical groupings.
```text
[lb]
%{ for ip in lb_ip ~}
${ip} domain=odennav.com
%{ endfor ~}
[ws]
%{ for ip in ws_ip ~}
${ip} domain=odennav.com
%{ endfor ~}
```
-----
6. **ANSIBLE INSTALLATION and MACHINE CONFIGURATION**
**Install Ansible**
Configure the PPA on your system and install Ansible
```bash
sudo apt update
sudo apt install software-properties-common
sudo add-apt-repository --yes --update ppa:ansible/ansible
sudo apt install ansible
```
**Run ansible playbooks**
Bootstrap each web server
Default user `ubuntu` used as `remote_user` in `on-prem-devops-vsphere/ansible/ansible.cfg` is not created yet
```bash
ansible-playbook -u ubuntu add_user.yaml
```
Remove Ubuntu default user from each web server
```bash
ansible-playbook remove_ubuntu.yaml
```
**Load-balanced application deployment with ansible**
Deploy and configure a load-balancer and multiple servers on vSphere VMs in datacenter cluster.
Install Nginx on vSphere Virtual Machines and configure as web servers
```bash
ansible-playbook install_nginx_ws.yaml
```
Install Nginx on vSphere Virtual Machine and configure as load balancer
```bash
ansible-playbook install_nginx_lb.yaml
```
**Verify web server and load balancer installation**
Make `http` requests to each webserver and the load balancer using `curl`
-----
## Automated Pipeline
1. **Jenkins Setup**
Jenkins is a self-contained, open source automation server which can be used to automate all sorts of tasks related to building, testing, and delivering or deploying software.
Our Jenkins Workflow:
- Define when to run job
- Download latest source code updates
- Run specified set of commands
- Create output artifacts(VM templates)
- Save console output for future debugging & troubleshooting
Generate ssh key-pair
```bash
ssh-keygen -t rsa -b 4096
```
Create local jenkins directories for volume mapping
```bash
sudo mkdir -p /opt/jenkins/bin
sudo mkdir -p /opt/jenkins/jenkins-docker-certs
```
Locate executable files of packer command
```bash
which packer
```
Locate executable files of terraform command
```bash
which terraform
```
Copy binaries to jenkins directory volume
```bash
sudo cp -rf /usr/bin/packer /opt/jenkins/bin
sudo cp -rf /usr/bin/terraform /opt/jenkins/bin
```
Confirm jenkins user permissions
```bash
sudo chown -R odennav:odennav /opt/jenkins
```
Run jenkins container
```bash
sudo docker run \
-d --name jenkins \
--restart always \
-v /opt/jenkins/jenkins-docker-certs:/certs/client \
-v /opt/jenkins:/var/jenkins_home \
-p 8080:8080 \
-p 5000:5000 \
-p 8600:8600 \
jenkins/jenkins:lts-jdk17
```
2. **Unlock Jenkins**
When you first access a new Jenkins instance, you are asked to unlock it using an automatically-generated password.
Browse to `http://192.168.149.8:8080` and wait until the Unlock Jenkins page appears.
Discover `Administrator password` to unlock jenkins
Copy and paste password into setup wizard
```bash
sudo cat /opt/jenkins/secrets/initialAdminPassword
```
After unlocking jenkins, click one of the options: `Install suggested plugins` to install the recommended set of plugins based on most common use cases.
3. **Create the First Administrator User**
Finally, after customizing Jenkins with plugins, Jenkins asks you to create your first administrator user.
Specify details for your administrator user, then save and continue setup.
When the `Jenkins is ready` page appears, click `Start using Jenkins`
Notes:
- This page may indicate Jenkins is almost ready! instead and if so, click Restart.
- If the page does not automatically refresh after a minute, use your web browser to refresh the page manually.
4. **Add Gogs Plugin to Jenkins**
We'll need to extend jenkins functionality with Gogs plugin.
- At Jenkins Dashboard appears, Go to **Manage Jenkins** > **Manage Plugins**
- Select **Available** tab and search for `gogs`
- Select box button of available gogs plugin
- Click on `Download now and install after restart` and select box button `Restart jenkins when installation is complete and no jobs are running`
- Wait while jenkins restarts.
5. **Configure Credentials**
Next step is to create vault credentials in Jenkins.
Login to Jenkins UI
Go to **Manage Jenkins** > **Manage Credentials**
Select `Jenkins` store
Under **System** tab, select `Global credentials(unrestricted)`
Check left tab and click on `Add Credentials`, then choose the following:
- Kind: *secret text*
- Secret:
- ID: *vault_token*
Get your ADMIN_TOKEN to fill in `Secret` field
```bash
echo $ADMIN_TOKEN
```
6. **Add Private Key as Gogs Credential to Jenkins**
Go to **Manage Jenkins** > **Manage Credentials**
Select `Jenkins` store
Under **System** tab, select `Global credentials(unrestricted)`
Check left tab and click on `Add Credentials`, then choose the following:
- Kind: *SSH Username with private key*
- ID: *id_rsa*
- Username: *odennav*
Under `Private Key`, select `Enter directly` radio button
Copy your private key from here:
```bash
cat ~/on-prem-devops-vsphere/keys/id_rsa
```
Then paste in `Key` field and click `OK`
7. **Add Public SSH Key to Gogs Server**
We'll add our public key to Gogs to ensure ssh authentication with Jenkins.
Go to gogs settings page at `http://192.168.149.8:3880/user/settings`
Select `SSH Keys` tab and click on `Add Key` on `Manage SSH Keys` tab
Enter `Key Name` as `id_rsa`
```bash
cat ~/on-prem-devops-vsphere/keys/id_rsa.pub | tr -d '\n'
```
Copy your public key, paste into `Content` field and click `Add Key`
8. **Automate CI/CD Pipeline**
Go to Jenkins Dashboard and select `New Item` on left tab.
Enter an item name, select `Freestyle project` and click `OK`
When `General` setup page appears, scroll down to `Source Code Management` under `Gogs Webhook` tab and select radio button for `Git`
Fill the `Repository URL` field with:
```bash
ssh://[email protected]:2222/odennav/on-prem-devops-vsphere.git
```
Select `odennav` credentials for SSH keys
Scroll down to `Build Environment` section under the `Build Triggers` tab
Select box button `Use secret text(s) or file(s)`
Add Binding for `Secret text`, enter variable name as `vault_token` and select credentials previously created `vault_token`
Sroll down to `Build Environment` section under the `Build Environment` tab
Click on `Add build step` drop down and select `Execute shell`
Fill in the following into `Command` field:
```bash
export VAULT_ADDR=http://192.168.149.8:8200"
export VAULT_TOKEN=$vault_token
export PACKER_LOG=1
cd "./packer/"
/var/jenkins_home/bin/packer init ubuntu20.pkr.hcl
/var/jenkins_home/bin/packer build -force ubuntu20.pkr.hcl
export TF_LOG=INFO
cd "./packer/terraform"
/var/jenkins_home/bin/terraform init
/var/jenkins_home/bin/terraform validate
/var/jenkins_home/bin/terraform plan
/var/jenkins_home/bin/terraform apply -auto-approve
cd "./packer/ansible"
/var/jenkins_home/bin/ansible ansible-playbook -u ubuntu add_user.yaml
/var/jenkins_home/bin/ansible ansible-playbook remove_ubuntu.yaml
/var/jenkins_home/bin/ansible ansible-playbook install_nginx_ws.yaml
/var/jenkins_home/bin/ansible ansible-playbook install_nginx_lb.yaml
```
Click `Save`
The Jenkins dashboard will show your project build just created.
Select `Build Now` at left tab and the build job will show up under `Build History` section
Click on this job, select `Console Output` at left tab and view the build job in real time.
When build job is completed, check the vSphere datacenter and confirm the following are deployed:
- VM template
- web01
- web02
- web03
- lb01
**Browse website deployed on web servers through load balancer**
View the Level website on your browser.
-----
Enjoy!