Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/old-xebis/infrastructure-template
Template to automate GitOps and IaC in a cloud. GitLab CI manages static and dynamic environments, which are created, updated, and destroyed by Terraform, then set up by cloud-init and Ansible.
https://github.com/old-xebis/infrastructure-template
automation continuous-delivery continuous-integration environments gitlab gitlab-ci gitops hetzner hetzner-cloud iac pull-request repository-template terraform terraform-hetzner-cloud
Last synced: about 2 months ago
JSON representation
Template to automate GitOps and IaC in a cloud. GitLab CI manages static and dynamic environments, which are created, updated, and destroyed by Terraform, then set up by cloud-init and Ansible.
- Host: GitHub
- URL: https://github.com/old-xebis/infrastructure-template
- Owner: old-xebis
- License: mit
- Created: 2021-11-06T17:03:47.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2024-01-06T22:26:40.000Z (12 months ago)
- Last Synced: 2024-08-02T18:42:43.516Z (5 months ago)
- Topics: automation, continuous-delivery, continuous-integration, environments, gitlab, gitlab-ci, gitops, hetzner, hetzner-cloud, iac, pull-request, repository-template, terraform, terraform-hetzner-cloud
- Language: Shell
- Homepage:
- Size: 859 KB
- Stars: 10
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
# Infrastructure Template
[](https://github.com/pre-commit/pre-commit)
[](https://conventionalcommits.org)
[](https://github.com/semantic-release/semantic-release)
[](https://gitlab.com/xebis/infrastructure-template/-/commits/main)
Template to automate GitOps and IaC in a cloud. GitLab CI manages static and dynamic environments, which are created, updated, and destroyed by Terraform, then set up by cloud-init and Ansible.
**The project is under active development.** The project is a fork of [xebis/repository-template](https://github.com/xebis/repository-template).
## The Goal
The goal is to have a GitOps repository to automatically handle environments life cycle - its creation, update, configuration, and eventually destroy as well.
> GitOps = IaC + MRs + CI/CD
_[GitLab: What is GitOps?](https://about.gitlab.com/topics/gitops/)_
## Table of Contents
- [Features](#features)
- [Environment](#environment)
- [Caveats](#caveats)
- [Images](#images)
- [Installation and Configuration](#installation-and-configuration)
- [Set up GitLab CI](#set-up-gitlab-ci)
- [Set up Local Usage](#set-up-local-usage)
- [Usage](#usage)
- [GitLab CI](#gitlab-ci)
- [Local Usage](#local-usage)
- [Terraform Configuration Documentation](#terraform-configuration-documentation)
- [Contributing](#contributing)
- [Testing](#testing)
- [Test at Docker Container](#test-at-docker-container)
- [To-Do list](#to-do-list)
- [Roadmap](#roadmap)
- [Credits and Acknowledgments](#credits-and-acknowledgments)
- [Copyright and Licensing](#copyright-and-licensing)
- [Changelog and News](#changelog-and-news)
- [Notes and References](#notes-and-references)
- [Dependencies](#dependencies)
- [Recommendations](#recommendations)
- [Suggestions](#suggestions)
- [Further Reading](#further-reading)
## Features
Optimized for [GitHub flow](https://guides.github.com/introduction/flow/), easily adjustable to [GitLab flow](https://docs.gitlab.com/ee/topics/gitlab_flow.html) or any other workflow.
Automatically checks conventional commits, validates Markdown, YAML, shell scripts, Terraform (HCL), runs static security analysis, terraform-doc, tests, deployments, releases, and so on. See [GitHub - xebis/repository-template: Well-manageable and well-maintainable repository template.](https://github.com/xebis/repository-template) and [Notes And References](#notes-and-references) for full feature list.
Environments are managed in stages:
- **_Deploy_**: overarching name for **provision** and **install** stages
- **Provision**: environment is provisioned by Terraform at Hetzner Cloud and pre-configured by Cloud-init
- **Install**: environment is installed by Ansible over SSH
- **Destroy** (only dynamic environments): environment is removed by Terraform from Hetzner Cloud
Automatically managed environments:
- On _release_ tag runs **production** environment stages
- On `main` branch commit runs **staging** environment stages
- Releases and creates _release_ tag when a commit starting `feat` or `fix` is present in the history from the previous release
- On _pre-release_ tag runs **testing/_tag_** environment stages, and plans automatic destroy after 1 week (earlier manual destruction possible)
- On _non-_`main` branch commit under certain conditions runs **development/_branch_** environment stages, and plans automatic destroy after 1 day (earlier manual destruction possible):
- It runs when the environment already exists or existed in the past (when Terraform backend returns HTTP status code `200 OK` for the environment state file)
- It runs when the pipeline is run by the _pipelines API_, _GitLab ChatOps_, created by using _trigger token_, created by using the **Run pipeline** _button in the GitLab UI_ or created by using the _GitLab WebIDE_
- It runs when the pipeline is by a _`git push` event_ or is _scheduled pipeline_, but only if there's non-empty the environment variable `ENV_CREATE` or `CREATE_ENV`
- It doesn't run when the environment variable `ENV_SKIP` or `SKIP_ENV` is present, or the commit message contains `[env skip]` or `[skip env]`, using any capitalization
**Development/_branch_** environment create or not decision:
Manually managed environments:
- All stages must be run manually and locally
### Environment
Creates one machine for development and testing environments, or intentionally zero machines for staging and production environments. Each machine uses [Xebis Ansible Collection](https://github.com/xebis/xebis-ansible-collection) roles:
- `xebis.ansible.system`: Well maintained operating system - updates and upgrades `deb` packages including autoremove and autoclean, reboots the system (when necessary), provides `Reboot machine` handler
- `xebis.ansible.firewall`: Extensible nftables firewall - installs `nftables` and sets up basic extensible nftables chains and rules, provides `Reload nftables` handler, see [GitHub: xebis/xebis-ansible-collection/README.md](https://github.com/xebis/xebis-ansible-collection/blob/main/README.md) for usage, configuration, and examples
- `xebis.ansible.fail2ban`: Fail2ban service - installs `fail2ban` and sets it up as a systemd service
- `xebis.ansible.iam`: IAM - creates user groups and users as regular users or admins, their public SSH keys, disables password remote logins, provides `Restart sshd` handler, see [GitHub: xebis/xebis-ansible-collection/README.md](https://github.com/xebis/xebis-ansible-collection/blob/main/README.md) for usage, configuration, and examples
- `xebis.ansible.bash`:Extensible Bash - installs `~/.bash_aliases` and sets up basic extensible Bash aliases, see [GitHub: xebis/xebis-ansible-collection/README.md](https://github.com/xebis/xebis-ansible-collection/blob/main/README.md) for usage, configuration, and examples
- `xebis.ansible.starship`: Starship CLI prompt - Installs `starship` and sets up improved PowerLine configuration
- `xebis.ansible.admin`: Administration essentials - installs and sets up `at`, `curl`, `htop`, `mc`, `screen`
Each machine uses [LabLabs RKE2 Ansible Role](https://github.com/lablabs/ansible-role-rke2):
- `lablabs.rke2`: Ansible Role to install RKE2 Kubernetes.
### Caveats
One Hetzner cloud project is used for all environments, which brings a few caveats to keep in one's mind:
- To distinguish machines between environments and to separate them from manually created machines they are named with prefix `env-slug-` and labeled `env=env-slug` by Terraform
- To use Ansible, inventory file `hcloud.yml` must have replaced `env-slug` with an **environment slug** before any local manual use
- Use Ansible group `env` instead of groups `all` or `hcloud`, as these groups contain all machines from all environments and eventually manually created machines as well
### Images
- [Git workflow examples & template](images/workflow.drawio) - inspired by [diagrams.net: Blog - How to create a gitflow diagram](https://www.diagrams.net/blog/gitflow-diagram)
- [Example of the full workflow](images/workflow-full.png)
- [Deploy in more detail](images/deploy.png)
## Installation and Configuration
Prepare Hetzner Cloud API token and GitLab CI SSH keys:
- [Hetzner Cloud - referral link with €20 credit](https://hetzner.cloud/?ref=arhwlvW4nCxX)
- Hetzner Cloud Console -> Projects -> _Your Project_ -> Security -> API Tokens -> Generate API Token `Read & Write`
- Generate GitLab CI SSH keys `ssh-keygen -t rsa` (no passphrase, _to your secret file_, **do not commit it!**), file with `.pub` extension will be generated automatically, put `*.pub` file contents at [`cloud-config.yml`](cloud-config.yml) under section `users:name=gitlab-ci` to the `ssh_authorized_keys` as the first element, and commit it
### Set up GitLab CI
- GitLab -> Settings
- General > Visibility, project features, permissions > Operations: **on**
- CI/CD > Variables:
- Add variable: Key `HCLOUD_TOKEN`, Value ``
- Add variable: Key `GL_CI_SSH_KEY`, Value _contents of your secret file_ created by `ssh-keygen -t rsa` above
### Set up Local Usage
Make sure **GL_TOKEN**: [GitLab Personal Access Token](https://gitlab.com/-/profile/personal_access_tokens) with scope `api` is present, otherwise `gitlab-ci-linter` is skipped. To run Terraform provisioning Hetzner Cloud you have to set up `TF_HTTP_PASSWORD`, `HCLOUD_TOKEN`, `TF_VAR_ENV_NAME`, and `TF_VAR_ENV_SLUG` required by Terraform configuration.
To load secrets you can use shell extension like [direnv](https://direnv.net/), encryption like [SOPS](https://github.com/getsops/sops), or secrets manager [HashiCorp Vault](https://www.vaultproject.io/), **please make sure you won't commit your secrets**.
```shell
export GL_TOKEN="" # Your GitLab's personal access token with the api scope
export TF_HTTP_PASSWORD="$GL_TOKEN" # Set password for Terraform HTTP backend
export HCLOUD_TOKEN="" # Your Hetzner API token
export TF_VAR_ENV_NAME="" # Replace with the environment name
export TF_VAR_ENV_SLUG="" # Replace with the environment slug
```
- Install repository dependencies by `sudo scripts/bootstrap` script, setup repository by `scripts/setup`, update repository by `scripts/update` script.
- Set up all admins and users, including public SSH key at [`ansible/group_vars/all.yml`](ansible/group_vars/all.yml) under section `users`, see documentation there. Do not forget to commit it 😀
## Usage
### GitLab CI
- Commit and push to run validations
- Push a _non-_`main` branch
- To create a **development/_branch_** environment you have to create a new pipeline for the branch using API, GitLab ChatOps, trigger token, or by using the **Run pipeline** button in the GitLab UI
- Alternatively, you can create a **development/_branch_** environment directly by pushing or scheduling with `ENV_CREATE` or `CREATE_ENV` environment variable present, for example by running `git push -o ci.variable="CREATE_ENV=true"`
- Once created, the environment will be updated (or recreated if it was destroyed) with each subsequent pipeline on the branch
- Environment deploy is skipped when the environment variable `ENV_SKIP` or `SKIP_ENV` is present, or commit message contains `[env skip]` or `[skip env]`, using any capitalization, or when CI pipeline is skipped altogether, for example using `git push -o ci.skip`
- Destroy **development/_branch_** environment manually, or wait until auto-stop (1 day from the last commit in the branch in GitLab, could be overridden in GitLab UI)
- Create a _pre-release_ tag to create **testing/_tag_** environment
- Pre-release tag must match regex `^v(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(?:-((?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))(?:\+([0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$`, see
- Destroy **testing/_tag_** environment manually, or wait until auto-stop (1 week, could be overridden in GitLab UI)
- Merge to the `main` branch to create or update the **staging** environment
- Creates or updates **production** environment when a commit starting `feat` or `fix` is present in the history from the previous release
- _Release_ tag must match regex `^v(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)?(?:\+([0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$`, see
_Release_ and _pre-release_ tags must follow SemVer string, see [Semantic Versioning 2.0.0: Is there a suggested regular expression (RegEx) to check a SemVer string?](https://semver.org/#is-there-a-suggested-regular-expression-regex-to-check-a-semver-string)
### Local Usage
Initialize local workspace if not yet initialized:
```shell
# Init local workspace
pushd terraform
terraform init -reconfigure \
-backend-config="address=https://gitlab.com/api/v4/projects/31099306/terraform/state/$TF_VAR_ENV_SLUG" \
-backend-config="lock_address=https://gitlab.com/api/v4/projects/31099306/terraform/state/$TF_VAR_ENV_SLUG/lock" \
-backend-config="unlock_address=https://gitlab.com/api/v4/projects/31099306/terraform/state/$TF_VAR_ENV_SLUG/lock"
```
- Create or update environment by `terraform apply` or `terraform apply -auto-approve`
- Get nodes IP addresses by `terraform output nodes_ipv4_addresses`
- Direct SSH by `ssh user@$(terraform output -raw nodes_ipv4_addresses)`
- Ansible:
- Change to Ansible configuration directory `pushd ../ansible`
- First replace `hcloud.yml` string `env-slug` with `$TF_VAR_ENV_SLUG`: `sed -i "s/env-slug/$TF_VAR_ENV_SLUG/" hcloud.yml`
- List or graph inventory: `ansible-inventory -i hcloud.yml --list # or --graph`
- Ping: `ansible -u user -i hcloud.yml env -m ansible.builtin.ping`
- Get all facts: `ansible -u user -i hcloud.yml env -m ansible.builtin.setup`
- Configure with playbook: `ansible-playbook -u user -i hcloud.yml playbook.yml`
- Change back to Terraform configuration directory `popd`
- Destroy environment by `terraform destroy` or `terraform destroy -auto-approve`, and go back to the repository root directory`popd`
Uninitialize local workspace if you wish:
```shell
rm -rf terraform/.terraform # Uninit local workspace, this step is required if you would like to work with another environment
```
Commit and push to run validations.
### Terraform Configuration Documentation
- [Terraform Root module: `terraform/README.md`](terraform/README.md)
- [Terraform Nodes module: `terraform/modules/nodes/README.md`](terraform/modules/nodes/README.md)
## Contributing
Please read [CONTRIBUTING](CONTRIBUTING.md) for details on our code of conduct, and the process for submitting merge requests to us.
### Testing
- Git hooks check a lot of things for you, including running automated tests `scripts/test full`
- Make sure all `scripts/*`, git hooks, and GitLab pipelines work as expected, testing checklist:
- `scripts/*` scripts - covered by unit tests `tests/*`
- [ ] [`scripts/bootstrap`](scripts/bootstrap)
- [ ] [`scripts/deploy-env`](scripts/deploy-env)
- [ ] [`scripts/pre-commit`](scripts/pre-commit)
- [ ] [`scripts/pre-push`](scripts/pre-push)
- [ ] [`scripts/setup`](scripts/setup)
- [ ] [`scripts/test`](scripts/test)
- [ ] [`scripts/update`](scripts/update)
- Local working directory
- [ ] `git commit` runs `pre-commit` hook-type `commit-msg` and [`scripts/pre-commit`](scripts/pre-commit)
- [ ] `git merge`
- [ ] Fast-forward shouldn't run any hooks or scripts
- [ ] Automatically resolved `merge commit` runs `pre-commit` hook-type `commit-msg` and [`scripts/pre-commit`](scripts/pre-commit)
- [ ] Manually resolved `merge commit` runs `pre-commit` hook-type `commit-msg` and [`scripts/pre-commit`](scripts/pre-commit)
- [ ] `git push` runs [`scripts/pre-push`](scripts/pre-push)
- Terraform and Ansible
- [ ] `terraform init`
- [ ] `terraform plan`
- [ ] `terraform apply`
- [ ] `ansible ... ping`
- [ ] `ansible-playbook`
- [ ] `terraform destroy`
- GitLab CI
- [ ] Commit on a new _non-_`main` branch runs `validate:lint` and `validate:test-full`
- [ ] Without any environment variables, runs `provision:provision-dev`, `install:install-dev`, and prepares `destroy:destroy-dev`
- [ ] With non-empty environment variable `ENV_CREATE` or `CREATE_ENV`, runs `provision:provision-dev`, `install:install-dev`, and prepares `destroy:destroy-dev`
- [ ] Commit on an existing _non-_`main` branch within 24 hours runs `provision:provision-dev`, `install:install-dev`, and prepares `destroy:destroy-dev`
- [ ] Absence of commit on an existing _non-_`main` branch within 24 hours auto-stops **development/_branch_** environment
- [ ] _Pre-release_ tag on a _non-_`main` branch commit runs `validate:lint`, `validate:test-full`, `provision:provision-test`, `install:install-test`, and prepares `destroy:destroy-test`
- [ ] After a week auto-stops **testing/_tag_** environment
- [ ] Merge to the `main` branch runs `validate:lint`, `validate:test-full`, `provision:provision-stage`, `install:install-stage`, and `release:release`
- [ ] With a new `feat` or `fix`, commit releases a new version
- [ ] _Release_ tag on the `main` branch commit runs `validate:lint`, `validate:test-full`, `provision:provision-prod`, and `install:install-prod`
- [ ] Without a new feature or fix commit does not release a new version
- [ ] Scheduled (nightly) pipeline runs `validate:lint` and `validate:test-nightly`
#### Test at Docker Container
To test your changes in a different environment, you might try to run a Docker container and test it from there.
Run a disposal Docker container:
- `sudo docker run -it --rm -v "$(pwd)":/infrastructure-template alpine:latest`
- `sudo docker run -it --rm -v "$(pwd)":/infrastructure-template --entrypoint sh hashicorp/terraform:light`
- `sudo docker run -it --rm -v "$(pwd)":/infrastructure-template --entrypoint sh gableroux/ansible:latest`
- `sudo docker run -it --rm -v "$(pwd)":/infrastructure-template --entrypoint sh node:alpine`
In the container:
```bash
cd infrastructure-template
# Set variables GL_TOKEN and GH_TOKEN when needed
# Put here commands from .gitlab-ci.yml job:before_script and job:script
# For example job test-full:
apk -U upgrade
apk add bats
bats tests
# Result is similar to:
# 1..1
# ok 1 dummy test
```
## To-Do list
- [ ] Fix workaround for pre-commit `jumanjihouse/pre-commit-hooks` hook `script-must-have-extension` - `*.bats` shouldn't be excluded
- [ ] Fix workaround for pre-commit `local` hook `shellcheck` - shellcheck has duplicated parameters from `.shellcheckrc`, because these are not taken into account
## Roadmap
- [ ] Find a satisfactory way how to manage (list, install, update) dependencies across various distributions and package managers
- [ ] Add [pre-commit meta hooks](https://pre-commit.com/#meta-hooks)
- [ ] Add [jumanjihouse/pre-commit-hooks hook protect-first-parent](https://github.com/jumanjihouse/pre-commit-hooks#protect-first-parent)
- [ ] Speed up CI/CD by preparing a set of Docker images with pre-installed dependencies for each CI/CD stage, or by cache for `apk`, `pip`, and `npm`
## Credits and Acknowledgments
- [Martin Bružina](https://bruzina.cz/) - Author
## Copyright and Licensing
- [MIT License](LICENSE)
- Copyright © 2021 Martin Bružina
## Changelog and News
- [Changelog](CHANGELOG.md)
## Notes and References
### Dependencies
- [Hetzner Cloud - referral link with €20 credit](https://hetzner.cloud/?ref=arhwlvW4nCxX)
- [Terraform](https://www.terraform.io/)
- [Terraform: Hetzner Cloud Provider](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs)
- [GitHub - antonbabenko/pre-commit-terraform: pre-commit git hooks to take care of Terraform configurations](https://github.com/antonbabenko/pre-commit-terraform)
- [GitHub - terraform-linters/tflint: A Pluggable Terraform Linter](https://github.com/terraform-linters/tflint)
- [checkov](https://www.checkov.io/)
- [terraform-docs: Generate Terraform modules documentation in various formats](https://terraform-docs.io/)
- [Ansible](https://www.ansible.com/)
- [GitHub - ansible-community/ansible-lint: Best practices checker for Ansible](https://github.com/ansible-community/ansible-lint)
- [GitHub - ansible-collections/hetzner.hcloud: A collection containing modules to manage resources on the Hetzner Cloud.](https://github.com/ansible-collections/hetzner.hcloud)
- [GitHub - hetznercloud/hcloud-python: hcloud-python is a library for the Hetzner Cloud API.](https://github.com/hetznercloud/hcloud-python)
- [Ansible Documentation: Ansible.Posix](https://docs.ansible.com/ansible/latest/collections/ansible/posix/index.html)
- [cloud-init](https://cloud-init.io/)
- [Docker Hub - hashicorp/terraform](https://hub.docker.com/r/hashicorp/terraform/)
- [Docker Hub - gableroux/ansible](https://hub.docker.com/r/gableroux/ansible)
- [jq: jq is a lightweight and flexible command-line JSON processor](https://stedolan.github.io/jq/)
- [GitHub - xebis/repository-template: Well-manageable and well-maintainable repository template.](https://github.com/xebis/repository-template) - contains GitLab CI/CD, set of useful scripts, `pre-commit`, `semantic-release`, and `Visual Studio Code` suggested extensions
- [GitHub - xebis/xebis-ansible-collection: A collection of Xebis shared Ansible roles.](https://github.com/xebis/xebis-ansible-collection)
- [GitHub - lablabs/ansible-role-rke2: Ansible Role to install RKE2 Kubernetes.](https://github.com/lablabs/ansible-role-rke2) and [Ansible Galaxy - lablabs.rke2](https://galaxy.ansible.com/ui/standalone/roles/lablabs/rke2/)
### Recommendations
- [GitHub - shuaibiyy/awesome-terraform: Curated list of resources on HashiCorp's Terraform](https://github.com/shuaibiyy/awesome-terraform)
- [GitHub - KeyboardInterrupt/awesome-ansible: Awesome Ansible List](https://github.com/KeyboardInterrupt/awesome-ansible)
- [GitHub - hetznercloud/awesome-hcloud: A curated list of awesome libraries, tools, and integrations for Hetzner Cloud](https://github.com/hetznercloud/awesome-hcloud)
### Suggestions
- [Visual Studio Code](https://code.visualstudio.com/) with [Extensions for Visual Studio Code](https://marketplace.visualstudio.com/VSCode):
- [HashiCorp Terraform](https://marketplace.visualstudio.com/items?itemName=HashiCorp.terraform)
- [Ansible](https://marketplace.visualstudio.com/items?itemName=redhat.ansible) - includes potentially unwanted extensions [Python](https://marketplace.visualstudio.com/items?itemName=ms-python.python) (Ansible dependency) and [Pylance](https://marketplace.visualstudio.com/items?itemName=ms-python.vscode-pylance) (could be uninstalled)
### Further Reading
- [GitLab: What is GitOps?](https://about.gitlab.com/topics/gitops/)
- [Semantic Versioning - Semantic Versioning 2.0.0](https://semver.org/)
- [RegEx101: Valid Semantic Versions](https://regex101.com/r/vkijKf/1/)
- [RegEx101: Valid Release Semantic Versions](https://regex101.com/r/9DFqb3/1)
- [RegEx101: Valid Pre-release Semantic Versions](https://regex101.com/r/G1OFXH/1)
- [Conventional Commits - Conventional Commits 1.0.0](https://www.conventionalcommits.org/en/v1.0.0/)