Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/hibernate/hibernate-infra

Configuration of Hibernate project infrastructure: CI, websites, bot deployment.
https://github.com/hibernate/hibernate-infra

ansible hibernate

Last synced: 8 days ago
JSON representation

Configuration of Hibernate project infrastructure: CI, websites, bot deployment.

Awesome Lists containing this project

README 

        
# Hibernate infrastructure



## What is this?



This is an Ansible playbook to set up the Hibernate infrastructure:

Continuous Integration, website, and bot deployments.



The Ansible playbook does not make extensive usage of variables as we don't expect to need that: feel free to take inspiration from these but don't expect this to be a general purpose framework to set up a CI environment.



We prefer to make some assumptions and keep this simple;

among others, we expect to run the public facing services on Red Hat Enterprise Linux 8,

and run some Jenkins worker nodes on Fedora Cloud.



The websites and the Jenkins coordinator node will run on permanent instances on Amazon AWS,

while most Jenkins worker nodes will run on AWS EC2 Spot instances launched by the Jenkins AWS EC2 plugin,

and some Jenkins worker nodes will be managed by partners.



## First setup



### Preparations: AWS launch templates



If your SSH key was never used to build the servers,

you will need to add the public key to the `cloud-init` file at the root of this git repository.

Then update the following launch templates owned by the `hibernate` organization on AWS accordingly,

by setting the "user data" to the content of the `cloud-init` file:



 - `jenkins-ci-coordinator`

 - `jenkins-ci-worker-ami-building`



You can use a different organization and launch templates of course, but we won't pay your AWS bills.



### Boot the servers



You should run:

- 1 instance to host the websites.

  Start it on AWS and figure out the launch template yourself (details have been lost to time, I think it should use a RHEL 8 image).

 - 1 instance to host the Jenkins coordinator node.

   Start it on AWS and use the `jenkins-ci-coordinator` launch template (a RHEL 8 image).

 - 1 instance to create an AMI for the various Jenkins worker nodes,

   which will then be used by the Jenkins AWS EC2 plugin to spawn workers on demand.

   Start it on AWS and use the `jenkins-ci-worker-ami-building` launch template.

   Note worker node instances are expected to provide an "instance storage" volume

   that the node will mount as `/mnt/workdir` and use for docker and jenkins data.

   Failing that, you're likely to see errors in your builds, e.g. "no space left on device".



Boot them using the provided 'cloud-init' script.

When booting machines from the UI, you can paste the content of 'cloud-init' into the "Customisation Script" section on the AWS console.



### Update the inventory file (server addresses and keys)



You will need to update the inventory file `hosts` to point to the servers you just launched.

Gather the public IP address or public DNS for each server,

and paste it in the 'hosts' file in the appropriate section:



- The address of the AWS Jenkins CI coordinator node in `jenkins-coordinator`

- The address of the AWS Jenkins CI worker node in `jenkins-worker`



Make sure to update the paths to the private keys as necessary.



Do not commit these changes unless your changes may be useful to other users.



### Let it configure your servers



Now install Ansible, then the required collections:



	ansible-galaxy collection install -r requirements.yml



Then run the Ansible playbook like this:



	ansible-playbook -i hosts site.yml



#### Performance Tip



When only updating the worker nodes (which run on Fedora), it is recommended to enable SSH pipelining which will make things go quite a bit faster. To do so, specify pipelining = True in ansible.cfg. (This couldn't work on RHEL 7 for security reasons, it might work on RHEL 8)



You can also run the playbook on a subset of the hosts in the file using the parameter "--limit":



    ansible-playbook -i hosts site.yml --limit jenkins-worker



If you want to have the list of IP affected without running the playbook you can use the option "--list-hosts":



    ansible-playbook -i hosts site.yml --limit jenkins-worker --list-hosts



It is also possible to execute specific tasks using tags:



    ansible-playbook -i hosts site.yml --limit jenkins-worker --tags "generate-script"



More details about tags can be found the ansible documentation.



### Finishing touches



The Jenkins coordinator node is now running, updates installed.



Jenkins is not configured however: you'll need to manually transfer a copy of the configuration

from a previous coordinator node or reconfigure it using the web UI.



The Jenkins worker node is also running and up to date, but needs to be turned into an AMI

so that Jenkins can spin up worker nodes dynamically.

Create an image from the `Instances` panel of the AWS EC2 console:

select the instance, then click `Actions > Image > Create Image`.

Do not forget to update the AMI in the Jenkins AWS EC2 plugin configuration.



## Maintenance of existing nodes



### Updating the website and CI coordinator nodes



Ansible should run a `dnf update` automatically, so just do this:



```shell

ansible-playbook -i hosts site.yml --limit jenkins-coordinator

```



or:



```shell

ansible-playbook -i hosts site.yml --limit websites

```



### Making changes to the Jenkins worker nodes



The Ansible playbook is designed so it can be re-run on your existing infrastructure without undoing configuration you did on the previous step.



To make changes to the configuration of a Jenkins worker node:



1. Update the playbook.

2. Start an instance on AWS (see first setup) using the current Jenkins worker node AMI. Alternatively, you can start from a fresh AMI.

3. Run the ansible-playbook command again (see first setup).

4. If it doesn't work, iterate.

5. When it works, commit and push the changes so that next time we'll need to rebuild nodes they will include your changes.