Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/coderefinery/gitlab-openstack-deploy

An Ansible playbook to deploy a Gitlab Server on OpenStack (Specifically CSC's Pouta)
https://github.com/coderefinery/gitlab-openstack-deploy

Last synced: 5 months ago
JSON representation

An Ansible playbook to deploy a Gitlab Server on OpenStack (Specifically CSC's Pouta)

Awesome Lists containing this project

README 

          
# Gitlab Devops



This repository contains Ansible playbooks for administering the gitlab

installations on OpenStack.. It should not be kept on the gitlab installation

to avoid obvious chicken-egg problems.



## Setting up the Python environment



This assumes you have virtualenvwrapper, virtualenv and in general Python

installed.



    $ mkvirtualenv cr-gitlab-devops

    (cr-gitlab-devops)$ pip install -r requirements.txt



On subsequent uses it suffices to activate the virtual environment



    $ workon cr-gitlab-devops

    (cr-gitlab-devops)$



## Setting up environment variables



This repository does not contain per-installation data.



The data is expected to be found thus



```

repo/

  playbook.yml

  roles/...

  environment/ # here you can have multiple environments

    coderefinery-gitlab/ # symlink to a separate repo

      hosts

      group_vars/

        all/

          vars.yml

          vault.yml

          ansible.cfg

          other.yml

    another-gitlab/

      hosts

      group_vars

        all/

          vars.yml

          vault.yml

          something_completely_different.yml

```



To create a new set of environment variables it is suggested to copy an

existing set for simplicity.



## Playbook



There roles are run with playbook called playbook.yml. The playbook will provision

the necessary resources from OpenStack and configure the system as much as

possible.



It requires certain environment variables for OpenStack authentication called

an OpenRC.



    (cr-gitlab-devops)$ source project-openrc.sh

    Please enter your OpenStack Password:

    (cr-gitlab-devops)$



Then you can run the actual playbook provided you have the vault password.



    (cr-gitlab-devops)$ export ANSIBLE\_CONFIG=~/path/to/coderefinery-gitlab/ansible.cfg

    ansible-playbook playbook.yml -i environments/coderefinery-gitlab/hosts

    Vault password:



and Bob is your uncle!



### Ansible vault



To de-crypt the encrypted secrets found in group\_vars/all/vault.yml

one needs a vault password. It is distributed separately.



To view the contents of the vault file run



    (cr-gitlab-devops)$ ansible-vault view group\_vars/all/vault.yml



And to edit it run



    (cr-gitlab-devops)$ ansible-vault edit group\_vars/all/vault.yml



For more information check out  [Ansible vault

documentation](http://docs.ansible.com/ansible/playbooks_vault.html)



## Provisioning



Step one of the playbook creates the following servers



* the actual gitlab host with a public ip

* a separate backup-host with a volume for backups and a cron-job that runs

  backups on gitlab-host

* a gitlab-runner instance that runs gitlab-runner inside docker against the gitlab instance

  * there was a chicken and egg issue with getting the authentication key for

    runner so after installing gitlab you will need to get the files



After the hosts are created the system creates ssh.cfg in the root of your

installation, which is used to use the gitlab installation as bastion so that

the gitlab installation can be the only one of the three items with a public

IP address.



To communicate with the other two parts run



    $ ssh -F ssh.{{gitlab_name}}.cfg gitlab-runner / gitlab-backup



## Configuring



Most configuration is in group\_vars/all/vars.yml or the vault.yml that was

already covered.



Most values should be self-explanatory. If in doubt try grepping for how they

are used.



When adding a variable to the vault make a line in vars.yml that copies the

variable to another variable without the prefix vaulted\_ . This makes it

easier to check the names of variables in the vault without decrypting it all the time..



Initial installation root password is stored inside the vault. Check that to

log in and start managing the server if installing from scratch.



### SSL



SSL keys were created by using certbot. There is a part called letsencrypt

that will in theory re-generate the certificates using a brand new ansible

module. This is doubtable in practice and someone should re-generate the

certificates manually before summer holidays 2017.



The playbook for letsencrypt is included but disabled in case there is time to

work on it.



### Runner config



Multiple virtual machines can be created to host runners for GitLab CI/CD

pipelines. Their specs are specified in a dict called runner_vms like so:



```

runner_vms:

  centos-dedicated:

    image: "CentOS-7.0"

    flavor: "io.70GB"

    runner_configs:

      - RUNNER_NAME: "centos-dedicated"

        DOCKER_IMAGE: "centos"

        REGISTER_LOCKED: "true"

        state: "started"

  ubuntu-dedicated:

    image: "CentOS-7.0"

    flavor: "io.70GB"

    runner_configs:

      - RUNNER_NAME: "ubuntu-dedicated"

        DOCKER_IMAGE: "ubuntu"

        REGISTER_LOCKED: "true"

        REGISTRATION_TOKEN: "{{ vaulted_ubuntu_dedicated_reg_token }}"

        state: "started"

  shared:

    image: "CentOS-7.0"

    flavor: "io.70GB"

    runner_configs:

      - RUNNER_NAME: "ubuntu-shared"

        DOCKER_IMAGE: "ubuntu"

        REGISTRATION_TOKEN: "{{ initial_shared_runners_registration_token }}"

        state: "started"

      - RUNNER_NAME: "alpine"

        DOCKER_IMAGE: "alpine"

        REGISTRATION_TOKEN: "{{ initial_shared_runners_registration_token }}"

        state: "started"

```



Runner configuration consists of setting an image and a flavor for the virtual

machine and a list of environment variable dicts (one per runner) that

configure runners at registration time. A list of these environment variables

can be retrieved by running "docker exec gitlab-runner gitlab-ci-multi-runner

help register" as root on a runner machine.



If a registration token is not initially specified for a runner, then it will

not register initially. This is useful when you want to precreate a dedicated

runner whose registration token will only be known once a project is created

that will use the runner. Once the registration token is known, you can use it

to register these precreated runners as runners by adding it into the dict.



In the example above, the "shared" runner gets a special value for

REGISTRATION_TOKEN. This is an initial token that is configured in GitLab's

config file and can be used to register shared runners right away.



For runners that should only be usable by a single project you can set

REGISTER_LOCKED: "true" in the list of environment variables.



The state variable can be any of the [states accepted by the docker

module](https://docs.ansible.com/ansible/docker_container_module.html). As

containers are identified by name only it is important that you do change

RUNNER_NAME without first setting it's state to stopped or absent. If you do,

you will have a ghost container and have to stop and remove it manually.



## Recovering backups



1) obtain files and copy them to remote machine, e.g.



```

[cloud-user@gitlab-internal tmp]$ ls -alhZ

drwxrwxrwt. root       root       system_u:object_r:tmp_t:s0       .

drwxr-xr-x. root       root       system_u:object_r:root_t:s0      ..

-rw-------. cloud-user cloud-user unconfined_u:object_r:user_tmp_t:s0

1495065614_2017_05_18_gitlab_backup.tar

-rw-------. cloud-user cloud-user unconfined_u:object_r:user_tmp_t:s0

etc-gitlab-1495065602.tgz

```



2)  copy  xxxx_gitlab_backup.tar to /srv/gitlab/data/backups

    chmod 0755 /srv/gitlab/data/backups/xxx_gitlab_backup.tar



  copy etc-gitlab-XXX.tgz to /srv/gitlab/config/backups/ (optional, you can

  pack it wherever you like)



3) shutdown stuff that uses the database



```

docker exec -it gitlab gitlab-ctl stop unicorn

docker exec -it gitlab gitlab-ctl stop sidekiq

```



4) unpack the config .tgz, copy files you wish to replace (everything except

gitlab.rb probably, unless you changed certificates)



5) run



```

docker exec -it gitlab gitlab-rake gitlab:backup:restore \

BACKUP=1495065614_2017_05_18

```



with the timestamp of the backup. Answer YES to everything.



## Updating



1. Notify on system

2. Verify that a recent backup exists

3. Edit gitlab_tag in environments/[your_env]/group_vars/all/vars.yml

4. Set up environment variables (like ANSIBLE_CONFIG, OpenStack credentials

   etc.)

5. Run playbook



If you suspect something fishy about the migrations not having been run, run



```

$ ssh -F ssh.[your_environment].cfg gitlab

$ su

# docker exec -it gitlab bash

# gitlab-rake db:migrate

```



Migrations should be run automatically at the startup of the container,

though.