{"id":13588164,"url":"https://github.com/jonashackt/gitlab-ci-stack","last_synced_at":"2025-08-20T03:30:57.841Z","repository":{"id":147269844,"uuid":"126224275","full_name":"jonashackt/gitlab-ci-stack","owner":"jonashackt","description":"Full CI pipeline project based on Gitlab \u0026 Gitlab CI running Docker, completely automated setup by Vagrant \u0026 Ansible, providing Let´s Encrypt certificates for private Servers, multiple Gitlab-Runners and the Gitlab Container Registry, incl. GitLab Pages","archived":false,"fork":false,"pushed_at":"2020-01-15T12:19:06.000Z","size":3952,"stargazers_count":179,"open_issues_count":4,"forks_count":60,"subscribers_count":14,"default_branch":"master","last_synced_at":"2024-12-09T04:22:43.882Z","etag":null,"topics":["ansible","ci-pipeline","container-registry","docker","encrypt-certificates","gitlab","gitlab-ci","gitlab-pages","gitlab-registry","gitlab-runner","gitlab-server","letsencrypt","vagrant"],"latest_commit_sha":null,"homepage":"https://blog.codecentric.de/en/2018/05/gitlab-ci-pipeline/","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jonashackt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2018-03-21T18:35:39.000Z","updated_at":"2024-11-26T14:26:35.000Z","dependencies_parsed_at":null,"dependency_job_id":"fc1937c6-2d10-46ee-b535-62a0812c5057","html_url":"https://github.com/jonashackt/gitlab-ci-stack","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonashackt%2Fgitlab-ci-stack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonashackt%2Fgitlab-ci-stack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonashackt%2Fgitlab-ci-stack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonashackt%2Fgitlab-ci-stack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jonashackt","download_url":"https://codeload.github.com/jonashackt/gitlab-ci-stack/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230388131,"owners_count":18217755,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ansible","ci-pipeline","container-registry","docker","encrypt-certificates","gitlab","gitlab-ci","gitlab-pages","gitlab-registry","gitlab-runner","gitlab-server","letsencrypt","vagrant"],"created_at":"2024-08-01T15:06:32.728Z","updated_at":"2024-12-19T06:09:16.714Z","avatar_url":"https://github.com/jonashackt.png","language":"Shell","funding_links":[],"categories":["Shell"],"sub_categories":[],"readme":"gitlab-ci-stack\n======================================================================================\n[![Build Status](https://travis-ci.org/jonashackt/gitlab-ci-stack.svg?branch=master)](https://travis-ci.org/jonashackt/gitlab-ci-stack)\n\nFull CI pipeline project based on Gitlab \u0026 Gitlab CI running Docker, completely automated setup by Vagrant \u0026 Ansible, providing Let´s Encrypt certificates for private Servers, multiple Gitlab-Runners and the Gitlab Container Registry\n\n[![asciicast](https://asciinema.org/a/205171.svg)](https://asciinema.org/a/205171)\n\nThis project is somehow based on the thought of https://github.com/marcelbirkner/docker-ci-tool-stack. But since the good old days of Jenkins times changed \"a bit\". Maybe today Jenkins incl. 2.x/Pipeline-plugin isn´t the way to go - or it´s just the way, if you really want to have a hard time. Why? Here are some points:\n\n\"Jenkins servers become snowflakes\"\n\n\"Jenkins 2.0 tries to address this by promoting a Pipeline plugin (plus another plugin to visualize it), but it kind of misses the point.\"\n\nhttps://content.pivotal.io/blog/comparing-bosh-ansible-chef-part-1\n\n\nI heard from so many colleagues: \n\n\u003e \"Hey Jonas you Jenkins fanboy. Have a look on all those cool new CI servers like Concourse, Circle CI oder even Gitlab CI! We don´t know, why you´re messing around with Jenkins...\" .\n\n\n## Table of Contents  \n* [Which one to choose?](#which-one-to-choose)\n* [Gitlab CI](#gitlab-ci)\n* [Why not just use Docker Compose to fire everything up?](#why-not-just-use-docker-compose-to-fire-everything-up)\n* [Prerequisites](#prerequisites)\n* [Let´s install \u0026 run Gitlab inside our Server/VagrantBox with Ansible](#lets-install--run-gitlab-inside-our-servervagrantbox-with-ansible)\n* [Install \u0026 Configure Docker](#install--configure-docker)\n* [Nice Gitlab URL with DNS configuration](#nice-gitlab-url-with-dns-configuration)\n* [Enable https for Gitlab on public accessable server](#enable-https-for-gitlab-on-public-accessable-server)\n* [Let´s Encrypt for our Gitlab on VirtualBox/Vagrant](#lets-encrypt-for-our-gitlab-on-virtualboxvagrant)\n* [Configure the \"self-created\" Let´s Encrypt certificates in Gitlab (it´s the same process for other certificates then Let´s Encrypt)](#configure-the-self-created-lets-encrypt-certificates-in-gitlab-its-the-same-process-for-other-certificates-then-lets-encrypt)\n* [Install Gitlab itself](#install-gitlab-itself)\n* [Gitlab Container Registry](#gitlab-container-registry)\n* [Ubuntu \u0026 Docker don´t know Let´s Encrypt so we need to copy the fullchain.pem instead of just the cert.pem!](#ubuntu--docker-dont-know-lets-encrypt-so-we-need-to-copy-the-fullchainpem-instead-of-just-the-certpem)\n* [Install Gitlab Runner](#install-gitlab-runner)\n* [Configure Gitlab Runner with shell executor](#configure-gitlab-runner-with-shell-executor)\n* [Configure a Docker-in-Docker enabled gitlab-runner with the docker executor](#configure-a-docker-in-docker-enabled-gitlab-runner-with-the-docker-executor)\n* [Changes needed in .gitlab-ci.yml for Docker-in-Docker compared to using a shell runner](#changes-needed-in-gitlab-ciyml-for-docker-in-docker-compared-to-using-a-shell-runner)\n* [Configure a Docker socket binding enabled gitlab-runner with the docker executor](#configure-a-docker-socket-binding-enabled-gitlab-runner-with-the-docker-executor)\n* [Changes needed in .gitlab-ci.yml for Docker-in-Docker compared to using a shell runner](#changes-needed-in-gitlab-ciyml-for-docker-in-docker-compared-to-using-a-shell-runner-1)\n* [Configure .gitlab-ci.yml Jobs to run only on specific gitlab-runners](#configure-gitlab-ciyml-jobs-to-run-only-on-specific-gitlab-runners)\n* [Using the Gitlab Container Registry](#using-the-gitlab-container-registry)\n* [Namespaces (username, group or subgroup)](#namespaces-username-group-or-subgroup)\n* [Import example project with .gitlab-ci.yml and run Gitlab CI pipeline](#import-example-project-with-gitlab-ciyml-and-run-gitlab-ci-pipeline)\n* [GitLab Pages on self-hosted GitLab-Instance](#gitlab-pages-on-self-hosted-gitlab-instance)\n* [Define another domain name for Pages](#define-another-domain-name-for-pages)\n* [Let´s Encrypt Wildcard certificates with dehydrated \u0026 lexicon](#lets-encrypt-wildcard-certificates-with-dehydrated--lexicon)\n* [Configure Pages in GitLab with Ansible](#configure-pages-in-gitlab-with-ansible)\n* [A new Jekyll site](#a-new-jekyll-site)\n* [Deploy Jekyll as GitLab Pages with GitLab CI \u0026 Docker](#deploy-jekyll-as-gitlab-pages-with-gitlab-ci--docker)\n* [Links](#links)\n\n\n## Which one to choose?\n\nWell, ok then. Let´s give it a try. And because of all this here:\n\nhttps://www.digitalocean.com/community/tutorials/ci-cd-tools-comparison-jenkins-gitlab-ci-buildbot-drone-and-concourse\n\nhttps://www.slant.co/versus/2482/10699/~gitlab-ci_vs_concourse-ci\n\nhttps://www.reddit.com/r/devops/comments/6cuj0s/concourse_jenkins_ci/\n\nhttps://concourse-ci.org/concourse-vs.html\n\nTherefore I wanted to have a deeper look into Gitlab.\n\n## Gitlab CI\n\nToday Gitlab not only offers a alternative to Bitbucket Server or GitHub Enterprise, they also offer an [alternate CI-Implementation](https://docs.gitlab.com/ce/ci/README.html):\n\n![cicd_pipeline_infograph](screenshots/cicd_pipeline_infograph.png)\n\nSource: https://docs.gitlab.com/ce/ci/README.html\n\n\n## Why not just use Docker Compose to fire everything up?\n\nBack in 2015 Marcel already fired up everything with Docker Compose (as I mentioned https://github.com/marcelbirkner/docker-ci-tool-stack). It is also easy to fire up Gitlab with Docker Compose locally, just fire up inside this repo:\n\n`docker-compose up -d`\n\n__BUT__: There are some prerequisites needed for Gitlab CI on your Machine: Docker \u0026 Compose have to be installed, and you need to manually install and configure Gitlab Runners - amongst some other things.\n\nWhat I really love is to achieve comprehensible solutions that are usable as is in your project. And you won´t run your Companie´s Gitlab on your local machine, would you?! You will always try to get a server and install Gitlab there. \n\nTo achieve a fully comprehensible setup here, we some DevOps tools FTW:\n\n* Ansible: This shiny pice will contain __ALL__ steps necessary to provision a Gitlab server with __everything__ needed. It´s also a great documentation what´s needed to setup a Gitlab server.\n* Vagrant: To just fire up a server locally that is based on a certain OS - because that´s needed to craft a Ansible playbook. But this is just for demonstration purposes - you can switch over to your Gitlab server by just editing the [hosts](hosts) and adding `[yourcompanygitlab]` together with it´s IP.\n\n\n## Prerequisites\n\nBe sure to have the following tools installed: Ansible, VirtualBox \u0026 Vagrant. On a Mac this is simple:\n\n```\nbrew install ansible\nbrew cask install virtualbox\nbrew cask install vagrant\n```\n\nOur setup uses the [vagrant-dns Plugin](https://github.com/BerlinVagrant/vagrant-dns). Just install it with:\n\n```\nvagrant plugin install vagrant-dns\n```\n\nNow be sure to add your domain name into the [Vagrantfile](Vagrantfile). As I own the domain [jonashackt.io](jonashackt.io), I added the following:\n\n```\n    config.vm.hostname = \"jonashackt\"\n    config.dns.tld = \"io\"\n```\n\nBe sure to configure your local DNS with:\n\n```\nvagrant dns --install\n```\n\nThat´s all, we´re ready to fire up our server with:\n\n```\nvagrant up\n```\n\nIf the server is up and running (this may take a while when doing it for the first time), we can execute Ansible. \n\nLet´s do a connection check first:\n\n```\nansible gitlab-ci-stack -i hosts -m ping\n```\n\nIf this gave a `SUCCESS`, we can move on to really execute our Ansible playbooks.\n\n\u003e All the Installation process is based upon the \"Omnibus GitLab installation\" (NOT the from source option)\n\n\n## Let´s install \u0026 run Gitlab inside our Server/VagrantBox with Ansible\n\n\nTo just execute the playbook and install everything needed to have a fully functional Gitlab-Instance, you only need to run the following Ansible Playbook.\n \nJust be sure to configure your Domain name inside [prepare-gitlab.yml](prepare-gitlab.yml):\n\n```\n  vars:\n    gitlab_domain: \"gitlab.jonashackt.io\"\n``` \n \nand provide `providername`, `providerusername` \u0026 `providertoken` for your DNS Providers´s API in `--extra-vars` (and maybe whitelist your current Internet IP):\n\n```\nansible-playbook -i hosts prepare-gitlab.yml --extra-vars \"providername=yourProviderNameHere providerusername=yourUserNameHere providertoken=yourProviderTokenHere\"\n```\n\nOnly, if you don´t use Vagrant or an only internally accessible Server, you can ignore the extra-vars - Gitlab will handle Let´s Encrypt for you then:\n\n```\nansible-playbook -i hosts prepare-gitlab.yml\n```\n\nIf you want to know more about the installation Process, just read on:\n\n\n\n## Install \u0026 Configure Docker \n\nThe [prepare-docker-ubuntu.yml](prepare-docker-ubuntu.yml) just walks through the [standard Docker docs guide on how to install Docker on Ubuntu](https://docs.docker.com/install/linux/docker-ce/ubuntu/). If you use another Distro, you can simply change modules etc. to match your Linux.\n\nThe only thing special here is how we install Docker Compose, which [is described in the Docs](https://docs.docker.com/compose/install/#install-compose) with that unappealing way of always using a hard-coded version in the curl. Therefore the hint: \n\n\u003e Use the latest Compose release number in the download command.\n\nBut there´s a much nicer way, because [Python PIP](https://pypi.org/project/pip/) has the current package ready for us:\n\n```\n  - name: Install pip\n    apt:\n      name: python3-pip\n      state: latest\n\n  - name: Install Docker Compose\n    pip:\n      name: docker-compose\n```\n\nNow we don´t need to mess with maintaining the Docker Compose version number \u0026 the upgrade process any more!\n\n\n\n## Nice Gitlab URL with DNS configuration\n\nWe don´t want to access Gitlab via a http://locahost:30080 call - instead we want to have something like http://docker.gitlab.ci!\n\nTo enable that on the Host machine, we need the [vagrant-dns Plugin](https://github.com/BerlinVagrant/vagrant-dns). Just install it with:\n\n```\nvagrant plugin install vagrant-dns\n```\n\nNow we configure a domain name such as `jonashackt` and a TLD like `io` in our Vagrantfile:\n\n```\nconfig.vm.hostname = \"jonashackt\"\n\nconfig.dns.tld = \"io\"\n```\n\nNow we need to register the vagrant-dns Server with the TLD `io` as a DNS resolver:\n\n```\nvagrant dns --install\n```\n\nNow check with `scutil --dns` (on a Mac), if the resolver is part of your DNS configuration:\n\n```\n...\n\nresolver #10\n  domain   : io\n  nameserver[0] : 127.0.0.1\n  port     : 5300\n  flags    : Request A records, Request AAAA records\n  reach    : 0x00030002 (Reachable,Local Address,Directly Reachable Address)\n\n...\n```\n\nThis looks good! Now after the usual `vagrant up`, try if you´re able to reach our Vagrant Box using our defined domain by typing e.g. `dscacheutil -q host -a name gitlab.jonashackt.io`:\n\n```\n$:gitlab-ci-stack jonashecht$ dscacheutil -q host -a name gitlab.jonashackt.io\n  name: gitlab.jonashackt.io\n  ip_address: 172.16.2.15\n```\n\n\nBut as we want to have the nice `docker.gitlab.ci` also available inside our Vagrant Box and the [vagrant-dns Plugin](https://github.com/BerlinVagrant/vagrant-dns) doesn´t support propagating the host´s DNS resolver to the Vagrant Boxes, we have a problem.\n \nBut luckily we have [VirtualBox as a virtualization provider for Vagrant](https://www.vagrantup.com/docs/virtualbox/common-issues.html), which supports the propagation of the host´s DNS resolver to the guest machines. All we have to do, is to use [this suggestion on serverfault](https://serverfault.com/a/506206/326340):, which will 'Using the host's resolver as a DNS proxy in NAT mode':\n\n```\n# Forward DNS resolver from host (vagrant dns) to box\nvirtualbox.customize [\"modifyvm\", :id, \"--natdnshostresolver1\", \"on\"]\n```\n\nAfter we configured that, we can do our well-known `vagrant up`.\n\n\nNow we should be able to ping `gitlab.jonashackt.io`.\n\n\n\n## Enable https for Gitlab on public accessable server\n\nhttps://docs.gitlab.com/omnibus/settings/nginx.html#enable-https\n\n\u003e From 10.7 we will automatically use [Let's Encrypt certificates if the external_url specifies https](https://docs.gitlab.com/omnibus/settings/ssl.html#let-39-s-encrypt-integration)), the certificate files are absent, and the embedded nginx will be used to terminate ssl connections.\n\nIf you have an externally accessable server and provision it with these Ansible scripts, you don´t have to worry about the process of obtaining Let´s Encrypt certificates and configuring them for Gitlab. Everything is just done for you by the Gitlab Installation process.\n\n\n## Let´s Encrypt for our Gitlab on VirtualBox/Vagrant\n\n__BUT__: The problem is our local setup here: Let´s Encrypt wont be able to validate the certificate for our domain, since it´s just a local DNS installation.\n\nThat sounds like we´re in need of a different way. Because if we just use our domain with https like https://gitlab.jonashackt.io/, our Browser will complain:\n\n![insecure-https](screenshots/insecure-https.png)\n\nand a `git push` will result into the following problem:\n\n```\n$ git push\nfatal: unable to access 'https://gitlab.jonashackt.io/root/yourRepoNameHere/': SSL certificate problem: self signed certificate\n```\n\nAlthough Let´s Encrypt was designed to be used with public accessable websites, there are ways to create these Certificates for non-public servers also. All you need to have is a __regularly registered domain__ - which maybe sounds like a big issue, but isn´t really a problem! (don´t try to use already registered ones, this won´t work!)\n\nIf you don´t mind about the tld, choose something like `yourDomainName.yxz` or `yourDomainName.online`, which are available from 1$/year. Just be sure to pick [one from this provider list](https://github.com/AnalogJ/lexicon#providers). \n\n__YOU NEED API ACCESS TO PROCEED!__: What all the blog posts only tell you at the end: Besides your regularly registered/purchased domain you´ll need API-access to your provider, which often isn´t included in the standard price of your domain! I choose https://www.namecheap.com/ and to use the [namecheap api](https://www.namecheap.com/support/api), you´ll need to [upgrade your balance to 50$](https://www.namecheap.com/support/knowledgebase/article.aspx/9739/63/api--faq#c).\nAnd you often also need to whitelist the IP your approaching the DNS provider API from - in our case, use a tool like http://www.whatsmyip.org/ to get the IP and add it to your DNS providers API access IP whitelist (just be sure to double check this IP before you call the playbook!)\n\n\n\u003e There has been done some great work in the field of generating Let´s Encrypt certificates for private servers (see https://blog.thesparktree.com/generating-intranet-and-private-network-ssl)\n\nThe playbook [obtain-letsencrypt-certs-dehydrated-lexicon.yml](obtain-letsencrypt-certs-dehydrated-lexicon.yml) just automates all the steps described in the mentioned post. It uses [dehydrated](https://github.com/lukas2511/dehydrated) as an alternative Let´s Encrypt client togehter with [lexicon](https://github.com/AnalogJ/lexicon), which is standardises the way how to manipulate DNS records via their API [on multiple DNS providers](https://github.com/AnalogJ/lexicon#providers). We install both tools with Ansible:\n\n```\n  # This playbook should work for servers, that aren´t accessable from the internet (like our local Vagrant setup here)\n  # Be sure to use a real/purchased domain!\n\n  # The playbook automates all the steps mentioned here https://blog.thesparktree.com/generating-intranet-and-private-network-ssl\n  - name: Update apt\n    apt:\n      update_cache: yes\n\n  - name: Install openssl, curl, sed, grep, mktemp, git\n    apt:\n      name:\n        - openssl\n        - curl\n        - sed\n        - grep\n        - mktemp\n        - git\n      state: latest\n\n  # install this neat tool https://github.com/lukas2511/dehydrated\n  - name: Install dehydrated\n    git:\n      repo: 'https://github.com/lukas2511/dehydrated.git'\n      dest: /srv/dehydrated\n\n  - name: Make dehydrated executable\n    file:\n      path: /srv/dehydrated/dehydrated\n      mode: \"+x\"\n\n  - name: Specify our internal domain\n    shell: \"echo '{{ gitlab_domain }}' \u003e /srv/dehydrated/domains.txt\"\n\n  - name: Install build-essential, python-dev, libffi-dev, python3-pip\n    apt:\n      name:\n        - build-essential\n        - python-dev\n        - libffi-dev\n        - libssl-dev\n        - python3-pip\n      state: latest\n\n  - name: Install requests[security]\n    pip:\n      name: \"requests[security]\"\n\n  # install this neat tool https://github.com/AnalogJ/lexicon\n  - name: Install dns-lexicon with correct provider (dns-lexicon[providernamehere])\n    pip:\n      name: \"dns-lexicon[{{providername|lower}}]\"\n```\n\nAs we don´t have a publicly accessable server, we need to use `dns-01` challenges instead of the Let´s Encrypt standard `http-01`. Therefor dehydrated needs a hook file to work with `dns-01`. [lexicon](https://github.com/AnalogJ/lexicon) has such a file for us [/examples/dehydrated.default.sh](https://github.com/AnalogJ/lexicon/blob/master/examples/dehydrated.default.sh) and we copy it simply inside our playbook:\n\n```\n  - name: Configure lexicon with Dehydrated hook for dns-01 challenge\n    get_url:\n      url: https://raw.githubusercontent.com/AnalogJ/lexicon/master/examples/dehydrated.default.sh\n      dest: /srv/dehydrated/dehydrated.default.sh\n      mode: \"+x\"\n```\n\nAt that point we need to use some private information about your DNS provider - because remember, the whole process could __only be done, if you have access to a real domain__. In order to grant lexicon access to your DNS provider´s API, we set some environment variables and then execute dehydrated:\n\n```\n  # be sure to check https://github.com/AnalogJ/lexicon#providers\n  # the env variables are constructed with LEXICON_{DNS Provider Name}_{Auth Type}\n  # since, the dynamic key name like LEXICON_{DNS Provider Name}_{Auth Type}, we can´t use the standard approach (http://docs.ansible.com/ansible/latest/user_guide/playbooks_environment.html)\n  # because our Environment variable key names are dynamic based on the Provider name. Therefor we use the hint in https://stackoverflow.com/a/44570290/4964553\n  # and construct the variables with \"{'dynamic environment variable key name inkl. {{ vars }}':'{{ dynamic environment variable value}}'}\"\n  # If everything went fine, this should place the new Let´s Encrypt Certificates into /srv/dehydrated/certs/{{ gitlab_domain }}\n  - name: Generate Certificates\n    shell: \"/srv/dehydrated/dehydrated --cron --hook /srv/dehydrated/dehydrated.default.sh --challenge dns-01 --accept-terms\"\n    environment:\n      - PROVIDER: \"{{providername|lower}}\"\n      - \"{'LEXICON_{{providername|upper}}_USERNAME':'{{providerusername}}'}\"\n      - \"{'LEXICON_{{providername|upper}}_TOKEN':'{{providertoken}}'}\"\n    ignore_errors: true\n```\n\nAs you can see, all environment variables are set with the help of Ansible´s `--extra-vars` CLI like this:\n\n```\nansible-playbook -i hosts prepare-gitlab.yml --skip-tags \"docker,install_gitlab,gitlab_runner\" --extra-vars \"providername=yourProviderNameHere providerusername=yourUserNameHere providertoken=yourProviderTokenHere\"\n```\n\nJust change `yourProviderNameHere`, `yourUserNameHere` and `yourProviderTokenHere` accordingly!\n\n\n### Configure the \"self-created\" Let´s Encrypt certificates in Gitlab (it´s the same process for other certificates then Let´s Encrypt)\n\n\u003e Please don´t get confused with this part of the docs: https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-certificate-authorities, since that´s only needed, if you want to install a custom certificate authority and not necessary for correctly created Let´s Encrypt certificates (since the Let´s Encrypt authority is already trusted!)\n\n[According to the docs there are 2 ways to configure HTTPS in Gitlab](https://docs.gitlab.com/omnibus/settings/nginx.html): The [automatic Let´s Encrypt way](https://docs.gitlab.com/omnibus/settings/ssl.html#let-39-s-encrypt-integration), which we sadly can´t use in our scenario here, where our Vagrant Box isn´t publically accessible - and [the way to manually configure HTTPS](https://docs.gitlab.com/omnibus/settings/nginx.html#manually-configuring-https), the one we need to choose here.\n\nTherefore we set the `external_url` via the Environment variable `EXTERNAL_URL: \"{{gitlab_url}}\"` at the Gitlab Omnibus installation process to contain an `https`, in our example here: `https://gitlab.jonashackt.io` (remember you need to own the domain or at least need access to the DNS provider for the [Let´s Encrypt process for internal servers](https://github.com/jonashackt/gitlab-ci-stack#lets-encrypt-for-our-gitlab-on-virtualboxvagrant)).\n\nAfter that, Gitlab Omnibus installation will look for certificates named `/etc/gitlab/ssl/gitlab.jonashackt.io.key` \u0026 `/etc/gitlab/ssl/gitlab.jonashackt.io.crt` - note that both file names are derived from the domain name `gitlab.jonashackt.io`.\n\nEverything needed is done by the [letsencrypt.yml](letsencrypt.yml), it will just copy the generated certificates with the correct name to the correct location:\n\n```\n  - name: Create Gitlab cert import folder /etc/gitlab/ssl for later Gitlab Installation usage\n    file:\n      path: /etc/gitlab/ssl\n      state: directory\n    when: success\n\n  - name: Copy certificate files to Gitlab cert import folder /etc/gitlab/ssl (see https://docs.gitlab.com/omnibus/settings/ssl.html#details-on-how-gitlab-and-ssl-work)\n    copy:\n      src: \"{{ item.src }}\"\n      dest: \"{{ item.dest }}\"\n      remote_src: yes\n    with_items:\n      - src: \"/srv/dehydrated/certs/{{ gitlab_domain }}/fullchain.pem\"\n        dest: \"/etc/gitlab/ssl/{{ gitlab_domain }}.crt\"\n\n      - src: \"/srv/dehydrated/certs/{{ gitlab_domain }}/privkey.pem\"\n        dest: \"/etc/gitlab/ssl/{{ gitlab_domain }}.key\"\n\n    when: success\n```\n\nNow you can use your Gitlab without cryptic error messages because of self-signed certificates:\n\n![complete_https_letsencrypt_gitlab](screenshots/complete_https_letsencrypt_gitlab.png) \n\n\n\n## Install Gitlab itself\n\nThe playbook [install-gitlab.yml](install-gitlab.yml) will walk through the standard Gitlab installation guide for Ubuntu - just automatically: https://about.gitlab.com/installation/#ubuntu (others are available also [like CentOS](https://about.gitlab.com/installation/#centos-6):\n\n```\n  - name: Update apt and autoremove\n    apt:\n      update_cache: yes\n      cache_valid_time: 3600\n      autoremove: yes\n\n  - name: Install curl, openssh-server, ca-certificates \u0026 postfix\n    apt:\n      name:\n        - curl\n        - openssh-server\n        - ca-certificates\n        - postfix\n      state: latest\n\n  - name: Add the GitLab package repository\n    shell: \"curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | sudo bash\"\n\n  - name: Update apt and autoremove\n    apt:\n      update_cache: yes\n      cache_valid_time: 3600\n      autoremove: yes\n\n  - name: Install Gitlab with Omnibus-Installer\n    apt:\n      name: gitlab-ce\n      state: latest\n    environment:\n      EXTERNAL_URL: \"{{gitlab_url}}\"\n    ignore_errors: true\n    register: gitlab_install_result\n\n  - name: Gitlab Omnibus is based on Chef and will give many insight, what it does in the background\n    debug:\n      msg:\n       - \"The installation process said the following: \"\n       - \"{{gitlab_install_result.stdout_lines}}\"\n\n  - name: Wait for Gitlab to start up\n    wait_for:\n      port: 443\n      delay: 10\n      sleep: 5\n\n  - name: Let´s check, if Gitlab is up and running\n    uri:\n      url: \"{{gitlab_url}}\"\n\n```\n\n\n\n\n## Gitlab Container Registry\n\nhttps://docs.gitlab.com/ee/user/project/container_registry.html\n\n[Gitlab Container Registry domain configuration](https://docs.gitlab.com/ee/administration/container_registry.html#container-registry-domain-configuration)\n\nIf we just use our configured domain, we can follow the docs here: https://docs.gitlab.com/ee/administration/container_registry.html#configure-container-registry-under-an-existing-gitlab-domain\n\n```\n  - name: Activate Container Registry in /etc/gitlab/gitlab.rb\n    lineinfile:\n      path: /etc/gitlab/gitlab.rb\n      line: \" registry_external_url '{{ gitlab_registry_url }}'\"\n\n  - name: Reconfigure Gitlab to activate Container Registry\n    shell: \"gitlab-ctl reconfigure\"\n    register: reconfigure_result\n\n  - name: Let´s see what Omnibus/Chef does\n    debug:\n      msg:\n       - \"The reconfiguration process gave the following: \"\n       - \"{{reconfigure_result.stdout_lines}}\"\n```\n\nThe playbook [configure-gitlab-registry.yml](configure-gitlab-registry.yml) inserts the following needed config into the gitlab.rb: `registry_external_url 'https://gitlab.jonashackt.io:4567'` and this follows after a `sudo gitlab-ctl reconfigure`:\n\n![configuring-gitlab-docker-registry](screenshots/configuring-gitlab-docker-registry.png)\n\n\n\n### Ubuntu \u0026 Docker don´t know Let´s Encrypt so we need to copy the fullchain.pem instead of just the cert.pem!\n\nThe next problem was the following error while registering the gitlab-runners in [letsencrypt.yml](letsencrypt.yml):\n\n```\nERROR: Registering runner... failed\nrunner=gyy8axxP status=couldn't execute POST against https://gitlab.jonashackt.io/api/v4/runners: Post https://gitlab.jonashackt.io/api/v4/runners: x509: certificate signed by unknown authority\nPANIC: Failed to register this runner. Perhaps you are having network problems\n```\n\nTo avoid that for now, I added the `--tls-ca-file` option like so: `--tls-ca-file=/etc/gitlab/ssl/gitlab.jonashackt.io.crt` which is described here: https://gitlab.com/gitlab-org/gitlab-runner/issues/230\n      \nBut, then the next problem occured inside the Pipeline at the first step, where we want to use the Gitlab Container Registry:\n\n```\nError response from daemon: Get https://gitlab.jonashackt.io:5000/v2/: x509: certificate signed by unknown authority\nERROR: Job failed: exit status 1\n```\n \nLuckily this problem was already solved by https://github.com/bkcsfi in https://github.com/moby/moby/issues/31602#issuecomment-379955352, where he states that the \n\n```\nI think you probably want to use fullchain.pem instead of cert.pem because neither docker (go lib) nor (ubuntu in my case) have LE root cert built in at this time\n\nI have gitlab setup with LE certificate. Browser works fine, but docker fails to push to registry.\n```\n\nSo all we have to do to tell the Gitlab Container Registry about our Let´s Encrypt certificates is to copy `/srv/dehydrated/certs/{{ gitlab_domain }}/fullchain.pem` instead of `/srv/dehydrated/certs/{{ gitlab_domain }}/cert.pem` to Gitlab certificates. \n\nThis configures also the Gitlab Container Registry, although it seems to be an configuration option for Gitlab itself only - just have a [look into the docs](https://docs.gitlab.com/ee/administration/container_registry.html#configure-container-registry-under-an-existing-gitlab-domain):\n\n\u003e If your TLS certificate is not in /etc/gitlab/ssl/gitlab.example.com.crt and key not in /etc/gitlab/ssl/gitlab.example.com.key uncomment the lines below ...\n\n--\u003e As our certiticates are named accordingly with the correct domain name, the Gitlab Container Registry also uses these certificates (and now our fullchain.pem). You can observe that while running an `sudo gitlab-ctl reconfigure` after you manually activated the `registry_external_url 'https://gitlab.jonashackt.io:4567'` inside the `/etc/gitlab/gitlab.rb`:\n\n```\n    ...\n\n    - create new file /var/opt/gitlab/nginx/conf/gitlab-registry.conf\n    - update content in file /var/opt/gitlab/nginx/conf/gitlab-registry.conf from none to 38ba8d\n    --- /var/opt/gitlab/nginx/conf/gitlab-registry.conf\t2018-05-23 07:06:18.857687999 +0000\n    +++ /var/opt/gitlab/nginx/conf/.chef-gitlab-registry20180523-13668-614sno.conf\t2018-05-23 07:06:18.857687999 +0000\n    @@ -1 +1,59 @@\n    +# This file is managed by gitlab-ctl. Manual changes will be\n    +# erased! To change the contents below, edit /etc/gitlab/gitlab.rb\n    +# and run `sudo gitlab-ctl reconfigure`.\n    +\n    +## Lines starting with two hashes (##) are comments with information.\n    +## Lines starting with one hash (#) are configuration parameters that can be uncommented.\n    +##\n    +###################################\n    +##         configuration         ##\n    +###################################\n    +\n    +\n    +server {\n    +  listen *:4567 ssl;\n    +  server_name  gitlab.jonashackt.io;\n    +  server_tokens off; ## Don't show the nginx version number, a security best practice\n    +\n    +  client_max_body_size 0;\n    +  chunked_transfer_encoding on;\n    +\n    +  ## Strong SSL Security\n    +  ## https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html \u0026 https://cipherli.st/\n    +  ssl on;\n    +  ssl_certificate /etc/gitlab/ssl/gitlab.jonashackt.io.crt;\n    +  ssl_certificate_key /etc/gitlab/ssl/gitlab.jonashackt.io.key;\n    \n    ...\n```\n\nWith this, we also don´t need to use the `--tls-ca-file` option to configure our gitlab-runners in [letsencrypt.yml](letsencrypt.yml) - the corresponding error is also gone now! \n\n\n\n\n## Install Gitlab Runner\n\nThe [gitlab-runner.yml](gitlab-runner.yml) shows how to install and register the Gitlab Docker Runner in non-interactive mode:\n\nAs https://docs.gitlab.com/runner/install/linux-repository.html#installing-the-runner states, we need to add the Gitlab Runner package repository and install the Runner via apt-get.\n\nBefore we´re able to register the Runner, we need to extract the Registration Token somehow automatically from our Gitlab instance. Since there´s no API at the moment (see https://gitlab.com/gitlab-org/gitlab-ce/issues/24030, https://gitlab.com/gitlab-org/gitlab-runner/issues/1727), we need to obtain it quite hacky through a database call.\n\n```\n  # To register the Gitlab Runner, we need to obtain the Registration Token from our Gitlab instance\n  # Because this will change every time we start up Gitlab (and/or Vagrant Box/Ansible setup, see https://gitlab.com/gitlab-org/gitlab-ce/issues/3703)\n  # we need to access it somehow. Sadly there´s no API atm (see https://gitlab.com/gitlab-org/gitlab-ce/issues/24030,\n  # https://gitlab.com/gitlab-org/gitlab-runner/issues/1727), so we have to dive directly into the Gitlab database :(\n  - name: Extract Runner Registration Token via SQL query\n    become: true\n    become_user: gitlab-psql\n    vars:\n        ansible_ssh_pipelining: true\n        query: \"SELECT runners_registration_token FROM application_settings ORDER BY id DESC LIMIT 1\"\n        psql_exec: \"/opt/gitlab/embedded/bin/psql\"\n        gitlab_db_name: \"gitlabhq_production\"\n    shell: '{{ psql_exec }} -h /var/opt/gitlab/postgresql/ -d {{ gitlab_db_name }} -t -A -c \"{{ query }}\"'\n    register: gitlab_runner_registration_token_result\n```\n\n\n## Configure Gitlab Runner with shell executor\n\nAs [the docs state](https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#runner-configuration):\n\n\u003e There are three methods to enable the use of docker build and docker run during jobs; each with their own tradeoffs.\n\n__Don't get confused:__ those three methods could be either applied to a the GitLab runner package installed directly on the host OS - or as an alternative to a GitLab runner pre-installed inside [a GitLab-provided Docker image called gitlab/gitlab-runner](https://hub.docker.com/r/gitlab/gitlab-runner/) - [see the docs for details](https://docs.gitlab.com/runner/register/#one-line-registration-command)\n\nAs \"The simplest approach is to install GitLab Runner in shell execution mode\", we use the `shell` executor for our setup primarily. The following ASCII art shows the simplicity of this approach:\n\n```\n                                    +----------------------------------------+\n                                    |               shell runner             |\n                                    |                                        |\n                                    |                   uses                 |\n                                    |                                        |\n                                    |            HOST Docker-engine          |\n                                    |                                        |\n                                    |       with local image repository      |                                                                    \n                                    +----------------------------------------+\n```\n\nTo register the Gitlab Docker Runner in [non-interactive mode](https://gitlab.com/gitlab-org/gitlab-runner/blob/master/docs/commands/README.md#non-interactive-registration), we do the following inside our playbook:\n\n```\n  # see https://docs.gitlab.com/ce/ci/docker/using_docker_images.html#register-docker-runner\n  # and this for non-interactive mode:\n  # https://gitlab.com/gitlab-org/gitlab-runner/blob/master/docs/commands/README.md#non-interactive-registration\n  - name: Register Gitlab-Runners using shell executor\n    shell: \"gitlab-runner register --non-interactive --url '{{gitlab_url}}' --registration-token '{{gitlab_runner_registration_token}}' --description 'shell-runner-{{ item }}' --executor shell --tag-list shell\"\n    loop: \"{{ range(1,gitlab_runner_count + 1)|list }}\"\n```\n\n__Attention!__ Do not confuse these runner configurations with the \"Non-Docker-in-Docker\" gitlab-runner also named \"docker\"!\n\nIf you don't want to go with the flexible and locally testable solution using a Dockerfile and docker commands directly inside your `.gitlab-ci.yml` (be aware of the fact, that you can't develop your pipeline locally right now because of the missing pieces in the `gitlab-runner exec` implementation! (see https://gist.github.com/jonashackt/2cfbf366a6a6b70a78068ab043edb8f7 for details)), then there's another - sadly widly used - way of how to register GitLab runners described here: https://docs.gitlab.com/ce/ci/docker/using_docker_images.html#register-docker-runner __But as with its predecessors like Jenkins, GitLab must not be the goto CI solution in the future - and if you want to be able to change your CI system fast, I would advice you to NOT USE this way of GitLab CI!__.\n\n\n### Configure a Docker-in-Docker enabled gitlab-runner with the docker executor\n\nThe second option on how to use standard Docker commands inside your `.gitlab-ci.yml`, is to use Docker-in-Docker (Dind) gitlab-runners - see https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#use-docker-in-docker-workflow-with-docker-executor\n\nUsing this option, __the setup get's much more complex!__ In [this issue Tomasz Maczukin explains the setup very well](https://gitlab.com/gitlab-org/gitlab-ce/issues/41227):\n\n```\n                                    +----------------------------------------+\n                                    |     HOST (docker-engine 1st level)     |\n                                    |                                        |\n                                    | /volume/for/builds # this is mounted   |\n                                    |                      to all containers |\n                                    |                      related to a job  |\n                                    +----------------------------------------+\n                                         ^                             ^\n                                         |                             |\n+----------------------------------------------+                 +----------------------------------------------+\n|             docker:dind container            |       link      |         job container (docker:latest)        |\n|                                              |----------------\u003e|                                              |\n| /       #containers root directory           | srvs_cnt:docker | /       #containers root directory           |\n| /builds # builds directory mounted from host |                 | /builds # builds directory mounted from host |\n|                                              |                 +----------------------------------------------+\n|    with container local image repository     | \n+----------------------------------------------+                 \n\n```\n\nThere is also [a good blog post on the fallacies of Docker-in-Docker](https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/). If you still want to go that way though, read on.\n\nTherefore we register our Dind runner like this - [incl. TLS enablement](https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#tls-enabled) mounting the host certs therefore with ` --docker-volumes '/certs/client'` and as stated in the docs we also pin to `--docker-image 'docker:19.03.1'` the Docker version to prevent \"unpredictable behavior, especially when new versions are released\".\n\nA downside of the Docker-in-Docker approach is also the usage of `--docker-privileged`, which can lead to security implications because we disable the security mechanisms of containers:\n\n```\n  - name: Register Gitlab-Runners using docker executor too\n    shell: \"gitlab-runner register --non-interactive --url '{{gitlab_url}}' --registration-token '{{gitlab_runner_registration_token}}' --description 'docker-in-docker-runner-{{ item }}' --executor docker --docker-image 'docker:19.03.1' --docker-privileged --docker-volumes '/certs/client' --tag-list dind\"\n    loop: \"{{ range(1,gitlab_runner_count + 1)|list }}\"\n\n```\n\n### Changes needed in .gitlab-ci.yml for Docker-in-Docker compared to using a shell runner\n\nThe example project https://github.com/jonashackt/gitlab-ci-dind-example provides a fully comprehensible example on how to use the Docker-in-Docker GitLab runner inside a [.gitlab-ci.yml](https://github.com/jonashackt/gitlab-ci-dind-example/blob/master/.gitlab-ci.yml):\n\n```\n# This .gitlab-ci.yml is an extension of the example provided in: \n# https://github.com/jonashackt/restexamples/blob/master/.gitlab-ci.yml\n# We use Docker in Docker here with a docker executor instead of the shell one\n# --\u003e Pinning the right Docker version for the service\nimage: docker:19.03.1\n\n# Pinning the right Docker version for the service also\nservices:\n  - docker:19.03.1-dind\n\nvariables:\n  # see https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#tls-enabled for Dind configuration\n  # DOCKER_HOST: tcp://docker:2375 --\u003e this should only be configured when using Kubernetes runners\n  # When using dind, it's wise to use the overlayfs driver for\n  # improved performance.\n  DOCKER_DRIVER: overlay2\n  # Specify to Docker where to create the certificates, Docker will\n  # create them automatically on boot, and will create\n  # `/certs/client` that will be shared between the service and job\n  # container, thanks to volume mount from config.toml\n  DOCKER_TLS_CERTDIR: \"/certs\"\n  # see usage of Namespaces at https://docs.gitlab.com/ee/user/group/#namespaces\n  REGISTRY_GROUP_PROJECT: $CI_REGISTRY/root/gitlab-ci-dind-example\n\n# One of the new trends in Continuous Integration/Deployment is to:\n# (see https://docs.gitlab.com/ee/ci/docker/using_docker_build.html)\n#\n# 1. Create an application image\n# 2. Run tests against the created image\n# 3. Push image to a remote registry\n# 4. Deploy to a server from the pushed image\n\nstages:\n  - build\n  - test\n  - push\n  - deploy\n\n# see how to login at https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#using-the-gitlab-container-registry\nbefore_script:\n  - docker login -u $CI_REGISTRY_USER -p $CI_JOB_TOKEN $CI_REGISTRY\n\nbuild-image:\n  stage: build\n  # the tag 'dind' advices only GitLab runners using this tag to pick up that job\n  tags: \n    - dind\n  script:\n    - docker build . --tag $REGISTRY_GROUP_PROJECT/gitlab-ci-dind-example:latest\n    \n...\n```\n\nBe sure to use a version-pinned Docker image with `image: docker:19.03.1` - since the configuration of the `.gitlab-ci.yml` depends on which Docker version you use. This setup here uses Docker `19.03.1`, from which on TLS-secured communication with the Docker daemon is the default (see https://about.gitlab.com/2019/07/31/docker-in-docker-with-docker-19-dot-03/ for more details).\n\nThe corresponding `dind` GitLab CI service should also be defined: \n\n```\nservices:\n  - docker:19.03.1-dind\n```\n\nNow if you don't use Kubernetes, than __you don't__ need to define the `DOCKER_HOST` variable starting from Docker `19.03.` on!\n\nTo speed up building speed, it's a good advice to use the overlayfs driver with `DOCKER_DRIVER: overlay2`.\n\nUsing `DOCKER_TLS_CERTDIR: \"/certs\"` tells Docker-in-Docker where to find the Docker generated certificates and establish a secured TLS connection to the Daemon.\n\nThe last step is to use `tags` to define the correct Gitlab runner for the `dind` jobs (see `Configure .gitlab-ci.yml Jobs to run only on specific gitlab-runners` paragraph).\n\n\n### Configure a Docker socket binding enabled gitlab-runner with the docker executor\n\nThe [third option to to enable the use of docker build and docker run during jobs is to a Docker socket binding](https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#use-docker-socket-binding). It's also [often referred to a the \"better Docker-in-Docker\" option](https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/).\n\nSo let's also configure 2 GitLab runners with Docker socket binding enabled. We don't need to use `--docker-privileged` here, we just need to bind-mount the Docker socket:\n\n```\n  - name: Register Gitlab-Runners using docker executor too\n    shell: \"gitlab-runner register --non-interactive --url '{{gitlab_url}}' --registration-token '{{gitlab_runner_registration_token}}' --description 'docker-socket-runner-{{ item }}' --executor docker --docker-image 'docker:stable' --docker-volumes /var/run/docker.sock:/var/run/docker.sock --tag-list socket\"\n    loop: \"{{ range(1,gitlab_runner_count + 1)|list }}\"\n\n```\n\nWe also tag this runner as the `socket` one, so we can explicitly let our Jobs run on that one.\n\n\n### Changes needed in .gitlab-ci.yml for Docker-in-Docker compared to using a shell runner\n\nThe example project https://github.com/jonashackt/gitlab-ci-docker-socket-binding-example provides a fully comprehensible example on how to use the a Docker socket bound GitLab runner inside a [.gitlab-ci.yml](https://github.com/jonashackt/gitlab-ci-dind-example/blob/master/.gitlab-ci.yml):\n\n```\n# see https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#use-docker-socket-binding\nimage: docker:stable\n\n# One of the new trends in Continuous Integration/Deployment is to:\n# (see https://docs.gitlab.com/ee/ci/docker/using_docker_build.html)\n#\n# 1. Create an application image\n# 2. Run tests against the created image\n# 3. Push image to a remote registry\n# 4. Deploy to a server from the pushed image\n\nstages:\n  - build\n  - test\n  - push\n  - deploy\n\n# see usage of Namespaces at https://docs.gitlab.com/ee/user/group/#namespaces\nvariables:\n  REGISTRY_GROUP_PROJECT: $CI_REGISTRY/root/gitlab-ci-shell-example\n\n# see how to login at https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#using-the-gitlab-container-registry\nbefore_script:\n  - docker login -u $CI_REGISTRY_USER -p $CI_JOB_TOKEN $CI_REGISTRY\n\nbuild-image:\n  stage: build\n  # the tag 'shell' advices only GitLab runners using this tag to pick up that job\n  tags:\n    - socket\n  script:\n    - docker build . --tag $REGISTRY_GROUP_PROJECT/gitlab-ci-shell-example:latest\n\n...\n```\n\nBesides the project name, there are only 2 differences to the shell runner needed here: Using `image: docker:stable` as the image for this pipeline and defining `socket` tag so that GitLab will only pick the correct Docker socket binding enabled runners for these jobs.\n\n\n### Configure .gitlab-ci.yml Jobs to run only on specific gitlab-runners\n\nAs we register our gitlab-runners with tags like `dind` and `shell` - as we can see inside the runners configuration in the GitLab GUI also:\n\n![gitlab-runners-with-tags](screenshots/gitlab-runners-with-tags.png)\n\nSo we can now configure our GitLab CI jobs to only run on these specific gitlab-runners - see https://docs.gitlab.com/ee/ci/yaml/#tags:\n\nUse gitlab-runner 'shell':\n\n```\nbuild-image:\n  stage: build\n  tags: \n    - shell\n  script:\n    - mvn clean install\n```\n\nOr use gitlab-runner 'dind':\n\n```\nbuild-image:\n  stage: build\n  tags: \n    - dind\n  script:\n    - mvn clean install\n```\n\nand the Jobs will only be picked by the specifically tagges runners.\n\n\n### Using the Gitlab Container Registry\n\nhttps://docs.gitlab.com/ee/ci/docker/using_docker_build.html#using-the-gitlab-container-registry\n\nhttps://docs.gitlab.com/ee/user/project/container_registry.html#build-and-push-images\n\n\u003e Caution: Mind the Namespaces when working with Gitlab Container Registry!!!\n\n### Namespaces (username, group or subgroup)\n\nhttps://docs.gitlab.com/ee/user/group/#namespaces\n\n\n## Import example project with .gitlab-ci.yml and run Gitlab CI pipeline\n\nNow we´re nearly there. Just add a new password for the root user and login with that credentials. Then head over to to `Create a project` and there click on `Import Project` / `Repo by URL`:\n\n![import-project](screenshots/import-project.png)\n\nPaste the example Projects git URL into __Git repository URL__ field: `https://github.com/jonashackt/gitlab-ci-shell-example.git`, change Visibility Level to __Internal__ and hit __Create Project__.\n\nNow at __CI / CD__ / __Pipelines__ fire up the Pipeline once (only this time manually since we didn´t push something new) and it should build a simple Spring Boot example project and push the resulting Image into our branch new Gitlab Container Registry:\n\n![successful-first-pipeline-run](screenshots/successful-first-pipeline-run.png)\n\nThe example project [gitlab-ci-shell-example](https://github.com/jonashackt/gitlab-ci-shell-example) has a [prepared .gitlab-ci.yml already](https://github.com/jonashackt/gitlab-ci-shell-example/blob/master/.gitlab-ci.yml), so it should do everything smoothly:\n\n```\n# One of the new trends in Continuous Integration/Deployment is to:\n# (see https://docs.gitlab.com/ee/ci/docker/using_docker_build.html)\n#\n# 1. Create an application image\n# 2. Run tests against the created image\n# 3. Push image to a remote registry\n# 4. Deploy to a server from the pushed image\n\nstages:\n  - build\n  - test\n  - push\n  - deploy\n\n# see usage of Namespaces at https://docs.gitlab.com/ee/user/group/#namespaces\nvariables:\n  REGISTRY_GROUP_PROJECT: $CI_REGISTRY/root/restexamples\n\n# see how to login at https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#using-the-gitlab-container-registry\nbefore_script:\n  - docker login -u $CI_REGISTRY_USER -p $CI_JOB_TOKEN $CI_REGISTRY\n\nbuild-image:\n  stage: build\n  script:\n    - docker build . --tag $REGISTRY_GROUP_PROJECT/restexamples:latest\n\ntest-image:\n  stage: test\n  script:\n    - echo Insert fancy API test here!\n\npush-image:\n  stage: push\n  script:\n    - docker push $REGISTRY_GROUP_PROJECT/restexamples:latest\n\ndeploy-2-dev:\n  stage: deploy\n  script:\n    - echo You should use Ansible here!\n  environment:\n    name: dev\n    url: https://dev.jonashackt.io\n```\n\nAnd you should be able to see your newly pushed Image in the Gitlab Registry overview:\n\n![gitlab-registry-overview](screenshots/gitlab-registry-overview.png)\n\nAnd there´s also the Environments tab, witch is a great view on your Pipeline. Here you can track which Deployment went to which infrastructural stage:\n\n![gitlab-environments](screenshots/gitlab-environments.png)\n\n\n\n## GitLab Pages on self-hosted GitLab-Instance\n\nGitLab supports to publish (and host) websites that are generated with a static site generator like Jekyll (https://docs.gitlab.com/ee/user/project/pages/).\n\nhttps://docs.gitlab.com/ee/administration/pages/index.html\n\n\n### Define another domain name for Pages\n\n[As the docs state](https://docs.gitlab.com/ee/administration/pages/index.html#prerequisites):\n\n\u003e Have an exclusive root domain for serving GitLab Pages. Note that you cannot use a subdomain of your GitLab’s instance domain.\n\nWhich is mainly because of security reasons: `You should strongly consider running GitLab pages under a different hostname than GitLab to prevent XSS attacks`. GitLab and GitLab Pages would also share the same cookies and so on...\n\nBut: You don´t need a \"real\" root domain (which I interpret as seperate [Top-level domain](https://en.wikipedia.org/wiki/Top-level_domain) (TLD) ). I really struggled in the first place trying to use another TLD here. But we can simple serve our GitLab Pages at `pages.jonashackt.io`, which is simply another subdomain of the same top-level domain!\n\nThe great Vagrant DNS is [able to use multiple TLDs](https://github.com/BerlinVagrant/vagrant-dns#vm-options) with the `vm.dns.tlds` configuration key. Our [Vagrantfile](Vagrantfile) would then look like this:\n\n```\n# instead of\nconfig.dns.tld = \"io\"\n\n# we now use\nconfig.dns.tlds = [\"io\", \"me\"]\n```\n\nA `scutil --dns` would reveal a new resolver together with our `io` one:\n\n```\nresolver #11\n  domain   : io\n  nameserver[0] : 127.0.0.1\n  port     : 5300\n  flags    : Request A records, Request AAAA records\n  reach    : 0x00030002 (Reachable,Local Address,Directly Reachable Address)\n\nresolver #12\n  domain   : me\n  nameserver[0] : 127.0.0.1\n  port     : 5300\n  flags    : Request A records, Request AAAA records\n  reach    : 0x00030002 (Reachable,Local Address,Directly Reachable Address)\n```\n\nBut it's nice, we don't need that here!\n\n\n### Let´s Encrypt Wildcard certificates with dehydrated \u0026 lexicon\n\nLet´s Encrypt is able to issue wildcard certificates [since early 2018](https://www.heise.de/security/meldung/Let-s-Encrypt-stellt-ab-sofort-Wildcard-Zertifikate-aus-3994552.html).\n\nAnd our Let´s Encrypt client dehydrated is on [the list of ACME v2 supporting clients](https://letsencrypt.org/docs/client-options/), so it should be possible to [get a Wildcard certificate using dehydraded](https://github.com/lukas2511/dehydrated/blob/master/docs/domains_txt.md#wildcards):\n\n\u003e Support for wildcards was added by the ACME v2 protocol. \n\nWith this we simply need to create an `domains.txt` file containing the following:\n\n```\ngitlab.jonashackt.io\npages.jonashackt.io *.pages.jonashackt.io\n```\n\n\u003e Be sure to verify if your DNS provider supports multiple TXT records for one domain! (see https://github.com/lukas2511/dehydrated/blob/master/docs/troubleshooting.md#dns-invalid-challenge-since-dehydrated-060--why-are-dns-challenges-deployed-first-and-verified-later)\n\nWe´re doing that right inside our [obtain-letsencrypt-certs-dehydrated-lexicon.yml](obtain-letsencrypt-certs-dehydrated-lexicon.yml) playbook using the Ansible copy module together with the `content` configuration:\n\n```yaml\n # In addition to the GitLab domain, we need to issue a wildcard certificate for GitLab Pages\n  # see https://github.com/lukas2511/dehydrated/blob/master/docs/domains_txt.md#wildcards\n  # in the format service.example.com *.service.example.com\n  - name: Specify our domains\n    copy:\n      dest: \"/srv/dehydrated/domains.txt\"\n      content: |\n        {{ gitlab_domain }}\n        {{ gitlab_pages_domain }} *.{{ gitlab_pages_domain }}\n```\n\nThe later executed dehydrated command will pick up the `domains.txt` file and retrieve all certificates for the domains mentioned inside the file:\n\n```bash\n/srv/dehydrated/dehydrated --cron --hook /srv/dehydrated/dehydrated.default.sh --challenge dns-01 --accept-terms\n``` \n\n\n### Configure Pages in GitLab with Ansible\n\nSimply run all the playbooks again or\n \n```\n# freshly generate certificates for GitLab pages with \nansible-playbook -i hosts prepare-gitlab.yml --tags \"letsencrypt\" --extra-vars \"providername=yourProviderNameHere providerusername=yourUserNameHere providertoken=yourProviderTokenHere\"\n\n# then configure GitLab pages with \nansible-playbook -i hosts prepare-gitlab.yml --tags \"pages\"\n```\n\nAfter the successful playbook run you should be able to spot the new __Pages__ menu:\n\n![gitlab-pages](screenshots/gitlab-pages.png)\n\n\n\n### A new Jekyll site\n\nThere is a huge list of possible static site generators for the use with GitLab Pages. Just have a look here: https://gitlab.com/pages\n\nI took Jekyll just out because I already know it from my personal site leveraging GitHub Pages: https://github.com/jonashackt/jonashackt.github.io\n\nUsing an example project would be easy - just pick https://gitlab.com/pages/jekyll and import it into your GitLab instance (__New Project__ / __Import__ etc).\n\nBut for comprehensibility reasons, let´s create a new [Jeykll](https://jekyllrb.com/) project from scratch. You´ll need some steps to accomplish this, since there are prerequisites (I'am using a Mac here, see this page for other OSses: https://jekyllrb.com/docs/installation/#requirements)\n\n\u003e make sure your ruby installation works and doesn´t interfer with Apple's system-wide ruby installation (which will lead to bad errors!)\n\n```\n# you need to have a ruby installation in place:\nbrew install ruby\n\n# be sure to use the correct (brew installed) ruby version on your Mac - and not the system wide Apple version!\n# therefore upgrade your user/.bash_profile file with adding:\nexport PATH=\"/usr/local/opt/ruby/bin:$PATH\"\n\n# upgrade to latest RubyGems (don´t do this with sudo(!), which would again alter your system wide Apple ruby version)\ngem update --system\n\n# then we need to install the packages bundler and jekyll\ngem install bundler jekyll\n```\n\nLet's create a new git repository inside GitLab:\n\n![gitlab-pages-empty-repository](screenshots/gitlab-pages-empty-repository.png)\n\nThen clone it locally and then you should be able to create a new Jekyll blog within it:\n\n```\njekyll new blog\n```\n\nIf your ruby installation didn´t work, is broken or you just don´t want to hassle with that, you could also use the Docker ruby image and do everything inside a container:\n```\ndocker run --rm -v \"$PWD\":/usr/src/app -w /usr/src/app ruby:2.5 bash -c \"gem install bundler jekyll; jekyll new blog; ls -l\"\n```\n\nThis will bootstrap a new Jekyll skeleton which should be buildalbe and shippable right out-of-the-box. Simply `cd` into the `blog` directory and run \n\n```\nbundle exec jekyll serve\n```\n\nIf that brings some error like `Could not find foobar-package-3.0.3 in any of the sources` you might need to install dependencies with `bundle install` or `bundle install --path vendor/bundle`.\n\nNow open your Browser and head to http://127.0.0.1:4000/ - you should see our new Blog freshly backed by Jekyll:\n\n![gitlab-pages-local-jekyll](screenshots/gitlab-pages-local-jekyll.png)\n\n\n\n### Deploy Jekyll as GitLab Pages with GitLab CI \u0026 Docker\n\nEvery GitLab Pages repository needs a `.gitlab-ci.yml`, that will use GitLab CI to run the Ruby based build process of Jekyll and publish your Static site. \n\nI have prepared a working `docker run command` based on the [official ruby Docker image](https://hub.docker.com/_/ruby/), which will output the resulting site into `public` directory inside your Jekyll site:\n\n```\ndocker run --rm -v \"$PWD\":/usr/src/app -w /usr/src/app ruby:2.5 bash -c \"bundle install; bundle exec jekyll build -d public\"\n```\n\nNow with this it is easy to craft our `.gitlab-ci.yml`:\n\n```\npages:\n  stage: deploy\n  script:\n  - docker run --rm -v \"$PWD\":/usr/src/app -w /usr/src/app ruby:2.5 bash -c \"bundle install; bundle exec jekyll build -d public; ls -l\"\n  artifacts:\n    paths:\n    - public\n  only:\n  - master\n```\n\nAdd all the files incl. the `.gitlab-ci.yml` to your Git repository - just be sure to extend your `.gitignore` like this:\n\n```\n_site\n.sass-cache\n.jekyll-metadata\npublic\nvendor\n.bundle\n```\n\nThen commit and push into GitLab. Your CI/CD pipeline should run successfully:\n\n![gitlab-pages-successful-first-jekyll-build](screenshots/gitlab-pages-successful-first-jekyll-build.png)\n\nNow go to __Settings/Pages__ inside your repository to find the URL where you can access your GitLab Page for this repository.\n\n\n# Links\n\n* Gitlab CI REFERENCE docs: https://docs.gitlab.com/ce/ci/yaml/README.html\n\n* Install Gitlab with Docker: https://docs.gitlab.com/omnibus/docker/\n\n* How to build Docker containers with Gitlab: https://docs.gitlab.com/ee/ci/docker/using_docker_build.html\n\n* Why to bind mount docker.sock into your Gitlab Docker Container (instead of using Docker-in-Docker): https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/\n\n* Good, but little old post abt installing Gitlab https://gitlabfan.com/setting-up-your-own-fully-functional-gitlab-https-registry-ci-runners-79901ac617c0\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonashackt%2Fgitlab-ci-stack","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjonashackt%2Fgitlab-ci-stack","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonashackt%2Fgitlab-ci-stack/lists"}