{"id":21134115,"url":"https://github.com/horgix/click-count-infra","last_synced_at":"2025-07-10T12:17:21.547Z","repository":{"id":95050908,"uuid":"67555986","full_name":"Horgix/click-count-infra","owner":"Horgix","description":"Click-Count infrastructure","archived":false,"fork":false,"pushed_at":"2016-11-13T11:57:00.000Z","size":614,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-05T17:46:06.852Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Horgix.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-09-07T00:04:38.000Z","updated_at":"2016-09-07T00:05:08.000Z","dependencies_parsed_at":"2023-06-11T22:48:03.155Z","dependency_job_id":null,"html_url":"https://github.com/Horgix/click-count-infra","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Horgix%2Fclick-count-infra","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Horgix%2Fclick-count-infra/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Horgix%2Fclick-count-infra/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Horgix%2Fclick-count-infra/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Horgix","download_url":"https://codeload.github.com/Horgix/click-count-infra/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243581060,"owners_count":20314162,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-20T06:23:32.561Z","updated_at":"2025-03-14T12:41:29.079Z","avatar_url":"https://github.com/Horgix.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"#**This README is a 'plain' version, you will find the real documentation in a readable format on https://clickcount.horgix.fr**\n\n\n# Introduction\n\n**If you want to directly jump to the \"do it\" part of this documentation, head\nover to the [\"How To\" page](howto.md).**\n\n# This project\n\nThis project aims at building to the CI/CD infrastructure for the [Click-Count\napplication](TODO), as asked by the following statement:\n\n\u003e Le livrable attendu doit être un repository sur GitHub contenant la\n\u003e **description de l’infrastructure** et du pipeline de livraison continu\n\u003e entièrement automatisés afin que la solution puisse être déployée sur un\n\u003e environnement différent.\n\nThis repository will allow you to spawn the so-said infrastucture, with the\nfollowing properties:\n\n- Deployed with Ansible\n- Based on AWS EC2\n- Running **entirely** on Docker\n- Using Zookeeper, Mesos and Marathon to deploy and orchestrate services and\n  the application itself\n\nThe EC2 part can of course be replaced by any other IAAS provider, may it be\npublic or private, admitting some adaptations.\n\n# Requirements\n\n## On your machine\n\nYou will need the AWS key with the privilege described in the [\"AWS\nside\"](index.md#aws-side) part at this path : `~/.ssh/ansible-click-count.pem`\n\nThe following software/modules are required to be able to use this repo:\n\n- Ansible (2.2, latest)\n- Python 2.7 (Ansible sadly doesn't work with Python 3)\n- pyapi-gitlab (python module for GitLab API)\n- boto (python module for AWS API)\n\n**If you don't want to install all of these, it's fine; there is already a\n[Docker image](https://hub.docker.com/r/horgix/ansible-aws-gitlab/) packaging\nthem right there for you so just make sure you have the docker daemon running**\n\n## AWS side\n\nTheses parts are also required on AWS side:\n\n- An AWS user with the following policies:\n    - AdministratorAccess\n    - AmazonEC2FullAccess\n    - AmazonRoute53FullAccess\n- A DNS Zone managed by Route 53\n\n# What you end up with\n\n## The cluster itself\n\n- 3 nodes cluster on AWS:\n    - One \"services\" node (m4.large)\n    - One \"staging\" node (t2.micro)\n    - One \"production\" node (t2.micro)\n- Each one of them runs:\n    - Zookeeper\n    - Mesos Slave\n    - Mesos Master\n    - Marathon\n- The \"services\" node also runs the following services as Marathon apps:\n    - [GitLab](https://about.gitlab.com/)\n    - a [GitLab CI runner](https://gitlab.com/gitlab-org/gitlab-ci-multi-runner)\n    - [Traefik](http://traefik.io/)\n    - a pretty [alternative Mesos UI](https://github.com/Capgemini/mesos-ui/)\n- The application [click-count](https://github.com/Horgix/click-count-app) is\n  deployed, depending on the targeted environment, on the staging and\n  production nodes\n\n# How it works\n\n![Steps](images/steps.png)\n\n# How To\n\n# 4 steps to rule them all\n\nIdeally, you should be able to have the infrastructure to start building in\n5min, and totally built after 20min (15min for Ansible to run everything)\n\n- Make sure to have your AWS key in `~/.ssh/ansible-click-count.pem`\n- `source ./init_credentials.sh` or export the environment variables by hand\n- Tweak what you want in `inventory/groups_vars/all`, probably:\n    - `domain_name`\n    - `vpc_subnet_id`\n    - `vpc_id`\n- Run `make docker` ; the image pull will be long, and the Ansible run too, but\n  you should be able to grab a coffee while it runs.\n\n# Configuration\n\nBefore taking the time to configure anything, make sure you meet the\n[requirements announced on the home page](index.md#requirements).\n\n## Using docker\n\nIf you decided to use the \"ansible-aws-gitlab\" docker image which provides\neverything, you should be ok with the above 4 steps instructions.\n\n## Ansible python interpreter\n\nIf you decided to run Ansible from your machine, and if you're not on\nArchlinux, you might want to comment lines 7-8 and 22-23 in\nplaybooks/deploy.yml:\n\n  vars:\n      ansible_python_interpreter: python2\n\nthese lines are here to be able to run Ansible tasks delegated to `localhost`\non an Archlinux host.\n\n## Credentials\n\nTo be able to do its job, this project require a login/password for 2 things :\n\n- AWS\n- Docker Hub\n\nYou have to provide them as environment variable; the script\n`init_credentials.sh` can also be used to load them from\n[`pass`](https://git.zx2c4.com/password-store/) (if you don't know what it is,\nI invite you to try it, it's a really simple way to store passwords, based on\nGPG, git, and tree).\n\n### AWS\n\n#### Why is it needed ?\n\n- Create EC2 instances and Route 53 DNS records\n- Execute the dynamic inventory\n\n#### What to define\n\n2 environment variable are required :\n\n- AWS\\_ACCESS\\_KEY\\_ID\n- AWS\\_SECRET\\_ACCESS\\_KEY\n\nYou can also export them by running `init_credentials.sh` if they are stored in\npass under `xebia/aws/ansible-key-id` and `xebia/aws/ansible-key-secret`.\n\n### Docker Hub\n\n#### Why is it needed ?\n\nThe GitLab CI `build` step builds a Docker image locally and then pushes it to\nthe [Docker Hub](https://hub.docker.com/) using `docker login` then `docker\npush`.\n\n#### What to define\n\n2 environment variable are required :\n\n- DOCKER\\_HUB\\_USERNAME\n- DOCKER\\_HUB\\_PASSWORD=`\n\nYou can also export them by running `init_credentials.sh` if they are stored in\npass under `dockerhub/username` and `dockerhub/password`.\n\n# Run\n\n- `make deploy` will create the EC2 instances and Route 53 DNS records\n- `make base` will install the base cluster on these instances\n- `make services` will deploy services, includind GitLab and its Runner which\n  power the CI for the Click-Count application\n\nYou can also simply call `make` to deploy the entire stack.\n\nFinaly, it's possible to prefix every make rule by `docker_` to make it run\ninside the all-provisionned Docker container.\n\n# Demo\n\nIf you want to see the result running, it's possible, but will probably not be\nhere forever since it's running on AWS with my own money.\n\n# Services\n\n- [Marathon](http://deploy.coffee:8080); credentials: xebia/verysecure\n- [GitLab](https://gitlab.deploy.coffee); credentials: xebia/verysecure\n- [Traefik](https://traefik.deploy.coffee)\n- [Mesos UI](https://cluster.deploy.coffee)\n- [This documentation](https://clickcount.horgix.fr)\n\n# Example run\n\n```none\n$ time make\n\nansible-playbook playbooks/deploy.yml -t deploy\nstatically included: /home/horgix/work/xebia/ansible/roles/deploy/tasks/secgroup.yml\nstatically included: /home/horgix/work/xebia/ansible/roles/deploy/tasks/ec2.yml\n\nPLAY [Base setup] **************************************************************\n\nTASK [deploy : EC2 - Create security group] ************************************\nchanged: [localhost]\n\nTASK [deploy : EC2 - Create instances] *****************************************\nchanged: [localhost] =\u003e (item={u'type': u'm4.large', u'tags': {u'zkid': 1, u'type': u'svc', u'Name': u'services'}})\nchanged: [localhost] =\u003e (item={u'type': u't2.micro', u'tags': {u'zkid': 2, u'type': u'staging', u'Name': u'staging'}})\nchanged: [localhost] =\u003e (item={u'type': u't2.micro', u'tags': {u'zkid': 3, u'type': u'prod', u'Name': u'production'}})\n\nTASK [deploy : Wait for SSH to come up] ****************************************\nok: [localhost] =\u003e (item={'_ansible_parsed': True, u'changed': True, '_ansible_no_log': False, u'instances': [{u'kernel': None, u'root_device_type': u'ebs', u'private_dns_name': u'ip-172-31-4-225.eu-west-1.compute.internal', u'public_ip': u'52.209.138.214', u'private_ip': u'172.31.4.225', u'id': u'i-5446a165', u'ebs_optimized': False, u'state': u'running', u'virtualization_type': u'hvm', u'architecture': u'x86_64', u'ramdisk': None, u'block_device_mapping': {u'/dev/xvda': {u'status': u'attached', u'delete_on_termination': False, u'volume_id': u'vol-e439d466'}}, u'key_name': u'aws', u'image_id': u'ami-f9dd458a', u'tenancy': u'default', u'groups': {u'sg-5b717d3c': u'Click-Count'}, u'public_dns_name': u'ec2-52-209-138-214.eu-west-1.compute.amazonaws.com', u'state_code': 16, u'tags': {u'zkid': u'1', u'type': u'svc', u'Name': u'services'}, u'placement': u'eu-west-1a', u'ami_launch_index': u'0', u'dns_name': u'ec2-52-209-138-214.eu-west-1.compute.amazonaws.com', u'region': u'eu-west-1', u'launch_time': u'2016-09-06T19:57:36.000Z', u'instance_type': u'm4.large', u'root_device_name': u'/dev/xvda', u'hypervisor': u'xen'}], '_ansible_item_result': True, u'instance_ids': [u'i-5446a165'], 'item': {u'type': u'm4.large', u'tags': {u'zkid': 1, u'type': u'svc', u'Name': u'services'}}, u'tagged_instances': [{u'kernel': None, u'root_device_type': u'ebs', u'private_dns_name': u'ip-172-31-4-225.eu-west-1.compute.internal', u'public_ip': u'52.209.138.214', u'private_ip': u'172.31.4.225', u'id': u'i-5446a165', u'ebs_optimized': False, u'state': u'running', u'virtualization_type': u'hvm', u'architecture': u'x86_64', u'ramdisk': None, u'block_device_mapping': {u'/dev/xvda': {u'status': u'attached', u'delete_on_termination': False, u'volume_id': u'vol-e439d466'}}, u'key_name': u'aws', u'image_id': u'ami-f9dd458a', u'tenancy': u'default', u'groups': {u'sg-5b717d3c': u'Click-Count'}, u'public_dns_name': u'ec2-52-209-138-214.eu-west-1.compute.amazonaws.com', u'state_code': 16, u'tags': {u'zkid': u'1', u'type': u'svc', u'Name': u'services'}, u'placement': u'eu-west-1a', u'ami_launch_index': u'0', u'dns_name': u'ec2-52-209-138-214.eu-west-1.compute.amazonaws.com', u'region': u'eu-west-1', u'launch_time': u'2016-09-06T19:57:36.000Z', u'instance_type': u'm4.large', u'root_device_name': u'/dev/xvda', u'hypervisor': u'xen'}], 'invocation': {'module_name': u'ec2', u'module_args': {u'kernel': None, u'image': u'ami-f9dd458a', u'monitoring': False, u'user_data': None, u'termination_protection': False, u'private_ip': None, u'spot_type': u'one-time', u'ec2_url': None, u'id': None, u'source_dest_check': True, u'aws_secret_key': None, u'spot_wait_timeout': u'600', u'spot_price': None, u'zone': None, u'exact_count': 1, u'ebs_optimized': False, u'state': u'present', u'placement_group': None, u'count_tag': u\"{'type': 'svc'}\", u'ramdisk': None, u'key_name': u'aws', u'spot_launch_group': None, u'vpc_subnet_id': u'subnet-3adebc5e', u'instance_ids': None, u'tenancy': u'default', u'profile': None, u'assign_public_ip': True, u'group': [u'Click-Count'], u'wait': True, u'count': 1, u'aws_access_key': None, u'security_token': None, u'instance_profile_name': None, u'region': u'eu-west-1', u'network_interfaces': None, u'instance_initiated_shutdown_behavior': u'stop', u'instance_type': u'm4.large', u'wait_timeout': u'300', u'volumes': [{u'volume_size': 20, u'volume_type': u'gp2', u'device_name': u'/dev/xvda'}], u'instance_tags': {u'zkid': 1, u'type': u'svc', u'Name': u'services'}, u'group_id': None, u'validate_certs': True}}})\nok: [localhost] =\u003e (item={'_ansible_parsed': True, u'changed': True, '_ansible_no_log': False, u'instances': [{u'kernel': None, u'root_device_type': u'ebs', u'private_dns_name': u'ip-172-31-5-225.eu-west-1.compute.internal', u'public_ip': u'52.210.28.188', u'private_ip': u'172.31.5.225', u'id': u'i-5e45a26f', u'ebs_optimized': False, u'state': u'running', u'virtualization_type': u'hvm', u'architecture': u'x86_64', u'ramdisk': None, u'block_device_mapping': {u'/dev/xvda': {u'status': u'attached', u'delete_on_termination': False, u'volume_id': u'vol-1e39d49c'}}, u'key_name': u'aws', u'image_id': u'ami-f9dd458a', u'tenancy': u'default', u'groups': {u'sg-5b717d3c': u'Click-Count'}, u'public_dns_name': u'ec2-52-210-28-188.eu-west-1.compute.amazonaws.com', u'state_code': 16, u'tags': {u'zkid': u'2', u'type': u'staging', u'Name': u'staging'}, u'placement': u'eu-west-1a', u'ami_launch_index': u'0', u'dns_name': u'ec2-52-210-28-188.eu-west-1.compute.amazonaws.com', u'region': u'eu-west-1', u'launch_time': u'2016-09-06T19:57:57.000Z', u'instance_type': u't2.micro', u'root_device_name': u'/dev/xvda', u'hypervisor': u'xen'}], '_ansible_item_result': True, u'instance_ids': [u'i-5e45a26f'], 'item': {u'type': u't2.micro', u'tags': {u'zkid': 2, u'type': u'staging', u'Name': u'staging'}}, u'tagged_instances': [{u'kernel': None, u'root_device_type': u'ebs', u'private_dns_name': u'ip-172-31-5-225.eu-west-1.compute.internal', u'public_ip': u'52.210.28.188', u'private_ip': u'172.31.5.225', u'id': u'i-5e45a26f', u'ebs_optimized': False, u'state': u'running', u'virtualization_type': u'hvm', u'architecture': u'x86_64', u'ramdisk': None, u'block_device_mapping': {u'/dev/xvda': {u'status': u'attached', u'delete_on_termination': False, u'volume_id': u'vol-1e39d49c'}}, u'key_name': u'aws', u'image_id': u'ami-f9dd458a', u'tenancy': u'default', u'groups': {u'sg-5b717d3c': u'Click-Count'}, u'public_dns_name': u'ec2-52-210-28-188.eu-west-1.compute.amazonaws.com', u'state_code': 16, u'tags': {u'zkid': u'2', u'type': u'staging', u'Name': u'staging'}, u'placement': u'eu-west-1a', u'ami_launch_index': u'0', u'dns_name': u'ec2-52-210-28-188.eu-west-1.compute.amazonaws.com', u'region': u'eu-west-1', u'launch_time': u'2016-09-06T19:57:57.000Z', u'instance_type': u't2.micro', u'root_device_name': u'/dev/xvda', u'hypervisor': u'xen'}], 'invocation': {'module_name': u'ec2', u'module_args': {u'kernel': None, u'image': u'ami-f9dd458a', u'monitoring': False, u'user_data': None, u'termination_protection': False, u'private_ip': None, u'spot_type': u'one-time', u'ec2_url': None, u'id': None, u'source_dest_check': True, u'aws_secret_key': None, u'spot_wait_timeout': u'600', u'spot_price': None, u'zone': None, u'exact_count': 1, u'ebs_optimized': False, u'state': u'present', u'placement_group': None, u'count_tag': u\"{'type': 'staging'}\", u'ramdisk': None, u'key_name': u'aws', u'spot_launch_group': None, u'vpc_subnet_id': u'subnet-3adebc5e', u'instance_ids': None, u'tenancy': u'default', u'profile': None, u'assign_public_ip': True, u'group': [u'Click-Count'], u'wait': True, u'count': 1, u'aws_access_key': None, u'security_token': None, u'instance_profile_name': None, u'region': u'eu-west-1', u'network_interfaces': None, u'instance_initiated_shutdown_behavior': u'stop', u'instance_type': u't2.micro', u'wait_timeout': u'300', u'volumes': [{u'volume_size': 20, u'volume_type': u'gp2', u'device_name': u'/dev/xvda'}], u'instance_tags': {u'zkid': 2, u'type': u'staging', u'Name': u'staging'}, u'group_id': None, u'validate_certs': True}}})\nok: [localhost] =\u003e (item={'_ansible_parsed': True, u'changed': True, '_ansible_no_log': False, u'instances': [{u'kernel': None, u'root_device_type': u'ebs', u'private_dns_name': u'ip-172-31-4-36.eu-west-1.compute.internal', u'public_ip': u'52.210.134.157', u'private_ip': u'172.31.4.36', u'id': u'i-1d44a32c', u'ebs_optimized': False, u'state': u'running', u'virtualization_type': u'hvm', u'architecture': u'x86_64', u'ramdisk': None, u'block_device_mapping': {u'/dev/xvda': {u'status': u'attached', u'delete_on_termination': False, u'volume_id': u'vol-e839d46a'}}, u'key_name': u'aws', u'image_id': u'ami-f9dd458a', u'tenancy': u'default', u'groups': {u'sg-5b717d3c': u'Click-Count'}, u'public_dns_name': u'ec2-52-210-134-157.eu-west-1.compute.amazonaws.com', u'state_code': 16, u'tags': {u'zkid': u'3', u'type': u'prod', u'Name': u'production'}, u'placement': u'eu-west-1a', u'ami_launch_index': u'0', u'dns_name': u'ec2-52-210-134-157.eu-west-1.compute.amazonaws.com', u'region': u'eu-west-1', u'launch_time': u'2016-09-06T19:58:19.000Z', u'instance_type': u't2.micro', u'root_device_name': u'/dev/xvda', u'hypervisor': u'xen'}], '_ansible_item_result': True, u'instance_ids': [u'i-1d44a32c'], 'item': {u'type': u't2.micro', u'tags': {u'zkid': 3, u'type': u'prod', u'Name': u'production'}}, u'tagged_instances': [{u'kernel': None, u'root_device_type': u'ebs', u'private_dns_name': u'ip-172-31-4-36.eu-west-1.compute.internal', u'public_ip': u'52.210.134.157', u'private_ip': u'172.31.4.36', u'id': u'i-1d44a32c', u'ebs_optimized': False, u'state': u'running', u'virtualization_type': u'hvm', u'architecture': u'x86_64', u'ramdisk': None, u'block_device_mapping': {u'/dev/xvda': {u'status': u'attached', u'delete_on_termination': False, u'volume_id': u'vol-e839d46a'}}, u'key_name': u'aws', u'image_id': u'ami-f9dd458a', u'tenancy': u'default', u'groups': {u'sg-5b717d3c': u'Click-Count'}, u'public_dns_name': u'ec2-52-210-134-157.eu-west-1.compute.amazonaws.com', u'state_code': 16, u'tags': {u'zkid': u'3', u'type': u'prod', u'Name': u'production'}, u'placement': u'eu-west-1a', u'ami_launch_index': u'0', u'dns_name': u'ec2-52-210-134-157.eu-west-1.compute.amazonaws.com', u'region': u'eu-west-1', u'launch_time': u'2016-09-06T19:58:19.000Z', u'instance_type': u't2.micro', u'root_device_name': u'/dev/xvda', u'hypervisor': u'xen'}], 'invocation': {'module_name': u'ec2', u'module_args': {u'kernel': None, u'image': u'ami-f9dd458a', u'monitoring': False, u'user_data': None, u'termination_protection': False, u'private_ip': None, u'spot_type': u'one-time', u'ec2_url': None, u'id': None, u'source_dest_check': True, u'aws_secret_key': None, u'spot_wait_timeout': u'600', u'spot_price': None, u'zone': None, u'exact_count': 1, u'ebs_optimized': False, u'state': u'present', u'placement_group': None, u'count_tag': u\"{'type': 'prod'}\", u'ramdisk': None, u'key_name': u'aws', u'spot_launch_group': None, u'vpc_subnet_id': u'subnet-3adebc5e', u'instance_ids': None, u'tenancy': u'default', u'profile': None, u'assign_public_ip': True, u'group': [u'Click-Count'], u'wait': True, u'count': 1, u'aws_access_key': None, u'security_token': None, u'instance_profile_name': None, u'region': u'eu-west-1', u'network_interfaces': None, u'instance_initiated_shutdown_behavior': u'stop', u'instance_type': u't2.micro', u'wait_timeout': u'300', u'volumes': [{u'volume_size': 20, u'volume_type': u'gp2', u'device_name': u'/dev/xvda'}], u'instance_tags': {u'zkid': 3, u'type': u'prod', u'Name': u'production'}, u'group_id': None, u'validate_certs': True}}})\n\nPLAY [Route 53] ****************************************************************\n\nPLAY RECAP *********************************************************************\nlocalhost                  : ok=3    changed=2    unreachable=0    failed=0\n\n./inventory/ec2.py --refresh-cache \u003e /dev/null\nansible-playbook playbooks/deploy.yml -t route53\nstatically included: /home/horgix/work/xebia/ansible/roles/deploy/tasks/secgroup.yml\nstatically included: /home/horgix/work/xebia/ansible/roles/deploy/tasks/ec2.yml\n\nPLAY [Base setup] **************************************************************\n\nPLAY [Route 53] ****************************************************************\n\nTASK [route53 : Add Route 53 host records] *************************************\nchanged: [production -\u003e 127.0.0.1]\nchanged: [services -\u003e 127.0.0.1]\nchanged: [staging -\u003e 127.0.0.1]\n\nTASK [route53 : Add Route 53 main record] **************************************\nskipping: [production]\nskipping: [staging]\nchanged: [services -\u003e 127.0.0.1]\n\nTASK [route53 : Add Route 53 services records] *********************************\nskipping: [production]\nskipping: [staging]\nok: [services -\u003e 127.0.0.1]\n\nPLAY RECAP *********************************************************************\nproduction                 : ok=1    changed=1    unreachable=0    failed=0\nservices                   : ok=3    changed=2    unreachable=0    failed=0\nstaging                    : ok=1    changed=1    unreachable=0    failed=0\n\nansible-playbook playbooks/master.yml -b -t base\nstatically included: /home/horgix/work/xebia/ansible/roles/base/tasks/requirements.yml\nstatically included: /home/horgix/work/xebia/ansible/roles/base/tasks/zookeeper.yml\nstatically included: /home/horgix/work/xebia/ansible/roles/base/tasks/mesos.yml\nstatically included: /home/horgix/work/xebia/ansible/roles/base/tasks/marathon.yml\nstatically included: /home/horgix/work/xebia/ansible/roles/services/tasks/mesos-ui.yml\nstatically included: /home/horgix/work/xebia/ansible/roles/services/tasks/traefik.yml\nstatically included: /home/horgix/work/xebia/ansible/roles/services/tasks/gitlab.yml\nstatically included: /home/horgix/work/xebia/ansible/roles/services/tasks/gitlab-content.yml\nstatically included: /home/horgix/work/xebia/ansible/roles/services/tasks/gitlab-runner.yml\n\nPLAY [Base setup] **************************************************************\n\nTASK [setup] *******************************************************************\nok: [staging]\nok: [production]\nok: [services]\n\nTASK [base : Set Zookeeper Nodes fact] *****************************************\nok: [staging]\nok: [production]\nok: [services]\n\nTASK [base : Zookeeper nodes] **************************************************\nok: [production] =\u003e {\n    \"msg\": \"3:172.31.4.36,2:172.31.5.225,1:172.31.4.225\"\n}\nok: [staging] =\u003e {\n    \"msg\": \"3:172.31.4.36,2:172.31.5.225,1:172.31.4.225\"\n}\nok: [services] =\u003e {\n    \"msg\": \"3:172.31.4.36,2:172.31.5.225,1:172.31.4.225\"\n}\n\nTASK [base : Upgrade every package] ********************************************\nchanged: [staging]\nchanged: [services]\nchanged: [production]\n\nTASK [base : Docker - Install package] *****************************************\nchanged: [services]\nchanged: [staging]\nchanged: [production]\n\nTASK [base : Docker - Install python lib] **************************************\nchanged: [staging]\nchanged: [production]\nchanged: [services]\n\nTASK [base : Docker - start and enable service] ********************************\nchanged: [staging]\nchanged: [production]\nchanged: [services]\n\nTASK [base : Docker - Pull base images] ****************************************\nchanged: [services] =\u003e (item=horgix/zookeeper)\nchanged: [staging] =\u003e (item=horgix/zookeeper)\nchanged: [production] =\u003e (item=horgix/zookeeper)\nchanged: [services] =\u003e (item=horgix/mesos-slave)\nchanged: [production] =\u003e (item=horgix/mesos-slave)\nchanged: [staging] =\u003e (item=horgix/mesos-slave)\nchanged: [services] =\u003e (item=mesoscloud/mesos-master)\nchanged: [production] =\u003e (item=mesoscloud/mesos-master)\nchanged: [staging] =\u003e (item=mesoscloud/mesos-master)\nchanged: [services] =\u003e (item=mesoscloud/marathon)\nchanged: [production] =\u003e (item=mesoscloud/marathon)\nchanged: [staging] =\u003e (item=mesoscloud/marathon)\n\nTASK [base : Zookeeper - Start] ************************************************\nchanged: [services]\nchanged: [production]\nchanged: [staging]\n\nTASK [base : Mesos - Start slave] **********************************************\nchanged: [production]\nchanged: [services]\nchanged: [staging]\n\nTASK [base : Mesos - Start master] *********************************************\nchanged: [staging]\nchanged: [services]\nchanged: [production]\n\nTASK [base : Marathon - Start] *************************************************\nchanged: [staging]\nchanged: [services]\nchanged: [production]\n\nPLAY [Setup Services] **********************************************************\n\nTASK [setup] *******************************************************************\nok: [services]\n\nPLAY RECAP *********************************************************************\nproduction                 : ok=12   changed=9    unreachable=0    failed=0\nservices                   : ok=13   changed=9    unreachable=0    failed=0\nstaging                    : ok=12   changed=9    unreachable=0    failed=0\n\nansible-playbook playbooks/master.yml -b -t services\nstatically included: /home/horgix/work/xebia/ansible/roles/base/tasks/requirements.yml\nstatically included: /home/horgix/work/xebia/ansible/roles/base/tasks/zookeeper.yml\nstatically included: /home/horgix/work/xebia/ansible/roles/base/tasks/mesos.yml\nstatically included: /home/horgix/work/xebia/ansible/roles/base/tasks/marathon.yml\nstatically included: /home/horgix/work/xebia/ansible/roles/services/tasks/mesos-ui.yml\nstatically included: /home/horgix/work/xebia/ansible/roles/services/tasks/traefik.yml\nstatically included: /home/horgix/work/xebia/ansible/roles/services/tasks/gitlab.yml\nstatically included: /home/horgix/work/xebia/ansible/roles/services/tasks/gitlab-content.yml\nstatically included: /home/horgix/work/xebia/ansible/roles/services/tasks/gitlab-runner.yml\n\nPLAY [Base setup] **************************************************************\n\nTASK [setup] *******************************************************************\nok: [staging]\nok: [production]\nok: [services]\n\nTASK [base : Set Zookeeper Nodes fact] *****************************************\nok: [production]\nok: [services]\nok: [staging]\n\nTASK [base : Zookeeper nodes] **************************************************\nok: [production] =\u003e {\n    \"msg\": \"3:172.31.4.36,2:172.31.5.225,1:172.31.4.225\"\n}\nok: [services] =\u003e {\n    \"msg\": \"3:172.31.4.36,2:172.31.5.225,1:172.31.4.225\"\n}\nok: [staging] =\u003e {\n    \"msg\": \"3:172.31.4.36,2:172.31.5.225,1:172.31.4.225\"\n}\n\nPLAY [Setup Services] **********************************************************\n\nTASK [setup] *******************************************************************\nok: [services]\n\nTASK [services : Docker - Pull service images] *********************************\nchanged: [services] =\u003e (item=gitlab/gitlab-ce)\nchanged: [services] =\u003e (item=horgix/gitlab-runner)\nchanged: [services] =\u003e (item=traefik)\nchanged: [services] =\u003e (item=jetty)\nchanged: [services] =\u003e (item=maven)\n\nTASK [services : MesosUI - Generate Marathon JSON description] *****************\nchanged: [services -\u003e localhost]\n\nTASK [services : MesosUI - Run Marathon app] ***********************************\nok: [services]\n\nTASK [services : Create volumes directories] ***********************************\nchanged: [services]\n\nTASK [services : Make sure acme.json exists] ***********************************\nok: [services]\n\nTASK [services : Configure] ****************************************************\nchanged: [services]\n\nTASK [services : Generate Marathon JSON description] ***************************\nchanged: [services -\u003e localhost]\n\nTASK [services : Run Marathon app] *********************************************\nok: [services]\n\nTASK [services : Create volumes directories] ***********************************\nchanged: [services] =\u003e (item=/srv/gitlab)\nchanged: [services] =\u003e (item=/srv/gitlab/config)\nchanged: [services] =\u003e (item=/srv/gitlab/logs)\nchanged: [services] =\u003e (item=/srv/gitlab/data)\nchanged: [services] =\u003e (item=/srv/gitlab/logs/reconfigure)\n\nTASK [services : Generate Marathon JSON description] ***************************\nchanged: [services -\u003e localhost]\n\nTASK [services : Run Marathon app] *********************************************\nok: [services]\n\nTASK [services : Wait for GitLab to come up] ***********************************\nFAILED - RETRYING: TASK: services : Wait for GitLab to come up (20 retries left).\nFAILED - RETRYING: TASK: services : Wait for GitLab to come up (19 retries left).\nFAILED - RETRYING: TASK: services : Wait for GitLab to come up (18 retries left).\nFAILED - RETRYING: TASK: services : Wait for GitLab to come up (17 retries left).\nFAILED - RETRYING: TASK: services : Wait for GitLab to come up (16 retries left).\nFAILED - RETRYING: TASK: services : Wait for GitLab to come up (15 retries left).\nok: [services]\n [WARNING]: Consider using get_url or uri module rather than running curl\n\n\nTASK [services : Get GitLab private token] *************************************\nok: [services]\n\nTASK [services : Set GitLab private token as fact] *****************************\nok: [services]\n\nTASK [services : Create Gitlab Group] ******************************************\nchanged: [services -\u003e 127.0.0.1]\n\nTASK [services : Create Gitlab User] *******************************************\nchanged: [services -\u003e 127.0.0.1]\n\nTASK [services : Create Gitlab Project] ****************************************\nchanged: [services -\u003e 127.0.0.1]\n\nTASK [services : Set variables] ************************************************\nok: [services] =\u003e (item=key=HUB_LOGIN\u0026value=\u003cstripped\u003e)\nok: [services] =\u003e (item=key=HUB_PASSWORD\u0026value=\u003cstripped\u003e)\nok: [services] =\u003e (item=key=MARATHON_URL\u0026value=http://xebia.horgix.fr:8080)\nok: [services] =\u003e (item=key=MARATHON_USERNAME\u0026value=\u003cstripped\u003e)\nok: [services] =\u003e (item=key=MARATHON_PASSWORD\u0026value=\u003cstripped\u003e)\nok: [services] =\u003e (item=key=STAGING_ENDPOINT\u0026value=staging.xebia.horgix.fr)\nok: [services] =\u003e (item=key=PRODUCTION_ENDPOINT\u0026value=prod.xebia.horgix.fr)\nok: [services] =\u003e (item=key=STAGING_REDIS\u0026value=\u003cstripped\u003e)\nok: [services] =\u003e (item=key=PRODUCTION_REDIS\u0026value=\u003cstripped\u003e)\n\nTASK [services : GitLab Runner - Get runners token] ****************************\nok: [services]\n\nTASK [services : GitLab Runner - Set runners token as fact] ********************\nok: [services]\n\nTASK [services : GitLab Runner - Generate Marathon JSON description] ***********\nchanged: [services -\u003e localhost]\n\nTASK [services : Runner - Run Marathon app] ************************************\nok: [services]\n\nPLAY RECAP *********************************************************************\nproduction                 : ok=3    changed=0    unreachable=0    failed=0\nservices                   : ok=26   changed=11   unreachable=0    failed=0\nstaging                    : ok=3    changed=0    unreachable=0    failed=0\n\nmake  51.26s user 8.01s system 8% cpu 12:03.27 total\n```\n\n# Screenshots\n\n## Mesos Dashboard\n\n![Mesos Dashboard](images/mesos_dashboard.png)\n\n## Mesos Nodes\n\n![Mesos Nodes](images/mesos_nodes.png)\n\n## Mesos Tasks\n\n![Mesos Tasks](images/mesos_tasks.png)\n\n## Marathon\n\n![Marathon](images/marathon.png)\n\n## Traefik\n\n![Traefik](Images/traefik.png)\n\n# Improvements ideas\n\n**Since this project as been made from scratch in a week, there is a room for a\nlot of improvements. This is a list of some ideas that could be implemented\nwith a bit more time and depending on the needs.**\n\n## Zookeeper IDs\n\nCurrently, Zookeeper IDs are taken from the tag `zkid` on the instances, which\nwill be kind of a pain to handle if we introduce autoscaling of the\ninfrastructure. It would be better if each Zookeeper could discover its ID.\n\n## Gitlab Container Registry\n\nSince the [8.8\nrelease](https://about.gitlab.com/2016/05/23/gitlab-container-registry/),\nGitLab can provide a Docker private registry. It would be really nice to use\nit; however, it has not been integrated in this project/demo, as it requires\npassing SSL certificates to it, which are currently only known to Traefik.\n\n## GitLab Runner health checks\n\nCurrently, there is no health checks on Marathon side for GitLab runners.\nSince they don't listen for connections and directly connect to GitLab, there\nis no easy way to check for its aliveness; something based on the health check\ntype \"command\" which would hit GitLab API to detect if the runners is\nreferenced as active or not would be possible, but currently not implemented.\n\n## Docker images based on Alpine\n\nCurrently, this project is using the most possible \"standard\" images, mainly\njust using the \"latest\" tag. Reducing the global size of containers using ones\nbased on Alpine would be nice, but would maybe have to be benchmarked before.\n\n## Rolling upgrade without service interruption\n\nCurrently, if we try to run the app with less than 300MB of memory, it simply\nends up being killed by the oom killer:\n\n    $ docker run -it --rm -m 200m horgix/click-count\n    $ dmesg\n    [ 5662.970129] Memory cgroup out of memory: Kill process 13119 (java) score 986 or sacrifice child\n    [ 5662.970139] Killed process 13119 (java) total-vm:3725396kB, anon-rss:201312kB, file-rss:0kB, shmem-rss:0kB\n\nFurthermore, the t2.micro EC2 are given \"0.5GB\" of memory, actually reported as\n\"497\" by Mesos.\n\nSo, at the end, it's impossible to have 2 instances of the click-count\napplication running side by side on the same t2.micro instance.\n\nSo we have 2 solutions to allow Marathon to update the application:\n\n1. Set the `minimumHealthCapacity` to 0, allowing Marathon to completely\n   shutdown the \"old\" application before starting the new one\n2. Set the `mem` resource limitation to 0, asking Mesos to not limit the memory\n   used by the running container, which will allow it to make offers to\n   Marathon so it could do a proper rolling upgrade\n\nThe first solution has been chosen, since the second one would introduce the\nrisk of anyway having one or both instances of the application killed by the\noom killer. The best case, of course, would be to run on something better than\nt2.micro instances, or having more of them.\n\n## GitLab Runners autoscaling and deregistering\n\nCurrently not implemented but would be nice to have.\n\n## Contribute a Marathon Ansible module\n\nThe lack of an Ansible module dedicated to running Marathon applications forces\nto use the `uri` module. It would be better and cleaner to have a dedicated\nmodule for Ansible, which could probably be created easily.\n\n## Tailor needed AWS key policies\n\nCurrently, the required policies listed for the AWS user are the following :\n\n- AdministratorAccess\n- AmazonEC2FullAccess\n- AmazonRoute53FullAccess\n\nIt could probably be restricted a bit more.\n\n## Improve security groups\n\nCurrently, the security groups are probably the worst part of this project.\nThe one created by Ansible allow everything, both inbound and outbound.\n\nI had a first try with the following opened ports, but still had issues with\nZookeeper so I postponed that to focus on the rest of this project.\n\n- 80 / 443 : HTTP(S)\n- 2888 / 3888 / 2181 : Zookeeper\n- 8080 : Marathon\n- 31000 - 32000 : Mesos Docker containers\n- 5050 : Mesos\n- 22 : SSH\n- ICMP\n\n## Discover host keys\n\nCurrently, the host key checking is disabled in Ansible's configuration to\nallow running on newly spawned instances without having to confirm anything. It\nwould be better to discover these keys.\n\n## Work without Route 53\n\nTo allow people to test this infrastructure more easily, it would be nice if it\ndidn't depend on some of the records like `gitlab.\u003cdomain\u003e`.\n\n## Tests...\n\nA real test step in the CI Pipeline would be a must have in a real usecase,\nsince until know I focused on the delivery part.\n\n# Fun parts\n\nThis project was really fun, and I thought I would detail some of the\nsurprising and interesting points I encountered during this week.\n\n# Zookeeper SERVERS\n\nWhile trying to use the [mesoscloud/zookeeper Docker\nimage](https://hub.docker.com/r/mesoscloud/zookeeper/), I originally used\nhardcoded IP to configure Zookeeper with the list of its nodes.\n\nThe zookeeper image can indeed be configured at runtime with the \"SERVERS\"\nenvironment variable, under the following form:\n\n    SERVERS=first_node_ip,second_node_ip,third_node_ip\n\nObviously, since this approach is absolutely neither flexible nor scalable, I\nthen decided to use the dynamic inventory from EC2 to populate this list of\nservers, which looks like that in Jinja:\n\n    zk_nodes: \"{% for host in groups['ec2'] %}{{ hostvars[host]['ec2_private_ip_address'] }}{% if not loop.last %},{% endif %}{% endfor %}\"\n\nWe just loop through the hosts referenced under the 'ec2' group, take each host\nIP address, and separate them with comas.\n\nWell, this should work, shouldn't it ?\n\nAt this point, it's worth mentioning that we also pass a \"MYID\" environment\nvariable to the zookeeper container, referencing... well, its ID.\n\nHow does this matter ?\n\nThe SERVERS and MYID environment variables are used by the\n[entrypoint.sh](https://github.com/mesoscloud/zookeeper/blob/master/3.4.8/centos/7/entrypoint.sh)\nscript used as entrypoint for the image.\n\nThe following parts are what interests us :\n\n    echo \"${MYID:-1}\" \u003e /tmp/zookeeper/myid\n\nIt just fills a file with the Zookeeper ID, everything looks good here. Then, a\nfew lines later:\n\n    printf '%s' \"$SERVERS\" | awk 'BEGIN { RS = \",\" }; { printf \"server.%i=%s:2888:3888\\n\", NR, $0 }' \u003e\u003e /opt/zookeeper/conf/zoo.cfg\n\nThe important part here is the `server.%i`, which increments at each node in\nthe list.\n\nSo, this environment variable:\n\n    SERVERS=first_node_ip,second_node_ip,third_node_ip\n\nis transformed into this configuration :\n\n    server.1=first_node_ip\n    server.2=second_node_ip\n    server.3=third_node_ip\n\nAnd the `X` in `server.X` is assumed to be the node ID. But since their ID is\nset by the `zkid` tag, when using a dynamic inventory it probably won't build\nthe list in the increasing order of your ID !\n\nSo at the end I just ended up tweaking the entrypoint to give couples of ID/IP\nas SERVERS.\n\n# GitLab first login automation\n\nWhen you first install GitLab, it automatically sets a random password for the\nroot account and force you to change it at your first connection. For a\ntraditional use, it's not a big deal, but when you're trying to automate the\nGitLab spawn and its configuration... you get a problem.\n\nSo, since there is absolutely no documentation on this usecase, I thought about\nthese 3 ways to solve it :\n\n1. Reset the password through the API. Except it doesn't have any endpoint for\n   this initial password, and we wouldn't be able to authenticate to the API\n   anyway\n2. Just POST the answer as if we did it from a browser. Except GitLab is based\n   on rails, which as built-in `authenticity_token` handling to avoid CSRF; so\n   we would have to GET the page first, then parse it, the POST the answer...\n   no way.\n3. Play with the assets/wrapper that is used as entrypoint/CMD of the docker\n   image\n\nAfter investigating on this 3rd point, I finaly realized that it's possible to\nset the root password at installation through environment, which seems to be\ndocumented **nowhere**.\n\nSee [001_admin.rb from gitlab-ce\nsource](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/db/fixtures/production/001_admin.rb)\nfor the code :\n\n    if ENV['GITLAB_ROOT_PASSWORD'].blank?\n      user_args[:password_automatically_set] = true\n      user_args[:force_random_password] = true\n    else\n      user_args[:password] = ENV['GITLAB_ROOT_PASSWORD']\n    end\n\nI think I might submit a PR on the documentation to add this point.\n\n# GitLab CI runner token\n\nGitLab CI Runners need a token to register themselves on GitLab. However, there\nis absolutely no way to discover this token, so I ended up doing it through the\nGitLab API and then feeding it by environment variables to the GitLab Runner\nDocker image.\n\nThe Docker image is also not made for autoregistering so I quickly [implemented\nit](https://github.com/Horgix/dockerfiles/tree/master/gitlab-runner) and will\nprobably suggest it as an improvement to the official gitlab-runner image.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhorgix%2Fclick-count-infra","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhorgix%2Fclick-count-infra","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhorgix%2Fclick-count-infra/lists"}