{"id":14155217,"url":"https://github.com/bruj0/jenkins_pipeline","last_synced_at":"2025-04-10T21:21:07.358Z","repository":{"id":169550604,"uuid":"114324769","full_name":"bruj0/jenkins_pipeline","owner":"bruj0","description":"A lean Continuous Deployment, Testing and Integration Pipeline using CoreOS/Docker/Jenkins","archived":false,"fork":false,"pushed_at":"2017-12-22T03:35:40.000Z","size":528,"stargazers_count":45,"open_issues_count":0,"forks_count":10,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-24T18:49:40.181Z","etag":null,"topics":["coreos","django","docker","docker-compose","jenkins","jenkins-ci","jenkins-pipeline","python"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bruj0.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2017-12-15T03:50:51.000Z","updated_at":"2024-05-22T15:58:35.000Z","dependencies_parsed_at":"2024-02-01T06:55:16.858Z","dependency_job_id":"f5ca32b0-7121-4c71-a2b8-220008fe8e96","html_url":"https://github.com/bruj0/jenkins_pipeline","commit_stats":null,"previous_names":["bruj0/jenkins_pipeline"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bruj0%2Fjenkins_pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bruj0%2Fjenkins_pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bruj0%2Fjenkins_pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bruj0%2Fjenkins_pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bruj0","download_url":"https://codeload.github.com/bruj0/jenkins_pipeline/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248298958,"owners_count":21080440,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["coreos","django","docker","docker-compose","jenkins","jenkins-ci","jenkins-pipeline","python"],"created_at":"2024-08-17T08:02:33.109Z","updated_at":"2025-04-10T21:21:07.337Z","avatar_url":"https://github.com/bruj0.png","language":"Shell","funding_links":[],"categories":["python"],"sub_categories":[],"readme":"# A lean Continuous Deployment, Testing and Integration Pipeline using CoreOS/Docker/Jenkins\r\n\r\nBy Rodrigo A. Diaz Leven\r\n\u003c!-- TOC --\u003e\r\n\r\n- [A lean Continuous Deployment, Testing and Integration Pipeline using CoreOS/Docker/Jenkins](#a-lean-continuous-deployment-testing-and-integration-pipeline-using-coreosdockerjenkins)\r\n    - [Disclaimer and prologue](#disclaimer-and-prologue)\r\n    - [Integration](#integration)\r\n    - [CoreOS install](#coreos-install)\r\n    - [Jenkins and Nginx](#jenkins-and-nginx)\r\n        - [Jenkins Dockerfile](#jenkins-dockerfile)\r\n        - [Nginx Dockerfile](#nginx-dockerfile)\r\n        - [Binding everything with docker-compose](#binding-everything-with-docker-compose)\r\n        - [Systemd service for Jenkins/Nginx](#systemd-service-for-jenkinsnginx)\r\n        - [Configuring Jenkins](#configuring-jenkins)\r\n        - [Jenkins Pipeline Code](#jenkins-pipeline-code)\r\n- [Unit Testing](#unit-testing)\r\n- [Deployment to Development](#deployment-to-development)\r\n    - [Deploying the image on build pass](#deploying-the-image-on-build-pass)\r\n        - [Installing Python in CoreOS](#installing-python-in-coreos)\r\n        - [Installing Docker puller](#installing-docker-puller)\r\n        - [Docker puller as an OS service](#docker-puller-as-an-os-service)\r\n        - [Example application Project1](#example-application-project1)\r\n        - [Dockerfile for our applications](#dockerfile-for-our-applications)\r\n            - [API application](#api-application)\r\n- [Deployment to Production in AWS](#deployment-to-production-in-aws)\r\n    - [AWS Configuration](#aws-configuration)\r\n    - [Instances configurations](#instances-configurations)\r\n    - [Jenkins](#jenkins)\r\n\r\n\u003c!-- /TOC --\u003e\r\n\r\n## Disclaimer and prologue\r\n\r\nWe are going to use CoreOS as the base of our pipeline because in my opinion, it's the best suited to build cloud immutable servers with the right Docker support at the kernel level.\r\n\r\nThe idea behind CoreOS is that an installation can be replicated many times without the use of an external application like Ansible or Puppet.\r\n\r\nThis is done by writing a configuration file that the installer reads and execute at install time to configure the OS as you see fit and also can be used to replicate the same server build many times.\r\n\r\nWe could say that this a way to have \"OS as code\" the same as we have “Infrastructure as code” with Terraform.\r\n\r\nMore documentation on this can be found here:[ https://coreos.com/os/docs/latest/provisioning.html](https://coreos.com/os/docs/latest/provisioning.html)\r\n\r\nHowever, we will **NOT** be using this technique besides the initial configuration because it's beyond the scope of this article and so its left as an exercise to the reader.\r\n\r\n## Integration\r\n\r\n![image alt text](image_0.png)\r\n\r\nThe idea behind this pipeline is to do continuous integration and deployment of applications using Docker as a way to virtualize the environments and Jenkins for the automatization side but any technology can be used to replace them like [RTK ](https://coreos.com/rkt/)and [CircleCI](https://circleci.com/).\r\n\r\nAnother requirement met is to use a simple and super lean infrastructure of a single server with a minimum of 4GB of ram.\r\n\r\nWe won't be using any clusters, like Kubernetes or Docker Swarm or any other type of container orchestration since it would be outside of the scope of this Article, but the same principles can be used.\r\n\r\nEverything will run under a Docker container, this way it will be an immutable server, even Jenkins and Nginx.\r\n\r\nWe will use a Django application as an example that has a frontend, backend and a database running in Postgres, something really common , but any type of application that can be run in a container can work in this pipeline.\r\n\r\nJenkins will poll the branches for commits or pull requests and  will checkout the code,create a Docker image for the proper branch and optionally push it to the registry and tag it.\r\n\r\nHowever, this Docker image will first have to pass our Unit testing and then it will be deployed, if it fails we will be notified and the image will not be used.\r\n\r\nIt's very important that the developers use the same environments to the one that is put into production to minimize runtime errors.\r\n\r\nThis means that the developers should use the docker images we create to run the application while they are modifying it.\r\n\r\nThis way we make sure that there won't be any runtime errors when we deploy to production.\r\n\r\n## CoreOS install\r\n\r\nWe will deploy a local VM, you can use any type of deployment that CoreOS accepts:\r\n\r\n[https://coreos.com/os/docs/latest/](https://coreos.com/os/docs/latest/)\r\n\r\nFile: coreos/cloud-config.yml \r\n\r\n\r\n```yaml\r\n#cloud-config\r\nssh_authorized_keys:\r\n\r\n- ssh-rsa AAAAB3....\r\n\r\nhostname: Pipe1\r\nusers:\r\n- name: \"core\"\r\n  groups:\r\n  - sudo\r\n  - docker\r\n    coreos:\r\n     update:\r\n    reboot-strategy: off\r\n     units:\r\n  - name: systemd-networkd.service\r\n    command: stop\r\n  - name: static.network\r\n    runtime: true\r\n    content: |\r\n      [Match]\r\n      Name=eth0\r\n      [Network]\r\n      Address=192.168.X.X/24\r\n      Gateway=192.168.X.X\r\n      DNS=8.8.8.8\r\n  - name: down-interfaces.service\r\n    command: start\r\n    content: |\r\n      [Service]\r\n      Type=oneshot\r\n      ExecStart=/usr/bin/ip link set eth0 down\r\n      ExecStart=/usr/bin/ip addr flush dev eth0\r\n  - name: systemd-networkd.service\r\n    command: restart\r\n```\r\nThis will create a \"**core**\" user that will authenticate to the OS with a **SSH key**, remember that this should not be used on a production server.\r\n\r\nWe also add this user to the **docker **and **sudo **groups so we can use both tools.\r\n\r\nSince we are deploying in a VM we will configure a fixed IP but normally in a cloud environment DHCP will be used.\r\n\r\nMore info how to configure networking at [https://coreos.com/os/docs/latest/network-config-with-networkd.html](https://coreos.com/os/docs/latest/network-config-with-networkd.html)\r\n\r\n## Jenkins and Nginx\r\n\r\nWe will run Jenkins and Nginx under a Docker Compose project.\r\n\r\n### Jenkins Dockerfile \r\n\r\nFile: jenkins/Dockerfile\r\n\r\n```dockerfile\r\nFROM jenkins/jenkins:lts\r\nUSER root\r\n\r\nCOPY docker-engine.key /tmp/docker-engine.key\r\n\r\nRUN apt-get update \u0026\u0026 echo \"deb https://apt.dockerproject.org/repo debian-stretch main\" \u003e /etc/apt/sources.list.d/docker.list \\\r\n\r\n   \u0026\u0026 apt-get install apt-transport-https \u0026\u0026 apt-key add /tmp/docker-engine.key \u0026\u0026 apt-get update \u0026\u0026 apt-get install -y docker-engine\r\n\r\nRUN usermod -aG docker jenkins\r\n\r\nUSER jenkins\r\n\r\n```\r\n\r\nWe use the **jenkins:lts** image but in production you should use a tagged version like **jenkins:2.7.3-alpine**\r\n\r\nTo build the images from inside the Jenkins container, we will need to install the Docker client binary, however\r\n\r\nthis is not a docker-in-docker solution, we just use the docker client.\r\n\r\nTo do this we add the official deb repository for stretch and install the rest of the tools needed.\r\n\r\nIts very important to switch to the user **jenkins** after we are finishing installing so Jenkins doesn't run under root.\r\n\r\n### Nginx Dockerfile\r\n\r\nNginx is our main Ingress point and runs in a separate container:\r\n\r\nFile: nginx/Dockerfile\r\n\r\n```dockerfile\r\nFROM nginx:1.12.2-alpine\r\n#Add default configuration\r\n\r\nCOPY conf/wildcard.conf /etc/nginx/conf.d/wildcard.conf\r\n\r\nEXPOSE 80\r\n```\r\n\r\nWe add a configuration file to our container and use the official Nginx image for Alpine.\r\n\r\nFile: nginx/conf/wildcard.conf\r\n\r\n```nginx\r\nmap $sname $port {\r\n default 80;\r\n \"project1\" 80;\r\n \"project2\" 80;\r\n \"jenkins\" 8080;\r\n}\r\nserver {\r\n\r\n   listen 80;\r\n\r\n   server_name ~^(?\u003csname\u003e.+).devel.example.com$;\r\n\r\nlocation / {\r\n sendfile off;\r\n proxy_pass         http://$sname:$port;\r\n proxy_redirect     $scheme://$sname:$port/ /;\r\n proxy_http_version 1.1;\r\n resolver 127.0.0.11;\r\n\r\n proxy_set_header   Host              $host;\r\n proxy_set_header   X-Real-IP         $remote_addr;\r\n proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;\r\n proxy_set_header   X-Forwarded-Proto $scheme;\r\n proxy_max_temp_file_size 0;\r\n\r\n #this is the maximum upload size\r\n client_max_body_size       10m;\r\n client_body_buffer_size    128k;\r\n\r\n proxy_connect_timeout      90;\r\n proxy_send_timeout         90;\r\n proxy_read_timeout         90;\r\n proxy_request_buffering    off; # Required for HTTP CLI commands in Jenkins \u003e 2.54\r\n # workaround for https://issues.jenkins-ci.org/browse/JENKINS-45651\r\n add_header 'X-SSH-Endpoint' '$host.devel.example.com:50022' always;\r\n }\r\n}\r\n```\r\n\r\nAll this wildcard configuration does it connect our applications domain name with the proper service living inside a container.\r\n\r\nFor example the URL http://**jenkins**.devel.example.org will be proxied to [http://jenkins:80](http://project1:80) where Jenkins is the name of the service we declare in our docker-compose file.\r\n\r\nThis resolves to the correct container IP via the docker DNS system, more information: [https://docs.docker.com/engine/userguide/networking/](https://docs.docker.com/engine/userguide/networking/)\r\n\r\nSometimes we need to specify a different port than 80 and for this, we use a nginx map: map $sname $port\r\n\r\nMore information: [http://nginx.org/en/docs/http/ngx_http_map_module.html](http://nginx.org/en/docs/http/ngx_http_map_module.html)\r\n\r\n### Binding everything with docker-compose\r\n\r\nFile: jenkins/docker-compose.yml\r\n\r\n```dockerfile\r\nversion: \"3\"\r\nservices:\r\n master:\r\n   build: ./jenkins\r\n   image: jenkins-master\r\n   volumes:\r\n     - 'jenkins-data:/var/jenkins_home'\r\n     - 'jenkins-log:/var/log/jenkins'\r\n     - /var/run/docker.sock:/var/run/docker.sock\r\n   ports:\r\n     - \"50000:50000\"\r\n   environment:\r\n     JAVA_OPTS: \"-Djava.awt.headless=true\"\r\n nginx:\r\n   build: ./nginx\r\n   image: jenkins-nginx\r\n   ports:\r\n   - \"80:80\"\r\n   links:\r\n    - master:jenkins\r\nvolumes:\r\n jenkins-data:\r\n jenkins-log:\r\nnetworks:\r\n default:\r\n   external:\r\n     name: jenkins_default\r\n```\r\n\r\n* We use 2 volumes to persist configuration and logs for Jenkins: **jenkins-data** and **jenkins-log**\r\n\r\n* We bind the docker unix socket **/var/run/docker.sock** from the OS to the Jenkins container so we can send commands from inside it.\r\n\r\n* We open the port **50000 **so we can access Jenkins API in case we need to but we won't do need it in this article.\r\n\r\n* We forward port **80** on the OS to the port **80** on the Nginx container, here is where we can add port 443 for SSL offloading, also not covered on this article.\r\n\r\n### Systemd service for Jenkins/Nginx\r\n\r\nTo start this services we will use a unit file for Systemd which is the service manager used in CoreOS, again this should be done using Ignition configuration and not be done manually.\r\n\r\nFile: coreos/jenkins.service\r\n\r\n```ini\r\n[Unit]\r\nDescription=Jenkins container starter\r\nAfter=docker.service network-online.target\r\nRequires=docker.service network-online.target\r\n\r\n[Service]\r\nWorkingDirectory=/home/core/mypipeline/jenkins\r\nUser=core\r\nType=oneshot\r\nRemainAfterExit=yes\r\n\r\n#ExecStartPre=/opt/bin/docker-compose pull --quiet --parallel\r\nExecStart=/opt/bin/docker-compose up -d\r\n\r\nExecStop=/opt/bin/docker-compose down\r\n\r\n#ExecReload=/usr/local/bin/docker-compose pull --quiet --parallel\r\n#ExecReload=/usr/local/bin/docker-compose -d --force-recreate\r\n\r\n[Install]\r\nWantedBy=multi-user.target\r\n\r\n```\r\n\r\n* This file should be inside **/etc/systemd/system**\r\n\r\n* Change WorkingDirectory and User accordingly to your setup.\r\n\r\n* Reload systemd configuration with: **systemd daemon-reload**\r\n\r\n* Run the service with:** systemd start jenkins**\r\n\r\n* Add it to startup: **systemd enable jenkins**\r\n\r\n### Configuring Jenkins\r\n\r\nAfter we have Jenkins running, visit the UI at [http://jenkins.devel.example.com](http://jenkins.devel.example.com) \r\n\r\nYou will be asked for a generated one-time password that can be found in the docker-compose logs with:\r\n\r\n**$ docker-compose logs jenkins**\r\n\r\nOnce we are done setting up Jenkins for the first time we will add the Github Organizations plugin: [https://wiki.jenkins.io/display/JENKINS/GitHub+Organization+Folder+Plugin](https://wiki.jenkins.io/display/JENKINS/GitHub+Organization+Folder+Plugin)\r\n\r\nAnd configure a new Item using this Plugin:\r\n\r\n![image alt text](image_1.png)\r\n\r\nWe need to add the credentials of the user that will be used to connect to Github and the owner of the repository.\r\n\r\nThis can be two totally different users, usually, the owner is the name of the Organizations and the credentials a user within this Organization that has access to the repositories.\r\n\r\n![image alt text](image_2.png)\r\n\r\nHere you can also change the interval between commits checks:\r\n\r\n![image alt text](image_3.png)\r\n\r\n### Jenkins Pipeline Code\r\n\r\nFor Jenkins to add a project all you have to do is add a **Jenkinsfile **file to his repository at the root level.\r\n\r\nThis code will be run each time a new commit is detected.\r\n\r\nFile: project1/Jenkinsfile\r\n\r\n```groovy\r\nnode {\r\n\r\n stage 'Checkout'\r\n    checkout scm\r\n\r\n   stage ('Build') {\r\n   def imagename = imagename()\r\n   def tag\r\n    // Tag images created on master branch with 'latest'\r\n   if(env.BRANCH_NAME == \"master\"){\r\n     tag = \"latest\"\r\n   }else{\r\n     tag = env.BRANCH_NAME\r\n   }\r\n\r\n   if(!imagename){\r\n     echo \"You must define an imagename in .jenkins-env\"\r\n     currentBuild.result = 'FAILURE'\r\n    }\r\n  \r\n   def customImage = docker.build(\"${imagename}:${tag}\")\r\n }\r\n stage ('Deploy') {\r\n   sh \"curl -o /dev/null -s -X POST 'http://SERVER_INTERNAL_IP:9010/?token=7a5afff405e7bb61efc424026d6d38c1\u0026hook=project1\"\r\n }\r\n}\r\n\r\ndef imagename() {\r\n def matcher = readFile('.jenkins-env') =~ 'imagename=\"(.+)\"'\r\n matcher ? matcher[0][1] : null\r\n}\r\n\r\ndef handleError(String message){\r\n echo \"${message}\"\r\n currentBuild.setResult(\"FAILED\")\r\n slackSend color: 'danger', message: \"${message}\"\r\n sh 'exit 1'\r\n}\r\n\r\n```\r\n\r\n* Change **SERVER_INTERNAL_IP** with the internal IP of your server, this is how we tell the Docker engine to pull and restart the containers once a build has passed testing and building.\r\n\r\n* Create a .jenkins-env file that will hold variables specific to the repository such as the name of the image that this project will use, example:\r\n\r\nFile: project1/.jenkins-env\r\n\r\n\r\n```bash\r\nimagename=\"project1-front\"\r\n```\r\n\r\n\r\n\r\n* This will checkout the code and run the equivalent to: **docker build -t $imagename:$branch_name .**\r\n\r\n* For example: **docker build mycompany/project1:dev**\r\n* Here you could add a push command after testing has passed by adding a new stage ie:\r\n\r\n\r\n```groovy\r\n stage ('Push') {\r\n    customImage.push(“\"${imagename}:${tag}”);\r\n }\r\n```\r\n\r\n\r\n\r\n# Unit Testing\r\n\r\nAfter the Docker image was created the Unit Testing and Code Analysis can begin\r\n\r\n![image alt text](image_4.jpg)\r\n\r\nAfter the build passes all the tests it will be pushed to the repository to the corresponding branch and tagged accordingly.\r\n\r\nThis can be done by adding a new stage to the pipeline in the Jenkinsfile file.\r\n\r\n\r\n```groovy\r\n    stage('Unit Testing) {\r\n        /* Ideally, we would run a test framework against our image.*/\r\n        customImage.inside {\r\n            sh 'echo \"Tests passed\"'\r\n        }\r\n    }\r\n\r\n```\r\n\r\n\r\n\r\nThis will run inside the Docker container we just built.\r\n\r\n# Deployment to Development\r\n\r\n![image alt text](image_5.png)\r\n\r\n## Deploying the image on build pass\r\n\r\nAfter the build has passed testing and optionally pushed to the registry, we have to restart the service wherever is running.\r\n\r\nThis can be done with a multitude of tools but in this article, we will use a helper application that listens on the private IP of the server for a token and a tag identifier to run a set of commands for us, normally pull the new image and restart the service.\r\n\r\nThis is a great simple python app called docker-puller: [https://github.com/glowdigitalmedia/docker-puller](https://github.com/glowdigitalmedia/docker-puller)\r\n\r\nFor this we will need a new user called \"jenkins-service\" that has access to sudo but only to one command:\r\n\r\nFile: jenkins/coreos/jenkins-service.sudoers\r\n\r\n\r\n```ini\r\nCmnd_Alias RESTART_PROJECT1 = /bin/systemctl reload project1.service\r\njenkins-service ALL=(ALL:ALL) NOPASSWD: RESTART_PROJECT1\r\n```\r\n\r\n\r\n\r\nThis file should live under /etc/sudoers.d/ directory.\r\n\r\n### Installing Python in CoreOS\r\n\r\nThis will be used only to run our docker-puller helper application and is not needed to run the example project.\r\n\r\n\r\n```bash\r\nmkdir -p /opt/bin\r\ncd /opt\r\nwget http://downloads.activestate.com/ActivePython/releases/${VERSION}/${PACKAGE}.tar.gz\r\ntar -xzvf ${PACKAGE}.tar.gz\r\nmv ${PACKAGE} apy \u0026\u0026 cd apy \u0026\u0026 ./install.sh -I /opt/python/ \r\nln -sf /opt/python/bin/easy_install /opt/bin/easy_install\r\nln -sf /opt/python/bin/pip /opt/bin/pip\r\nln -sf /opt/python/bin/python /opt/bin/python\r\nln -sf /opt/python/bin/python /opt/bin/python2\r\nln -sf /opt/python/bin/virtualenv /opt/bin/virtualenv\r\n```\r\n\r\n### Installing Docker puller\r\n\r\nWe follow the simple steps to install docker puller:\r\n\r\n\r\n```bash\r\nsu - jenkins-service\r\ngit clone https://github.com/glowdigitalmedia/docker-puller.git\r\ncd docker-puller\r\nvirtualenv dockerpuller/\r\nsource dockerpuller/bin/activate\r\npip install -r requirements.txt\r\n```\r\nThe configuration will be:\r\n\r\nFile: docker-puller/config.json\r\n\r\n```json\r\n{\r\n   \"host\": \"SERVER_INTERNAL_IP\",\r\n   \"port\": 9010,\r\n   \"token\": \"7a5afff405e7bb61efc424026d6d38c1\",\r\n   \"hooks\": {\r\n       \"project1_restart\": \"scripts/restart-project1.sh\"\r\n   }\r\n}\r\n```\r\n\r\nFile: jenkins/docker-puller/restart-project1.sh\r\n\r\n```bash\r\n#!/bin/bash\r\nsudo /bin/systemctl reload project1.service \u0026\r\n```\r\n\r\nAn empty POST to the URL: \r\n\r\n[http://SERVER_INTERNAL_IP:9010/7a5afff405e7bb61efc424026d6d38c1/project1/project1_restart](http://SERVER_INTERNAL_IP:9010/7a5afff405e7bb61efc424026d6d38c1/project1/project1_restart)\r\n\r\nWill run our script at docker-puller/scripts/restart-project1.sh.\r\n\r\n### Docker puller as an OS service\r\n\r\nWe will configure docker-puller as a system service by using this systemd unit:\r\n\r\nFile: coreos/docker-puller.service\r\n\r\n```ini\r\n[Unit]\r\nDescription=Project1 reloader\r\nAfter=syslog.target network.target\r\n\r\n[Service]\r\nType=simple\r\nUser=jenkins-service\r\nWorkingDirectory=\r\nExecStart=/home/jenkins-service/docker-puller/dockerpuller/bin/python /home/jenkins-service/docker-puller/dockerpuller/app.py\r\nRestart=on-abort\r\n\r\n[Install]\r\nWantedBy=multi-user.target\r\n```\r\n\r\n\r\n\r\n### Example application Project1\r\n\r\nOur application runs under Django and consist of a frontend, backend, and a database.\r\n\r\nIt will be orchestrated using docker-compose and each one of the parts will have its own container.\r\n\r\nThis docker-compose file describes the \"dev\" branch of our application and can be used for any other branch like master or production.\r\n\r\nFile: project1/docker-compose.yml\r\n```dockerfile\r\nversion: '3'\r\n\r\nservices:\r\n postgres:\r\n   container_name: 'postgres'\r\n   image: 'postgres:9.5.3'\r\n   ports:\r\n     - '127.0.0.1:6432:5432'\r\n   volumes:\r\n     - 'pgdata:/var/lib/postgresql/data/'\r\n api:\r\n   container_name: project1-api\r\n   build: ./project1\r\n   image: project1-api:dev\r\n   command: [\"sh\", \"./deploy/container_start.sh\"]\r\n   ports:\r\n     - \"8002:8002\"\r\n   depends_on:\r\n     - postgres\r\n project1:\r\n   container_name: project1-front\r\n   build: ./project1-front\r\n   image: project1-front:dev\r\n   command: [ \"service\", \"nginx\", \"start\" ]\r\n   ports:\r\n     - 8080:80\r\n   depends_on:\r\n     - api\r\nvolumes:\r\n pgdata:\r\nnetworks:\r\n default:\r\n   external:\r\n     name: jenkins_default\r\n\r\n```\r\n\r\n* Each service has a build tag that points to the directory where the source of each part of our app resides, this directory must have a Dockerfile file that generates a proper image.\r\n\r\n* We bind port **6432 **on localhost to the Postgres container port **5432 **in case we need to access it using psql\r\n\r\n* **/deploy/container_start.sh** is our entrypoint for the API part of the application\r\n\r\n* Port **8002 **is exposed as the API port so the backend can access it.\r\n\r\n* The frontend part runs nginx inside the container serving the static files that connect to the API, ie AngularJS.\r\n\r\n* The frontend exposes the port **8080 **redirected to his internal **80 **port. \r\n\r\n* The service name **project1 **points to the frontend and can be accessed by our main Nginx as [http://project1:80](http://project1:80) , this is the name of the first part of our URL: [http://](http://project1.devel.example.com)**[project**1](http://project1.devel.example.com)[.devel.example.com](http://project1.devel.example.com) , and must match in both docker-compose and the URL.\r\n\r\n* We join both services (jenkins/nginx and our project app) by using a common network called \"**jenkins_default**\".\r\n\r\n### Dockerfile for our applications\r\n\r\n#### API application\r\n\r\nFile: project1/Dockerfile\r\n\r\n```dockerfile\r\n\r\n# Base python 2.7 build\r\n# Python 2.7 | Django\r\n\r\nFROM python:2.7\r\n\r\n\r\n##############################################################################\r\n# Environment variables\r\n##############################################################################\r\n# Get noninteractive frontend for Debian to avoid some problems:\r\n#    debconf: unable to initialize frontend: Dialog\r\nENV DEBIAN_FRONTEND noninteractive\r\n\r\n##############################################################################\r\n# OS Updates and Python packages\r\n##############################################################################\r\n\r\nRUN apt-get update \\\r\n   \u0026\u0026 apt-get upgrade -y \\\r\n   \u0026\u0026 apt-get install -y\r\n\r\nRUN apt-get install -y apt-utils\r\n# Libs required for geospatial libraries on Debian...\r\nRUN apt-get -y install binutils libproj-dev gdal-bin\r\n\r\n##############################################################################\r\n# A Few pip installs not commonly in requirements.txt\r\n##############################################################################\r\n\r\nRUN apt-get install -y nano wget\r\n# build dependencies for Postgres and image bindings\r\nRUN apt-get install -y python-imaging python-psycopg2\r\n\r\n##############################################################################\r\n# Install dependencies and run scripts.\r\n##############################################################################\r\n\r\nRUN mkdir -p /var/projects/project1\r\nCOPY requirements.txt /var/projects/project1/\r\nRUN pip install -r /var/projects/project1/requirements.txt\r\nCOPY . /var/projects/project1\r\nWORKDIR /var/projects/project1\r\n\r\n##############################################################################\r\n# Run start.sh script when the container starts.\r\n# Note: If you run migrations etc outside CMD, envs won't be available!\r\n##############################################################################\r\nCMD [\"sh\", \"./deploy/container_start.sh\"]\r\n\r\n# Expose listen ports\r\nEXPOSE 8002\r\n\r\nFront end application\r\n\r\nFile: project1-front/Dockerfile\r\nFROM node:carbon\r\n\r\n# Create app directory\r\nRUN mkdir -p /var/projects/project1 \u0026\u0026 mkdir -p /var/projects/project1/bower_components\r\nWORKDIR /var/projects/project1\r\n\r\n# Install app dependencies\r\n# A wildcard is used to ensure both package.json AND package-lock.json are copied\r\n# where available (npm@5+)\r\nCOPY package*.json ./\r\n\r\n# If you are building your code for production\r\n# RUN npm install --only=production\r\n\r\nRUN npm install\r\nRUN npm install -g grunt bower\r\n\r\nCOPY bower.json ./\r\nRUN bower install --config.interactive=false --allow-root\r\n\r\nCOPY ./app ./app\r\nCOPY Gruntfile.js ./\r\n\r\nRUN grunt build\r\n\r\nRUN apt-get update \u0026\u0026 apt-get install -y nginx\r\n\r\nRUN echo \"daemon off;\" \u003e\u003e /etc/nginx/nginx.conf\r\nCOPY nginx/default.conf /etc/nginx/sites-available/default\r\n\r\nEXPOSE 80\r\nCMD [\"service\",\"nginx\",\"start\"]\r\n```\r\n# Deployment to Production in AWS\r\nWe will use a two account setup to deploy to master.\r\n\r\nThis way we can have complete separation between Development and Production.\r\n\r\nThe idea is to allow only the Operations people to be able to access the Production servers.\r\n\r\nOne **Account A** will have the Jenkins/Development instance and **Account B** the Production instance.****\r\n\r\n## AWS Configuration\r\n\r\n* **Account A:** Give the instance where Jenkins is running permissions to access ECR:\r\n```\r\n{\r\n    \"Version\": \"2012-10-17\",\r\n    \"Statement\": [\r\n        {\r\n            \"Effect\": \"Allow\",\r\n            \"Action\": [\r\n                \"ecr:*\",\r\n                \"cloudtrail:LookupEvents\"\r\n            ],\r\n            \"Resource\": \"*\"\r\n        }\r\n    ]\r\n}\r\n```\r\n\r\n* **Account A:** Attach the role to the instance: \r\n\r\n![image alt text](image_6.png)\r\n\r\n* **Account B:** Configure the ECR repository in the production account, to permit the push/pull from the Jenkins instance by using the account ID from Account A in the Principal value.\r\n\r\n![image alt text](image_7.png)\r\n\r\n## Instances configurations\r\n\r\n* Login and run  this command to see if the permissions are ok (replace XXX with your Production account ID):\r\n\r\n```\r\n# su - jenkins_user\r\n$ docker run --rm -t -i brentley/awscli ecr get-login --registry-ids XXXXXXXXXXX --region us-east-1\r\ndocker login -u AWS -p eyJwYXlsb2FkIjoiS1o4MX...\r\n```\r\nIf successful you will see the command needed to execute to login to the repository in ECR.\r\n\r\n* In your Production instance open the port 9010 to access Docker puller **only** from your Jenkins instance public IP.\r\n  (This could also be done by using VPC Peering between the accounts and using private IPs)\r\n\r\n* In the Production instance configure Docker compose files to start the application\r\n\r\n* Configure docker puller to restart the application\r\n\r\n## Jenkins\r\n\r\n* Install the Amazon ECR and AWS Credentials plugins:\r\n\r\nhttps://wiki.jenkins-ci.org/display/JENKINS/Amazon+ECR\r\n\r\nhttps://wiki.jenkins.io/display/JENKINS/CloudBees+AWS+Credentials+Plugin\r\n\r\n\r\n* Configure the AWS credentials in Jenkins and name it \"**ec2role**\", use the ARN role you created in the first step.\r\n\r\nFollow this path: Jenkins -\u003e Your Org -\u003e Credentials -\u003e Folder -\u003e Global credentials (unrestricted)\r\n\r\n![image alt text](image_8.png)\r\n\r\n* We will a add a new step to our Pipeline, on the master or production branch, called \"Docker push\" after the build step.\r\n  (Replace with your repository URL)\r\n\r\n```groovy\r\n  steage ('Docker Push') {\r\n    docker.withRegistry(\"https://XXXXXXXX.dkr.ecr.us-east-1.amazonaws.com\",''ecr:us-east-1:ec2role') {\r\n    docker.image(\"${imagename}:${tag}\").push()\r\n  }\r\n```\r\n\r\n* Change the deploy URL to point to your production server\r\n```\r\n  stage ('Deploy') {\r\n    sh \"curl -o /dev/null -s -X POST 'http://PRODUCTION_PUBLIC_IP:9010/?token=7a5afff405e7bb61efc424026d6d38c1\u0026hook=project1'\"\r\n  }\r\n```\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbruj0%2Fjenkins_pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbruj0%2Fjenkins_pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbruj0%2Fjenkins_pipeline/lists"}