Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/roeeelnekave/kubernetes-ingress-circle-ci

Kubernetes with flask app with ingress enabled monitoring with grafana and prometheus ci-cd through circle ci
https://github.com/roeeelnekave/kubernetes-ingress-circle-ci

docker flask grafana ingress-nginx kubernetes prometheus

Last synced: 5 days ago
JSON representation

Kubernetes with flask app with ingress enabled monitoring with grafana and prometheus ci-cd through circle ci

Awesome Lists containing this project

README 

        
# Setup Grafana and prometheus monitoring in kubernetes with flask inside and deploy using circle ci

# Prerequities

- 64-bit chip in your system

- minikube

- docker

- docker-compose

- python

- helm

- nodejs

- make



# Setup directories 

```bash

mkdir -p app/gulp/assets/{css,images,js}

mkdir -p app/gulp/assets/js/modules

mkdir -p docker/{flask,nginx}

mkdir -p kube

mkdir -p templates

```



# Setup grafana and promethues

- Run the following to setup helm

```bash

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

helm repo update

```



- Create the custom-values `touch ./kube/custom-values.yaml` and paste the following

```yaml

# custom-values.yaml

prometheus:

  service:

    type: NodePort

grafana:

  service:

    type: NodePort

```

- Then, install the kube-prometheus-stack using Helm run the following:



```bash

helm upgrade --install kube-prometheus-stack prometheus-community/kube-prometheus-stack -f ./kube/custom-values.yaml

```



- Verify `kubectl get services`



Download images



Run the following command to download an image that you'll display in the website


wget https://blog.adobe.com/en/publish/2021/04/07/media_1460789842033a3aab3da4086a5abfd2326d59789.png -O app/gulp/assets/images/landscape.png




Create Flask application



As a first step, we'll create the flask application and once it is running we'll add all other functionalities


./requirements.txt


This is the file that stores the dependencies necessary for our small application
click==8.0.3

Flask==2.0.2

gunicorn==20.1.0

itsdangerous==2.0.1

Jinja2==3.0.3

MarkupSafe==2.0.1

Werkzeug==2.0.2




./app.py


This is the main flask application that holds the rules for our website.
#!/bin/env python3



import os

from flask import Flask, render_template



app = Flask(__name__)



@app.route("/")

def homepage():

    return render_template("homepage.html", content="Hello world")



if __name__ == '__main__':

    app.run(

        host=os.getenv('FLASK_IP', '0.0.0.0'),

        port=os.getenv('FLASK_PORT', 5000),

        debug=bool(os.getenv('FLASK_DEBUG', True))

    )




./templates/homepage.html


Let's just write a dummy html code that will be displayed by flask application. Don't worry about the missing css and js files yet. They'll be added later.
<!doctype html>

<html lang="en">

<head>

    <meta charset="UTF-8">

    <meta name="viewport"

          content="width=device-width, user-scalable=no, initial-scale=1.0, maximum-scale=1.0, minimum-scale=1.0">

    <meta http-equiv="X-UA-Compatible" content="ie=edge">

    <title>Document</title>

    <link rel="stylesheet" href="/css/external.css">

    <link rel="stylesheet" href="/css/app.css">

</head>

<body>



<h1>Hello world</h1>

<h3>welcome to my page</h3>



<img src="/images/bmw-r-1250-gs.jpg" width="500" alt="BMW R1250GS" />

<img src="/images/bmw-r-1250-gs.jpg" width="500" alt="BMW R1250GS" />

<img src="/images/bmw-r-1250-gs.jpg" width="500" alt="BMW R1250GS" />



<script src="/js/external.js"></script>

<script src="/js/app.js"></script>

</body>

</html>




Before we run our application, we'll have to create a virtual environment:


python3 -m venv myv

source myv/bin/activate




Then, let's install the dependencies for our little project:


pip install -r requirements.txt




Now, let's start our project and view it in the browser:
python app.py




Open the browser at url localhost:5000 and you should see the text from our html page,  but you won't be able to see the image displayed three times. 


The reason why you don't see the colored text and images is that they are not existing in the public directory and you don't have any server configured to serve those files.  We'll configure nginx inside a docker container a little bit later.


Create nodejs assets



To compile the assets files (css, js and images) we'll use gulp and a small npm package that I wrote to make life easier when it comes to frontend dependencies: kisphp-assets


If you haven't used gulp before, have a look at the gulp documentation.


Create the following files:


./app/gulp/gulpfile.js


This will be the main gulpfile.js file where you configure what tasks you want to run by gulp for your project.
const { task, series } = require('gulp');



const config = require('../../gulp-config');



function requireUncached(module) {

    delete require.cache[require.resolve(module)];

    return require(module);

}



const js = require('kisphp-assets/tasks/javascripts')(config.js.external);

const bsrf = requireUncached('kisphp-assets/tasks/browserify')(config.js.project);

const css = require('kisphp-assets/tasks/css')(config.css.external);

const incss = requireUncached('kisphp-assets/tasks/css')(config.css.project);

const files = require('kisphp-assets/tasks/copy_files')(config.files);



task('default', series(

    files.copy_files,

    css.css,

    incss.css,

    js.javascripts,

    bsrf.browserify,

));




./app/gulp/assets/css/main.css


Just a small styling for your page so you can see how it interacts with your application
h1 {

    color: #9C1A1C;

}



h3 {

    color: #3A7734;

}




./app/gulp/assets/js/modules/demo.js


This js file will not do much, but it will show you how to add custom js code. All you have to do, is to create a file per use case  and have init function for the exported object. You also can create functions, classes or everything you need in that file.


The files will be compiled by gulp with browserify plugin.
module.exports = {

    init: function() {

        console.log('file loaded');

    }

}




./app/gulp/assets/js/app.js


Here is the main javascript file for your application and will load all modules inside a document.ready jquery object.
$(document).ready(function(){

    require('./modules/demo').init();

    // add here more files that do one thing (Single Responsibility Principle)

});




./package.json


Let's create the dependencies list for our assets 
{

  "scripts": {

    "build": "gulp --gulpfile app/gulp/gulpfile.js --cwd ."

  },

  "dependencies": {

    "bootstrap": "^5.1.3",

    "jquery": "^3.6.0",

    "kisphp-assets": "^0.6.0"

  }

}




./gulp-config.js


The purpose of this file is to have the list for which files will be compiled by gulp tasks. You'll have the following configurations:
    

  • 

    js.external -> combine external javascript dependencies and combine them all into one external.js file
    • 

  • 

    js.project -> local javascript files written in require.js format and compiled by browserify into one app.js file
    • 

  • 

    css.external -> combine external css dependencies and save them into one external.css file
    • 

  • 

    css.project -> build local css files and save them into app.css file. Here you can use css, stylus or scss sources.
    • 

  • 

    files.xxxx -> here is the definition of the files you want to copy from dependencies to public directory. Usually you will copy images and fonts.
    • 



const settings = function(){

  this.root_path = __dirname;

  this.project_assets = __dirname + "/app/gulp/";



  this.settings = {

    "name": "kisphp demo",

    "root_path": this.root_path,

    "project_assets": this.project_assets,



    "js": {

      "external": {

        "sources": [

          'node_modules/jquery/dist/jquery.min.js',

          'node_modules/bootstrap/dist/js/bootstrap.min.js',

        ],

        "output_filename": "external.js",

        "output_dirname": "public/js/",

      },

      "project": {

        "sources": [

          this.project_assets + '/assets/js/app.js',

        ],

        "output_filename": "app.js",

        "output_dirname": "public/js/",

      }

    },

    "css": {

      "external": {

        "sources": [

          'node_modules/bootstrap/dist/css/bootstrap.min.css',

        ],

        "output_filename": "external.css",

        "output_dirname": "public/css/",

      },

      "project": {

        "sources": [

          this.project_assets + '/assets/css/main.css'

        ],

        "output_filename": "app.css",

        "output_dirname": "public/css/",

      }

    },

    "files": {

      "fonts": {

        "sources": [

          'node_modules/bootstrap/fonts/*.*',

        ],

        "output_dirname": "public/fonts"

      },

      "images": {

        "sources": [

          this.project_assets + '/assets/images/**/*.*'

        ],

        "output_dirname": "public/images"

      }

    }

  };



  return this.settings;

};



module.exports = settings();






Now that you have all these files created, let's install the dependencies:


npm install




Then let's generate our public directory with all required files in it:


npm run build




At this point, you should have a flask application and a generated public directory with two css files, two javascript files and one image


Again, if you run python app.py you still won't be able to load the assets files and the image. We'll do this in the next step.


Create Docker configuration


This file is optional but it will not add the generated directories into the docker context while you build the docker images
.dockerignore


myv

public




As you will see, I like to follow the convention of keeping the docker files inside the docker directory, even if I have one or mode docker images per project.  This helps me to have the projects a little bit more structured and clean.  


./docker/flask/Dockerfile


Let's create the dockerfile for the flask application. The installation of nodejs here is necessary only for the kubernetes use case which will be later.
FROM python:3.12 as base



COPY requirements.txt /requirements.txt



RUN pip install --upgrade pip \

&& pip install -r /requirements.txt



FROM base



COPY . /app/



RUN cp /usr/share/zoneinfo/Europe/Berlin /etc/localtime \

&& apt-get update \

&& apt-get install -y curl gcc g++ make \

&& curl -fsSL https://deb.nodesource.com/setup_16.x | bash - \

&& apt-get install -y nodejs



WORKDIR /app



CMD ["gunicorn", "--workers=2", "--chdir=.", "--bind", "0.0.0.0:5000", "--access-logfile=-", "--error-logfile=-", "app:app"]






./docker/nginx/Dockerfile


This is the dockerfile for the nginx container
FROM node:20 as base



COPY . /app



WORKDIR /app



RUN npm install --no-interaction

RUN npm run build



FROM nginx



COPY --from=base /app/public /app/public



COPY docker/nginx/proxy.conf /etc/nginx/conf.d/default.conf






./docker/nginx/proxy.conf


This is the nginx configuration for our application
server {

    listen 80;



    server_name _;



    location ~ \.(css|js|jpg|png|jpeg|webp|gif|svg) {

        root /app/public;

    }



    location / {

        proxy_set_header Host $host ;

        proxy_set_header X-Real-IP $remote_addr;

        proxy_set_header X-Forwarded-Proto: http;



        proxy_http_version 1.1;

        proxy_set_header Upgrade $http_upgrade;

        proxy_set_header Connection "upgrade";



        proxy_pass http://flask-app:5000;

        proxy_read_timeout 10;

    }

}




./docker-compose.yml



Here you configure your flask and nginx containers to work together and serve your application in the browser


version: "3"

services:

  flask-app:

    build:

      dockerfile: docker/flask/Dockerfile

      context: .

    ports:

      - 5000

    volumes:

      - ./:/app

      - ./public:/app/public



  flask-nginx:

    image: nginx

    volumes:

      - ./docker/nginx/proxy.conf:/etc/nginx/conf.d/default.conf

      - ./public:/app/public

    ports:

      - 80:80




Well, in this point, if you still have flask application running, press CTRL + C to stop it. 



Create kubernetes configuration



Now, that we have the application running on local, let's configure kubernetes.


For tests, we'll use minikube


Start minikube


minikube start




Make sure you have the following plugins installed and enabled:
    

  • dns
    • 

  • ingress
    • 

  • registry
    • 

  • storage-provisioner
    • 

  • metrics-server
    • 




List addons:


minikube addons list




Enable plugins if they are not enabled already


minikube addons enable registry 

minikube addons enable metrics-server 

minikube addons enable ingress 

minikube addons enable ingress-dns

minikube addons enable storage-provisioner




Let's go further and create our kubernetes manifests:


./kube/deployment.yaml


This is the deployment manifest where you configure the pods that will run the application.  In this setup, we'll create a pod with two containers (flask and nginx) and one initcontainer that will generate the fils in the public directory which will be defined as a shared volume between the containers.


apiVersion: apps/v1

kind: Deployment

metadata:

  name: flask

  labels:

    app: flask

spec:

  replicas: 2

  progressDeadlineSeconds: 120

  strategy:

    type: RollingUpdate

    rollingUpdate:

      maxUnavailable: 1

  selector:

    matchLabels:

      app: flask

  template:

    metadata:

      name: flask

      labels:

        app: flask

    spec:

      restartPolicy: Always

      containers:

        - name: flask

          image: localhost:5000/flask:__VERSION__

          imagePullPolicy: Always

          ports:

            - containerPort: 5000

          envFrom:

            - configMapRef:

                name: flask

          volumeMounts:

            - mountPath: /app/public

              name: public-dir

          livenessProbe:

            exec:

              command:

                - cat

                - /app/public/ready

            initialDelaySeconds: 5

            periodSeconds: 5

          readinessProbe:

            exec:

              command:

                - cat

                - /app/public/ready

            initialDelaySeconds: 5

            periodSeconds: 5

        - name: nginx

          image: nginx

          imagePullPolicy: Always

          ports:

            - containerPort: 80

          volumeMounts:

            - mountPath: /etc/nginx/conf.d

              name: nginx-config

            - mountPath: /app/public

              name: public-dir

          livenessProbe:

            exec:

              command:

                - cat

                - /app/public/ready

            initialDelaySeconds: 5

            periodSeconds: 5

          readinessProbe:

            exec:

              command:

                - cat

                - /app/public/ready

            initialDelaySeconds: 5

            periodSeconds: 5

      initContainers:

        - name: npm

          image: localhost:5000/flask:__VERSION__

          imagePullPolicy: Always

          workingDir: /app

          command:

            - bash

          args:

            - build.sh

          volumeMounts:

            - mountPath: /app/public

              name: public-dir

      volumes:

        - name: nginx-config

          configMap:

            name: flask-nginx

        - name: public-dir

          emptyDir: {}




./kube/config-map.yaml


We create two configurations maps. One for the flask application and one for the nginx container.  I think you have already noticed that in the deployment file, we don't use the a custom nginx container.  We could and that would have been easier, but let's do it like this so we don't create a docker image with static files.
apiVersion: v1

kind: ConfigMap

metadata:

  name: flask

  namespace: default

data:

  FLASK_PORT: "5000"

  FLASK_DEBUG: "0"

---

apiVersion: v1

kind: ConfigMap

metadata:

  name: flask-nginx

  namespace: default

data:

  default.conf: |

    server {

        listen 80;



        server_name _;



        location ~ \.(css|js|jpg|png|jpeg|webp|gif|svg) {

            root /app/public;

        }



        location / {

            proxy_set_header Host $host ;

            proxy_set_header X-Real-IP $remote_addr;

            proxy_set_header X-Forwarded-Proto: http;



            proxy_http_version 1.1;

            proxy_set_header Upgrade $http_upgrade;

            proxy_set_header Connection "upgrade";



            proxy_pass http://127.0.0.1:5000;

            proxy_read_timeout 10;

        }

    }




./kube/ingress.yaml


The ingress configuration will be used to access our application in the browser under the http://dev.k8s/ url
apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

  name: flask

  namespace: default

spec:

  rules:

    - host: dev.k8s

      http:

        paths:

          - path: /

            pathType: ImplementationSpecific

            backend:

              service:

                name: flask

                port:

                  number: 80



    - host: grafana.k8s

      http:

        paths:

          - path: /

            pathType: Prefix

            backend:

              service:

                name: kube-prometheus-stack-grafana 

                port:

                  number: 80

    



Postgresql



./kube/ps-claim.yaml


Copy and paste the following to create postgres volume claim




apiVersion: v1

kind: PersistentVolumeClaim

metadata:

  name: postgres-volume-claim

  labels:

    app: postgres

spec:

  storageClassName: manual

  accessModes:

    - ReadWriteMany

  resources:

    requests:

      storage: 10Gi




To create a configmap

./kube/ps-configmap.yaml





apiVersion: v1

kind: ConfigMap

metadata:

  name: postgres-secret

  labels:

    app: postgres

data:

  POSTGRES_DB: ps_db

  POSTGRES_USER: admin

  POSTGRES_PASSWORD: admin





To create deployment

./kube/ps-deployment





apiVersion: apps/v1

kind: Deployment

metadata:

  name: postgres

spec:

  replicas: 3

  selector:

    matchLabels:

      app: postgres

  template:

    metadata:

      labels:

        app: postgres

    spec:

      containers:

        - name: postgres

          image: 'postgres:16'

          imagePullPolicy: IfNotPresent

          ports:

            - containerPort: 5432

          envFrom:

            - configMapRef:

                name: postgres-secret

          volumeMounts:

            - mountPath: /var/lib/postgresql/data

              name: postgresdata

      volumes:

        - name: postgresdata

          persistentVolumeClaim:

            claimName: postgres-volume-claim







To create a presistent volume use the following code

./kube/ps-pv.yaml




apiVersion: v1

kind: PersistentVolume

metadata:

  name: postgres-volume

  labels:

    type: local

    app: postgres

spec:

  storageClassName: manual

  capacity:

    storage: 10Gi

  accessModes:

    - ReadWriteMany

  hostPath:

    path: /data/postgresql



To create postgres service use the following code and file

./kube/ps-service.yaml




apiVersion: v1

kind: Service

metadata:

  name: postgres

  labels:

    app: postgres

spec:

  type: NodePort

  ports:

    - port: 5432

  selector:

    app: postgres



For this run the following command in your terminal to add the minikube ip to your /etc/hosts:


sudo bash -c "echo \"$(minikube ip) dev.k8s\" >> /etc/hosts"




Also 



sudo bash -c "echo \"$(minikube ip) grafana.k8s\" >> /etc/hosts"



Then, if you run cat /etc/hosts you should see on the last line your minikube ip and dev.k8s


./kube/service.yaml


We'll create a service of type ClusterIP for our application that will connect to the port 80 on the nginx container in the deployment pod. 
apiVersion: v1

kind: Service

metadata:

  name: flask

  namespace: default

spec:

  type: ClusterIP

  selector:

    app: flask

  ports:

    - port: 80

      targetPort: 80




For a better understanding of what happens here, when you make a request to the url http://dev.k8s/, the browser will make a request to the minikube instance and will match the ingress with the url defined earlier which will connect to the flask service which will connect to the nginx container in the running pod of the flask application.


For the requests to the css, js or images files, nginx will directly deliver them, but for other types of requests, the nginx will proxy to the python application.


./Makefile



We'll use this makefile to simulate a real deployment to a real kubernetes cluster


.PHONY: run up svc



version = $(shell date +%H%M%S)



run:

	python3 app.py



up: dependencies

up:

	docker build -f docker/flask/Dockerfile -t localhost:5000/flask:$(version) .

	docker push localhost:5000/flask:$(version)

	cat kube/deployment.yaml | sed "s/__VERSION__/$(version)/g" | kubectl apply -f -



svc: dependencies

svc:

	cat kube/deployment.yaml | sed "s/__VERSION__/214714/g" | kubectl apply -f -



dependencies:

	kubectl apply -f kube/config-map.yaml

    kubectl apply -f kube/service.yaml

    kubectl apply -f kube/ingress.yaml

    kubectl apply -f kube/ps-claim.yaml

    kubectl apply -f kube/ps-configmap.yaml

    kubectl apply -f kube/ps-deployment.yaml

    kubectl apply -f kube/ps-service.yaml



clean:

	docker images | grep localhost | awk '{print $$3}' | uniq | xargs docker rmi -f




Please note that in makefiles, you MUST use tabs for the commands bellow every stage and not spaces


./build.sh


This file is used by the init container to generate the content of the public directory
#!/bin/bash



npm install --no-interaction

npm run build



touch public/ready




At this point, if you open the url http://dev.k8s/ in your browser, you should see a 404 Page Not Found error, which is fine.


Our setup, will use a private/local registry for the docker images and that is running on the minikube virtual machine.


Let's stop and delete the local docker containers that we used earlier:


docker stop $(docker ps -q)

docker rm $(docker ps -q)




Run the following command to connect to the minikube docker


eval $(minikube docker-env)



And then 

docker run -d -p 5000:5000 --name myregistry registry:2



again 

eval $(minikube docker-env)




To make sure you are using the docker from minikube, run docker images and you should see k8s docker images listed.


Run the following command to build the docker image, push it to the registry and deploy all resources to minikube kubernetes:


make up