{"id":16597093,"url":"https://github.com/trannhatnguyen2/thesis","last_synced_at":"2026-04-09T21:04:02.601Z","repository":{"id":226370844,"uuid":"768429758","full_name":"trannhatnguyen2/thesis","owner":"trannhatnguyen2","description":"Sentiment analysis model deployment - CI/CD Pipeline To Deploy To Kubernetes Cluster Using Jenkins","archived":false,"fork":false,"pushed_at":"2024-07-24T05:04:45.000Z","size":34267,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-17T04:44:01.111Z","etag":null,"topics":["ansible","fastapi","gcp","helm-charts","jenkins","k8s","monitoring","terraform"],"latest_commit_sha":null,"homepage":"","language":"Smarty","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/trannhatnguyen2.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-07T04:12:52.000Z","updated_at":"2024-07-24T05:04:48.000Z","dependencies_parsed_at":"2024-03-07T08:52:24.691Z","dependency_job_id":"c1a63931-3e2b-4c52-a00f-933963062519","html_url":"https://github.com/trannhatnguyen2/thesis","commit_stats":null,"previous_names":["trannhatnguyen2/thesis"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trannhatnguyen2%2Fthesis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trannhatnguyen2%2Fthesis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trannhatnguyen2%2Fthesis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trannhatnguyen2%2Fthesis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/trannhatnguyen2","download_url":"https://codeload.github.com/trannhatnguyen2/thesis/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242274430,"owners_count":20101043,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ansible","fastapi","gcp","helm-charts","jenkins","k8s","monitoring","terraform"],"created_at":"2024-10-11T23:54:57.829Z","updated_at":"2026-04-09T21:04:02.507Z","avatar_url":"https://github.com/trannhatnguyen2.png","language":"Smarty","funding_links":[],"categories":[],"sub_categories":[],"readme":"# THESIS GRADUATION - Sentiment analysis model deployment - CI/CD Pipeline To Deploy To Kubernetes Cluster Using Jenkins\n\n\u003c!-- @import \"[TOC]\" {cmd=\"toc\" depthFrom=1 depthTo=6 orderedList=false} --\u003e\n\u003c!-- code_chunk_output --\u003e\n\n- Contents:\n  - [Introduction](#introduction)\n  - [Repository structure](#repository-structure)\n  - [Prerequisites installation](#prerequisites-installation)\n  - [Component Preparation](#component-preparation)\n  - [Usage](#usage)\n  \u003c!-- /code_chunk_output --\u003e\n\n## Introduction:\n\nThis repo will help and guide you to build and serve ML model as in a production environment (Google Cloud Platform). I also used tool \u0026 technologies to quickly deploy the ML system into production and automate processes during the development and deployment of the ML system.\n\n ## Workflow graph: \n\n![systempipline](assets/project_pipeline.png)\n\n- Source control: Git/Github\n- CI/CD: Jenkins\n- Build API: FastAPI\n- Containerize application: Docker\n- Container orchestration system: Kubernetes/K8S\n- K8s's package manager: Helm\n- Monitoring tool: Prometheus \u0026 Grafana\n- Deliver infrastructure as code: Ansible \u0026 Terraform\n- Ingress controller: Nginx ingress\n- Cloud platform: Google cloud platform/GCP\n\n ### Kubernetes architecture:\n![k8sarchi](assets/Kubernetesarchi.png) \n\n## Repository structure:\n\n```txt\nThesis-Sentiment\n    ├── app/                                            /* application /*\n    │   ├── preprocess/                                 /* preprocessing dataset /*\n    │   └── model.py                                    /* load model and predict sentiment /*\n    ├── model_storage/                                  /* storage model /*\n    │   ├── phobert-base-v2/\n    │   └── phobert_fold5.pth\n    ├── helm_chart/                                     /* manage Kubernetes applications /*\n    │   ├── app/                                        /* templates for app /*\n    │   ├── nginx_ingress/                              /* templates for nginx-ingress /*\n    │   ├── prometheus-grafana/                         /* templates for prometheus \u0026 gafana (Metrics) /*\n    │   │   └── kube-prometheus-stack/\n    │   │       └── templates/\n    │   │           ├── alertmanager/                   /* templates for alertmanager /*\n    │   │           ├── exporters/                      /* templates for exporters /*\n    │   │           ├── prometheus/\n    │   │           ├── gafana/\n    │   │           └── ../\n    │   ├── elasticsearch/                              /* templates for elasticsearch (Logs) /*\n    │   └── jaeger-operator/                            /* templates for jaeger (Traces) /*\n    ├── ansible/\n    │   ├── deploy_jenkins/\n    │   │   ├── create_compute_instance.yaml            /* config compute instance /*\n    │   │   └── deploy_jenkins.yaml                     /* config deploy jenkins to instance /*\n    │   ├── secret_keys/                                /* service account in Google Cloud /*\n    │   │   └── mlops-416203-8f35c6f23ccf.json\n    │   └── inventory                                   /* host group /*\n    ├── terraform/                                      /* config GKE /*\n    │   ├── main.tf\n    │   ├── variables.tf\n    │   └── outputs.tf\n    ├── .gitignore\n    ├── Dockerfile                                      /* build image to package model /*\n    ├── docker-compose.yaml\n    ├── Jenkinsfile                                     /* config workflow CI/CD /*\n    ├── main.py\n    ├── client.py                                       /* testing request /*\n    ├── README.md                                       /* guide /*\n    ├── requirements.txt                                /* libraries for app (packing model) /*\n    └── requirements_dev.txt                            /* libraries for deployment /*\n```\n\n## Prerequisites installation:\n\n### Google Cloud Platform: Account Registration \u0026 Project Billing\n\nGoogle Cloud Platform will be the cloud we use in this project, so you should access [Google Console](https://console.cloud.google.com/) and register an account. (If you have a Gmail account, this should be easy)\n\nAfter creating GCP account, let's create your own `Project` now:\n\n![CreatenewproGCP](assets/CreatenewProjectGCP.png)\n\nFill Project name (for example, \"mlops\" ), and hit **Create**\n\n![CreatenewproGCP2](assets/CreateNewprojectGCP2.png)\n\n**Note**: Remember to create a `billing account` after creating the project, then linking that `billing account` to the newly created project (refer: [Create and Link Billing account](https://www.youtube.com/watch?v=uINleRduCWM)). If you've never used GCP before, choose \"START MY FREE TRIAL\" to try it out for 3 months for free.\n\nNext, navigate to [Compute Engine API UI](https://console.cloud.google.com/marketplace/product/google/compute.googleapis.com) to \"ENABLE\" **Compute Engine API**:\n\n![EnableComputeEngine](assets/EnableComputeEngineAPI.png)\n\nNavigate to [Kubernetes Engine API UI](https://console.cloud.google.com/marketplace/product/google/container.googleapis.com) to \"ENABLE\" **Kubernetes Engine API**:\n\n![Enablek8s](assets/enableK8s.png)\n\n### Install the gcloud CLI:\n\nWe can easily connect to GKE using the Gcloud CLI. Reading this guide to install gcloud CLI in local terminal [gcloud CLI](https://cloud.google.com/sdk/docs/install#deb).\n\nAfter that, initialize the gcloud CLI by typing `gcloud init`, then type \"Y\"\n\n```bash\ngcloud init\n```\n\n**Note**:\n\n- A pop-up to select your Google account will appear, select the one you used to register GCP, and click the button Allow\n- Now, go back to your terminal, in which you typed `gcloud init`, choose your project, and Enter.\n- Then type Y, and select the area that is ideal for you., then Enter.\n\n### Install dev environment:\n\n#### Requirements:\n\n```bash\npip install -r requirements_dev.txt\n```\n\n**Note**: Simply said, this is the setting when you code locally. The `requirements.txt` file specifies the application environment in detail.\n\n### Additional Installation (Skip if you have already installed):\n\n- [Docker](https://docs.docker.com/desktop/install/ubuntu/)\n- [Kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/)\n- [kubectx + kubens](https://github.com/ahmetb/kubectx#manual-installation-macos-and-linux) (Optional)\n- [Terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli#install-terraform)\n\n## Component Preparation:\n\n### Create Jenkins on google cloud VM:\n\nLet's create your Jenkins VM instance using `ansible`.\n\nBefore creating google cloud VM by ansbile, you must first prepare a few things to access the GCP like `service account`. You can refer to this link [Create service account](https://cloud.google.com/iam/docs/service-accounts-create)\n\n**Note**: When creating a service account, grant it `Compute admin` permission. And then:\n\n- Find the three dots icon in the service account's Actions column, then select Manage keys.\n- Click ADD KEY, then Create new key\n- Download a JSON file by selecting CREATE. Keep this file SAFE at all times.\n- Put your credentials under the folder `/local/ansible/secrets`\n\nCreate Jenkins VM instance on GCP.\n\n```bash\ncd ./local/ansible/deploy_jenkins\nansible-playbook create_compute_instance.yaml\n```\n\n**Note**: Please check the file `create_compute_instance.yaml`. The `project id` and `service account` should be changed to match yours (e.g., line 11 \u0026 line 14, line 43 \u0026 line 45).\n\nAfter creating your Jenkins VM instance on GCP, navigate to [VM instance UI](https://console.cloud.google.com/compute/instances) and COPY `external IP` corresponding with yours. I COPY `external IP` \"jenkins-instance\" for example:\n\n![ansibleIP](assets/AnsibleIP.png)\n\nModify the IP of the newly created instance to the `inventory` file, then run the following commands:\n\n```bash\nansible-playbook -i ../inventory deploy_jenkins.yml\n```\n\n**Note:** Please save this `Jenkins external IP`, we will use it later to access Jenkins again\n\n### Create GKE cluster:\n\nChange directory to `/terraform` folder and initializes a working directory containing Terraform configuration files.\n\n```bash\ncd ./terraform\nterraform init\n```\n\nThen you can creates an execution plan, which lets you preview the changes that Terraform plans to make to your infrastructure.\n\n```bash\nterraform plan\n```\n\nNote: Before creates an execution plan, you should authenticate with GCP first using the following command:\n\n```bash\ngcloud auth application-default login\n```\n\nCarries out the planned changes to each resource using the relevant infrastructure provider's API.\n\n**Note**: It will ask you for confirmation before making any changes. Type `yes` if you have checked the execution plan carefully.\n\n```bash\nterraform apply\n```\n\n### Connect to the GKE cluster:\n\nAfter `terraform apply` successfully, you have now initialized the gke cluster. Let's install [Helm](https://helm.sh/docs/intro/install/) to deploy application on the k8s cluster easily.\n\nThen navigate to [GKE UI](https://console.cloud.google.com/kubernetes):\n\n![GKEui](assets/GKEui.png)\n\nClick on the cluster \"mlops-416203-gke\" for example and select \"CONNECT\"\n\n![GKEconnect0](assets/GKEconnect0.png)\n\nA pop-up to CONNECT to your cluster will appear:\n\n![GKEconnect](assets/GKEconnect.png)\n\nCopy the line \"gcloud container ...\" into your local terminal:\n\n```bash\ngcloud container clusters get-credentials \u003cyour_gke_name\u003e --zone us-central1-c --project \u003cyour_project_id\u003e\n```\n\nWe should see the line \"kubeconfig entry generated for mlops-416203-gke\" after above command.\n\nThen, switch to your gke cluster using kubectx:\n\n```bash\nkubectx \u003cYOUR_GKE_CLUSTER\u003e\n```\n\nInstall the `nginx controller` on this new cluster right now to route traffic from outside to services within the cluster.\n\n```bash\nhelm upgrade --install ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx --namespace ingress-nginx --create-namespace\n```\n\n### Create Prometheus and Grafana monitoring:\n\nPrometheus and Grafana form a powerful combination for monitoring and observability. Therefore, I will utilize these two tools as my cluster's monitoring services.\n\nChange directory to /`prometheus-grafana` folder and using helm to install Prometheus and Grafana on newly created cluster:\n\n```bash\ncd ./prometheus-grafana\nhelm upgrade --install prometheus-grafana-stack -f values-prometheus.yaml kube-prometheus-stack --namespace monitoring --create-namespace\n```\n\n**Note:** View more information and get additional guide at [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack)\n\nNow both prometheus and grafana have been installed on GKE cluster (in namespace `monitoring`).\n\nLet's verify each matching monitoring service's host name and ingress IP to see if it has been installed successfully or not:\n\n```bash\nkubectl get ingress -n monitoring\n```\n\nWe should see our Ingresses after this command.\nIf you see host names for ingress like \"grafana.nguyentn.site,\" \"alertmanager.nguyentn.site\" and \"prometheus.nguyentn.site\" for example, with their corresponding addresses. That indicates that the installation was successful.\n\nSo we are going to do now is that we are going to take that addresses and in our `etc/hosts` file.\n\n```bash\nsudo vi /etc/hosts\n```\n\nAt the end of open file (below example image), we gonna define our mapping.\n\n![IPmapping](assets/mappingIPP.png)\n\nAnd this works locally if we are going type \"prometheus.nguyentn.site\" in the browser (below example image), and this will be the IP address that it's going to be mapped to. Do the same way when visiting \"alertmanager.nguyentn.site\" or \"prometheus.nguyentn.site\"\n\n![prometheusUIexample](assets/prometheusUIexample.png)\n\n**Note**: The domain names of the monitoring services can be altered to suit your preferences. To set them up, open the values-Prometheus.yaml file. Lines `364` for Alertmanager, `919` for Grafana, and `2726` for Prometheus are in particular.\n\n#### Sending Prometheus Alerts to Discord with Alertmanager:\n\nFirst, create an alerting rule with `additionalPrometheusRules` in `values-prometheus.yaml` file (line 154). You could also simply use the rule I've already built to stay an eye on Node memory.\n\nSetting up a webhook on `Discord`:\n\nI assume you're already using Discord and have a channel that you want to send alerts to (in this example, we're using #alerts).\n\nThen go to line 297 in `values-prometheus.yaml` file to replace the \u003cDISCORD_WEBHOOK_URL\u003e placeholder with the webhook URL you just copied from Discord. It should look something like this: https://discord.com/api/webhooks/XXX/YYY.\n\nThe config above will sends all alerts (grouped by alertname and job) to a single Discord receiver.\n\n## Usage:\n\n### CI/CD with Jenkins:\n\nFirst, check if we can connect to the External IP of Jenkins via port 22 by using telnet on your local terminal:\n\n```bash\ntelnet \u003cjenkins_external_IP\u003e 22\n```\n\nWe will see a notification that you have successfully connected if you did it correctly\n\nGenerate your SSH key first. Open your local terminal, type `ssh-keygen` and type Enter to die until Overwrite:\n\n```bash\nssh-keygen\n```\n\nNavigate to [METADATA](https://console.cloud.google.com/compute/metadata) and Select the tab SSH KEYS and click the button + ADD ITEM (or ADD SSH KEY if you don’t see the + ADD ITEM button):\n\nCopy the content of your file `~/.ssh/id_rsa.pub` to GCP and press the blue button SAVE at the bottom of the page:\n\n![sshkey](assets/sshkeyy.png)\n\n**Note**: To see the content of the file `~/.ssh/id_rsa.pub`, use the cat command\n\n```bash\ncat ~/.ssh/id_rsa.pub\n```\n\nNext, ssh to your jenkins VM:\n\n```bash\nssh -i ~/.ssh/id_rsa username@jenkins_externalIP\n```\n\nCheck if `jenkins` container is running:\n\n```bash\nsudo docker ps\n```\n\n![Jenkinscheck](assets/jenkincheck.png)\n\nOk! jenkins is running successfully. Let get the jenkins \"password\" now:\n\n```bash\nsudo docker exec -ti jenkins cat /var/jenkins_home/secrets/initialAdminPassword\n```\n\nOpen web browser jenkins UI through http://yourExternalIP:8081/ and paste jenkins password here:\n\n![Jenkinslogin](assets/jenkinslogin.png)\n\nAfter entering the password, install the \"set sugested plugin\". Information for user can be \"next/skiped\" -\u003e \"save and finish\" and so on ...\n\n![Jenkinsinstallsugest](assets/jenkinsInstallsugested.png)\n\nNow, we are in Jenkins UI:\n\n![JenkinsUI](assets/JenkinsUI.png)\n\n#### Install necessary plugins:\n\nNavigate to Dashboard \u003e Manage Jenkins \u003e Plugins \u003e Available plugin. And TYPE \"Docker, Docker pipeline, gcloud SDK, kubernetes\" on search bar. Then SELECT \"Install without restart\".\n\n![](assets/jenkinsInstallationplugin.gif)\n\nWhen the installation is complete, `ssh` to your Jenkins VM again and restart jenkins container:\n\n```bash\nssh -i ~/.ssh/id_rsa username@jenkins_externalIP # skip it if you are already in jenkins VM\nsudo docker restart jenkins\n```\n\n**Note:** When you go back to jenkins after restarting, it will force you to log in again. Enter \"admin\" in the account part to log in. The password part is the same as you took in the previous step.\n\n#### Connect and assign permissions so that Jenkins connect to the K8s cluster:\n\n- In local terminal, create `ClusterRoleBinding` to grant permissions that access cluster-wide (granting permissions across all namespaces):\n\n  ```bash\n  kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=system:anonymous\n\n  kubectl create clusterrolebinding cluster-admin-default-binding --clusterrole=cluster-admin --user=system:serviceaccount:model-serving:default\n  ```\n\n- Then, back to Jenkins UI. Navigate to Dashboard \u003e Manage Jenkins \u003e Node and Clouds.\n- SELECT \"Clouds\".\n- Add a new cloud \u003e Kubernetes.\n- Then fill in the cluster's information. (To get the \"Cluster CA certificate\", refer to [GKE UI](https://console.cloud.google.com/kubernetes)).\n  ![](assets/jenkinsConnectK8-crop.gif)\n\n#### Add dockerhub credential:\n\n- Navigate to Dashboard \u003e Manage Jenkins \u003e Credentials \u003e (global).\n- Hit \"Add Credentials\" blue box in the top right corner.\n- Then fill in the dockerhub information.\n  ![](assets/jenkinsAddDockerCredential-crop.gif)\n\n#### Generate github access tokens:\n\n- Go to your Github account [Github](https://github.com/)\n- Navigate Settings \u003e Developer Settings \u003e Personal access tokens\n- Create your access token. I'll give this token full permissions just to make things simple. It can be adjusted as desired.\n\nAnd now, we can create new Jenkins pipeline by following these step:\n\n- Click the New Item menu within Jenkins Classic UI left column\n  ![JenkinsNewitem](assets/Newitemjenkins.png)\n- Provide a name for your new item (e.g. My-Pipeline) and select Multibranch Pipeline\n  ![MultibranchJenkins](assets/multibranchJenkins.png)\n- Click the Add Source button, choose the type of repository you want to use and fill in the details (e.g. Github)\n  ![ChoosesourceJenkins](assets/choosetypeGithub.png)\n- Then, an expand UI to connect a GitHub Repository will appear. You can start using the github token you generated in the previous step.\n  ![ConnectGithub](assets/githubConnect.png)\n\n- After adding credential, remember to pick the credential you just added. Click the Save button and watch your first Pipeline run\n- You should see like the image below:\n  ![sucessPipeline](assets/sucessPipeline.png)\n\n#### Add webhook to your github repository:\n\n- Navigate to your repo \u003e Settings \u003e Webhooks\n- Hit \"Add webhook\" box in the top right corner.\n- Fill \"http://[JenkinsVMexternalIP]:8081/github-webhook/\"\n- Select Content type \"application/json\" \u003e \"Let me select individual events\" (Any event can be specified here to start the CI/CD pipeline. Meanwhile, I will decide which \"push\" and \"pull request\" events to set triggered.)\n\n- From now on, Jenkins will perform CI/CD as soon as you publish or pull a change to github automatically.\n\n### Test API:\n\nWe can now navigate to \"http://thesis.sentiment.com/docs\" in your web browser to test Sentiment Analysis API (Running on 2 pods).\n\n![testAPI](assets/testAPI.png)\n\n ### Using Prometheus and Grafana:\n\n#### Node exporter:\n\nThe Node Exporter will collect information such as CPU usage, memory usage, disk usage, and network usage. It help us to monitor the health of Kubernetes nodes and troubleshoot performance. Additionally, we already have a template dashboard for it that only needs to be reused.\n\n- Navigate to http://grafana.nguyentn.site/\n  ![NodeExporterStep1](assets/NodeExporterStep1.png)\n- Select \"Dashboard\"\n  ![NodeExporterStep2](assets/NodeExporterStep2.png)\n- Hit \"New\" blue box then select \"Import\":\n  ![NodeExporterStep3](assets/NodeExporterStep3.png)\n- Provide `Node exporter` Dashboard ID with \"1860\". (Any other already dashboard can be obtained through [Grafana Dashboard](https://grafana.com/grafana/dashboards/))\n  ![NodeExporterStep4](assets/NodeExporterStep4.png)\n- Select \"Prometheus\" data source and hit \"Import\"\n  ![NodeExporterStep5](assets/NodeExporterStep5.png)\n- Finally, you should see like the image below:\n  ![NodeExporterStep6](assets/NodeExporterStep6.png)\n\n#### Opentelemetry custom metrics dashboard:\n\nTo capture and export metrics from Sentiment Analysis API (`counter` for \"number of requests\" and `histogram` for \"response time\"). I utilized `Opentelemetry` module. Prometheus will then use port \"8099\" to scrape these metrics. Grafana will be set up to show these customized metrics on customized dashboard.\n\n- Go to [Prometheus Targets](http://prometheus.nguyentn.site/targets?search=) first to check if it has actually scraped Opentelemetry metrics from the Sentiment Analysis API.\n  ![prometheusCheckScrape](assets/prometheusCheckScrape.png)\n\n- Then navigate to [Grafana dashboard](http://grafana.nguyentn.site/dashboards) and create new dashboard:\n  ![OpenteleDashboardStep1](assets/OpenteleDashboardStep1.png)\n- Hit blue box \"Add visualization\"\n  ![OpenteleDashboardStep2](assets/OpenteleDashboardStep2.png)\n- A pop-up to select \"data source\" will appear. Then select \"Prometheus\".\n  ![OpenteleDashboardStep3](assets/OpenteleDashboardStep3.png)\n- Then add and decorate your new panel and dashboard:\n  ![OpenteleDashboardStep4](assets/OpenteleDashboardStep4.png)\n- Finally, you should see like the image below:\n  ![OpenteleDashboardStep5](assets/OpenteleDashboardStep5.png)\n\n**Note:**\n\n- You can customize other metrics from [Opentelemetry API metrics](https://opentelemetry.io/docs/specs/otel/metrics/api/) by yourself. Then just edit the `/app/main.py` file to wrap up.\n- Go to [Prometheus UI](http://prometheus.nguyentn.site/) to perform any expressions as you like. For instance, I want to know how many responses, on average, will come in within 5 minutes in 1 second through this expression \"`rate(diabetespred_response_histogram_seconds_count[5m])`\"\n  ![prometheusQueryExample](assets/prometheusQueryExample.png) --\u003e\n\n---\n\n\u003cp\u003e\u0026copy; 2024 NhatNguyen\u003c/p\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrannhatnguyen2%2Fthesis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftrannhatnguyen2%2Fthesis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrannhatnguyen2%2Fthesis/lists"}