{"id":24550005,"url":"https://github.com/ajithvcoder/emlo4-session-15-ajithvcoder","last_synced_at":"2025-03-16T12:38:32.047Z","repository":{"id":272936008,"uuid":"911628019","full_name":"ajithvcoder/emlo4-session-15-ajithvcoder","owner":"ajithvcoder","description":null,"archived":false,"fork":false,"pushed_at":"2025-01-17T14:01:52.000Z","size":382,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-17T15:24:29.863Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ajithvcoder.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-03T13:18:00.000Z","updated_at":"2025-01-17T14:01:53.000Z","dependencies_parsed_at":"2025-01-17T15:24:36.574Z","dependency_job_id":"54393ff5-dde9-4c6c-9ff6-13950eec84d9","html_url":"https://github.com/ajithvcoder/emlo4-session-15-ajithvcoder","commit_stats":null,"previous_names":["ajithvcoder/emlo4-session-15-ajithvcoder"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ajithvcoder%2Femlo4-session-15-ajithvcoder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ajithvcoder%2Femlo4-session-15-ajithvcoder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ajithvcoder%2Femlo4-session-15-ajithvcoder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ajithvcoder%2Femlo4-session-15-ajithvcoder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ajithvcoder","download_url":"https://codeload.github.com/ajithvcoder/emlo4-session-15-ajithvcoder/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243870339,"owners_count":20361184,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-23T00:18:23.346Z","updated_at":"2025-03-16T12:38:32.017Z","avatar_url":"https://github.com/ajithvcoder.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n## EMLOV4-Session-15 Assignment - Kubernetes - III: EKS, ALB, Auto Scaling\n\nUsing Helm charts, we are deploying a `cat-dog model service` hosted on a FastAPI server with loadbalancer and auto scaling capablity in `AWS EKS` platform. Alongside, a `backend service built with FastAPI` handles requests. The `NextJS UI service` functions as the front-end interface for users. Additionally, a `Redis caching service` is integrated to enhance performance. Finally its exposed to internet with `aws-loadbalancers`. Cluster autoscaling(`CA`) and horizontal pod autoscaling(`HPA`) is used for `model-server`\n\n### Contents\n\n- [Requirements](#requirements)\n- [Development Method](#development-method)\n    - [Architecture Diagram](#architecture-diagram)\n    - [Installation](#installation)\n    - [Cluster Creation](#cluster-creation)\n    - [Install loadbalancer and kubernetes dashboard](#install-loadbalancer-and-kubernetes-dashboard)\n    - [Cluster Autoscaler](#cluster-autoscaler)\n    - [Horizontal Pod Autoscaler](#horizontal-pod-autoscaler)\n    - [Testing](#testing)\n    - [Deletion Procedure](#deletion-procedure)\n- [Learnings](#learnings)\n- [Results Screenshots](#results-screenshots)\n\n### Requirements\n\n- You will be deploying the previous assignment with EKS with CA \u0026 HPA\n- Before you begin changing anything, first create an Architecture Diagram\n- Create HELM Chart for the entire deployment\n    - You must use helm to deploy this\n    - no k apply -f\n- You must use ClusterAutoScaler, and HorizontalPodScaler\n    - Create an artificial load on the service\n    - Print the Latency of the Inference\n    - Log the HPA metrics, like we did in the session\n    - Your Pods and Nodes should automatically scale out and scale in when needed.\n- What to Submit\n    - Github repository will all the above changes\n    - must have the helm chart\n    - Instructions to deploy your solution on EKS\n    - With all the EKSCTL commands\n    - must have the logs for HPA scaling out\n    - Outputs of\n        - kubectl get all -A -o yaml dump it to a single YAML file\n        - kubectl top pod before load\n        - kubectl top pod after load\n        - kubectl describe \u003cyour main ingress\u003e\n        - Inference latency before heavy load\n        - Inference latency while load\n\n### Architecture Diagram\n\n![](./assets/images/snap_a15_arch.png)\n\n![](./assets/images/snap_a15_arch_2.png)\n\nNote: You can refer [class-work](./eks-dev-class-work) and develop the deployments stage by stage similar in session-15 class\n\nRefer: [class-work-readme](./eks-dev-class-work/README.md) for proper usage of classwork files\n\nNote: it took 5$ for doing class work debugging and development and 6$ for assignment debugging and development.\n\nLocal installations (no need a new ec2 instance for doing below work)\n\n### Installation\n\n**AWS install**\n\n```\ncurl \"https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip\" -o \"awscliv2.zip\"\nunzip awscliv2.zip\nsudo ./aws/install\n```\n\n**Provide credentials**\n\n```\naws configure\n```\n\n**EKSCTL Install**\n\n```\n# for ARM systems, set ARCH to: `arm64`, `armv6` or `armv7`\nARCH=amd64\nPLATFORM=$(uname -s)_$ARCH\n\ncurl -sLO \"https://github.com/eksctl-io/eksctl/releases/latest/download/eksctl_$PLATFORM.tar.gz\"\n\n# (Optional) Verify checksum\ncurl -sL \"\u003chttps://github.com/eksctl-io/eksctl/releases/latest/download/eksctl_checksums.txt\u003e\" | grep $PLATFORM | sha256sum --check\n\ntar -xzf eksctl_$PLATFORM.tar.gz -C /tmp \u0026\u0026 rm eksctl_$PLATFORM.tar.gz\n\nsudo mv /tmp/eksctl /usr/local/bin\n```\n\n**Set the default ssh-gen key in local**\n\nThis default ssh key is used by aws for default ssh login\n\n```\nssh-keygen -t rsa -b 4096\n```\n\n**Install kubectl for aws eks in your local**\n\n```\ncurl -O https://s3.us-west-2.amazonaws.com/amazon-eks/1.32.0/2024-12-20/bin/linux/amd64/kubectl\n\nchmod +x ./kubectl\n\nmkdir -p $HOME/bin \u0026\u0026 cp ./kubectl $HOME/bin/kubectl \u0026\u0026 export PATH=$HOME/bin:$PATH\n```\n\n**Docker images to ECR**\n\nBuild and push docker images to AWS ECR \n\nModel server\n\n- `docker build -t model-server -f Dockerfile.model-server .`\n\nWeb server\n\n- `docker build -t web-server -f Dockerfile.web-server .`\n\nUI server\n\n- `docker build -t ui-server -f Dockerfile.ui-server .`\n\n**Note: Make sure you change your account number in all `.yaml` files**\n\n### Cluster Creation\n\nGo into `eks-setup/cluster-config` folder\n\n```\neksctl create cluster -f eks-cluster.yaml\n```\n\n\nCheck instances from ec2\n\n```\nssh ec2-user@43.204.212.5\nkubectl config view\nkubectl get all\n```\n\n#### Enable OIDC on your EKS Cluster\n\n- `eksctl utils associate-iam-oidc-provider --region ap-south-1 --cluster basic-cluster-ap --approve`\n\n- `curl -o iam-policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.11.0/docs/install/iam_policy.json`\n\n- `aws iam create-policy --policy-name AWSLoadBalancerControllerIAMPolicy --policy-document file://iam-policy.json`\n\n\nCreate IAM Role for Service Account (IRSA)\n\n```\neksctl create iamserviceaccount --cluster=basic-cluster-ap --namespace=kube-system --name=aws-load-balancer-controller  --attach-policy-arn=arn:aws:iam::306093656765:policy/AWSLoadBalancerControllerIAMPolicy --override-existing-serviceaccounts  --region ap-south-1  --approve\n```\n\n### Install loadbalancer and kubernetes dashboard\n\nInstall the AWS Load Balancer Controller using HELM\n\n```\nhelm repo add eks https://aws.github.io/eks-charts\nhelm repo update\nhelm install aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system --set clusterName=basic-cluster-ap --set serviceAccount.create=false --set serviceAccount.name=aws-load-balancer-controller\n```\n\nVerify load balancer\n\n```\nkubectl get all -n kube-system\n```\n\nOptional:  Control Scaling of nodegroup\n\n```\neksctl scale nodegroup --cluster=basic-cluster-ap --nodes=2 ng-dedicated-1 --nodes-max=4\n```\n\n\n**Kubernetes dashboard**\n\nUse this and fix metric-server api error if u use genric one\n\n`https://medium.com/@cloudspinx/fix-error-metrics-api-not-available-in-kubernetes-aa10766e1c2f`\n\nonly then u can get output for ```kubectl top node``\n\n\n```\nhelm repo add kubernetes-dashboard https://kubernetes.github.io/dashboard/\nhelm upgrade --install kubernetes-dashboard kubernetes-dashboard/kubernetes-dashboard --create-namespace --namespace kubernetes-dashboard\n```\n\n### Cluster Autoscaler \n\nGo into `eks-dev-cluster-config`\n\n```\neksctl utils associate-iam-oidc-provider --region ap-south-1 --cluster basic-cluster-ap --approve\n```\n\n```\naws iam create-policy --policy-name AWSClusterAutoScalerIAMPolicy --policy-document file://cas-iam-policy.json\n```\n\n```\neksctl create iamserviceaccount --cluster=basic-cluster-ap --namespace=kube-system  --name=cluster-autoscaler --attach-policy-arn=arn:aws:iam::306093656765:policy/AWSClusterAutoScalerIAMPolicy --override-existing-serviceaccounts --region ap-south-1 --approve\n```\n\n\n```\nwget https://raw.githubusercontent.com/kubernetes/autoscaler/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-autodiscover.yaml\n```\n\nDo necessary modification\n\nModified `cluster-autoscaler-autodiscover.yaml` is present in cluster-config folder\n\n```\nkubectl apply -f eks-setup/cluster-config/cluster-autoscaler-autodiscover.yaml\n```\n\nVerify it with\n\n```\neksctl get nodegroup --cluster basic-cluster-ap\n```\n\n### Horizontal Pod Autoscaler\n\nDo neccessary changes in `model-server.deployment.yaml` file for HPA configurations\nNote: `redis`, `ui-server` and `web-server` can also be scaled like this\n\n\nInstall `fastapi-release-prod` app procedure\n\n- `helm install fastapi-release-prod fastapi-helm --values fastapi-helm/values.yaml -f fastapi-helm/values-prod.yaml`\n\nCheck if all pods are running\n\n- `kubectl get pod -n prod`\n\nUninstall app\n\n- `helm uninstall  fastapi-release-prod`\n\nRecognise deployment\n\n- `kubectl describe hpa classifier-hpa -n prod`\n\n\nAdmin Dashboard\n\n```\nkubectl -n kubernetes-dashboard create token admin-user\n\nkubectl -n kubernetes-dashboard port-forward svc/kubernetes-dashboard-kong-proxy 8443:443\n```\n\n**Debugging model-server during HPA scaling \"pending\" error**\n\n\n- `kubectl get nodes --show-labels`\n\ncheck if some node is with role: spot only then ASG can fetch it\n\nRun the linking of cluster autoscaler commands\n\nHPA scaling \"pending\" error\n\n- `eksctl delete nodegroup --cluster basic-cluster-ap --name ng-spot-4`\n\nRecreating ng-spot-4 managednodegroup\n\n- `eksctl create nodegroup --config-file=eks-setup/cluster-config/eks-cluster.yaml`\n\nWait for some time like 5 minutes if the pods are pending. else recreate the nodegroup and proceed\n\n**Debugging ingress**\n\n- `kubectl get ingress -n prod`\n\n- The `scaleTargetRef.name` should match with `model-server.deployment` name in `classifer-hpa` yaml content in `model-server.service.yml`\n\n- Wait for 3 minutes after helm installation and then use the ADDRESS from above command in browser\n\n### Testing\n\n- `kubectl get ingress -n prod`\n\n- Wait for 3 minutes after helm installation and then use the ADDRESS from above command in browser\n\n- Go to \"http://k8s-default-classifi-18da2b317c-1112532225.ap-south-1.elb.amazonaws.com/\" loadbalancer url and go to \"/docs\" and verify infer and health\n\n- Also verify the UI server\n\n**HPA Testing**\n\n- `cd src/model-server`\n\n- `python test_requests.py --url http://k8s-prod-modelser-107450934f-179443958.ap-south-1.elb.amazonaws.com --requests 1 --workers 1`\n\n\nCheck these commands if its working fine. else google and fix it\neg: for `kubectl top pod` you need metric server api\n\n`kubectl top pod -n prod`\n\n`kubectl get hpa classifier-hpa`\n\n`kubectl get node -n prod`\n\n### Deletion Procedure\n\nUninstall helm charts\n\n- `helm uninstall  fastapi-release-prod`\n\n- `kubectl delete pod --field-selector=\"status.phase==Failed\"`\n\nDelete kubectl resources\n\n- `kubectl delete all --all`\n\nDelete cluster\n\n- `eksctl delete cluster --name basic-cluster-ap --region ap-south-1`\n\n\n- Go to cloudformation and check if all are deleted that are created when u started u r work.\n- Remeber dont believe \"Deletion in progress\" it may get failed so wait and make sure all the resources are completely deleted in cloud formation\n- Delete the loadbalancers\n- Check the ec2 instance dashboard, spotrequest, autocluster group, loadbalancers and enure all are closed/terminated/deleted\n\n### Learnings\n\n- Learnt about setting up Cluster autoscaling and horizontal scaling\n- The ingress should be modified with internetfacing method and other changes to be integrated with load balancer\n- Debugging model-server during HPA scaling \"pending\" error\n\n### Results Screenshots\n\n- Helm charts are used for deployment\n\n    ![arch_1](./assets/images/snap_a15_arch.png)\n\n    ![arch_2](./assets/images/snap_a15_arch_2.png)\n\n- Model server and UI server exposed via ingress\n\n    ![ui server](./assets/images/snap_a15_ui_server.png)\n\n    ![model server](./assets/images/snap_15_model_server.png)\n\n    ![ingress](./assets/images/snap_ap15_ingress.png)\n\n\n- Create an artificial load on the service and print the Latency of the Inference\n\n    ![latency Heavy load](./assets/images/snap_a15_latency_during_heavy_load.png)\n\n- HPA logs\n\n    - [hpa_rescale_logs file](./assets/logs/hpa_rescale_logs.txt)\n    - [cluster_autoscaler_logs file](./assets/logs/cluster_autoscaler_logs.txt)\n\n- Screenshots of Autoscale out and scale in\n\n    Pod: scale out\n\n    ![Pod Heavy load](./assets/images/snap_a15_scale_in_0.png)\n    ![Pod Heavy load](./assets/images/snap_a15_scale_in_1.png)\n    ![Pod Heavy load](./assets/images/snap_a15_scale_in.png)\n\n    Pod: scale in\n\n    Note: Didnt capture the screenshot so kindly check cluster logs and hpa logs\n\n    [hpa_logs_last_line](./assets/logs/hpa_rescale_logs.txt)\n\n    Node: scale out\n\n    ![Node Heavy load](./assets/images/snap_a15_node_scaling_during_heavy_load.png)\n\n    Node: scale in \n\n    ![Node Heavy load](./assets/images/snap_a15_nodegroup_after_heavyload.png)\n\n\n- Nodegroup eks\n\n    ![snap_a15_nodegroup_eksctl](./assets/images/snap_a15_nodegroup_eksctl.png)\n\n- `kubectl get all -A -o yaml` dump it to a single YAML file\n\n    - [kubectl all config file](./assets/logs/all_config.yaml)\n\n- `kubectl top pod` before load\n\n    ![snap_a15_pod_before_load](./assets/images/snap_a15_pod_before_load.png)\n\n- `kubectl top pod` after load\n\n    ![snap_a15_top_pod_during_hload_3](./assets/images/snap_a15_top_pod_during_hload_3.png)\n\n- `kubectl describe \u003cyour main ingress\u003e`\n\n    ![snap_a15_describe_ingress](./assets/images/snap_a15_describe_ingress.png)\n\n- Inference latency before heavy load\n    ![snap_a15_inference_before_hload](./assets/images/snap_a15_inference_before_hload.png)\n\n- Inference latency while load\n    ![snap_a15_latency_during_heavy_load](./assets/images/snap_a15_latency_during_heavy_load.png)\n\n### Group Members\n\n1. Ajith Kumar V (myself)\n2. Pravin Sagar\n3. Hema M\n4. Muthukamalan\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fajithvcoder%2Femlo4-session-15-ajithvcoder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fajithvcoder%2Femlo4-session-15-ajithvcoder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fajithvcoder%2Femlo4-session-15-ajithvcoder/lists"}