{"id":18010878,"url":"https://github.com/yokawasa/kubectl-tips","last_synced_at":"2025-03-26T14:32:19.367Z","repository":{"id":66860534,"uuid":"229496696","full_name":"yokawasa/kubectl-tips","owner":"yokawasa","description":"Tips on Kubernetes cluster management using kubectl command","archived":false,"fork":false,"pushed_at":"2022-12-28T12:18:01.000Z","size":26,"stargazers_count":11,"open_issues_count":0,"forks_count":3,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-21T23:33:49.721Z","etag":null,"topics":["kubectl","kubernetes","tips","tricks"],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"cc0-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yokawasa.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-12-21T23:46:22.000Z","updated_at":"2022-12-28T12:15:32.000Z","dependencies_parsed_at":"2023-03-11T00:24:20.819Z","dependency_job_id":null,"html_url":"https://github.com/yokawasa/kubectl-tips","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yokawasa%2Fkubectl-tips","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yokawasa%2Fkubectl-tips/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yokawasa%2Fkubectl-tips/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yokawasa%2Fkubectl-tips/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yokawasa","download_url":"https://codeload.github.com/yokawasa/kubectl-tips/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245670931,"owners_count":20653455,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kubectl","kubernetes","tips","tricks"],"created_at":"2024-10-30T02:15:41.972Z","updated_at":"2025-03-26T14:32:19.349Z","avatar_url":"https://github.com/yokawasa.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# kubectl-tips\nTips on Kubernetes cluster management using kubectl command. A goal of this repository is to allow you reversely lookup kubectl commands from what you want to do around kubernetes cluster management.\n\n\u003c!-- TOC --\u003e\n\n- [kubectl-tips](#kubectl-tips)\n\t- [Kubectl Version Manager](#kubectl-version-manager)\n\t- [Print Cluster Info](#print-cluster-info)\n\t- [Print the supported API resources](#print-the-supported-api-resources)\n\t- [Print the available API versions](#print-the-available-api-versions)\n\t- [Display Resource (CPU/Memory) usage of nodes/pods](#display-resource-cpumemory-usage-of-nodespods)\n\t- [Updating Kubernetes Deployments on a ConfigMap/Secrets Change](#updating-kubernetes-deployments-on-a-configmapsecrets-change)\n\t- [Deploy and rollback app using kubectl](#deploy-and-rollback-app-using-kubectl)\n\t- [Get all endpoints in the cluster](#get-all-endpoints-in-the-cluster)\n\t- [Execute shell commands inside the cluster](#execute-shell-commands-inside-the-cluster)\n\t- [Access k8s API endpoint via local proxy](#access-k8s-api-endpoint-via-local-proxy)\n\t- [Port forward a local port to a port on k8s resources](#port-forward-a-local-port-to-a-port-on-k8s-resources)\n\t- [Change the service type to LoadBalancer by patching](#change-the-service-type-to-loadbalancer-by-patching)\n\t- [Delete Kubernetes Resources](#delete-kubernetes-resources)\n\t- [Using finalizers to control deletion](#using-finalizers-to-control-deletion)\n\t- [Delete a worker node in the cluster](#delete-a-worker-node-in-the-cluster)\n\t- [Evicted all pods in a node for investigation](#evicted-all-pods-in-a-node-for-investigation)\n\t- [Get Pods Logs](#get-pods-logs)\n\t- [Get Kubernetes events](#get-kubernetes-events)\n\t- [Get Kubernetes Raw Metrics - Prometheus metrics endpoint](#get-kubernetes-raw-metrics---prometheus-metrics-endpoint)\n\t- [Get Kubernetes Raw Metrics - metrics API](#get-kubernetes-raw-metrics---metrics-api)\n\n\u003c!-- /TOC --\u003e\n\n\n## Kubectl Version Manager\n\nYou can manager multiple kubectl version with [asdf](https://asdf-vm.com/) and [asdf-kubectl](https://github.com/asdf-community/asdf-kubectl)\n\nInstall asdf and asdf-kubectl\n\n```bash\n# Homebrew on macOS\n# Install asdf\nbrew install asdf\n# Add asdf.sh to your ~/.zshrc\necho -e \"\\n. $(brew --prefix asdf)/libexec/asdf.sh\" \u003e\u003e ~/.zshrc\n```\n\nInstall asdf-kubectl\n\n```bash\n# Install asdf-kubectl plugin\nasdf plugin-add kubectl https://github.com/asdf-community/asdf-kubectl.git\n# check installed plugin\nasdf plugin list\n# -\u003e kubectl\n```\n\nThen, check available version\n\n```bash\nasdf list-all kubectl\n```\n\nLet's install 1.21.14\n\n```bash\nasdf install kubectl 1.21.14\n# Check install kubectl version\nasdf list kubectl\n#-\u003e *1.21.14\n```\n\nFinally configure asdf to use kubectl version 1.21.14\n\n```bash\nasdf global kubectl 1.21.14\n```\n\nLet's see how kubectl client version looks like\n\n```bash\nkubectl version --client\n\nClient Version: version.Info{Major:\"1\", Minor:\"21\", GitVersion:\"v1.21.14\", GitCommit:\"0f77da5bd4809927e15d1658fb4aa8f13ad890a5\", GitTreeState:\"clean\", BuildDate:\"2022-06-15T14:17:29Z\", GoVersion:\"go1.16.15\", Compiler:\"gc\", Platform:\"darwin/amd64\"}\n```\n\n\nSee also\n- https://asdf-vm.com/\n- https://asdf-vm.com/guide/getting-started.html\n- https://github.com/asdf-community/asdf-kubectl\n\n## Print Cluster Info\n\n```bash\nkubectl cluster-info\n\nKubernetes control plane is running at https://A8A4143BAA8FADD6BA355D6C2A12344.gr5.ap-northeast-1.eks.amazonaws.com\nCoreDNS is running at https://A8A4143BAA8FADD6BA355D6C2A12345.gr7.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy\nMetrics-server is running at https://A8A4143BAA8FADD6BA355D6C2A12345.gr7.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy\n```\n\n## Print the supported API resources\n\n```bash\nkubectl api-resources\nkubectl api-resources -o wide\n```\n\n\u003cdetails\u003e\u003csummary\u003esample output\u003c/summary\u003e\n\u003cp\u003e\n\n```\nNAME                              SHORTNAMES         APIGROUP                       NAMESPACED   KIND\nbindings                                                                            true         Binding\ncomponentstatuses                 cs                                                false        ComponentStatus\nconfigmaps                        cm                                                true         ConfigMap\nendpoints                         ep                                                true         Endpoints\nevents                            ev                                                true         Event\nlimitranges                       limits                                            true         LimitRange\nnamespaces                        ns                                                false        Namespace\nnodes                             no                                                false        Node\npersistentvolumeclaims            pvc                                               true         PersistentVolumeClaim\npersistentvolumes                 pv                                                false        PersistentVolume\npods                              po                                                true         Pod\npodtemplates                                                                        true         PodTemplate\nreplicationcontrollers            rc                                                true         ReplicationController\nresourcequotas                    quota                                             true         ResourceQuota\nsecrets                                                                             true         Secret\nserviceaccounts                   sa                                                true         ServiceAccount\nservices                          svc                                               true         Service\nmutatingwebhookconfigurations                        admissionregistration.k8s.io   false        MutatingWebhookConfiguration\nvalidatingwebhookconfigurations                      admissionregistration.k8s.io   false        ValidatingWebhookConfiguration\ncustomresourcedefinitions         crd,crds           apiextensions.k8s.io           false        CustomResourceDefinition\napiservices                                          apiregistration.k8s.io         false        APIService\ncontrollerrevisions                                  apps                           true         ControllerRevision\ndaemonsets                        ds                 apps                           true         DaemonSet\ndeployments                       deploy             apps                           true         Deployment\nreplicasets                       rs                 apps                           true         ReplicaSet\nstatefulsets                      sts                apps                           true         StatefulSet\napplications                      app,apps           argoproj.io                    true         Application\nappprojects                       appproj,appprojs   argoproj.io                    true         AppProject\ntokenreviews                                         authentication.k8s.io          false        TokenReview\nlocalsubjectaccessreviews                            authorization.k8s.io           true         LocalSubjectAccessReview\nselfsubjectaccessreviews                             authorization.k8s.io           false        SelfSubjectAccessReview\nselfsubjectrulesreviews                              authorization.k8s.io           false        SelfSubjectRulesReview\nsubjectaccessreviews                                 authorization.k8s.io           false        SubjectAccessReview\nhorizontalpodautoscalers          hpa                autoscaling                    true         HorizontalPodAutoscaler\ncronjobs                          cj                 batch                          true         CronJob\njobs                                                 batch                          true         Job\ncertificatesigningrequests        csr                certificates.k8s.io            false        CertificateSigningRequest\nleases                                               coordination.k8s.io            true         Lease\neniconfigs                                           crd.k8s.amazonaws.com          false        ENIConfig\nevents                            ev                 events.k8s.io                  true         Event\ndaemonsets                        ds                 extensions                     true         DaemonSet\ndeployments                       deploy             extensions                     true         Deployment\ningresses                         ing                extensions                     true         Ingress\nnetworkpolicies                   netpol             extensions                     true         NetworkPolicy\npodsecuritypolicies               psp                extensions                     false        PodSecurityPolicy\nreplicasets                       rs                 extensions                     true         ReplicaSet\nnetworkpolicies                   netpol             networking.k8s.io              true         NetworkPolicy\npoddisruptionbudgets              pdb                policy                         true         PodDisruptionBudget\npodsecuritypolicies               psp                policy                         false        PodSecurityPolicy\nclusterrolebindings                                  rbac.authorization.k8s.io      false        ClusterRoleBinding\nclusterroles                                         rbac.authorization.k8s.io      false        ClusterRole\nrolebindings                                         rbac.authorization.k8s.io      true         RoleBinding\nroles                                                rbac.authorization.k8s.io      true         Role\npriorityclasses                   pc                 scheduling.k8s.io              false        PriorityClass\nstorageclasses                    sc                 storage.k8s.io                 false        StorageClass\nvolumeattachments                                    storage.k8s.io                 false        VolumeAttachment\n\n```\n\n\u003c/p\u003e\n\u003c/details\u003e\n\n## Print the available API versions\n```bash\nkubectl get apiservices\n```\n\n\u003cdetails\u003e\u003csummary\u003esample output\u003c/summary\u003e\n\u003cp\u003e\n\n```\nNAME                                   SERVICE   AVAILABLE   AGE\nv1.                                    Local     True        97d\nv1.apps                                Local     True        97d\nv1.authentication.k8s.io               Local     True        97d\nv1.authorization.k8s.io                Local     True        97d\nv1.autoscaling                         Local     True        97d\nv1.batch                               Local     True        97d\nv1.networking.k8s.io                   Local     True        97d\nv1.rbac.authorization.k8s.io           Local     True        97d\nv1.storage.k8s.io                      Local     True        97d\nv1alpha1.argoproj.io                   Local     True        4d\nv1alpha1.crd.k8s.amazonaws.com         Local     True        6d\nv1beta1.admissionregistration.k8s.io   Local     True        97d\nv1beta1.apiextensions.k8s.io           Local     True        97d\nv1beta1.apps                           Local     True        97d\nv1beta1.authentication.k8s.io          Local     True        97d\nv1beta1.authorization.k8s.io           Local     True        97d\nv1beta1.batch                          Local     True        97d\nv1beta1.certificates.k8s.io            Local     True        97d\nv1beta1.coordination.k8s.io            Local     True        97d\nv1beta1.events.k8s.io                  Local     True        97d\nv1beta1.extensions                     Local     True        97d\nv1beta1.policy                         Local     True        97d\nv1beta1.rbac.authorization.k8s.io      Local     True        97d\nv1beta1.scheduling.k8s.io              Local     True        97d\nv1beta1.storage.k8s.io                 Local     True        97d\nv1beta2.apps                           Local     True        97d\nv2beta1.autoscaling                    Local     True        97d\nv2beta2.autoscaling                    Local     True        97d\n```\n\n\u003c/p\u003e\n\u003c/details\u003e\n\n\n## Display Resource (CPU/Memory) usage of nodes/pods\n\n\nDisplay Resource (CPU/Memory) usage of nodes\n```bash\nkubectl top node\n```\n\n\u003cdetails\u003e\u003csummary\u003esample output\u003c/summary\u003e\n\u003cp\u003e\n\n```\nNAME                          CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%\naks-node-28537427-0           281m         1%     10989Mi         39%\naks-node-28537427-1           123m         0%     6795Mi          24%\naks-node-28537427-2           234m         1%     7963Mi          28%\n```\n\u003c/p\u003e\n\u003c/details\u003e\n\n\nDisplay Resource (CPU/Memory) usage of pods\n```bash\nkubectl top po -A\nkubectl top po --all-namespaces\n```\n\n\u003cdetails\u003e\u003csummary\u003esample output\u003c/summary\u003e\n\u003cp\u003e\n\n```\nNAMESPACE                   NAME                                           CPU(cores)   MEMORY(bytes)\ndd-agent                    dd-agent-2nffb                                 55m          235Mi\ndd-agent                    dd-agent-kkxsq                                 26m          208Mi\ndd-agent                    dd-agent-srnlt                                 29m          210Mi\nkube-system                 azure-cni-networkmonitor-5k7ws                 1m           22Mi\nkube-system                 azure-cni-networkmonitor-72sxx                 1m           20Mi\nkube-system                 azure-cni-networkmonitor-wxqvm                 1m           22Mi\nkube-system                 azure-ip-masq-agent-gft8h                      1m           11Mi\nkube-system                 azure-ip-masq-agent-tc8jc                      1m           10Mi\nkube-system                 azure-ip-masq-agent-v54pm                      1m           11Mi\nkube-system                 coredns-6cb457974f-kth9q                       4m           25Mi\nkube-system                 coredns-6cb457974f-m9lth                       4m           24Mi\nkube-system                 coredns-autoscaler-66cdbfb8fc-9kklp            1m           11Mi\nkube-system                 kube-proxy-b4x7q                               5m           47Mi\nkube-system                 kube-proxy-gm8df                               6m           49Mi\nkube-system                 kube-proxy-vsgbs                               5m           50Mi\nkube-system                 kubernetes-dashboard-686c6f85dc-n5xgg          1m           19Mi\nkube-system                 metrics-server-5b9794db67-5rs25                1m           18Mi\nkube-system                 tunnelfront-f586b8b5c-lrfkm                    68m          52Mi\ncustom-app-00-dev1          custom-app-deployment-5db9d949dd-fnrqd         2m           1563Mi\ncustom-app-00-dev2          custom-app-deployment-c67497d95-rtggx          2m           1577Mi\ncustom-app-00-dev3          custom-app-deployment-958c59798-6vvwj          2m           1581Mi\ncustom-app-00-dev4          custom-app-deployment-5c7797bc85-mlppp         2m           1585Mi\ncustom-app-00-dev5          custom-app-deployment-bfd4596dd-d5q76          2m           1603Mi\ncustom-app-00-dev6          custom-app-deployment-6f9f56ffc6-tpngg         2m           1600Mi\ncustom-app-00-dev7          custom-app-deployment-7c6896ff98-4pmln         2m           1573Mi\n```\n\n\u003c/p\u003e\n\u003c/details\u003e\n\n\n## Updating Kubernetes Deployments on a ConfigMap/Secrets Change\n\n`kubectl v1.15+` provides a rollout restart command that allows you to restart Pods in a Deployment and allow them pick up changes to a referenced ConfigMap, Secret or similar.\n\n```bash\nkubectl rollout restart deploy/\u003cdeployment-name\u003e\nkubectl rollout restart deploy \u003cdeployment-name\u003e\n```\n\n\n## Deploy and rollback app using kubectl\n```bash\nkubectl run nginx --image nginx\nkubectl run nginx --image=nginx --port=80 --restart=Never\nkubectl expose deployment nginx --external-ip=\"10.0.47.10\" --port=8000 --target-port=80\nkubectl scale --replicas=3 deployment nginx\nkubectl set image deployment nginx nginx=nginx:1.8\nkubectl rollout status deploy nginx\nkubectl set image deployment nginx nginx=nginx:1.9\nkubectl rollout status deploy nginx\nkubectl rollout history deploy nginkubectl get --raw /metricsx\n```\n\nLet's check rollout history\n```bash\nkubectl rollout history deploy nginx\n\ndeployment.extensions/nginx\nREVISION  CHANGE-CAUSE\n1         \u003cnone\u003e\n2         \u003cnone\u003e\n3         \u003cnone\u003e\n```\n\nYou can undo the rollout of nginx deploy like this:\n```bash\nkubectl rollout undo deploy nginx\nkubectl describe deploy nginx |grep Image\n    Image:        nginx:1.8\n```\n\nMore specifically you can undo the rollout with `--to-revision` option\n```bash\nkubectl rollout undo deploy nginx --to-revision=1\nkubectl describe deploy nginx |grep Image\n    Image:        nginx\n```\n\n## Get all endpoints in the cluster\n```bash\nkubectl get endpoints [-A|-n \u003cnamespace\u003e]\nkubectl get ep [-A|-n \u003cnamespace\u003e]\n```\n\n## Execute shell commands inside the cluster\nYou can exec shell commands in a new creating Pod\n```bash\nkubectl run --generator=run-pod/v1 -it busybox --image=busybox --rm --restart=Never -- sh\n```\nIf you want to run `curl` from a pod (The busybox image above doesn't contain curl)\n```bash\nkubectl run --generator=run-pod/v1 -it --rm busybox --image=radial/busyboxplus:curl --restart=Never -- sh\n```\nIf you want to debug databases connections from a pod\n```bash\nkubectl run mysql -it --rm --image=mysql -- mysql -h \u003chost/ip\u003e -P \u003cport\u003e -u \u003cuser\u003e -p\u003cpassword\u003e\n```\n\nYou can also exec shell commands in existing Pod\n```bash\nkubectl exec -n \u003cnamespace\u003e -it \u003cpod-name\u003e -- /bin/sh\nkubectl exec -n \u003cnamespace\u003e -it \u003cpod-name\u003e -c \u003ccontainer-name\u003e -- /bin/sh\n```\n\nYou can change `kind` of the resource you're creating with option in `kubectl run`\n\n| Kind      | Option |\n| ----------- | ----------- |\n| deployment  | node       |\n| pod         | `--restart=Never` |\n| job         | `--restart=OnFailure` |\n| cronjob     | `--schedule='cron format(0/5 * * * ?'` |\n\n\n\u003e NOTE: kubernetes 1.18+\n\u003e kubectl run has removed previously deprecated flags not related to generator and pod creation. The kubectl run command now only creates pods. To create objects other than Pods, see the specific kubectl create subcommand.\n\u003e  So run `kubectl create deployment` in order to create deployment like this:\n\u003e```\n\u003e kubectl create deployment nginx --image=nginx\n\u003ekubectl create deployment nginx --image=nginx --dry-run -o yaml\n\u003e```\n\u003e see also [kubectl cheat sheet](https://kubernetes.io/docs/reference/kubectl/cheatsheet/)\n\n\n## Access k8s API endpoint via local proxy\n\nYou can access k8s API via local proxy\n```bash\n# Making local proxy\nkubectl proxy\nStarting to serve on 127.0.0.1:8001\n```\n\nAccess k8s API resources via local proxy\n\n```bash\n# Get pods list in namespace foo\ncurl http://localhost:8001/api/v1/namespaces/foo/pods\n# Access pod's endpoint\ncurl http://localhost:8001/api/v1/namespaces/foo/pods/\u003cpod-name\u003e/proxy/\u003cpodendpoint\u003e\ncurl http://localhost:8001/api/v1/namespaces/foo/pods/foo-86c498d84c-xbkn9/proxy/healthcheck\n```\n\n## Port forward a local port to a port on k8s resources\n\n```bash\n# kubectl port-forward -n \u003cnamespace\u003e \u003cresource\u003e LocalPort:TargetPort\nkubectl port-forward -n \u003cnamespace\u003e redis-master-765d459796-258hz 7000:6379\nkubectl port-forward -n \u003cnamespace\u003e pods/redis-master-765d459796-258hz 7000:6379\nkubectl port-forward -n \u003cnamespace\u003e deployment/redis-master 7000:6379\nkubectl port-forward -n \u003cnamespace\u003e rs/redis-master 7000:6379\nkubectl port-forward -n \u003cnamespace\u003e svc/redis-master 7000:6379\n```\nSee also [this](https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/) for more detail\n\n## Change the service type to LoadBalancer by patching \n\n```bash\n# kubectl patch svc SERVICE_NAME -p '{\"spec\": {\"type\": \"LoadBalancer\"}}'\nkubectl patch svc argocd-server -n argocd -p '{\"spec\": {\"type\": \"LoadBalancer\"}}'\n```\n\n## Delete Kubernetes Resources\n\n```bash\n# Delete resources that has name=\u003clabel\u003e label\nkubectl delete svc,cm,secrets,deploy -l name=\u003clabel\u003e -n \u003cnamespace\u003e\n# Delete all certain resources from a certain namespace\n# --all is used to delete every object of that resource type instead of specifying it using its name or label.\nkubectl delete svc,cm,secrets,deploy --all -n \u003cnamespace\u003e\n# Delete all resources (except crd) from a certain namespace\nkubectl delete all --all -n \u003cnamespace\u003e\n\n# Delete pods in namespace \u003cnamespace\u003e\nfor pod in $(kubectl get po -n \u003cnamespace\u003e --no-headers=true | cut -d ' ' -f 1); do\n  kubectl delete pod $pod -n \u003cnamespace\u003e\ndone\n\n# Delete pods with --grace-period=0 and --force option\n#    Add --grace-period=0 in order to delete pod as quickly as possible\n#    Add --force in case that pod stay terminating state and cannnot be deleted\nkubectl delete pod \u003cpod\u003e -n \u003cnamespace\u003e --grace-period=0 --force\n```\n\n\n## Using finalizers to control deletion\n\nYou can delete the k8s object by patching command to remove finalizers. Simply patch it on the command line to remove the finalizers, so the object will be deleted\n\nFor example, you want to delete `configmap/mymap`\n```bash\nkubectl patch configmap/mymap \\\n    --type json \\\n    --patch='[ { \"op\": \"remove\", \"path\": \"/metadata/finalizers\" } ]'\n```\n\n![](https://d33wubrfki0l68.cloudfront.net/2921aff96caba07229c862903fea89cbab9ad5a6/8e8fd/images/blog/2021-05-14-using-finalizers-to-control-deletion/state-diagram-finalize.png)\nref: https://kubernetes.io/blog/2021/05/14/using-finalizers-to-control-deletion/\n\nRead [Using Finalizers to Control Deletion](https://kubernetes.io/blog/2021/05/14/using-finalizers-to-control-deletion/) to understand how the object will be deleted by using finalizers\n## Delete a worker node in the cluster\n\nA point is to `cordon` a node at first, then to evict pods in the node with `drain`.\n```bash\nkubectl get nodes  # to get nodename to delete\nkubectl cordon \u003cnode-name\u003e\nkubectl drain --ignore-daemonsets \u003cnode-name\u003e\nkubectl delete node \u003cnode-name\u003e\n```\n\u003e [NOTE] Add `--ignore-daemonsets` if you want to ignore DaemonSet for eviction\n\n## Evicted all pods in a node for investigation\n\nA point is to `cordon` a node at first, then to evict pods in the node with `drain`, to `uncordon` after the investigation\n```bash\nkubectl get nodes  # to get nodename to delete\nkubectl cordon \u003cnode-name\u003e\nkubectl drain --ignore-daemonsets \u003cnode-name\u003e\n```\nNow that you evicted all pods in the node, you can do investigation in the node. After the investigation, you can `uncordon` the node with the following command.\n```\nkubectl uncordon \u003cnode-name\u003e\n```\n\u003e [NOTE] Add `--ignore-daemonsets` if you want to ignore DaemonSet for eviction\n\n## Get Pods Logs\nYou can get Kubernetes pods logs by running the following commands\n```bash\nkubectl logs \u003cpod-name\u003e -n \u003cnamespace\u003e\n\n# Add -f option if the logs should be streamed\nkubectl logs \u003cpod-name\u003e -n \u003cnamespace\u003e -f \nkubectl logs \u003cpod-name\u003e -n \u003cnamespace\u003e -f --tail 0\n```\n\nOr you can use 3rd party OSS tools. For example, you can get `stern` that allows you to aggregate logs of all pods and to filter them by using regular expression like `stern \u003cexpression\u003e`\n```bash\nstern \u003ckeyword\u003e -n \u003cnamespace\u003e\n```\n\n## Get Kubernetes events \n\nYou can Pods' events with the following commands which will show events at the end of the output for the pod (Relatively recent events will appear).\n\n```bash\nkubectl describe pod \u003cpodname\u003e -n \u003cnamespace\u003e\n```\n\n```bash\n# Get recent events in a specific namespace\nkubectl get events -n \u003cnamespaces\u003e\nkubectl get events -n kube-system\n\n# Get recent events for all resources in the system\n# Get events from all namespaces with either --all-namespaces or -A \nkubectl get event --all-namespaces\nkubectl get event -A\nkubectl get event -A -o wide\n\n# No Pod events\nkubectl get events --field-selector involvedObject.kind!=Pod\n# Events from a specific Pod\nkubectl get events --field-selector involvedObject.kind=Pod,involvedObject.name=\u003cpodname\u003e\n# Events from a specific Node\nkubectl get events --field-selector involvedObject.kind=Node,involvedObject.name=\u003cnodename\u003e\n\n# Warning events (from all namespaces)\nkubectl get events --field-selector type=Warning -A\n# NOT normal events (from all namespaces)\nkubectl get events --field-selector type!=Normal -A\n```\n\n## Get Kubernetes Raw Metrics - Prometheus metrics endpoint\n\nMany Kubernetes components exposes their metrics via the `/metrics` endpoint, including API server, etcd and many other add-ons. These metrics are in [Prometheus format](https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md), and can be defined and exposed using [Prometheus client libs](https://prometheus.io/docs/instrumenting/clientlibs/)\n\n```\nkubectl get --raw /metrics\n```\n\n\u003cdetails\u003e\u003csummary\u003esample output\u003c/summary\u003e\n\u003cp\u003e\n\n```\nAPIServiceOpenAPIAggregationControllerQueue1_adds 282663\nAPIServiceOpenAPIAggregationControllerQueue1_depth 0\nAPIServiceOpenAPIAggregationControllerQueue1_longest_running_processor_microseconds 0\nAPIServiceOpenAPIAggregationControllerQueue1_queue_latency{quantile=\"0.5\"} 63\nAPIServiceOpenAPIAggregationControllerQueue1_queue_latency{quantile=\"0.9\"} 105\nAPIServiceOpenAPIAggregationControllerQueue1_queue_latency{quantile=\"0.99\"} 126\nAPIServiceOpenAPIAggregationControllerQueue1_queue_latency_sum 1.4331448e+07\nAPIServiceOpenAPIAggregationControllerQueue1_queue_latency_count 282663\nAPIServiceOpenAPIAggregationControllerQueue1_retries 282861\nAPIServiceOpenAPIAggregationControllerQueue1_unfinished_work_seconds 0\nAPIServiceOpenAPIAggregationControllerQueue1_work_duration{quantile=\"0.5\"} 59\nAPIServiceOpenAPIAggregationControllerQueue1_work_duration{quantile=\"0.9\"} 98\nAPIServiceOpenAPIAggregationControllerQueue1_work_duration{quantile=\"0.99\"} 2003\nAPIServiceOpenAPIAggregationControllerQueue1_work_duration_sum 2.1373689e+07\nAPIServiceOpenAPIAggregationControllerQueue1_work_duration_count 282663\n...\nworkqueue_work_duration_seconds_bucket{name=\"non_structural_schema_condition_controller\",le=\"1e-08\"} 0\nworkqueue_work_duration_seconds_bucket{name=\"non_structural_schema_condition_controller\",le=\"1e-07\"} 0\nworkqueue_work_duration_seconds_bucket{name=\"non_structural_schema_condition_controller\",le=\"1e-06\"} 0\nworkqueue_work_duration_seconds_bucket{name=\"non_structural_schema_condition_controller\",le=\"9.999999999999999e-06\"} 459\nworkqueue_work_duration_seconds_bucket{name=\"non_structural_schema_condition_controller\",le=\"9.999999999999999e-05\"} 473\nworkqueue_work_duration_seconds_bucket{name=\"non_structural_schema_condition_controller\",le=\"0.001\"} 924\nworkqueue_work_duration_seconds_bucket{name=\"non_structural_schema_condition_controller\",le=\"0.01\"} 927\nworkqueue_work_duration_seconds_bucket{name=\"non_structural_schema_condition_controller\",le=\"0.1\"} 928\nworkqueue_work_duration_seconds_bucket{name=\"non_structural_schema_condition_controller\",le=\"1\"} 928\nworkqueue_work_duration_seconds_bucket{name=\"non_structural_schema_condition_controller\",le=\"10\"} 928\nworkqueue_work_duration_seconds_bucket{name=\"non_structural_schema_condition_controller\",le=\"+Inf\"} 928\nworkqueue_work_duration_seconds_sum{name=\"non_structural_schema_condition_controller\"} 0.09712353499999991\n...\n```\n\u003c/p\u003e\n\u003c/details\u003e\n\n\n## Get Kubernetes Raw Metrics - metrics API\n\nYou can access [Metrics API](https://github.com/kubernetes/metrics) via kubectl proxy like this:\n\n```\nkubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes\nkubectl get --raw /apis/metrics.k8s.io/v1beta1/pods\nkubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes/\u003cnode-name\u003e\nkubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces/\u003cnamespace-name\u003e/pods/\u003cpod-name\u003e\n```\n(ref: [feiskyer/kubernetes-handbook](https://github.com/feiskyer/kubernetes-handbook/blob/master/en/addons/metrics.md#metrics-api))\n\nOutputs are unformated JSON. It's good to use `jq` to parse it.\n\n\u003cdetails\u003e\u003csummary\u003esample output -  /apis/metrics.k8s.io/v1beta1/nodes\u003c/summary\u003e\n\u003cp\u003e\n\n```json\nkubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes | jq\n\n{\n  \"kind\": \"NodeMetricsList\",\n  \"apiVersion\": \"metrics.k8s.io/v1beta1\",\n  \"metadata\": {\n    \"selfLink\": \"/apis/metrics.k8s.io/v1beta1/nodes\"\n  },\n  \"items\": [\n    {\n      \"metadata\": {\n        \"name\": \"ip-xxxxxxxxxx.ap-northeast-1.compute.internal\",\n        \"selfLink\": \"/apis/metrics.k8s.io/v1beta1/nodes/ip-xxxxxxxxxx.ap-northeast-1.compute.internal\",\n        \"creationTimestamp\": \"2020-05-24T01:29:05Z\"\n      },\n      \"timestamp\": \"2020-05-24T01:28:58Z\",\n      \"window\": \"30s\",\n      \"usage\": {\n        \"cpu\": \"105698348n\",\n        \"memory\": \"819184Ki\"\n      }\n    },\n    {\n      \"metadata\": {\n        \"name\": \"ip-yyyyyyyyyy.ap-northeast-1.compute.internal\",\n        \"selfLink\": \"/apis/metrics.k8s.io/v1beta1/nodes/ip-yyyyyyyyyy.ap-northeast-1.compute.internal\",\n        \"creationTimestamp\": \"2020-05-24T01:29:05Z\"\n      },\n      \"timestamp\": \"2020-05-24T01:29:01Z\",\n      \"window\": \"30s\",\n      \"usage\": {\n        \"cpu\": \"71606060n\",\n        \"memory\": \"678944Ki\"\n      }\n    }\n  ]\n}\n\n```\n\n\u003c/p\u003e\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\u003csummary\u003esample output - /apis/metrics.k8s.io/v1beta1/namespaces/NAMESPACE/pods/PODNAME \u003c/summary\u003e\n\u003cp\u003e\n\n```json\nkubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/cluster-autoscaler-7d8d69668c-5rcmt | jq\n\n{\n  \"kind\": \"PodMetrics\",\n  \"apiVersion\": \"metrics.k8s.io/v1beta1\",\n  \"metadata\": {\n    \"name\": \"cluster-autoscaler-7d8d69668c-5rcmt\",\n    \"namespace\": \"kube-system\",\n    \"selfLink\": \"/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/cluster-autoscaler-7d8d69668c-5rcmt\",\n    \"creationTimestamp\": \"2020-05-24T01:33:17Z\"\n  },\n  \"timestamp\": \"2020-05-24T01:32:58Z\",\n  \"window\": \"30s\",\n  \"containers\": [\n    {\n      \"name\": \"cluster-autoscaler\",\n      \"usage\": {\n        \"cpu\": \"1030416n\",\n        \"memory\": \"27784Ki\"\n      }\n    }\n  ]\n}\n\n```\n\u003c/p\u003e\n\u003c/details\u003e\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyokawasa%2Fkubectl-tips","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyokawasa%2Fkubectl-tips","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyokawasa%2Fkubectl-tips/lists"}