{"id":22874402,"url":"https://github.com/thesurlydev/kubernetes-bare-metal","last_synced_at":"2025-03-31T12:43:13.873Z","repository":{"id":105886004,"uuid":"272076563","full_name":"thesurlydev/kubernetes-bare-metal","owner":"thesurlydev","description":"Personal notes for setting up a Kubernetes cluster on bare metal.","archived":false,"fork":false,"pushed_at":"2020-07-05T15:58:43.000Z","size":99,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-06T18:52:28.501Z","etag":null,"topics":["k8s","k8s-cluster","k8s-cluster-ubuntu","kubernetes","kubernetes-install"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thesurlydev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2020-06-13T19:55:32.000Z","updated_at":"2023-10-25T11:03:06.000Z","dependencies_parsed_at":"2023-04-13T04:24:10.198Z","dependency_job_id":null,"html_url":"https://github.com/thesurlydev/kubernetes-bare-metal","commit_stats":null,"previous_names":["thesurlydev/kubernetes-bare-metal"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thesurlydev%2Fkubernetes-bare-metal","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thesurlydev%2Fkubernetes-bare-metal/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thesurlydev%2Fkubernetes-bare-metal/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thesurlydev%2Fkubernetes-bare-metal/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thesurlydev","download_url":"https://codeload.github.com/thesurlydev/kubernetes-bare-metal/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246472582,"owners_count":20783225,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["k8s","k8s-cluster","k8s-cluster-ubuntu","kubernetes","kubernetes-install"],"created_at":"2024-12-13T14:35:56.301Z","updated_at":"2025-03-31T12:43:13.849Z","avatar_url":"https://github.com/thesurlydev.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# kubernetes-bare-metal\n\nKubernetes setup for Ubuntu on bare metal.\n\n\nThe intent is to outline the steps necessary to install a multi-node Kubernetes 1.18 cluster on Ubuntu 18.04.\n\n\u003c!--ts--\u003e   \n  * [Components](#components)  \n  * [Gotchas](#gotchas)\n  * [Docker](#docker)\n      * [Installation](#installation)\n      * [Post-install configuration](#post-install-configuration)\n  * [kubectl, kubeadm, kubelet](#kubectl-kubeadm-kubelet)\n  * [kubelet service config](#kubelet-service-config)\n  * [Disable Swap](#disable-swap)\n  * [Install via kubeadm](#install-via-kubeadm)\n  * [Troubleshooting kubelet](#troubleshooting-kubelet)\n  * [Post Install](#post-install)\n  * [Weave](#weave)\n  * [Join Additional Nodes (workers)](#join-additional-nodes-workers)\n  * [Dashboard](#dashboard)\n      * [Create user](#create-user)\n      * [Install Dashboard](#install-dashboard)\n      * [Start Dashboard Proxy](#start-dashboard-proxy)\n      * [Login](#login)\n  * [Pulling from Private Docker Registries](#pulling-from-private-docker-registries)\n  * [Updating a deployment](#updating-a-deployment)\n  * [Get a list of all containers running in all pods in all namespaces](#get-a-list-of-all-containers-running-in-all-pods-in-all-namespaces)\n  * [Port forwarding](#port-forwarding)  \n  * [Multiple Clusters](#multiple-clusters)\n  * [Troubleshooting](#troubleshooting)  \n\n\u003c!-- Added by: shane, at: Mon 30 Mar 2020 09:21:29 AM PDT --\u003e\n\n\u003c!--te--\u003e\n\n\n## Components\n\n* Ubuntu 18.04\n* Kubernetes 1.18 with `cgroups-driver=systemd`\n* Docker 19.03.11\n* Docker Hub Registry\n* Weave 2.6.2\n* Web UI (kubernetes dashboard)\n\n\n## Gotchas\n\n* Versions\n* Ensuring cgroups-driver for Docker and Kubelet match\n* Starting over:\n   * kubeadm reset\n   * sudo rm -rf /etc/kubernetes\n* Making changes to kubelet:\n   * systemctl daemon-reload\n   * systemctl restart kubelet \n\n\n## Docker\n\n### Installation\n\nFirst, update existing list of packages:\n```\nsudo apt update\n```\n\nNext, install prerequisites:\n\n```\nsudo apt install apt-transport-https ca-certificates curl software-properties-common\n```\nAdd the GPG key for the official Docker repository:\n```\ncurl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -`\n```\nAdd the Docker repository to APT sources:\n```\nsudo add-apt-repository \"deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable\"\n```\nUpdate package database with the Docker packages:\n```\nsudo apt update\n```\nMake sure you are about to install from the Docker repo instead of the default Ubuntu repo:\n```\napt-cache policy docker-ce\n```\nNow, install:\n```\nsudo apt install docker-ce\n```\n\n\n\nConfirm install:\n```\nsudo systemctl status docker\n```\nshould return somethink like:\n```\ndocker.service - Docker Application Container Engine\n   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)\n   Active: active (running) since Fri 2020-06-19 15:27:09 UTC; 4s ago\n     Docs: https://docs.docker.com\n Main PID: 17823 (dockerd)\n    Tasks: 22\n   CGroup: /system.slice/docker.service\n           └─17823 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock\n```           \n\n\n\n```\ndocker version\n```\n\n```bash\nClient: Docker Engine - Community\n Version:           19.03.11\n API version:       1.40\n Go version:        go1.13.10\n Git commit:        42e35e61f3\n Built:             Mon Jun  1 09:12:22 2020\n OS/Arch:           linux/amd64\n Experimental:      false\n\nServer: Docker Engine - Community\n Engine:\n  Version:          19.03.11\n  API version:      1.40 (minimum version 1.12)\n  Go version:       go1.13.10\n  Git commit:       42e35e61f3\n  Built:            Mon Jun  1 09:10:54 2020\n  OS/Arch:          linux/amd64\n  Experimental:     false\n containerd:\n  Version:          1.2.13\n  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429\n runc:\n  Version:          1.0.0-rc10\n  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd\n docker-init:\n  Version:          0.18.0\n  GitCommit:        fec3683\n```\n\n### Post-install Configuration\n\nUpdate cgroupdriver in `/etc/docker/daemon.json` to use systemd:\n```\n{\n  \"exec-opts\": [\n    \"native.cgroupdriver=systemd\"\n  ],\n  \"log-driver\": \"json-file\",\n  \"log-opts\": {\n    \"max-size\": \"100m\"\n  },\n  \"storage-driver\": \"overlay2\"\n}\n```\n\nRestart Docker daemon via:\n```\nsystemctl restart docker\n```\n\n\nVerify Docker cgroup driver via:\n```\ndocker info -f {{.CgroupDriver}}\n```\n\n## kubectl, kubeadm, kubelet\n\n```bash\nsudo apt-get update \u0026\u0026 sudo apt-get install -y apt-transport-https curl\ncurl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -\ncat \u003c\u003cEOF | sudo tee /etc/apt/sources.list.d/kubernetes.list\ndeb https://apt.kubernetes.io/ kubernetes-xenial main\nEOF\nsudo apt-get update\nsudo apt-get install -y kubelet kubeadm kubectl\nsudo apt-mark hold kubelet kubeadm kubectl\n```\n\n\n```kubectl version -o json```\n\n```json\n{\n  \"clientVersion\": {\n    \"major\": \"1\",\n    \"minor\": \"18\",\n    \"gitVersion\": \"v1.18.0\",\n    \"gitCommit\": \"9e991415386e4cf155a24b1da15becaa390438d8\",\n    \"gitTreeState\": \"clean\",\n    \"buildDate\": \"2020-03-25T14:58:59Z\",\n    \"goVersion\": \"go1.13.8\",\n    \"compiler\": \"gc\",\n    \"platform\": \"linux/amd64\"\n  },\n  \"serverVersion\": {\n    \"major\": \"1\",\n    \"minor\": \"18\",\n    \"gitVersion\": \"v1.18.0\",\n    \"gitCommit\": \"9e991415386e4cf155a24b1da15becaa390438d8\",\n    \"gitTreeState\": \"clean\",\n    \"buildDate\": \"2020-03-25T14:50:46Z\",\n    \"goVersion\": \"go1.13.8\",\n    \"compiler\": \"gc\",\n    \"platform\": \"linux/amd64\"\n  }\n}\n````\n\nAdd gpg key:\n```bash\ncurl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -\n```\n\n## Configuration\n\n\n### Disable Swap\n\n```bash\nsudo swapoff -a\n```\n\nTo permanently disable swap (to survive reboot) then remove any **swap** entry from `/etc/fstab`.\n\n\n### Configure kubeadm and kubelet\n\n```\nsudo apt update\nsudo apt install -y apt-transport-https ca-certificates curl software-properties-common gnupg2\n```\n\nCreate dir if it doesn't exist:\n```bash\nsudo mkdir -p /var/lib/kubelet\n```\n\nAdd the following to `/var/lib/kubelet/kubeadm-flags.env`:\n```\nKUBELET_KUBEADM_ARGS=\"--cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.2 --resolv-conf=/run/systemd/resolve/resolv.conf\"\n```\n\nInstall the cluster via:\n```sudo kubeadm init -v=6```\n\n\n\n## Troubleshooting kubelet\n\nThe following are your friends when troubleshooting kubelet startup issues:\n\nGet the status of the kubelet service:\n```systemctl status kubelet```\n\nLook at the logs of the kubelet service (scroll to bottom):\n```journalctl -xeu kubelet```\n\n\n## Post Install\n\n```\nsudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config\nsudo chown $(id -u):$(id -g) $HOME/.kube/config\n```\n\n\n## Weave\n\nInstall via:\n```\nkubectl apply -f \"https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\\n')\"\n```\n\nTo disable NPC:\n\n```\nkubectl apply -f \"https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\\n')\u0026disable-npc=true\"\n```\n\n\n## Join Additional Nodes (workers)\n\nAfter running `kubeadm init` on the master node, there will be output including something like the below snipper:\n\n```\nThen you can join any number of worker nodes by running the following on each as root:\n\nkubeadm join 192.168.7.40:6443 --token th6npu.fmuxapxjzsx8k7zt \\\n    --discovery-token-ca-cert-hash sha256:e947ea9db4ef3182ac870618d300c16cac09d6de45c28ffab73b74099f673870\n```\n\nIf you need to add nodes later on and need the join command, use the following command to print it from the master node:\n```\nkubeadm token create --print-join-command\n```\n\n\n\n## Dashboard\n\n### Create user\n\nTaken from: https://github.com/kubernetes/dashboard/blob/master/docs/user/access-control/creating-sample-user.md\n\nCreate a file called `dashboard-adminuser.yaml` with the following content:\n\n```yaml\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n  name: admin-user\n  namespace: kubernetes-dashboard  \n---\napiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRoleBinding\nmetadata:\n  name: admin-user\nroleRef:  \n  kind: ClusterRole\n  name: cluster-admin\n  apiGroup: rbac.authorization.k8s.io\nsubjects:\n- kind: ServiceAccount\n  name: admin-user\n  namespace: kubernetes-dashboard\n- kind: ServiceAccount\n  name: admin-user\n  namespace: default\n```\n\nUse the following to apply/create:\n\n```\nkubectl apply -f dashboard-adminuser.yaml\n```\n\n\n\n### Install Dashboard\n\n```kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-rc6/aio/deploy/recommended.yaml```\n\n### Start Dashboard Proxy\n\n```kubectl proxy```\n\n### Login\n\nFirst, get the bearer token via:\n\n```bash\nkubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}')\n```\n\nSecond, visit http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/ and login with the bearer token.\n\n\n## Pulling from Private Docker Registries\n\nIn order to all Kubernetes to pull from private Docker registries you need to create a secret with the registry credentials then add the secret to the `default` service account.\n\n```bash\nkubectl create secret docker-registry dockerhubregistrykey --docker-server=https://index.docker.io/v1/ --docker-username=digitalsanctum --docker-password=YOUR_PASSWORD --docker-email=shane@digitalsanctum.com\n```\n```bash\nkubectl edit serviceaccounts default\n```\n\nAdd the following to the file after `secrets`:\n\n```yaml\nimagePullSecrets:\n- name: dockerhubregistrykey\n```\nSave the file as `sa.yaml` then update the service account via:\n\n```bash\nkubectl replace serviceaccount default -f ./sa.yaml\n```\n\nFinally, update deployment.yaml with `imagePullSecrets`:\n\n```\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: kgen\n  namespace: default\n  labels:\n    app: kgen\nspec:\n  replicas: 1\n  selector:\n    matchLabels:\n      app: kgen\n  strategy:\n    type: RollingUpdate\n    rollingUpdate:\n      maxUnavailable: 25%\n      maxSurge: 25%\n  revisionHistoryLimit: 10\n  progressDeadlineSeconds: 600\n  template:\n    metadata:\n      labels:\n        app: kgen\n        cluster-name: kubernetes\n    spec:\n      serviceAccountName: kgen-sa\n      imagePullSecrets:\n        - name: dockerhubregistrykey\n      containers:\n        - name: kgen\n          image: digitalsanctum/kgen-service:0.1.0\n          imagePullPolicy: IfNotPresent\n          ports:\n            - name: http\n              containerPort: 8080\n              protocol: TCP\n          env:\n            - name: SPRING_PROFILES_ACTIVE\n              value: local\n            - name: POD_NAME\n              value: metadata.name\n          readinessProbe:\n            httpGet:\n              path: /actuator/health\n              port: 8080\n            initialDelaySeconds: 5\n            periodSeconds: 60\n            timeoutSeconds: 2\n            failureThreshold: 1\n            successThreshold: 1\n          livenessProbe:\n            httpGet:\n              path: /actuator/health\n              port: 8080\n            initialDelaySeconds: 15\n            periodSeconds: 70\n            timeoutSeconds: 5\n            failureThreshold: 2\n            successThreshold: 1\n          resources:\n            limits:\n              cpu: 500m\n              memory: 1Gi\n            requests:\n              cpu: 250m\n              memory: 500Mi\n```\n\n## Updating a deployment\n\nFirst, grab the deployment via:\n```bash\nkubectl get deployments kgen-service -o yaml \u003e kgen-service-deployment.yaml\n```\nSecond, edit the file as necessary.\n\nLastly, update the deployment with your changes:\n```bash\nkubectl replace deployments kgen-service -f ./kgen-service-deployment.yaml\n```\n-OR-\n\nJust edit the deployment. When it's saved the deployment will be replaced:\n\n```bash\nkubectl edit deployments kgen-service\n```\n\n## Get a list of all containers running in all pods in all namespaces\n\n```bash\nkubectl get pods --all-namespaces -o jsonpath={..image} | tr -s '[[:space:]]' '\\n' | sort -u\n```\n\n## Port forwarding\n\nForward one or more local ports to a pod. This command requires the node to have 'socat' installed.\n\nAs an example to port forward to an application listening on port `8080`:\n\n```bash\nkubectl port-forward deployment/kgen-service 8080\n```\n\n## Multiple Clusters\n\nTo view the available contexts/clusters:\n\n```\nkubectl config get-contexts\n```\n\nTo switch from one cluster to another:\n\n```\nkubectl config use-context kubernetes-admin@kubernetes\n```\n\n## Custom Namespaces\n\n```\nkubectl get pods -n \u003ccustom-namespace\u003e\n```\n\n## Scaling your deployments\n\n```\nkubectl scale deployment \u003capp-name\u003e --replicas \u003creplica-count\u003e\n```\n\n## Zero downtime deployments\n\nThe default way to update a running application in Kubernetes is to deploy a new image tag to your Docker registry and then deploy it using:\n\n```\nkubectl set image deployment/\u003capp-name\u003e-app \u003capp-name\u003e=\u003cnew-image\u003e\n```\n\nUsing livenessProbes and readinessProbe allows you to tell Kubernetes about the state of your applications, in order to ensure availability of your services. You will need a minimum of 2 replicas for every application if you want to have zero downtime deployment. This is because the rolling upgrade strategy first stops a running replica in order to place a new one. Running only one replica will cause a short downtime during upgrades.\n\n## Troubleshooting\n\n\n\n### Deploy Verification\n\nWait for deploy to complete:\n```\nkubectl rollout status -n default deployments/springboot-microservice --watch\n```\n-or-\n```\n`curl --silent --fail --retry 60 --retry-delay 5 --retry-connrefused --insecure --output /dev/null $ENDPOINT/health`\n```\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthesurlydev%2Fkubernetes-bare-metal","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthesurlydev%2Fkubernetes-bare-metal","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthesurlydev%2Fkubernetes-bare-metal/lists"}