{"id":30047630,"url":"https://github.com/tmax-cloud/install-k8s","last_synced_at":"2025-08-07T09:57:09.733Z","repository":{"id":55481733,"uuid":"315201266","full_name":"tmax-cloud/install-k8s","owner":"tmax-cloud","description":null,"archived":false,"fork":false,"pushed_at":"2023-06-07T04:50:29.000Z","size":674,"stargazers_count":2,"open_issues_count":0,"forks_count":10,"subscribers_count":5,"default_branch":"5.0","last_synced_at":"2025-04-19T10:10:08.671Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tmax-cloud.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-11-23T04:36:42.000Z","updated_at":"2024-11-29T13:50:24.000Z","dependencies_parsed_at":"2023-01-30T02:55:23.199Z","dependency_job_id":null,"html_url":"https://github.com/tmax-cloud/install-k8s","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/tmax-cloud/install-k8s","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tmax-cloud%2Finstall-k8s","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tmax-cloud%2Finstall-k8s/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tmax-cloud%2Finstall-k8s/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tmax-cloud%2Finstall-k8s/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tmax-cloud","download_url":"https://codeload.github.com/tmax-cloud/install-k8s/tar.gz/refs/heads/5.0","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tmax-cloud%2Finstall-k8s/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269236777,"owners_count":24383241,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-07T02:00:09.698Z","response_time":73,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-08-07T09:56:52.429Z","updated_at":"2025-08-07T09:57:09.719Z","avatar_url":"https://github.com/tmax-cloud.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"## 구성 요소 및 버전\n* cri-o (v1.19.1) or docker-ce(v20.10.5)\n* kubeadm, kubelet, kubectl (v1.19.4)\n* k8s.gcr.io/kube-apiserver:v1.19.4\n* k8s.gcr.io/kube-proxy:v1.19.4\n* k8s.gcr.io/kube-scheduler:v1.19.4\n* k8s.gcr.io/kube-controller-manager:v1.19.4\n* k8s.gcr.io/etcd:3.4.13-0\n* k8s.gcr.io/pause:3.2\n* k8s.gcr.io/coredns:1.7.0\n\n## Prerequisites\n* 클러스터 구성전 master, worker node 최소 스팩\n  * master node (controll plane node) - CPU : 2Core 이상\n  * master/worker node - RAM : 2GiB 이상\n\n## 폐쇄망 구축 가이드 \n1. **폐쇄망에서 설치하는 경우** 아래 가이드를 참고 하여 image registry를 먼저 구축한다.\n    * https://github.com/tmax-cloud/install-registry/tree/5.0  \n2. 사용하는 image repository에 k8s 설치 시 필요한 이미지를 push한다. \n    * 작업 디렉토리 생성 및 환경 설정\n    ```bash\n    $ mkdir -p ~/k8s-install\n    $ cd ~/k8s-install\n    $ export REGISTRY={ImageRegistryIP:Port}\n      ex) $ export REGISTRY=172.22.5.2:5000\n    ```\n    * 외부 네트워크 통신이 가능한 환경에서 필요한 이미지를 다운받는다.\n    ```bash\n    $ sudo docker pull k8s.gcr.io/kube-proxy:v1.19.4\n    $ sudo docker pull k8s.gcr.io/kube-apiserver:v1.19.4\n    $ sudo docker pull k8s.gcr.io/kube-controller-manager:v1.19.4\n    $ sudo docker pull k8s.gcr.io/kube-scheduler:v1.19.4\n    $ sudo docker pull k8s.gcr.io/etcd:3.4.13-0\n    $ sudo docker pull k8s.gcr.io/coredns:1.7.0\n    $ sudo docker pull k8s.gcr.io/pause:3.2\n    ```\n    ![image](figure/dockerimages.PNG)\n    * docker image를 tar로 저장한다.\n    ```bash\n    $ sudo docker save -o kube-proxy.tar k8s.gcr.io/kube-proxy:v1.19.4\n    $ sudo docker save -o kube-controller-manager.tar k8s.gcr.io/kube-controller-manager:v1.19.4\n    $ sudo docker save -o etcd.tar k8s.gcr.io/etcd:3.4.13-0\n    $ sudo docker save -o coredns.tar k8s.gcr.io/coredns:1.7.0\n    $ sudo docker save -o kube-scheduler.tar k8s.gcr.io/kube-scheduler:v1.19.4\n    $ sudo docker save -o kube-apiserver.tar k8s.gcr.io/kube-apiserver:v1.19.4\n    $ sudo docker save -o pause.tar k8s.gcr.io/pause:3.2\n    ```\n    ![image](figure/dockersave.PNG)\n3. 위의 과정에서 생성한 tar 파일들을 폐쇄망 환경으로 이동시킨 뒤 사용하려는 registry에 이미지를 push한다.\n    ```bash\n    $ sudo docker load -i kube-apiserver.tar\n    $ sudo docker load -i kube-scheduler.tar\n    $ sudo docker load -i kube-controller-manager.tar \n    $ sudo docker load -i kube-proxy.tar\n    $ sudo docker load -i etcd.tar\n    $ sudo docker load -i coredns.tar\n    $ sudo docker load -i pause.tar\n    ```\n    ![image](figure/dockerload.PNG)\n    ```bash\n    $ sudo docker tag k8s.gcr.io/kube-apiserver:v1.19.4 ${REGISTRY}/k8s.gcr.io/kube-apiserver:v1.19.4\n    $ sudo docker tag k8s.gcr.io/kube-proxy:v1.19.4 ${REGISTRY}/k8s.gcr.io/kube-proxy:v1.19.4\n    $ sudo docker tag k8s.gcr.io/kube-controller-manager:v1.19.4 ${REGISTRY}/k8s.gcr.io/kube-controller-manager:v1.19.4\n    $ sudo docker tag k8s.gcr.io/etcd:3.4.13-0 ${REGISTRY}/k8s.gcr.io/etcd:3.4.13-0\n    $ sudo docker tag k8s.gcr.io/coredns:1.7.0 ${REGISTRY}/k8s.gcr.io/coredns:1.7.0\n    $ sudo docker tag k8s.gcr.io/kube-scheduler:v1.19.4 ${REGISTRY}/k8s.gcr.io/kube-scheduler:v1.19.4\n    $ sudo docker tag k8s.gcr.io/pause:3.2 ${REGISTRY}/k8s.gcr.io/pause:3.2\n    ```\n    ![image](figure/tag.PNG)\n    ```bash\n    $ sudo docker push ${REGISTRY}/k8s.gcr.io/kube-apiserver:v1.19.4\n    $ sudo docker push ${REGISTRY}/k8s.gcr.io/kube-proxy:v1.19.4\n    $ sudo docker push ${REGISTRY}/k8s.gcr.io/kube-controller-manager:v1.19.4\n    $ sudo docker push ${REGISTRY}/k8s.gcr.io/etcd:3.4.13-0\n    $ sudo docker push ${REGISTRY}/k8s.gcr.io/coredns:1.7.0\n    $ sudo docker push ${REGISTRY}/k8s.gcr.io/kube-scheduler:v1.19.4\n    $ sudo docker push ${REGISTRY}/k8s.gcr.io/pause:3.2\n    ```\n    ![image](figure/push.PNG)\n    ```bash\n    $ curl ${REGISTRY}/v2/_catalog\n    ```    \n    ![image](figure/check.PNG)\n* 비고 :\n    * 위 내용은 2개이상의 마스터 구축시 마스터 1개에서만 진행한다.\n\n## 설치 가이드\n0. [(Master/Worker 공통) 환경 설정](/README.md#step0-환경-설정-masterworker-공통)\n1. [(Master/Worker 공통) runtime 설치]\ncase1. [cri-o 설치](/README.md#case-1-cri-o-설치-및-설정-masterworker-공통)\ncase2. [docker-ce 설치](/README.md#case-2-docker-ce-설치-및-설정-masterworker-공통)\n2. [(Master/Worker 공통) kubeadm, kubelet, kubectl 설치](/README.md#step-2-kubeadm-kubelet-kubectl-설치-masterworker-공통)\n3. [(Master) kubernetes cluster 구성]\ncase1. [ 단일 master cluser 구성](/README.md#case-1-단일-contorl-plain-구성-master-1개)\ncase2. [ 다중화 master cluser 구성](/README.md##case-2-다중-contorl-plain-구성-master-2개-이상)\n4. [(Worker) kubernetes cluster join](/README.md#step-4-cluster-join-worker)\n\n## 삭제 가이드\n0. [(Master/Worker 공통) kubernetes cluster reset](/README.md#step-1-cluster-reset-masterworker-공통)\n1. [(Master/Worker 공통) 패키지 삭제](/README.md#step-2-설치-패키지-삭제-masterworker-공통)\n\n## 설치 가이드\n## Step0. 환경 설정 (Master/Worker 공통)\n* 목적 : `k8s 설치 진행을 위한 os 환경 설정`\n* 순서 : \n    * os hostname을 설정한다.\n\t```bash\n\tsudo hostnamectl set-hostname k8s-master\n\t```\n    * hostname과 ip를 등록한다.\n      * sudo vi /etc/hosts\n\t```bash\n\t127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4\n\t::1         localhost localhost.localdomain localhost6 localhost6.localdomain6\n\n\t172.22.5.2 k8s-master\n\t```\n    * 방화벽(firewall)을 해제한다. \n\t```bash\n\tsudo systemctl stop firewalld\n\tsudo systemctl disable firewalld\n\t```\t\n    * 스왑 메모리를 비활성화 한다. \n\t```bash\n\tsudo swapoff -a\n\t```\n    * 스왑 메모리 비활성화 영구설정\n      * sudo vi /etc/fstab \n\t```bash\n\tswap 관련 부분 주석처리\n\t# /dev/mapper/centos-swap swap                    swap    defaults        0\n\t```\n    ![image](figure/fstab.PNG)\n    * SELinux 설정을 해제한다. \n\t```bash\n\tsudo setenforce 0\n\tsudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config\n\t```\n\t* (cri-o) 사용 전 환경 설정\n\t```bash\n\tsudo modprobe overlay\n\tsudo modprobe br_netfilter\n\t\n\tsudo cat \u003c\u003c \"EOF\" | sudo tee -a /etc/sysctl.d/99-kubernetes-cri.conf\n\tnet.bridge.bridge-nf-call-iptables  = 1\n\tnet.ipv4.ip_forward                 = 1\n\tnet.bridge.bridge-nf-call-ip6tables = 1\n\tEOF\n\t\n\tsudo sysctl --system\n\t```\n\t* (docker) 사용 전 환경 설정\n\t```bash\n\tsudo cat \u003c\u003c \"EOF\" | sudo tee -a /etc/sysctl.d/k8s.conf\n\tnet.bridge.bridge-nf-call-iptables  = 1\n\tnet.ipv4.ip_forward                 = 1\n\tnet.bridge.bridge-nf-call-ip6tables = 1\n\tEOF\n\t\n\tsudo sysctl --system\n\t```\n    * xfs filesystem을 사용하는 경우, 'xfs_info /' 커맨드가 ftype=1이 출력되는지 확인한다.\n    \n## Step 1. runtime 설치 및 설정 (Master/Worker 공통)\n### Case 1. cri-o 설치 및 설정 (Master/Worker 공통)\n* 목적 : `k8s container cri-o runtime 설치`\n* 순서 :\n    * cri-o(v1.19.1)를 설치한다. \n     * (폐쇄망) 아래 주소를 참조하여 패키지 레포를 등록 후 crio를 설치한다.\n        * https://github.com/tmax-cloud/install-pkg-repo/tree/5.0\n\t```bash\n\tsudo yum -y install cri-o\n\tsudo systemctl enable crio\n\tsudo systemctl start crio\n\t```\n     * (외부망) crio 버전 지정 및 레포를 등록 후 crio를 설치한다.\n\t```bash\n\tVERSION=1.19:1.19.1\n\tsudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/CentOS_7/devel:kubic:libcontainers:stable.repo\n\tsudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable:cri-o:${VERSION}.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:${VERSION}/CentOS_7/devel:kubic:libcontainers:stable:cri-o:${VERSION}.repo\n  \n\tsudo yum -y install cri-o\n\tsudo systemctl enable crio\n\tsudo systemctl start crio\n\t```\t\n    * cri-o 설치를 확인한다.\n\t```bash\n\tsudo systemctl status crio\n\trpm -qi cri-o\n\t```\n    ![image](figure/crio-v(v1.19.1).PNG)\n* 비고 :\n    * 추후 설치예정인 network plugin과 crio의 가상 인터페이스 충돌을 막기위해 cri-o의 default 인터페이스 설정을 제거한다.\n\t```bash\n\tsudo rm -rf  /etc/cni/net.d/100-crio-bridge.conf\n \tsudo rm -rf  /etc/cni/net.d/200-loopback.conf\n\tsudo rm -rf /etc/cni/net.d/87-podman-bridge.conflist\n\t``` \n    * crio.conf 내용을 수정한다. ( sudo vi /etc/crio/crio.conf )\n      * (폐쇄망) insecure_registries = [\"{registry}:{port}\"]\n      * (폐쇄망) pause_image : \"k8s.gcr.io/pause:3.1\" 을 \"{registry}:{port}/k8s.gcr.io/pause:3.1\" 로 변경\n      ![image](figure/crio_config.PNG)\n    * pid cgroup의 max pid limit 설정이 필요한 경우 pids_limit 개수를 수정한다.\n      * default : pids_limit = 1024\n      * 시스템의 제한값인 `/proc/sys/kernel/pid_max`의 값 이하로 설정한다.\n\t```bash\n\tpids_limit = 32768\n\t```     \n    * private image registry를 사용할 경우 registries.conf 내용을 수정한다.\n      * sudo vi /etc/containers/registries.conf\n\t```bash\n\tunqualified-search-registries = ['registry.fedoraproject.org', 'registry.access.redhat.com', 'registry.centos.org', 'docker.io', '{registry}:{port}']\n\tex) unqualified-search-registries = ['registry.fedoraproject.org', 'registry.access.redhat.com', 'registry.centos.org', 'docker.io', '172.22.5.2:5000']\n\t``` \t      \n    * crio를 재시작 한다.\n\t```bash\n\tsudo systemctl restart crio\n\t```\n\t\n### Case 2. docker-ce 설치 및 설정 (Master/Worker 공통)\n* 목적 : `k8s container docker runtime 설치`\n* 생성 순서 :\n    * docker-ce를 설치한다.\n    * (폐쇄망) 아래 주소를 참조하여 패키지 레포를 등록 후 docker-ce를 설치한다. (image registry 구축 시에 docker를 이미 설치하였고 daemon.json 내용이 같다면 아래 과정은 생략한다.)\n      * https://github.com/tmax-cloud/install-pkg-repo/tree/5.0 \n    ```bash\n    sudo yum install -y docker-ce\n    sudo systemctl start docker\n    sudo systemctl enable docker\n    ```  \n    * (외부망) docker 의존성 패키지를 설치와 docker-ce.repo를 등록 후 docker-ce를 설치한다.(image registry 구축 시에 docker를 이미 설치하였고 daemon.json 내용이 같다면 아래 과정은 생략한다.)\n    ```bash\n    yum -y install yum-utils device-mapper-persistent-data lvm2\n    yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo\n\n    sudo yum install -y docker-ce\n    sudo systemctl start docker\n    sudo systemctl enable docker\n    ```\n    * docker-ce 설치 시 특정 버전을 설치할 경우 버전을 명시하여 설치한다.\n    ```bash\n    yum list docker-ce.x86-64 --showduplicates\n    ex) yum install -y docker-ce-18.09.7.ce\n    ```  \n    * (폐쇄망) private registry 접근을 위해 daemon.json 내용을 수정한다.\n      * sudo vi /etc/docker/daemon.json\n    ```bash\n    {\n      \"exec-opts\": [\"native.cgroupdriver=systemd\"],\n      \"log-driver\": \"json-file\",\n      \"log-opts\": { \"max-size\": \"100m\" },\n      \"storage-driver\": \"overlay2\",\n      \"insecure-registries\": [\"{registry}:{port}\"]\n    }\n    ```\n    * docker를 재실행하고 status를 확인한다.\n    ```bash\n      sudo systemctl restart docker\n      sudo systemctl status docker\n    ```\n* 비고 :\n    * docker-ce 설치 시 runtime confilct 에러가 발생하는 경우 아래와 같이 우회하여 docker 설치를 진행한다.\n      *  ex) Error: containerd.io conflicts with 2:runc-1.0.0-377.rc93.el7.8.1.x86_64\n\t ```bash\n    $ sudo yum list | grep runc\n    $ sudo yum remove runc\n    $ sudo yum install docker-ce\n\t ```  \n    * private image registry를 사용할 경우 registries.conf 내용을 수정한다.\n      * sudo vi /etc/containers/registries.conf\n\t```bash\n\tunqualified-search-registries = ['registry.fedoraproject.org', 'registry.access.redhat.com', 'registry.centos.org', 'docker.io', '{registry}:{port}']\n\tex) unqualified-search-registries = ['registry.fedoraproject.org', 'registry.access.redhat.com', 'registry.centos.org', 'docker.io', '172.22.5.2:5000']\n\t``` \t      \n    * docker를 재시작 한다.\n\t```bash\n\tsudo systemctl restart docker\n\t```\t\n## Step 2. kubeadm, kubelet, kubectl 설치 (Master/Worker 공통)\n* 목적 : `Kubernetes 구성을 위한 kubeadm, kubelet, kubectl 설치한다.`\n* 순서:\n    * CRI-O 메이저와 마이너 버전은 쿠버네티스 메이저와 마이너 버전이 일치해야 한다.\n    * (폐쇄망) kubeadm, kubectl, kubelet 설치 (v1.19.4)\n\t```bash\n\tsudo yum install -y kubeadm-1.19.4-0 kubelet-1.19.4-0 kubectl-1.19.4-0\n\t\n\tsudo systemctl enable kubelet\n\t```  \t\n    * (외부망) 레포 등록 후 kubeadm, kubectl, kubelet 설치 (v1.19.4)\n\t```bash\n\tsudo cat \u003c\u003c \"EOF\" | sudo tee -a /etc/yum.repos.d/kubernetes.repo\n\t[kubernetes]\n\tname=Kubernetes\n\tbaseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64\n\tenabled=1\n\tgpgcheck=1\n\trepo_gpgcheck=1\n\tgpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg\n\tEOF\n\n\tsudo yum install -y kubeadm-1.19.4-0 kubelet-1.19.4-0 kubectl-1.19.4-0\n\t\n\tsudo systemctl enable kubelet\n\t```  \n\n## Step 3. kubernetes cluster 구성 (Master)\n### Case 1. 단일 contorl plain 구성 (Master 1개)\n* 목적 : `kubernetes master를 구축한다.`\n* 순서 :\n    * 쿠버네티스 설치시 필요한 kubeadm-config를 작성한다.\n        * vi kubeadm-config.yaml\n\t```bash\n\tapiVersion: kubeadm.k8s.io/v1beta2\n\tkind: InitConfiguration\n\tlocalAPIEndpoint:\n  \t\tadvertiseAddress: {api server IP}\n  \t\tbindPort: 6443\n\tnodeRegistration:\n  \t\tcriSocket: /var/run/crio/crio.sock\n\t---\n\tapiVersion: kubeadm.k8s.io/v1beta2\n\tkind: ClusterConfiguration\n\tkubernetesVersion: {k8s version}\n\tcontrolPlaneEndpoint: {endpoint IP}:6443\n\timageRepository: {registry}/k8s.gcr.io\n\tnetworking:\n \t\tserviceSubnet: {SERVICE_IP_POOL}/{CIDR}\n  \t\tpodSubnet: {POD_IP_POOL}/{CIDR}\n\t---\n\tapiVersion: kubelet.config.k8s.io/v1beta1\n\tkind: KubeletConfiguration\n\tcgroupDriver: systemd\n\t```\n      * kubernetesVersion : kubernetes version\n      * advertiseAddress : API server IP ( master IP )\n        * 해당 master 노드의 IP\n      * controlPlaneEndpoint : endpoint IP ( master IP ) , port는 반드시 6443으로 설정\n      * serviceSubnet : ${SERVICE_IP_POOL}/${CIDR}\n      * podSubnet : ${POD_IP_POOL}/${CIDR}\n      * imageRepository : ${registry}/docker_hub_name\n      * cgroupDriver: cgroup driver 설정\n      \n     * kubeadm init\n\t```bash\n\tsudo kubeadm init --config=kubeadm-config.yaml\n\t```\n\t![image](figure/kubeinit.PNG)\n     * 듀얼 스택 클러스터 구축 시에는 아래의 [비고] yaml을 참조하여 진행한다.                \n    * kubernetes config \n\t```bash\n\tmkdir -p $HOME/.kube\n\tsudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config\n\tsudo chown $(id -u):$(id -g) $HOME/.kube/config\n\t```\n\t![image](figure/success.PNG)\n    * 확인\n\t```bash\n\tkubectl get nodes\n\t```\n\t![image](figure/nodes.PNG)\n\t```bash\n\tkubectl get pods -A -o wide\n\t```\n\t![image](figure/pods.PNG)\t\n* 비고 : \n    * master에도 pod 스케줄을 가능하게 하려면 master taint를 제거한다\n\t```bash\n\tkubectl taint node [master hostname] node-role.kubernetes.io/master:NoSchedule-\n\tex) kubectl taint node k8s-master1 node-role.kubernetes.io/master:NoSchedule- \n\t```\n    * join시에 apiserver ip 변경이 필요할 경우 --apiserver-advertise-address 옵션을 추가한다.\n\t```bash\n\tkubeadm join 172.22.5.2:6443 --token 2cks7n.yvojnnnq1lyz1qud \\ --discovery-token-ca-cert-hash sha256:efba18bb4862cbcb54fb643a1b7f91c25e08cfc1640e5a6fffa6de83e4c76f07 \\ --control-plane --certificate-key f822617fcbfde09dff35c10e388bc881904b5b6c4da28f3ea8891db2d0bd3a62 --apiserver-advertise-address=172.22.4.3 --cri-socket=/var/run/crio/crio.sock\n\t```  \t\n    * 듀얼 스택 클러스터 구축 시에는 아래의 kubeadm-config.yaml을 참조한다.\n      * vi kubeadm-config.yaml\n      ```bash\n      apiVersion: kubeadm.k8s.io/v1beta2\n      kind: InitConfiguration\n      localAPIEndpoint:\n      advertiseAddress: {api server IP}\n  \t\tbindPort: 6443\n      nodeRegistration:\n  \t\tcriSocket: /var/run/crio/crio.sock\n      ---\n      apiVersion: kubeadm.k8s.io/v1beta2\n      kind: ClusterConfiguration\n      kubernetesVersion: {k8s version}\n      controlPlaneEndpoint: {endpoint IP}:6443\n      imageRepository: {registry}/k8s.gcr.io\n      networking:\n \t\tserviceSubnet: ${SERVICE_IPV4_POOL}/${CIDR},${SERVICE_IPV6_POOL}/${CIDR}\n  \t\tpodSubnet: ${POD_IPV4_POOL}/${CIDR},${POD_IPV6_POOL}/${CIDR}\n      featureGates:\n    \t\tIPv6DualStack: true\n      ---\n      apiVersion: kubelet.config.k8s.io/v1beta1\n      kind: KubeletConfiguration\n      cgroupDriver: systemd\n      ---\n      apiVersion: kubeproxy.config.k8s.io/v1alpha1\n      kind: KubeProxyConfiguration\n      mode: ipvs\n\t```\n     * kubernetesVersion : kubernetes version\n     * advertiseAddress : API server IP ( master IP )\n        * 해당 master 노드의 IP\n     * controlPlaneEndpoint : endpoint IP ( master IP or virtual IP) , port는 반드시 6443으로 설정\n        * 1개의 마스터 : master IP , 2개 이상의 마스터 구축시 : virtual IP\n     * serviceSubnet : ${SERVICE_IPV4_POOL}/${CIDR},${SERVICE_IPV6_POOL}/${CIDR}\n     * podSubnet : ${POD_IPV4_POOL}/${CIDR},${POD_IPV6_POOL}/${CIDR}\n     * imageRepository : ${registry}/docker_hub_name\n     * IPv6DualStack: true, dual stack 기능이 베타 버전이라서 기본값으로는 비활성화이기 때문에 활성화 시키는 세팅(추후 기본으로 활성화 되면 빠져야 함)\n     * cgroupDriver: cgroup driver \n     * mode: ipvs, dual stack 기능은 kube-proxy ipvs 모드에서만 동작\n      \n### Case 2. 다중 contorl plain 구성 (Master 2개 이상)\n* 목적 : `kubernetes master를 구축한다.`\n* 순서 : \n    * Keepalived 설치\n    ```bash\n    sudo yum install -y keepalived\n    ```\n    * Keepalived 설정\n    ```bash\n\tsudo vi /etc/keepalived/keepalived.conf\n\t\n\tvrrp_instance VI_1 {    \n\tstate {MASTER or BACKUP}   \n\tinterface {network interface}    \n\tvirtual_router_id {virtual router id}    \n\tpriority {priority}    \n\tadvert_int 1    \n\tnopreempt    \n\tauthentication {        \n\t\tauth_type PASS        \n\t\tauth_pass {password}  \n\t\t}   \n\tvirtual_ipaddress {        \n\t\t{VIP}  \n\t\t} \n\t}\n    ```\t\n    ![image](figure/keepalived.PNG)\n\t* interface : network interface 이름 확인 (ip a 명령어로 확인) ex) enp0s8\n\t* state : master or backup으로 설정, 하나의 master에만 master를 설정하고 나머지 master에는 backup으로 설정\n\t* priority : Master 우선순위  \n\t    * priority 값이 높으면 최우선적으로 Master 역할 수행\n\t    * 각 Master마다 다른 priority 값으로 수정\n\t    * ex) master1 priority 100, master2 priority 99, master3 priority 98 \n\t* virtual_ipaddress : virtual ip(VIP) 설정\n\t* virtual_router_id : vritual router id ex) 50\n\t* auth_pass : 패스워드 설정\n\t\n    * keepalived 재시작 및 상태 확인\n    ```bash\n    sudo systemctl restart keepalived\n    sudo systemctl enable keepalived\n    sudo systemctl status keepalived\n    ```\t\n    * network interface 확인\n    ```bash\n    ip a\n    ```\n    * 설정한 VIP 확인 가능, 여러 마스터 중 하나만 보임.\n    * inet {VIP}/32 scope global eno1\n    ![image](figure/ipa.PNG)\n\n    * 쿠버네티스 설치시 필요한 kubeadm-config를 작성한다.\n        * vi kubeadm-config.yaml\n\t```bash\n\tapiVersion: kubeadm.k8s.io/v1beta2\n\tkind: InitConfiguration\n\tlocalAPIEndpoint:\n  \t\tadvertiseAddress: {api server IP}\n  \t\tbindPort: 6443\n\tnodeRegistration:\n  \t\tcriSocket: /var/run/crio/crio.sock\n\t---\n\tapiVersion: kubeadm.k8s.io/v1beta2\n\tkind: ClusterConfiguration\n\tkubernetesVersion: {k8s version}\n\tcontrolPlaneEndpoint: {endpoint IP}:6443\n\timageRepository: {registry}/k8s.gcr.io\n\tnetworking:\n \t\tserviceSubnet: {SERVICE_IP_POOL}/{CIDR}\n  \t\tpodSubnet: {POD_IP_POOL}/{CIDR}\n\t---\n\tapiVersion: kubelet.config.k8s.io/v1beta1\n\tkind: KubeletConfiguration\n\tcgroupDriver: systemd\n\t```\n      * kubernetesVersion : kubernetes version\n      * advertiseAddress : API server IP ( master IP )\n        * 해당 master 노드의 IP\n      * controlPlaneEndpoint : endpoint IP ( virtual IP ) , port는 반드시 6443으로 설정\n      * serviceSubnet : ${SERVICE_IP_POOL}/${CIDR}\n      * podSubnet : ${POD_IP_POOL}/${CIDR}\n      * imageRepository : ${registry}/docker_hub_name\n      * cgroupDriver: cgroup driver \n      \n     * kubeadm init\n       * Master 다중구성시 --upload-certs 옵션은 반드시 필요.\n\t```bash\n\tsudo kubeadm init --config=kubeadm-config.yaml --upload-certs\n\t```\n\t![image](figure/kubeinit.PNG)\n\t\n    * kubernetes config \n\t```bash\n\tmkdir -p $HOME/.kube\n\tsudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config\n\tsudo chown $(id -u):$(id -g) $HOME/.kube/config\n\t```\n\t![image](figure/master2.PNG)\n\t\n    * 아래 내용을 참조하여 다른 control plane 노드(master)를 구성한다.\n        * runtime cri-o 사용할 경우, join 시에 --cri-socket=/var/run/crio/crio.sock 옵션을 추가하여 실행한다.\n\t    ```bash\n\t    sudo kubeadm init --config=kubeadm-config.yaml --upload-certs \n\t    sudo kubeadm join {IP}:{PORT} --token ~~ discovery-token-ca-cert-hash --control-plane --certificate-key ~~ --cri-socket=/var/run/crio/crio.sock (1)\n\t    sudo kubeadm join {IP}:{PORT} --token ~~ discovery-token-ca-cert-hash --cri-socket=/var/run/crio/crio.sock (2)\n\t    ``` \n\t* 해당 옵션은 certificates를 control-plane으로 upload하는 옵션\n\t* 해당 옵션을 설정하지 않을 경우, 모든 Master 노드에서 key를 복사해야 함\n\t* Master 단일구성과는 다르게, --control-plane --certificate-key 옵션이 추가된 명령어가 출력됨\n\t* (1)처럼 Master 다중구성을 위한 hash 값을 포함한 kubeadm join 명령어가 출력되므로 해당 명령어를 복사하여 다중구성에 포함시킬 다른 Master에서 실행\n\t* (2)처럼 Worker의 join을 위한 명령어도 출력되므로 Worker 노드 join시 사용, crio 사용시 --cri-socket 옵션 추가\n\t   ```bash\n\t     kubeadm join 172.22.5.2:6443 --token 2cks7n.yvojnnnq1lyz1qud \\ --discovery-token-ca-cert-hash sha256:efba18bb4862cbcb54fb643a1b7f91c25e08cfc1640e5a6fffa6de83e4c76f07 \\ --control-plane --certificate-key f822617fcbfde09dff35c10e388bc881904b5b6c4da28f3ea8891db2d0bd3a62 --cri-socket=/var/run/crio/crio.sock\n\t   ```\n    * 확인\n\t```bash\n\tkubectl get nodes\n\t```\n\t![image](figure/nodes.PNG)\n\t```bash\n\tkubectl get pods -A -o wide\n\t```\n\t![image](figure/pods.PNG)\t\n* 비고 : \n    * master에도 pod 스케줄을 가능하게 하려면 master taint를 제거한다\n\t```bash\n\tkubectl taint node [master hostname] node-role.kubernetes.io/master:NoSchedule-\n\tex) kubectl taint node k8s-master1 node-role.kubernetes.io/master:NoSchedule- \n\t```\n* 비고 : \t\n    * join시에 apiserver ip 변경이 필요할 경우 --apiserver-advertise-address 옵션을 추가한다.\n\t```bash\n\tkubeadm join 172.22.5.2:6443 --token 2cks7n.yvojnnnq1lyz1qud \\ --discovery-token-ca-cert-hash sha256:efba18bb4862cbcb54fb643a1b7f91c25e08cfc1640e5a6fffa6de83e4c76f07 \\ --control-plane --certificate-key f822617fcbfde09dff35c10e388bc881904b5b6c4da28f3ea8891db2d0bd3a62 --apiserver-advertise-address=172.22.4.3 --cri-socket=/var/run/crio/crio.sock\n\t```    \n\t\n## Step 4. Cluster join (Worker)\n* 목적 : `kubernetes cluster에 join한다.`\n* 순서 :\n    * kubernetes master 구축시 생성된 join token을 worker node에서 실행한다.\n    * kubeadm join\n       * (cri-o) join token에 --cri-socket=/var/run/crio/crio.sock 옵션을 추가하여 실행한다.\n       ```bash\n       kubeadm join 172.22.5.2:6443 --token r5ks9p.q0ifuz5pcphqvc14 \\ --discovery-token-ca-cert-hash sha256:90751da5966ad69a49f2454c20a7b97cdca7f125b8980cf25250a6ee6c804d88 --cri-socket=/var/run/crio/crio.sock\n       ```\n       * (docker) join token을 그대로 실행한다.\n       ```bash\n       kubeadm join 172.22.5.2:6443 --token r5ks9p.q0ifuz5pcphqvc14 \\ --discovery-token-ca-cert-hash sha256:90751da5966ad69a49f2454c20a7b97cdca7f125b8980cf25250a6ee6c804d88\n       ```\t\n    ![image](figure/noding.PNG)\n* 비고 : \n    * kubeadm join command를 저장해놓지 못한 경우, master node에서 아래 명령어를 통해 token 재생성이 가능하다.\n\t```bash\n\tkubeadm token create --print-join-command\n\t```\t\n    \n## 삭제 가이드\n## Step 1. Cluster reset (Master/Worker 공통)\n* 목적 : `kubernetes cluster를 reset한다.`\n* 순서 :\n    * kubeadm reset\n       * (cri-o) --cri-socket=/var/run/crio/crio.sock 옵션을 추가하여 kubeadm reset을 실행한다.\n       ```bash\n       kubeadm reset --cri-socket=/var/run/crio/crio.sock\n       ```\n       * (docker) kubeadm reset을 실행한다.\n       ```bash\n       kubeadm reset\n       ```\n\n## Step 2. 설치 패키지 삭제 (Master/Worker 공통)\n* 목적 : `kubernetes에 필요한 패키지들을 삭제한다`\n* 순서 :\n    * 설치했던 패키지들을 삭제 한다.\n      ```bash\n      sudo yum remove -y kubeadm-1.19.4-0 kubelet-1.19.4-0 kubectl-1.19.4-0\n      \n      sudo yum remove -y crio  \n      or  \n      sudo yum remove -y docker-ce\n      \n      sudo yum remove -y keepalived\n      ```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftmax-cloud%2Finstall-k8s","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftmax-cloud%2Finstall-k8s","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftmax-cloud%2Finstall-k8s/lists"}