{"id":13576768,"url":"https://github.com/yanghongfei/Kubernetes","last_synced_at":"2025-04-05T08:33:14.221Z","repository":{"id":132333197,"uuid":"136032794","full_name":"yanghongfei/Kubernetes","owner":"yanghongfei","description":"K8S Kubeadm和二进制部署高可用集群","archived":false,"fork":false,"pushed_at":"2019-04-08T02:09:47.000Z","size":59808,"stargazers_count":56,"open_issues_count":2,"forks_count":30,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-11-05T13:41:54.111Z","etag":null,"topics":["etcd-cluster","etcd-server","ingress","ipvs","k8s-cluster","kubernetes","kubernetes-service","pod","prometheus-alertmanager","prometheus-exporter","traefik"],"latest_commit_sha":null,"homepage":"http://yanghongfei.me/","language":"Jsonnet","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yanghongfei.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2018-06-04T13:55:23.000Z","updated_at":"2024-09-16T14:25:23.000Z","dependencies_parsed_at":"2024-01-14T04:07:38.392Z","dependency_job_id":null,"html_url":"https://github.com/yanghongfei/Kubernetes","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yanghongfei%2FKubernetes","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yanghongfei%2FKubernetes/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yanghongfei%2FKubernetes/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yanghongfei%2FKubernetes/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yanghongfei","download_url":"https://codeload.github.com/yanghongfei/Kubernetes/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247312065,"owners_count":20918340,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["etcd-cluster","etcd-server","ingress","ipvs","k8s-cluster","kubernetes","kubernetes-service","pod","prometheus-alertmanager","prometheus-exporter","traefik"],"created_at":"2024-08-01T15:01:13.859Z","updated_at":"2025-04-05T08:33:14.197Z","avatar_url":"https://github.com/yanghongfei.png","language":"Jsonnet","funding_links":[],"categories":["Jsonnet"],"sub_categories":[],"readme":"Table of Contents\n=================\n   * [目录](#目录)\n      * [基于Kubeadm部署Kubernetes1.13.3 HA 高可用集群](#基于kubeadm部署kubernetes1133-ha-高可用集群)\n         * [01. 部署目的](#01-部署目的)\n            * [1.1 Kubernetes的特性](#11-kubernetes的特性)\n            * [1.2 贴微服务，开发环境快速部署](#12-贴微服务开发环境快速部署)\n         * [02. 环境说明](#02-环境说明)\n            * [2.1 集群说明](#21-集群说明)\n         * [03. K8S集群名词说明](#03-k8s集群名词说明)\n            * [3.1 Kubernetes](#31-kubernetes)\n            * [3.2 Docker](#32-docker)\n            * [3.3 Etcd](#33-etcd)\n            * [3.4 Calico](#34-calico)\n         * [04. 开始部署Kubernetes集群](#04-开始部署kubernetes集群)\n            * [4.1 安装前准备](#41-安装前准备)\n            * [4.2 优化系统和集群准备](#42-优化系统和集群准备)\n            * [4.3 安装Docker-CE](#43-安装docker-ce)\n            * [4.4 所有节点配置Docker镜像加速](#44-所有节点配置docker镜像加速)\n         * [05. 生成TLS证书和秘钥](#05-生成tls证书和秘钥)\n            * [5.1 Kubernetes 集群所需证书](#51-kubernetes-集群所需证书)\n            * [5.2 安装CFSSL](#52-安装cfssl)\n            * [5.3 创建CA文件,生成etcd证书](#53-创建ca文件生成etcd证书)\n            * [5.4 分发证书到所有节点](#54-分发证书到所有节点)\n         * [06. 安装配置etcd](#06-安装配置etcd)\n            * [6.1 安装etcd](#61-安装etcd)\n            * [6.2 配置etcd](#62-配置etcd)\n            * [6.3  启动etcd](#63--启动etcd)\n            * [6.4 集群状态检查(维护)](#64-集群状态检查维护)\n         * [07 安装配置Kubeadm](#07-安装配置kubeadm)\n            * [7.1 配置kubelet](#71-配置kubelet)\n            * [7.2 加载配置文件](#72-加载配置文件)\n         * [08 Master节点高可用](#08-master节点高可用)\n            * [8.1 安装keepalived](#81-安装keepalived)\n            * [8.2 配置Keepalived](#82-配置keepalived)\n            * [8.3 启动Keepalived](#83-启动keepalived)\n            * [8.4 测试Keepalived可用性](#84-测试keepalived可用性)\n         * [09 初始化集群](#09-初始化集群)\n            * [9.1 初始化集群](#91-初始化集群)\n            * [9.2 初始化失败解决办法](#92-初始化失败解决办法)\n         * [10. 部署网络组件](#10-部署网络组件)\n            * [10.1 Calico组件部署（二选一）](#101-calico组件部署二选一)\n            * [10.2 Flannel组件部署(二选一)](#102-flannel组件部署二选一)\n            * [10.3 查看集群状态](#103-查看集群状态)\n            * [10.4 测试Calico DNS网络问题](#104-测试calico-dns网络问题)\n         * [11. 部署Dashboard](#11-部署dashboard)\n            * [11.1 登陆访问](#111-登陆访问)\n         * [12 部署Traefik Ingress](#12-部署traefik-ingress)\n            * [12.1 配置Tracefik HTTPS](#121-配置tracefik-https)\n            * [12.2 部署Traefik](#122-部署traefik)\n            * [12.3 访问Traefik](#123-访问traefik)\n         * [13. Prometheus-Operator](#13-prometheus-operator)\n            * [13.1 快速开始](#131-快速开始)\n            * [13.2使用Traefik 代理出来Prometheus Grafana](#132使用traefik-代理出来prometheus-grafana)\n            * [13.3配置Promethe-Operator自定义报警](#133配置promethe-operator自定义报警)\n            * [13.4使用Promethues监控外接ETCD集群](#134-使用promethues监控外接etcd集群)\n         * [维护相关](#维护相关)\n            * [01. 强制删除Pod](#01-强制删除pod)\n            * [02. 重新获取集群Token](#02-重新获取集群token)\n            * [03. Master节点也允许Pod容器](#03-master节点也允许pod容器)\n            * [04. modprobe: FATAL: Module br_netfilter not found](#04-modprobe-fatal-module-br_netfilter-not-found)\n            * [05. Node配置Keepalived，避免整个Node挂掉，Tarefik解析不可用问题](#05-node配置keepalived避免整个node挂掉tarefik解析不可用问题)\n            * [06. POD容器不能访问外网问题](#06-pod容器不能访问外网问题)\n         * [参考文档](#参考文档)\n\n\n# 目录\n\n## 基于Kubeadm部署Kubernetes1.13.3 HA 高可用集群 \n\n\u003e K8S 1.13版本中默认：\n\u003e\n\u003e  \tDNS：CoreDNS\n\u003e\n\u003e   Kube-proxy： IPVS模板    #非iptables\n\u003e\n\u003e   Network： Calico\n\u003e\n\u003e 注意：\n\u003e\n\u003e 默认kubeadm一键部署`etcd`非集群高可用，\n\u003e\n\u003e 本示例使用`外接etcd`实现高可用集群，Master APIserver使用Keeplived；\n\n### 01. 部署目的\n\n#### 1.1 Kubernetes的特性\n\n- 分布式部署\n- 服务发现\n- 服务实例保护\n- 滚动升级\n- 节点无痕维护\n- 负载平衡\n- 动态扩缩容\n\n从而能够贴合未来微服部署维护的需求\n\n#### 1.2 贴微服务，开发环境快速部署\n\n\u003e 01通过docker镜像，可以快速并且一致的为每个开发人员提供相同的linux开发环境，省去了每个员工自行部署开发环境带来的疑问和冗余，让开发人员能够能专注于code本身，节省大量时间\n\n\n\n### 02. 环境说明\n\n- 五台机器进行部署K8S v1.13+集群环境，一台内网Harbor。其中`etcd`为所有节点部署\n- Kubernetes中所有数据都是存储在etcd中的，etcd必须高可用集群\n- Master使用keepalived高可用，Master主要是分发用户操作指令等操作；\n- Master官方给出是用keepalived进行集群，建议也可以使用自建LB/商业AWS的（ALB ELB ）\n- Node节点为搭配Taaefik解析IP高可用，也使用Keepalived，Tarefik解析 VIP上\n\n|            System             |   Roles   |   IP Address    |\n| :---------------------------: | :-------: | :-------------: |\n|     Master Keepalived VIP     |    VIP    |   172.16.1.49   |\n|      Node Keepalived VIP      |    VIP    |   172.16.1.59   |\n| CentOS Linux release 7.4.1708 | Master01  |   172.16.1.50   |\n| CentOS Linux release 7.4.1708 | Master02  |   172.16.1.51   |\n| CentOS Linux release 7.4.1708 |  Node01   |   172.16.1.52   |\n| CentOS Linux release 7.4.1708 |  Node02   |   172.16.1.53   |\n| CentOS Linux release 7.4.1708 |  Node01   |   172.16.1.54   |\n| CentOS Linux release 7.4.1708 | VM Harbor | xxx.xxx.xxx.xxx |\n\n#### 2.1 集群说明\n\n|      Software       | Version |\n| :-----------------: | :-----: |\n|     Kubernetes      | 1.13.3  |\n|      Docker-CE      | 18.06.1 |\n|        Etcd         | 3.3.11  |\n|       Calico        |  3.1.4  |\n|      Dashboard      | v1.10.0 |\n|   Traefik Ingress   |  1.7.9  |\n| prometheus-operator | 0.29.0  |\n\n\n\n### 03. K8S集群名词说明\n\n#### 3.1 Kubernetes\n\n\u003e  Kubernetes 是 Google 团队发起并维护的基于Docker的开源容器集群管理系统，它不仅支持常见的云平台，而且支持内部数据中心。建于 Docker 之上的 Kubernetes 可以构建一个容器的调度服务，其目的是让用户透过Kubernetes集群来进行云端容器集群的管理，而无需用户进行复杂的设置工作。系统会自动选取合适的工作节点来执行具体的容器集群调度处理工作。其核心概念是Container Pod（容器仓）。一个Pod是有一组工作于同一物理工作节点的容器构成的。这些组容器拥有相同的网络命名空间/IP以及存储配额，可以根据实际情况对每一个Pod进行端口映射。此外，Kubernetes工作节点会由主系统进行管理，节点包含了能够运行Docker容器所用到的服务。\n\n####  3.2 Docker\n\n\u003e  Docker是一个开源的引擎，可以轻松的为任何应用创建一个轻量级的、可移植的、自给自足的容器。开发者在笔记本上编译测试通过的容器可以批量地在生产环境中部署，包括VMs（虚拟机）、bare metal、OpenStack 集群和其他的基础应用平台。\n\n####  3.3 Etcd\n\n\u003e ETCD是用于共享配置和服务发现的分布式，一致性的KV存储系统。\n\n#### 3.4 Calico\n\n\u003e 同Flannel,用于解决 docker 容器直接跨主机的通信问题，Calico的实现是非侵入式的，不封包解包，直接通过iptables转发，基本没有消耗，flannel需要封包解包，有cpu消耗，效率不如calico，calico基本和原机差不多了 \n\n\n\n### 04. 开始部署Kubernetes集群\n\n#### 4.1 安装前准备\n\n截至2019年2月，Kubernetes目前文档版本：v1.13+  官方版本迭代很快，我们选择目前文档版本搭建\n\n**K8S所有节点配置主机名**\n\n```shell\n# 设置主机名\nhostnamectl set-hostname K8S01-Master01\nhostnamectl set-hostname K8S01-Master02\nhostnamectl set-hostname K8S01-Node01\nhostnamectl set-hostname K8S01-Node02\nhostnamectl set-hostname K8S01-Node03\n\n# 配置hosts\ncat \u003c\u003cEOF \u003e /etc/hosts\n127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4\n::1         localhost localhost.localdomain localhost6 localhost6.localdomain6\n172.16.1.50 master01 K8S01-Master01\n172.16.1.51 master02 K8S01-Master02\n172.16.1.52 node01 K8S01-Node01\n172.16.1.53 node02 K8S01-Node02\n172.16.1.54 node03 K8S01-Node03\nEOF\n\n#配置免密钥登陆\nssh-keygen   #一直回车\nssh-copy-id   master01\nssh-copy-id   master02\nssh-copy-id   node01\nssh-copy-id   node02\n```\n\n\n\n#### 4.2 优化系统和集群准备\n\n```shell\n#关闭防火墙\nsystemctl stop firewalld\nsystemctl disable firewalld\n\n###关闭Swap\nswapoff -a \nsed -i 's/.*swap.*/#\u0026/' /etc/fstab\n\n###禁用Selinux\nsetenforce  0 \nsed -i \"s/^SELINUX=enforcing/SELINUX=disabled/g\" /etc/sysconfig/selinux \nsed -i \"s/^SELINUX=enforcing/SELINUX=disabled/g\" /etc/selinux/config \nsed -i \"s/^SELINUX=permissive/SELINUX=disabled/g\" /etc/sysconfig/selinux \nsed -i \"s/^SELINUX=permissive/SELINUX=disabled/g\" /etc/selinux/config  \n\n###报错请参考下面报错处理\nmodprobe br_netfilter   \ncat \u003c\u003cEOF \u003e  /etc/sysctl.d/k8s.conf\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.bridge.bridge-nf-call-iptables = 1\nnet.ipv4.ip_forward = 1\nvm.swappiness=0\nEOF\nsysctl -p /etc/sysctl.d/k8s.conf\nls /proc/sys/net/bridge\n\n###K8S源\ncat \u003c\u003cEOF \u003e /etc/yum.repos.d/kubernetes.repo\n[kubernetes]\nname=Kubernetes\nbaseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/\nenabled=1\ngpgcheck=1\nrepo_gpgcheck=1\ngpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg\nEOF\n\n###内核优化\necho \"* soft nofile 204800\" \u003e\u003e /etc/security/limits.conf\necho \"* hard nofile 204800\" \u003e\u003e /etc/security/limits.conf\necho \"* soft nproc 204800\"  \u003e\u003e /etc/security/limits.conf\necho \"* hard nproc 204800\"  \u003e\u003e /etc/security/limits.conf\necho \"* soft  memlock  unlimited\"  \u003e\u003e /etc/security/limits.conf\necho \"* hard memlock  unlimited\"  \u003e\u003e /etc/security/limits.conf\n\n###kube-proxy开启ipvs的前置条件\n# 原文：https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/README.md\n# 参考：https://www.qikqiak.com/post/how-to-use-ipvs-in-kubernetes/\n\n# 加载模块 \u003cmodule_name\u003e\nmodprobe -- ip_vs\nmodprobe -- ip_vs_rr\nmodprobe -- ip_vs_wrr\nmodprobe -- ip_vs_sh\nmodprobe -- nf_conntrack_ipv4\n\n# 检查加载的模块\nlsmod | grep -e ipvs -e nf_conntrack_ipv4\n# 或者\ncut -f1 -d \" \"  /proc/modules | grep -e ip_vs -e nf_conntrack_ipv4\n\n#所有node节点安装ipvsadm\nyum install ipvsadm -y\nipvsadm -l -n\n# Version INFO: IP Virtual Server version 1.2.1 (size=4096)\n```\n\n\n\n#### 4.3 安装Docker-CE\n\n```shell\nyum install -y yum-utils device-mapper-persistent-data lvm2\nyum-config-manager \\\n    --add-repo \\\n    https://download.docker.com/linux/centos/docker-ce.repo\n\nyum makecache fast\nyum install -y --setopt=obsoletes=0 \\\n  docker-ce-18.06.1.ce-3.el7\n\nsystemctl start docker\nsystemctl enable docker\n```\n\n\n\n#### 4.4 所有节点配置Docker镜像加速 \n\n阿里云容器镜像加速器配置地址\u003chttps://dev.aliyun.com/search.html\u003e  登录管理中心获取个人专属加速器地址\n\n```shell\nsudo mkdir -p /etc/docker\nsudo tee /etc/docker/daemon.json \u003c\u003c-'EOF'\n{\n  \"registry-mirrors\": [\"https://3csy84rx.mirror.aliyuncs.com\"]\n}\nEOF\nsudo systemctl daemon-reload\nsudo systemctl restart docker\n```\n\n\n\n\n\n### 05. 生成TLS证书和秘钥\n\n#### 5.1 Kubernetes 集群所需证书\n\n\u003e `ca`证书为集群admin证书。\n\u003e\n\u003e `etcd`证书为etcd集群使用。\n\u003e\n\u003e `shinezone`证书为Harbor使用。\n\n|      CA\u0026Key       | etcd | api-server | proxy | kebectl | Calico | harbor |\n| :---------------: | :--: | :--------: | :---: | :-----: | :----: | :----: |\n|      ca.csr       |  √   |     √      |   √   |    √    |   √    |        |\n|      ca.pem       |  √   |     √      |   √   |    √    |   √    |        |\n|    ca-key.pem     |  √   |     √      |   √   |    √    |   √    |        |\n|      ca.pem       |  √   |            |       |         |        |        |\n|     etcd.csr      |  √   |            |       |         |        |        |\n|   etcd-key.pem    |  √   |            |       |         |        |        |\n| shinezone.com.crt |      |            |       |         |        |   √    |\n| shinezone.com.key |      |            |       |         |        |   √    |\n\n\n\n#### 5.2 安装CFSSL\n\n- K8S01执行\n\n```\nyum install wget -y\nwget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64\nwget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64\nwget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64\nchmod +x cfssl_linux-amd64\nmv cfssl_linux-amd64 /usr/local/bin/cfssl\nchmod +x cfssljson_linux-amd64\nmv cfssljson_linux-amd64 /usr/local/bin/cfssljson\nchmod +x cfssl-certinfo_linux-amd64\nmv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo\nexport PATH=/usr/local/bin:$PATH\n```\n\n#### 5.3 创建CA文件,生成etcd证书\n\n```\nmkdir /root/ssl\ncd /root/ssl\ncat \u003e  ca-config.json \u003c\u003cEOF\n{\n\"signing\": {\n\"default\": {\n  \"expiry\": \"8760h\"\n},\n\"profiles\": {\n  \"kubernetes-Soulmate\": {\n    \"usages\": [\n        \"signing\",\n        \"key encipherment\",\n        \"server auth\",\n        \"client auth\"\n    ],\n    \"expiry\": \"8760h\"\n  }\n}\n}\n}\nEOF\n\ncat \u003e  ca-csr.json \u003c\u003cEOF\n{\n\"CN\": \"kubernetes-Soulmate\",\n\"key\": {\n\"algo\": \"rsa\",\n\"size\": 2048\n},\n\"names\": [\n{\n  \"C\": \"CN\",\n  \"ST\": \"shanghai\",\n  \"L\": \"shanghai\",\n  \"O\": \"k8s\",\n  \"OU\": \"System\"\n}\n]\n}\nEOF\n\ncfssl gencert -initca ca-csr.json | cfssljson -bare ca\n\n#hosts项需要加入所有etcd集群节点，建议将所有node也加入，便于扩容etcd集群。\ncat \u003e etcd-csr.json \u003c\u003cEOF\n{\n  \"CN\": \"etcd\",\n  \"hosts\": [\n    \"127.0.0.1\",\n    \"172.16.1.50\",\n    \"172.16.1.51\",\n    \"172.16.1.52\",\n    \"172.16.1.53\",\n    \"172.16.1.54\"\n  ],\n  \"key\": {\n    \"algo\": \"rsa\",\n    \"size\": 2048\n  },\n  \"names\": [\n    {\n      \"C\": \"CN\",\n      \"ST\": \"shanghai\",\n      \"L\": \"shanghai\",\n      \"O\": \"k8s\",\n      \"OU\": \"System\"\n    }\n  ]\n}\nEOF\n\ncfssl gencert -ca=ca.pem \\\n  -ca-key=ca-key.pem \\\n  -config=ca-config.json \\\n  -profile=kubernetes-Soulmate etcd-csr.json | cfssljson -bare etcd\n```\n\n字段说明\n\n- 如果 hosts 字段不为空则需要指定授权使用该证书的?**IP 或域名列表**\n\n- `ca-config.json`：可以定义多个 profiles，分别指定不同的过期时间、使用场景等参数；后续在签名证书时使用某个 profile；\n- `signing`：表示该证书可用于签名其它证书；生成的 ca.pem 证书中?`CA=TRUE`；\n- `server auth`：表示client可以用该 CA 对server提供的证书进行验证；\n- `client auth`：表示server可以用该CA对client提供的证书进行验证；\n- \"CN\"：`Common Name`，kube-apiserver 从证书中提取该字段作为请求的用户名 (User Name)；浏览器使用该字段验证网站是否合法；\n- \"O\"：`Organization`，kube-apiserver 从证书中提取该字段作为请求用户所属的组 (Group)；\n\n\n\n#### 5.4 分发证书到所有节点\n\n- Master01执行\n\n\u003e 本集群所有所有节点安装etcd，因此需要证书分发所有节点。\n\n```shell\nmkdir -p /etc/etcd/ssl\ncp etcd.pem etcd-key.pem ca.pem /etc/etcd/ssl/\nscp -r /etc/etcd/ master02:/etc/\nscp -r /etc/etcd/ node01:/etc/\nscp -r /etc/etcd/ node02:/etc/\nscp -r /etc/etcd/ node03:/etc/\n```\n\n### 06. 安装配置etcd\n\n#### 6.1 安装etcd\n\n- 所有节点执行\n\n```shell\nyum install etcd -y   \nmkdir -p /var/lib/etcd\n```\n\n#### 6.2 配置etcd\n\nmaster01的`etcd.service`\n\n```shell\ncat \u003c\u003cEOF \u003e/usr/lib/systemd/system/etcd.service\n[Unit]\nDescription=Etcd Server\nAfter=network.target\nAfter=network-online.target\nWants=network-online.target\nDocumentation=https://github.com/coreos\n\n[Service]\nType=notify\nWorkingDirectory=/var/lib/etcd/\nExecStart=/usr/bin/etcd \\\n  --name k8s01 \\\n  --cert-file=/etc/etcd/ssl/etcd.pem \\\n  --key-file=/etc/etcd/ssl/etcd-key.pem \\\n  --peer-cert-file=/etc/etcd/ssl/etcd.pem \\\n  --peer-key-file=/etc/etcd/ssl/etcd-key.pem \\\n  --trusted-ca-file=/etc/etcd/ssl/ca.pem \\\n  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \\\n  --initial-advertise-peer-urls https://172.16.1.50:2380 \\\n  --listen-peer-urls https://172.16.1.50:2380 \\\n  --listen-client-urls https://172.16.1.50:2379,http://127.0.0.1:2379 \\\n  --advertise-client-urls https://172.16.1.50:2379 \\\n  --initial-cluster-token etcd-cluster-0 \\\n  --initial-cluster k8s01=https://172.16.1.50:2380,k8s02=https://172.16.1.51:2380,k8s03=https://172.16.1.52:2380,k8s04=https://172.16.1.53:2380,k8s05=https://172.16.1.54:2380 \\\n  --initial-cluster-state new \\\n  --data-dir=/var/lib/etcd\nRestart=on-failure\nRestartSec=5\nLimitNOFILE=65536\n\n[Install]\nWantedBy=multi-user.target\nEOF\n```\n\nmaster02的`etcd.service`\n\n```shell\ncat \u003c\u003cEOF \u003e/usr/lib/systemd/system/etcd.service\n[Unit]\nDescription=Etcd Server\nAfter=network.target\nAfter=network-online.target\nWants=network-online.target\nDocumentation=https://github.com/coreos\n\n[Service]\nType=notify\nWorkingDirectory=/var/lib/etcd/\nExecStart=/usr/bin/etcd \\\n  --name k8s02 \\\n  --cert-file=/etc/etcd/ssl/etcd.pem \\\n  --key-file=/etc/etcd/ssl/etcd-key.pem \\\n  --peer-cert-file=/etc/etcd/ssl/etcd.pem \\\n  --peer-key-file=/etc/etcd/ssl/etcd-key.pem \\\n  --trusted-ca-file=/etc/etcd/ssl/ca.pem \\\n  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \\\n  --initial-advertise-peer-urls https://172.16.1.51:2380 \\\n  --listen-peer-urls https://172.16.1.51:2380 \\\n  --listen-client-urls https://172.16.1.51:2379,http://127.0.0.1:2379 \\\n  --advertise-client-urls https://172.16.1.51:2379 \\\n  --initial-cluster-token etcd-cluster-0 \\\n  --initial-cluster k8s01=https://172.16.1.50:2380,k8s02=https://172.16.1.51:2380,k8s03=https://172.16.1.52:2380,k8s04=https://172.16.1.53:2380,k8s05=https://172.16.1.54:2380 \\\n  --initial-cluster-state new \\\n  --data-dir=/var/lib/etcd\nRestart=on-failure\nRestartSec=5\nLimitNOFILE=65536\n\n[Install]\nWantedBy=multi-user.target\nEOF\n```\n\nnode01的`etcd.service`\n\n```shell\ncat \u003c\u003cEOF \u003e/usr/lib/systemd/system/etcd.service\n[Unit]\nDescription=Etcd Server\nAfter=network.target\nAfter=network-online.target\nWants=network-online.target\nDocumentation=https://github.com/coreos\n\n[Service]\nType=notify\nWorkingDirectory=/var/lib/etcd/\nExecStart=/usr/bin/etcd \\\n  --name k8s03 \\\n  --cert-file=/etc/etcd/ssl/etcd.pem \\\n  --key-file=/etc/etcd/ssl/etcd-key.pem \\\n  --peer-cert-file=/etc/etcd/ssl/etcd.pem \\\n  --peer-key-file=/etc/etcd/ssl/etcd-key.pem \\\n  --trusted-ca-file=/etc/etcd/ssl/ca.pem \\\n  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \\\n  --initial-advertise-peer-urls https://172.16.1.52:2380 \\\n  --listen-peer-urls https://172.16.1.52:2380 \\\n  --listen-client-urls https://172.16.1.52:2379,http://127.0.0.1:2379 \\\n  --advertise-client-urls https://172.16.1.52:2379 \\\n  --initial-cluster-token etcd-cluster-0 \\\n  --initial-cluster k8s01=https://172.16.1.50:2380,k8s02=https://172.16.1.51:2380,k8s03=https://172.16.1.52:2380,k8s04=https://172.16.1.53:2380,k8s05=https://172.16.1.54:2380 \\\n  --initial-cluster-state new \\\n  --data-dir=/var/lib/etcd\nRestart=on-failure\nRestartSec=5\nLimitNOFILE=65536\n\n[Install]\nWantedBy=multi-user.target\nEOF\n```\n\nnode02的`etcd.service`\n\n```shell\ncat \u003c\u003cEOF \u003e/usr/lib/systemd/system/etcd.service\n[Unit]\nDescription=Etcd Server\nAfter=network.target\nAfter=network-online.target\nWants=network-online.target\nDocumentation=https://github.com/coreos\n\n[Service]\nType=notify\nWorkingDirectory=/var/lib/etcd/\nExecStart=/usr/bin/etcd \\\n  --name k8s04 \\\n  --cert-file=/etc/etcd/ssl/etcd.pem \\\n  --key-file=/etc/etcd/ssl/etcd-key.pem \\\n  --peer-cert-file=/etc/etcd/ssl/etcd.pem \\\n  --peer-key-file=/etc/etcd/ssl/etcd-key.pem \\\n  --trusted-ca-file=/etc/etcd/ssl/ca.pem \\\n  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \\\n  --initial-advertise-peer-urls https://172.16.1.53:2380 \\\n  --listen-peer-urls https://172.16.1.53:2380 \\\n  --listen-client-urls https://172.16.1.53:2379,http://127.0.0.1:2379 \\\n  --advertise-client-urls https://172.16.1.53:2379 \\\n  --initial-cluster-token etcd-cluster-0 \\\n  --initial-cluster k8s01=https://172.16.1.50:2380,k8s02=https://172.16.1.51:2380,k8s03=https://172.16.1.52:2380,k8s04=https://172.16.1.53:2380,k8s05=https://172.16.1.54:2380 \\\n  --initial-cluster-state new \\\n  --data-dir=/var/lib/etcd\nRestart=on-failure\nRestartSec=5\nLimitNOFILE=65536\n\n[Install]\nWantedBy=multi-user.target\nEOF\n```\n\nnode03的`etcd.service`\n\n```shell\ncat \u003c\u003cEOF \u003e/usr/lib/systemd/system/etcd.service\n[Unit]\nDescription=Etcd Server\nAfter=network.target\nAfter=network-online.target\nWants=network-online.target\nDocumentation=https://github.com/coreos\n\n[Service]\nType=notify\nWorkingDirectory=/var/lib/etcd/\nExecStart=/usr/bin/etcd \\\n  --name k8s05 \\\n  --cert-file=/etc/etcd/ssl/etcd.pem \\\n  --key-file=/etc/etcd/ssl/etcd-key.pem \\\n  --peer-cert-file=/etc/etcd/ssl/etcd.pem \\\n  --peer-key-file=/etc/etcd/ssl/etcd-key.pem \\\n  --trusted-ca-file=/etc/etcd/ssl/ca.pem \\\n  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \\\n  --initial-advertise-peer-urls https://172.16.1.54:2380 \\\n  --listen-peer-urls https://172.16.1.54:2380 \\\n  --listen-client-urls https://172.16.1.54:2379,http://127.0.0.1:2379 \\\n  --advertise-client-urls https://172.16.1.54:2379 \\\n  --initial-cluster-token etcd-cluster-0 \\\n  --initial-cluster k8s01=https://172.16.1.50:2380,k8s02=https://172.16.1.51:2380,k8s03=https://172.16.1.52:2380,k8s04=https://172.16.1.53:2380,k8s05=https://172.16.1.54:2380 \\\n  --initial-cluster-state new \\\n  --data-dir=/var/lib/etcd\nRestart=on-failure\nRestartSec=5\nLimitNOFILE=65536\n\n[Install]\nWantedBy=multi-user.target\nEOF\n```\n\n#### 6.3  启动etcd\n\n```shell\n systemctl daemon-reload\n systemctl enable etcd\n systemctl start etcd\n systemctl status etcd\n```\n\n#### 6.4 集群状态检查(维护)\n\n\u003e 使用v3版本API\n\n```shell\n$ echo \"export ETCDCTL_API=3\" \u003e\u003e/etc/profile  \u0026\u0026 source /etc/profile\n$ etcdctl version\netcdctl version: 3.2.18\nAPI version: 3.2\n```\n\n\u003e 查看集群健康状态\n\n```shell\n$ etcdctl --endpoints=https://172.16.1.50:2379,https://172.16.1.51:2379,https://172.16.1.52:2379,https://172.16.1.53:2379,https://172.16.1.54:2379 --cacert=/etc/etcd/ssl/ca.pem   --cert=/etc/etcd/ssl/etcd.pem   --key=/etc/etcd/ssl/etcd-key.pem   endpoint health\n//输出信息如下：\nhttps://172.16.1.54:2379 is healthy: successfully committed proposal: took = 1.911784ms\nhttps://172.16.1.50:2379 is healthy: successfully committed proposal: took = 2.648385ms\nhttps://172.16.1.52:2379 is healthy: successfully committed proposal: took = 3.472479ms\nhttps://172.16.1.51:2379 is healthy: successfully committed proposal: took = 2.850887ms\nhttps://172.16.1.53:2379 is healthy: successfully committed proposal: took = 3.711259ms\n```\n\n\u003e 查询所有key\n\n```shell\n$ etcdctl --endpoints=https://172.16.1.50:2379,https://172.16.1.51:2379,https://172.16.1.52:2379,https://172.16.1.53:2379,https://172.16.1.54:2379 --cacert=/etc/etcd/ssl/ca.pem   --cert=/etc/etcd/ssl/etcd.pem   --key=/etc/etcd/ssl/etcd-key.pem    get / --prefix --keys-only\n\n// kubeadm初始化之前是没有任何信息的，初始化完成后查询得到的信息如：\n/registry/apiregistration.k8s.io/apiservices/v1.\n/registry/apiregistration.k8s.io/apiservices/v1.apps\n/registry/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io\n/registry/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io\n/registry/apiregistration.k8s.io/apiservices/v1.autoscaling\n/registry/apiregistration.k8s.io/apiservices/v1.batch\n........................\n```\n\n\u003e 清除`所有/指定`key(**生成环境慎用**)\n\u003e\n\u003e 线上环境如有k8s组件出现问题,需要针对特定问题key进行清除操作。\n\n```shell\n etcdctl --endpoints=https://172.16.1.50:2379,https://172.16.1.51:2379,https://172.16.1.52:2379,https://172.16.1.53:2379,https://172.16.1.54:2379 --cacert=/etc/etcd/ssl/ca.pem   --cert=/etc/etcd/ssl/etcd.pem   --key=/etc/etcd/ssl/etcd-key.pem    del /registry/apiregistration.k8s.io/apiservices/v1.batch --prefix \n```\n\n\n\n\n\n### 07 安装配置Kubeadm\n\n\u003e 集群所有节点安装`kebelet` `kubeadm` `kebectl`\n\n```shell\nyum install -y kubelet kubeadm kubectl\n###暂不启动，未初始化前启动也会报错\nsystemctl enable kubelet   \n```\n\n#### 7.1 配置kubelet\n\n\u003e 所有节点修改，kubelet类似Agent，每台Node上必须要安装的组件\n\n```shell\n// 所有机器执行\nsed -i s#systemd#cgroupfs#g /etc/systemd/system/kubelet.service.d/10-kubeadm.conf\necho 'Environment=\"KUBELET_EXTRA_ARGS=--v=2 --fail-swap-on=false --pod-infra-container-image=harbor.domain.com/shinezonetest/pause-amd64:3.1\"' \u003e\u003e  /etc/systemd/system/kubelet.service.d/10-kubeadm.conf\n```\n\n#### 7.2 加载配置文件\n\n```shell\nsystemctl daemon-reload\nsystemctl enable kubelet\n```\n\n\n\n### 08 Master节点高可用\n\n\u003e Master节点高可用使用keepalived，也可以使用商业ELB ALB SLB，或自建N'gin'x负载均衡。\n\n#### 8.1 安装keepalived\n\n- master节点执行\n\n```shell\nyum install -y keepalived\nsystemctl enable keepalived\n```\n\n#### 8.2 配置Keepalived\n\n\u003e 注意修改`interface`网卡名，`priority`权重值，`unicast_peer`\n\n Master01 配置文件\n\n```shell\ncat \u003c\u003cEOF \u003e/etc/keepalived/keepalived.conf\nglobal_defs {\n   router_id LVS_k8s\n}\n\nvrrp_script CheckK8sMaster {\n    script \"curl -k https://172.16.1.49:6443\"    #VIP Address\n    interval 3\n    timeout 9\n    fall 2\n    rise 2\n}\n\nvrrp_instance VI_1 {\n    state MASTER\n    interface ens32       #Your Network Interface Name\n    virtual_router_id 61\n    priority 120          #权重，数字大的为主，数字一样则选择第一台为Master\n    advert_int 1\n    mcast_src_ip 172.16.1.50  #local IP\n    nopreempt\n    authentication {\n        auth_type PASS\n        auth_pass sqP05dQgMSlzrxHj\n    }\n    unicast_peer {\n        #172.16.1.50\n        172.16.1.51    #另外一台masterIP\n    }\n    virtual_ipaddress {\n        172.16.1.49/24    # VIP\n    }\n    track_script {\n        CheckK8sMaster\n    }\n\n}\nEOF\n```\n\nMaster02配置文件\n\n```shell\ncat \u003c\u003cEOF \u003e/etc/keepalived/keepalived.conf\nglobal_defs {\n   router_id LVS_k8s\n}\n\nvrrp_script CheckK8sMaster {\n    script \"curl -k https://172.16.1.49:6443\"\n    interval 3\n    timeout 9\n    fall 2\n    rise 2\n}\n\nvrrp_instance VI_1 {\n    state MASTER\n    interface ens32\n    virtual_router_id 61\n    priority 110\n    advert_int 1\n    mcast_src_ip 172.16.1.51\n    nopreempt\n    authentication {\n        auth_type PASS\n        auth_pass sqP05dQgMSlzrxHj\n    }\n    unicast_peer {\n        172.16.1.50\n        #172.16.1.51\n    }\n    virtual_ipaddress {\n        172.16.1.49/24\n    }\n    track_script {\n        CheckK8sMaster\n    }\n\n}\nEOF\n```\n\n#### 8.3 启动Keepalived\n\n```shell\nsed s#'KEEPALIVED_OPTIONS=\"-D\"'#'KEEPALIVED_OPTIONS=\"-D -d -S 0\"'#g /etc/sysconfig/keepalived -i   //配置日志文件\necho \"local0.*    /var/log/keepalived.log\" \u003e\u003e /etc/rsyslog.conf\nservice rsyslog restart\nsystemctl start keepalived\nsystemctl status keepalived\n```\n\n#### 8.4 测试Keepalived可用性\n\n\u003e 测试：关闭一台Master机器，看IP是否漂移，API是否可用。\n\n```shell\n//确认VIP在Master01上\n$ ip a | grep inet |grep \"172.16\"\n    inet 172.16.1.50/21 brd 172.16.7.255 scope global ens32\n    inet 172.16.1.49/24 scope global ens32   //VIP\n    \n// 关闭Master01机器，确认VIP是否飘逸\n$ ip a |grep inet |grep \"172.16\"\n    inet 172.16.1.51/21 brd 172.16.7.255 scope global ens32\n    inet 172.16.1.49/24 scope global ens32  //可以看到瞬间就偏移到了Master02机器上\n    \n// 确认APi服务可用性,也可在下步初始化集群后测试，直接访问dashboard看效果。\n$ curl https://your_dashboard_address/ -I\nHTTP/1.1 200 OK\nServer: nginx/1.10.0\nDate: Wed, 06 Jun 2018 05:58:22 GMT\nContent-Type: text/html; charset=utf-8\nContent-Length: 990\nConnection: keep-alive\nAccept-Ranges: bytes\nCache-Control: no-store\nLast-Modified: Tue, 13 Feb 2018 11:17:03 GMT\n```\n\n### 09 初始化集群\n\n\u003e Master机器添加初始化配置文件, V1.13版本后version版本使用v1beta1 语法变化很大\n\n```shell\n#cat EOF方式格式会乱掉，直接vim 复制粘贴进去，保持格式不变\n#vim config.yaml   :set paste  \n\napiVersion: kubeproxy.config.k8s.io/v1alpha1\nkind: KubeProxyConfiguration\nmode: \"ipvs\"  #使用IPVS模式，非iptables\n---\napiVersion: kubeadm.k8s.io/v1beta1  #v1beta1版本,非v1alpha版本，语法会有变化\ncertificatesDir: /etc/kubernetes/pki   \nclusterName: kubernetes\ncontrolPlaneEndpoint: 172.16.1.49:6443  #Keeplived 虚拟IP地址\ncontrollerManager: {}\ndns:\n  type: CoreDNS  #默认DNS：CoreDNS\nimageRepository: k8s.gcr.io   #官方镜像\n#imageRepository: harbor.domain.com/k8s   #可修改为自己的Har镜像库\nkind: ClusterConfiguration\nkubernetesVersion: v1.13.3   #K8S版本\nnetworking:\n  dnsDomain: cluster.local\n  serviceSubnet: 10.96.0.0/12  #SVC网络段 \n  podSubnet: 100.64.0.0/10     #POD网络段\napiServer:\n        certSANs:  \n        - 172.16.1.50\n        - 172.16.1.51\n        - 172.16.1.52\n        - 172.16.1.53\n        - 172.16.1.54\n        extraArgs: \n           etcd-cafile: /etc/etcd/ssl/ca.pem\n           etcd-certfile: /etc/etcd/ssl/etcd.pem\n           etcd-keyfile: /etc/etcd/ssl/etcd-key.pem\netcd:  #使用外接etcd高可用\n    external:\n        caFile: /etc/etcd/ssl/ca.pem\n        certFile: /etc/etcd/ssl/etcd.pem\n        keyFile: /etc/etcd/ssl/etcd-key.pem\n        endpoints:\n        - https://172.16.1.50:2379\n        - https://172.16.1.51:2379\n        - https://172.16.1.52:2379\n        - https://172.16.1.53:2379\n        - https://172.16.1.54:2379\n```\n\n\n\n#### 9.1 初始化集群\n\n- Master01操作\n\n```shell\nkubeadm init --config config.yaml  \n\n# 初始化过程中可以journalctl -u kubelet -f查看log，可能会报错cni网络问题，因为我们指定了网络，calico还没安装，所以报错，等calico安装完成就好了\n```\n\n初始化成功后输出信息：\n\n```shell\nYour Kubernetes master has initialized successfully!\n\nTo start using your cluster, you need to run the following as a regular user:\n\n  mkdir -p $HOME/.kube\n  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config\n  sudo chown $(id -u):$(id -g) $HOME/.kube/config\n\nYou should now deploy a pod network to the cluster.\nRun \"kubectl apply -f [podnetwork].yaml\" with one of the options listed at:\n  https://kubernetes.io/docs/concepts/cluster-administration/addons/\n\nYou can now join any number of machines by running the following on each node\nas root:\n\n  kubeadm join 172.16.1.49:6443 --token b99a00.a144ef80536d4344 --discovery-token-ca-cert-hash sha256:8c\n```\n\n初始化完成后执行相关命令\n\n```shell\nmkdir -p $HOME/.kube\nsudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config\nsudo chown $(id -u):$(id -g) $HOME/.kube/config\n```\n\n查看状态\n\n```shell\n$ kubectl get cs\nNAME                 STATUS    MESSAGE             ERROR\ncontroller-manager   Healthy   ok                  \nscheduler            Healthy   ok                  \netcd-0               Healthy   {\"health\":\"true\"}   \netcd-2               Healthy   {\"health\":\"true\"}   \netcd-3               Healthy   {\"health\":\"true\"}   \netcd-4               Healthy   {\"health\":\"true\"}   \netcd-1               Healthy   {\"health\":\"true\"}\n```\n\n\u003e 注意，多台Master的主机都要进行初始化，第二台master初始化也需要config.yaml文件，前提需要将\n\n- Master02操作\n\n```shell\n#分发kebeadm生成的证书文件和密码文件\n#每台Master机器的证书和密码文件都是相同的，有新的Master加入，直接分发初始化即可。\n\n$ scp -r /etc/kubernetes/pki  master02:/etc/kubernetes/\n//然后初始化，执行命令\n$ kubeadm init --config config.yaml \n$ mkdir -p $HOME/.kube\n$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config\n$ sudo chown $(id -u):$(id -g) $HOME/.kube/config\n```\n\n#### 9.2 初始化失败解决办法\n\n```shell\n$ kubeadm reset\n// 或者删除相关文件和images\n$ rm -rf /etc/kubernetes/*.conf\n$ rm -rf /etc/kubernetes/manifests/*.yaml\n$ docker ps -a |awk '{print $1}' |xargs docker rm -f\n$ systemctl  stop kubelet\n```\n\n再次初始化前需要执行清除etcd所有数据的操作。\n\n```shell\n $ etcdctl --endpoints=https://172.16.1.50:2379,https://172.16.1.51:2379,https://172.16.1.52:2379,https://172.16.1.53:2379,https://172.16.1.54:2379 --cacert=/etc/etcd/ssl/ca.pem   --cert=/etc/etcd/ssl/etcd.pem   --key=/etc/etcd/ssl/etcd-key.pem    del / --prefix\n```\n\n\n\n### 10. 部署网络组件\n\n\u003e Flanneld和Calico都是解决容器通信组件，任选一个即可，这里使用DaemonSet部署，只在Master01执行\n\n#### 10.1 Calico组件部署（二选一）\n\n```shell\nwget https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml\nwget  https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml\n#修改calico.yaml里image镜像路径部分,已推内网harbor,有梯子的可以默认直接apply yaml文件\n$ grep image calico.yaml \n\n        - image: harbor.domain.com/shinezonetest/calico-typha:1.0\n          image: harbor.domain.com/shinezonetest/calico-node:1.0\n          image: harbor.domain.com/shinezonetest/calico-cni:1.0\n          \n$ kubectl apply -f rbac-kdd.yaml\n$ kubectl apply -f calico.yaml\n```\n\n#### 10.2 Flannel组件部署(二选一)\n\n- 目前都是使用Calico网络，二选一即可，Flannel不需要部署\n\n```shell\n$ mkdir -p /run/flannel/\n$ cat \u003e/run/flannel/subnet.env \u003c\u003cEOF\nFLANNEL_NETWORK=10.244.0.0/16\nFLANNEL_SUBNET=10.244.0.1/24\nFLANNEL_MTU=1450\nFLANNEL_IPMASQ=true\nEOF\n$ wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml\n  #版本信息：quay.io/coreos/flannel:v0.10.0-amd64\n$ kubectl create -f  kube-flannel.yml\n```\n\n#### 10.3 查看集群状态\n\n\u003e 集群中组件通信都要基于Calico/Flanneld，需要等到网络组件启动后才可以确认。\n\n```shell\n$ kubectl get nodes\nNAME             STATUS   ROLES    AGE     VERSION\nk8s01-master01   Ready    master   22h     v1.13.3\nk8s01-master02   Ready    master   6h47m   v1.13.3\nk8s01-node01     Ready    \u003cnone\u003e   22h     v1.13.3\nk8s01-node02     Ready    \u003cnone\u003e   22h     v1.13.3\nk8s01-node03     Ready    \u003cnone\u003e   22h     v1.13.3\n\n$ kubectl get pods -n kube-system\nNAME                                     READY   STATUS    RESTARTS   AGE\ncalico-node-bt2gq                        2/2     Running   0          22h\ncalico-node-gln4j                        2/2     Running   0          22h\ncalico-node-s4xj7                        2/2     Running   0          6h47m\ncalico-node-ttj6q                        2/2     Running   0          22h\ncalico-node-wsc7s                        2/2     Running   0          22h\ncoredns-86c58d9df4-n8tvw                 1/1     Running   0          22h\ncoredns-86c58d9df4-qlq2n                 1/1     Running   0          22h\nkube-apiserver-k8s01-master01            1/1     Running   0          22h\nkube-apiserver-k8s01-master02            1/1     Running   0          6h47m\nkube-controller-manager-k8s01-master01   1/1     Running   0          22h\nkube-controller-manager-k8s01-master02   1/1     Running   0          6h47m\nkube-proxy-89f5j                         1/1     Running   0          6h47m\nkube-proxy-fnc9t                         1/1     Running   0          22h\nkube-proxy-gf4xt                         1/1     Running   0          22h\nkube-proxy-j7ltr                         1/1     Running   0          22h\nkube-proxy-z4tbz                         1/1     Running   0          22h\nkube-scheduler-k8s01-master01            1/1     Running   0          22h\nkube-scheduler-k8s01-master02            1/1     Running   0          6h47m\n```\n\n#### 10.4 测试Calico DNS网络问题\n\n\u003e DNS使用CoreDNS，是集群初始化默认启用的，Node之间通信是依赖Calico的\n\n**坑一，Busybox镜像Bug**\n\n```shell\n$ kubectl run curl --image=radial/busyboxplus:curl   #启动busybox\n$ nslookup kubernetes  #非全路径则无法解析\nServer:    10.96.0.10\nAddress 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local  \nnslookup: can‘t resolve kubernetes\n\n$ nslookup kubernetes.default.svc.cluster.local     #nslookup解析全路径就可以解析\nServer:    10.96.0.10 \nAddress 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local\n\nName:      kubernetes.default.svc.cluster.local\nAddress 1: 10.96.0.1 kubernetes.default.svc.cluster.local\n\n# 使用DNSTOOLS工具\n$ kubectl run -it --rm --image=infoblox/dnstools dns-client\ndnstools# ping kubernetes   #确认可以解析内部\nPING kubernetes (10.96.0.1): 56 data bytes\n64 bytes from 10.96.0.1: seq=0 ttl=64 time=0.042 ms\n\ndnstools# ping www.qq.com   #确认pod可以访问外网\n```\n\n**坑二，pod请求外网，DNS报错io/timeout**\n\n\u003e 简单记录下这里的排查思路。\n\u003e\n\u003e 当我整个集群都部署完成，监控就位的时候，发现Alertmanager无法发送邮件，然后确认Alert信息从Prometheus递交给了Alertmanager。\n\u003e\n\u003e 查看POD日志，POD日志显示无法链接到SMTP服务器，这时候意识到可能DNS解析有问题，然后就启动了busybox测试DNS，What Fuck!  我CoreDNS 解析集群内部的SVC都解析不到，那我Tarefik怎么转发工作的。\n\u003e\n\u003e 最后排查半天原来是busybox镜像有bug，nslookup 全路径就可以解析，如以上**坑一** ，使用dnstools即可解决，内网通了很开心，然后外网还是不通，其余POD没有报错日志，唯有DNS一直报错 timeout。\n\u003e\n\u003e 到了这个时候一直认为是CoreDNS问题，先去了解了下K8S集群中DNS工作原理，参考链接：https://www.simpleapples.com/2018/07/15/solving-kubernetes-dns-problem/\n\u003e\n\u003e DNS解析外部地址需要借助上层DNS，CoreDNS里面也有配置依赖Node DNS，默认/etc/resolv.conf 里面的nameserver，接着使用`kuberctl exec \u003cdns_pod_name\u003e cat /etc/resolv.conf -n kube-system ` 查看我DNS确实依赖了我Node的DNS呀 参考文档：https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/，什么情况，为什么还不能解析外网。\n\u003e\n\u003e 然后将CoreDNS换成了Kube-DNS，还是不行，各种尝试，reset集群，重启机器，耗时了将近一天时间，集群也重新建立了多次，最后开始怀疑自己排查点好像出错了。\n\u003e\n\u003e 开始意识到应该不是DNS问题，可能是我的Calico网络有问题，但是我POD之间又是可以通信，很奇怪，终于皇天不负有心人，我开始将重心搜素放到了`pod无法上外网`让我找到了以下https://blog.csdn.net/kozazyh/article/details/80595782，这里我一直认为我kube-proxy使用的IPVS模块，不会和Iptable有关系，直到我加了2条POD和SVC的iptable NAT规则，发现网络居然通了。。。。 我擦，这可能不是官方的正确解法，可能是我Calico网络没配置好，或者我IPVS这块理解有问题，还是要多看官方文档。\n\n```shell\n#解决办法：\n\ncat config.yaml   #你的SVC和POD地址段，添加2条规则\n$ ps uax |grep kube-proxy  //确保加载了--cluster-cidr\n$ iptables -nvL |grep FORWARD  //查看FORWARD 是否为ACCEPT \n$ iptables -P FORWARD ACCEPT  //开启IPtableS转发ACCEPT\n$ sysctl -a | grep ip_forward //确认系统ip_forward开启\n\n$ /sbin/iptables -t nat -I POSTROUTING -s  100.64.0.0/10 -j MASQUERADE\n$ /sbin/iptables -t nat -I POSTROUTING -s  10.96.0.0/12 -j MASQUERADE\n\n$ kubectl run -it --rm --image=infoblox/dnstools dns-client #测试\ndnstools# ping qq.com  #可以ping通\n```\n\n\n\n### 11. 部署Dashboard\n\n- 暴露端口：Node Port：30000\n\n```yaml\n$ cat \u003c\u003cEOF \u003e kubernetes-dashboard.yaml\n# Copyright 2017 The Kubernetes Authors.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# Configuration to deploy release version of the Dashboard UI compatible with\n# Kubernetes 1.8.\n#\n# Example usage: kubectl create -f \u003cthis_file\u003e\n\n# ------------------- Dashboard Secret ------------------- #\n\napiVersion: v1\nkind: Secret\nmetadata:\n  labels:\n    k8s-app: kubernetes-dashboard\n  name: kubernetes-dashboard-certs\n  namespace: kube-system\ntype: Opaque\n\n---\n# ------------------- Dashboard Service Account ------------------- #\n\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n  labels:\n    k8s-app: kubernetes-dashboard\n  name: kubernetes-dashboard\n  namespace: kube-system\n\n---\n# ------------------- Dashboard Role \u0026 Role Binding ------------------- #\n\nkind: Role\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n  name: kubernetes-dashboard-minimal\n  namespace: kube-system\nrules:\n  # Allow Dashboard to create 'kubernetes-dashboard-key-holder' secret.\n- apiGroups: [\"\"]\n  resources: [\"secrets\"]\n  verbs: [\"create\"]\n  # Allow Dashboard to create 'kubernetes-dashboard-settings' config map.\n- apiGroups: [\"\"]\n  resources: [\"configmaps\"]\n  verbs: [\"create\"]\n  # Allow Dashboard to get, update and delete Dashboard exclusive secrets.\n- apiGroups: [\"\"]\n  resources: [\"secrets\"]\n  resourceNames: [\"kubernetes-dashboard-key-holder\", \"kubernetes-dashboard-certs\"]\n  verbs: [\"get\", \"update\", \"delete\"]\n  # Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.\n- apiGroups: [\"\"]\n  resources: [\"configmaps\"]\n  resourceNames: [\"kubernetes-dashboard-settings\"]\n  verbs: [\"get\", \"update\"]\n  # Allow Dashboard to get metrics from heapster.\n- apiGroups: [\"\"]\n  resources: [\"services\"]\n  resourceNames: [\"heapster\"]\n  verbs: [\"proxy\"]\n- apiGroups: [\"\"]\n  resources: [\"services/proxy\"]\n  resourceNames: [\"heapster\", \"http:heapster:\", \"https:heapster:\"]\n  verbs: [\"get\"]\n\n---\napiVersion: rbac.authorization.k8s.io/v1\nkind: RoleBinding\nmetadata:\n  name: kubernetes-dashboard-minimal\n  namespace: kube-system\nroleRef:\n  apiGroup: rbac.authorization.k8s.io\n  kind: Role\n  name: kubernetes-dashboard-minimal\nsubjects:\n- kind: ServiceAccount\n  name: kubernetes-dashboard\n  namespace: kube-system\n\n---\n# ------------------- Dashboard Deployment ------------------- #\n\nkind: Deployment\napiVersion: apps/v1beta2\nmetadata:\n  labels:\n    k8s-app: kubernetes-dashboard\n  name: kubernetes-dashboard\n  namespace: kube-system\nspec:\n  replicas: 1\n  revisionHistoryLimit: 10\n  selector:\n    matchLabels:\n      k8s-app: kubernetes-dashboard\n  template:\n    metadata:\n      labels:\n        k8s-app: kubernetes-dashboard\n    spec:\n      containers:\n      - name: kubernetes-dashboard\n        #image: harbor.shinezone.com/shinezonetest/kubernetes-dashboard-amd64:v1.8.3\n        image: k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.0\n        ports:\n        - containerPort: 8443\n          protocol: TCP\n        args:\n          - --auto-generate-certificates\n          # Uncomment the following line to manually specify Kubernetes API server Host\n          # If not specified, Dashboard will attempt to auto discover the API server and connect\n          # to it. Uncomment only if the default does not work.\n          # - --apiserver-host=http://my-address:port\n          #- --authentication-mode=basic\n        volumeMounts:\n        - name: kubernetes-dashboard-certs\n          mountPath: /certs\n          # Create on-disk volume to store exec logs\n        - mountPath: /tmp\n          name: tmp-volume\n        livenessProbe:\n          httpGet:\n            scheme: HTTPS\n            path: /\n            port: 8443\n          initialDelaySeconds: 30\n          timeoutSeconds: 30\n      volumes:\n      - name: kubernetes-dashboard-certs\n        secret:\n          secretName: kubernetes-dashboard-certs\n      - name: tmp-volume\n        emptyDir: {}\n      serviceAccountName: kubernetes-dashboard\n      # Comment the following tolerations if Dashboard must not be deployed on master\n      tolerations:\n      - key: node-role.kubernetes.io/master\n        effect: NoSchedule\n\n---\n# ------------------- Dashboard Service ------------------- #\n\nkind: Service\napiVersion: v1\nmetadata:\n  labels:\n    k8s-app: kubernetes-dashboard\n#    kubernetes.io/cluster-service: \"true\"\n#    addonmanager.kubernetes.io/mode: Reconcile\n  name: kubernetes-dashboard\n  namespace: kube-system\nspec:\n  type: NodePort\n  ports:\n    - port: 443\n      targetPort: 8443\n      nodePort: 30000\n  selector:\n    k8s-app: kubernetes-dashboard\n\n---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n  name: admin-user\n  namespace: kube-system\n\n---\napiVersion: rbac.authorization.k8s.io/v1beta1\nkind: ClusterRoleBinding\nmetadata:\n  name: admin-user\nroleRef:\n  apiGroup: rbac.authorization.k8s.io\n  kind: ClusterRole\n  name: cluster-admin\nsubjects:\n- kind: ServiceAccount\n  name: admin-user\n  namespace: kube-system\nEOF\n```\n\n```shell\n$ kubectl create -f kubernetes-dashboard.yaml\n$ kubectl create clusterrolebinding  login-on-dashboard-with-cluster-admin --clusterrole=cluster-admin --user=admin\n```\n\n #### 11.1 登陆访问\n\n`https://172.16.1.49:30000/ `\n\n获取token,通过令牌登陆\n\n```shell\n$ kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')\n```\n\n\n\n### 12 部署Traefik Ingress\n\n\u003e Traefik 使用DS方式进行部署\n\n```yaml\n$ cat \u003c\u003cEOF \u003e traefik-rbac.yaml\n---\nkind: ClusterRole\napiVersion: rbac.authorization.k8s.io/v1beta1\nmetadata:\n  name: traefik-ingress-controller\nrules:\n  - apiGroups:\n      - \"\"\n    resources:\n      - services\n      - endpoints\n      - secrets\n    verbs:\n      - get\n      - list\n      - watch\n  - apiGroups:\n      - extensions\n    resources:\n      - ingresses\n    verbs:\n      - get\n      - list\n      - watch\n---\nkind: ClusterRoleBinding\napiVersion: rbac.authorization.k8s.io/v1beta1\nmetadata:\n  name: traefik-ingress-controller\nroleRef:\n  apiGroup: rbac.authorization.k8s.io\n  kind: ClusterRole\n  name: traefik-ingress-controller\nsubjects:\n- kind: ServiceAccount\n  name: traefik-ingress-controller\n  namespace: default\nEOF\n```\n\n```yaml\n$ cat \u003c\u003cEOF \u003e traefik-ui.yaml\napiVersion: v1\nkind: Service\nmetadata:\n  name: traefik-web-ui\n  namespace: default\nspec:\n  selector:\n    k8s-app: traefik-ingress-lb\n  ports:\n  - port: 80\n    targetPort: 8080\n---\napiVersion: extensions/v1beta1\nkind: Ingress\nmetadata:\n  name: traefik-web-ui\n  namespace: default\n  annotations:\n    kubernetes.io/ingress.class: traefik\nspec:\n  rules:\n  - host: k8s-traefik.shinezone.com\n    http:\n      paths:\n      - backend:\n          serviceName: traefik-web-ui\n          servicePort: 80\nEOF\n```\n\n```yaml\n$ cat \u003c\u003cEOF \u003e traefik-ds.yaml\n---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n  name: traefik-ingress-controller\n  namespace: default\n---\nkind: DaemonSet\napiVersion: extensions/v1beta1\nmetadata:\n  name: traefik-ingress-controller\n  namespace: default\n  labels:\n    k8s-app: traefik-ingress-lb\nspec:\n  template:\n    metadata:\n      labels:\n        k8s-app: traefik-ingress-lb\n        name: traefik-ingress-lb\n    spec:\n      serviceAccountName: traefik-ingress-controller\n      terminationGracePeriodSeconds: 60\n      hostNetwork: true\n      restartPolicy: Always\n      volumes:\n      - name: ssl\n        secret:\n          secretName: traefik-cert\n      - name: config\n        configMap:\n          name: traefik-conf\n      containers:\n      - image: traefik\n        name: traefik-ingress-lb\n        volumeMounts:\n        - mountPath: \"/etc/kubernetes/ssl\"  #证书所在目录\n          name: \"ssl\"\n        - mountPath: \"/root/k8s-online/traefik\"        #traefik.toml文件所在目录\n          name: \"config\"\n        resources:\n          limits:\n            cpu: 200m\n            memory: 30Mi\n          requests:\n            cpu: 100m\n            memory: 20Mi\n        ports:\n        - name: http\n          containerPort: 80\n          hostPort: 80\n        - name: admin\n          containerPort: 8080\n        - name: https\n          containerPort: 443\n          hostPort: 443\n        securityContext:\n          privileged: true\n        args:\n        - --api\n        - --kubernetes\n        - --logLevel=INFO\n        - --configfile=/root/k8s-online/traefik/traefik.toml\n---\nkind: Service\napiVersion: v1\nmetadata:\n  name: traefik-ingress-service\n  namespace: default\nspec:\n  selector:\n    k8s-app: traefik-ingress-lb\n  ports:\n    - protocol: TCP\n      port: 80\n      name: http\n    - protocol: TCP\n      port: 443\n      name: https\n    - protocol: TCP\n      port: 8080\n      name: admin\n  type: NodePort\nEOF\n```\n\n#### 12.1 配置Tracefik HTTPS\n\n```yaml\n$ cat \u003c\u003cEOF \u003e traefik.toml\ndefaultEntryPoints = [\"http\", \"https\"]\n\n[entryPoints]\n  [entryPoints.http]\n  address = \":80\"\n    [entryPoints.http.redirect]\n    entryPoint = \"https\"\n  [entryPoints.https]\n  address = \":443\"\n    [entryPoints.https.tls]\n      [[entryPoints.https.tls.certificates]]\n      certFile = \"/etc/kubernetes/ssl/shinezone.com.crt\"\n      keyFile = \"/etc/kubernetes/ssl/shinezone.com.key\"\nEOF\n```\n\n```shell\n$ kubectl create configmap traefik-conf --from-file=traefik.toml    //生成配置字典\n$ kubectl create secret generic traefik-cert --from-file=/etc/kubernetes/ssl/shinezone.com.key --from-file=/etc/kubernetes/ssl/shinezone.com.crt                    //生成保密字典\n```\n\n#### 12.2 部署Traefik\n\n```shell\n$ kubectl create -f traefik-rbac.yaml \n$ kubectl create -f traefik-ds.yaml\n$ kubectl create -f traefik-ui.yaml\n$ kubectl get pods --all-namespaces |grep traefik\ndefault       traefik-ingress-controller-gwcjk         1/1     Running   0         \ndefault       traefik-ingress-controller-rphgz         1/1     Running   0         \ndefault       traefik-ingress-controller-sd5l7         1/1     Running   0\n$ kubectl get svc --all-namespaces |grep traefik\ndefault       traefik-ingress-service   NodePort    10.105.248.48    \u003cnone\u003e        80:30899/TCP,443:30942/TCP,8080:30686/TCP  \ndefault       traefik-web-ui            ClusterIP   10.109.104.153   \u003cnone\u003e        80/TCP                                    \n```\n\n#### 12.3 访问Traefik\n\n\u003e 默认端口Node IP+ NodePort,也可以使用Traefik代理出来 使用域名访问\n\n- NodePort：http://ip:node_port/dashboard/\n- Traefik：https://k8s-traefik.shinezone.com  #推荐使用traefik配置的域名方式\n\n\n\n\n\n### 13. Prometheus-Operator\n\n- [Prometheus Operator介绍](https://github.com/yanghongfei/Kubernetes#153-%E5%AE%89%E8%A3%85promethues-operator)\n- 官网：https://github.com/coreos/prometheus-operator\n\n- 参考文档：http://www.servicemesher.com/blog/prometheus-operator-manual/\n\n#### 13.1 快速开始\n\n\u003e 官方把所有文件都放在一起，这里我分类下\n\n```\ncd contrib/kube-prometheus/manifests/\nmkdir -p operator node-exporter alertmanager grafana kube-state-metrics prometheus serviceMonitor adapter\nmv *-serviceMonitor* serviceMonitor/\nmv 0prometheus-operator* operator/\nmv grafana-* grafana/\nmv kube-state-metrics-* kube-state-metrics/\nmv alertmanager-* alertmanager/\nmv node-exporter-* node-exporter/\nmv prometheus-adapter* adapter/\nmv prometheus-* prometheus/\n$ ll\ntotal 40\ndrwxr-xr-x 9 root root 4096 Jan  6 14:19 ./\ndrwxr-xr-x 9 root root 4096 Jan  6 14:15 ../\n-rw-r--r-- 1 root root   60 Jan  6 14:15 00namespace-namespace.yaml\ndrwxr-xr-x 3 root root 4096 Jan  6 14:19 adapter/\ndrwxr-xr-x 3 root root 4096 Jan  6 14:19 alertmanager/\ndrwxr-xr-x 2 root root 4096 Jan  6 14:17 grafana/\ndrwxr-xr-x 2 root root 4096 Jan  6 14:17 kube-state-metrics/\ndrwxr-xr-x 2 root root 4096 Jan  6 14:18 node-exporter/\ndrwxr-xr-x 2 root root 4096 Jan  6 14:17 operator/\ndrwxr-xr-x 2 root root 4096 Jan  6 14:19 prometheus/\ndrwxr-xr-x 2 root root 4096 Jan  6 14:17 serviceMonitor/\n```\n\n- 部署opertor\n\n\u003e 先创建ns和operator，quay.io仓库拉取慢，可以使用脚本拉取，其他镜像也可以这样去拉，不过在apply之前才能拉，一旦被docker接手拉取就只能漫长等\n\n```shell\nkubectl apply -f .\ncurl -s https://zhangguanzhang.github.io/bash/pull.sh | bash -s -- quay.io/coreos/prometheus-operator:v0.26.0\nkubectl apply -f operator/\n# 确认状态运行正常再往后执行，这里镜像是quay.io仓库的可能会很慢耐心等待或者自行修改成能拉取到的。\n$ kubectl -n monitoring get pod\nNAME                                   READY     STATUS    RESTARTS   AGE\nprometheus-operator-56954c76b5-qm9ww   1/1       Running   0          24s\n```\n\n- 若脚本访问不到，以下是脚本内容\n\n```shell\n#!/bin/bash\n[ -z \"$set_e\" ] \u0026\u0026 set -e\n\n[ -z \"$1\" ] \u0026\u0026 { echo '$1 is not set';exit 2; }\n\nif [ \"$1\" == search ];then\n    shift\n    which jq \u0026\u003e /dev/null || { echo 'search needs jq, please install the jq';exit 2; }\n    img=${1%/}\n    [[ $img == *:* ]] \u0026\u0026 img_name=${img/://} || img_name=$img\n    if [[ \"$img\" =~ ^gcr.io|^k8s.gcr.io ]];then\n        if [[ \"$img\" =~ ^k8s.gcr.io ]];then\n            img_name=${img_name/k8s.gcr.io\\//gcr.io/google_containers/}\n        elif [[ \"$img\" == *google-containers* ]]; then\n            img_name=${img_name/google-containers/google_containers}\n        fi\n        repository=gcr.io\n    elif [[ \"$img\" =~ ^quay.io ]];then\n            repository=quay.io \n    else \n        echo 'not sync the namespaces!';exit 0;\n    fi\n    #echo '查询用的github,GFW原因可能会比较久,请确保能访问到github'\n    curl -sX GET https://api.github.com/repos/zhangguanzhang/${repository}/contents/$img_name?ref=develop |\n        jq -r 'length as $len | if $len ==2 then .message elif $len ==12 then .name else .[].name  end'\nelse\n    img=$1\n    name=${img%:*}\n    [[ $img == *:* ]] \u0026\u0026 tag=${img##*:} || tag=latest\n\n    if [[ \"$img\" =~ ^gcr.io|^quay.io ]];then\n        repo=zhangguanzhang/${name//\\//.}\n        [[ \"$img\" == *google-containers* ]] \u0026\u0026 repo=${repo/google-containers/google_containers}\n        docker pull $repo:$tag\n        docker tag $repo:$tag $img\n        docker rmi $repo:$tag\n\n    elif [[ \"$img\" =~ ^k8s.gcr.io ]];then\n        repo=zhangguanzhang/${name/k8s.gcr.io\\//gcr.io.google_containers.}\n        docker pull $repo:$tag\n        docker tag $repo:$tag $img\n        docker rmi $repo:$tag\n    else\n        echo 'not sync the namespaces!';exit 0;\n    fi\nfi\n\n```\n\n\n\n- 接着部署整套CRD\n\n```shell\n# 由于网络原因，可能需要很长时间\nkubectl apply -f adapter/\nkubectl apply -f alertmanager/\nkubectl apply -f node-exporter/\nkubectl apply -f kube-state-metrics/\nkubectl apply -f grafana/\nkubectl apply -f prometheus/\nkubectl apply -f serviceMonitor/\n\n#查看进度\nkubectl -n monitoring get all\n```\n\n#### 13.2使用Traefik 代理出来Prometheus Grafana \n\n\u003e 将以下文件进行apply即可\n\u003e\n\u003e kubectl apply -f  traefik-grafana.yaml \n\u003e\n\u003e kubectl apply -f  traefik-prometheus.yaml \n\n- `traefik-grafana.yaml `\n\n```yaml\n\napiVersion: extensions/v1beta1\nkind: Ingress\nmetadata:\n  name: grafana\n  namespace: monitoring\nspec:\n  rules:\n  - host: k8s-grafana.shinezone.com\n    http:\n      paths:\n      - path: /\n        backend:\n          serviceName: grafana\n          servicePort: 3000\n```\n\n- ` traefik-prometheus.yaml `\n\n```yaml\napiVersion: extensions/v1beta1\nkind: Ingress\nmetadata:\n  name: prometheus-k8s\n  namespace: monitoring\nspec:\n  rules:\n  - host: k8s-prometheus.shinezone.com\n    http:\n      paths:\n      - path: /\n        backend:\n          serviceName: prometheus-k8s\n          servicePort: 9090     \n```\n\n- `alertmanager-ingress.yaml`\n\n```yaml\napiVersion: extensions/v1beta1\nkind: Ingress\nmetadata:\n  name: alertmanager-k8s\n  namespace: monitoring\nspec:\n  rules:\n  - host: k8s-alertmanager.shinezone.com\n    http:\n      paths:\n      - path: /\n        backend:\n          serviceName: alertmanager-main\n          servicePort: 9093\n```\n\n\n\n- 效果图\n\n  ![](./images/dashboard.png)\n\n![](./images/grafana-cluster.png)\n\n![](./images/grafana-nodes.png)\n\n\n\n![](./images/altermanager.png)\n\n\n\n#### 13.3配置Promethe-Operator自定义报警\n\n\u003e OK ，到了这一步监控组件已经安装完成，接下来就是要理解怎么自定义规则，规则怎么写？报警怎么触发。\n\n现在登陆到Prometheus的Dashboard发现已经有一些自带的规则了，还有已经触发的Alter，如下图\n\n![](./images/prometheus.png)\n\n\n\n\u003e但是这些报警信息是哪里来的呢？他们应该用怎样的方式通知我们呢？现在我们通过 Operator 部署的呢？我们可以在 Prometheus Dashboard 的 Config 页面下面**查看关于 AlertManager 的配置**：\n\n```yaml\nglobal:\n  scrape_interval: 30s\n  scrape_timeout: 10s\n  evaluation_interval: 30s\n  external_labels:\n    prometheus: monitoring/k8s\n    prometheus_replica: prometheus-k8s-0\nalerting:\n  alert_relabel_configs:\n  - separator: ;\n    regex: prometheus_replica\n    replacement: $1\n    action: labeldrop\n  alertmanagers:\n  - kubernetes_sd_configs:\n    - role: endpoints\n      namespaces:\n        names:\n        - monitoring\n    scheme: http\n    path_prefix: /\n    timeout: 10s\n    relabel_configs:\n    - source_labels: [__meta_kubernetes_service_name]\n      separator: ;\n      regex: alertmanager-main\n      replacement: $1\n      action: keep\n    - source_labels: [__meta_kubernetes_endpoint_port_name]\n      separator: ;\n      regex: web\n      replacement: $1\n      action: keep\nrule_files:\n- /etc/prometheus/rules/prometheus-k8s-rulefiles-0/*.yaml\n```\n\n上面 alertmanagers 实例的配置我们可以看到是通过角色为 endpoints 的 kubernetes 的服务发现机制获取的，匹配的是服务名为 alertmanager-main，端口名未 web 的 Service 服务，我们查看下 alertmanager-main 这个 Service：\n\n```shell\n$ kubectl describe svc alertmanager-main -n monitoring\nName:              alertmanager-main\nNamespace:         monitoring\nLabels:            alertmanager=main\nAnnotations:       kubectl.kubernetes.io/last-applied-configuration:\n                     {\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"labels\":{\"alertmanager\":\"main\"},\"name\":\"alertmanager-main\",\"namespace\":\"...\nSelector:          alertmanager=main,app=alertmanager\nType:              ClusterIP\nIP:                10.111.82.41\nPort:              web  9093/TCP\nTargetPort:        web/TCP\nEndpoints:         100.64.2.6:9093,100.64.2.8:9093,100.64.3.6:9093\nSession Affinity:  None\nEvents:            \u003cnone\u003e\n```\n\n可以看到服务名正是 alertmanager-main，Port 定义的名称也是 web，符合上面的规则，所以 Prometheus 和 AlertManager 组件就正确关联上了。而对应的报警规则文件位于：/etc/prometheus/rules/prometheus-k8s-rulefiles-0/目录下面所有的 YAML 文件。我们可以进入 Prometheus 的 Pod 中验证下该目录下面是否有 YAML 文件：\n\n```shell\n$ kubectl exec -it prometheus-k8s-0 /bin/sh -n monitoring\nDefaulting container name to prometheus.\nUse 'kubectl describe pod/prometheus-k8s-0 -n monitoring' to see all of the containers in this pod.\n/prometheus $ ls /etc/prometheus/rules/prometheus-k8s-rulefiles-0/\nmonitoring-prometheus-k8s-rules.yaml\n/prometheus $ cat /etc/prometheus/rules/prometheus-k8s-rulefiles-0/monitoring-pr\nometheus-k8s-rules.yaml\ngroups:\n- name: k8s.rules\n  rules:\n  - expr: |\n      sum(rate(container_cpu_usage_seconds_total{job=\"kubelet\", image!=\"\", container_name!=\"\"}[5m])) by (namespace)\n    record: namespace:container_cpu_usage_seconds_total:sum_rate\n......\n```\n\n这个 YAML 文件实际上就是我们之前创建的一个 PrometheusRule 文件包含的：\n\n```shell\n$ cat prometheus-rules.yaml\napiVersion: monitoring.coreos.com/v1\nkind: PrometheusRule\nmetadata:\n  labels:\n    prometheus: k8s\n    role: alert-rules\n  name: prometheus-k8s-rules\n  namespace: monitoring\nspec:\n  groups:\n  - name: k8s.rules\n    rules:\n    - expr: |\n        sum(rate(container_cpu_usage_seconds_total{job=\"kubelet\", image!=\"\", container_name!=\"\"}[5m])) by (namespace)\n      record: namespace:container_cpu_usage_seconds_total:sum_rate\n```\n\n我们这里的 PrometheusRule 的 name 为 prometheus-k8s-rules，namespace 为 monitoring，我们可以猜想到我们创建一个 PrometheusRule 资源对象后，会自动在上面的 prometheus-k8s-rulefiles-0 目录下面生成一个对应的\u003cnamespace\u003e-\u003cname\u003e.yaml文件，所以如果以后我们需要自定义一个报警选项的话，只需要定义一个 PrometheusRule 资源对象即可。至于为什么 Prometheus 能够识别这个 PrometheusRule 资源对象呢？这就需要查看我们创建的 prometheus 这个资源对象了，里面有非常重要的一个属性 ruleSelector，用来匹配 rule 规则的过滤器，要求匹配具有 prometheus=k8s 和 role=alert-rules 标签的 PrometheusRule 资源对象，现在明白了吧？\n\n```yaml\nruleSelector:\n  matchLabels:\n    prometheus: k8s\n    role: alert-rules\n```\n\n所以我们要想自定义一个报警规则，只需要创建一个具有 prometheus=k8s 和 role=alert-rules 标签的 PrometheusRule 对象就行了，这里简单测试一个rule规则文件，[更多rules文件参考](https://github.com/yanghongfei/Kubernetes/tree/master/kube-prometheus/manifests/prometheus/prometheus_rules)\n\n```yaml\n$ cat prometheus-cpu-rules.yaml \napiVersion: monitoring.coreos.com/v1\nkind: PrometheusRule\nmetadata:\n  labels:\n    prometheus: k8s\n    role: alert-rules\n  name: prometheus-cpu-rules\n  namespace: monitoring\nspec:\n  groups:\n  - name: 主机CPU监控\n    rules:\n    - alert: CPU利用率过高\n      annotations:\n        detail:  \"{{$labels.instance}}: CPU利用率过高于75% (当前值: {{ $value }})\"\n        summary: \"{{$labels.instance}}: CPU利用率过高\"\n      expr: |\n        100 - (avg by (instance) (irate(node_cpu_seconds_total{job=\"node-exporter\",mode=\"idle\"}[5m])) * 100) \u003e 75\n      for: 1m\n      labels:\n        severity: 严重\n    - name: 主机CPU Load15 监控\n    rules:\n    - alert: CPU Load 15分钟过高\n      annotations:\n        detail:  \"{{$labels.instance}}: 15分钟内CPU Load 过高，(当前值: {{ $value }})\"\n        summary: \"{{$labels.instance}}: 15分钟内CPU Load 过高\"\n      expr: |\n        (node_load15) \u003e 1     #根据你的主机核心数来定\n      for: 1m\n      labels:\n        severity: 严重\n```\n\n注意 label 标签一定至少要有 prometheus=k8s 和 role=alert-rules，创建完成后，隔一会儿再去容器中查看下 rules 文件夹\n\n![](./images/yanghongfei_rule.png)\n\n可以看到我们创建的 rule 文件已经被注入到了对应的 rulefiles 文件夹下面了，证明我们上面的设想是正确的。然后再去 Prometheus Dashboard 的 Alert 页面下面就可以查看到上面我们新建的报警规则了\n\n\n\n**配置报警**\n\n  我们知道了如何去添加一个报警规则配置项，但是这些报警信息用怎样的方式去发送呢？前面的课程中我们知道我们可以通过 AlertManager 的配置文件去配置各种报警接收器，现在我们是通过 Operator 提供的 alertmanager 资源对象创建的组件，应该怎样去修改配置呢？ \n\n  首先我们将 alertmanager-main 这个 Service 改为 NodePort 类型的 Service，修改完成后我们可以在页面上的 status 路径下面查看 AlertManager 的默认配置信息。\n\n这些配置信息实际上是来自于我们之前在prometheus-operator/contrib/kube-prometheus/manifests目录下面创建的 alertmanager-secret.yaml 文件：\n\n```shell\napiVersion: v1\ndata:\n  alertmanager.yaml: Imdsb2JhbCI6IAogICJyZXNvbHZlX3RpbWVvdXQiOiAiNW0iCiJyZWNlaXZlcnMiOiAKLSAibmFtZSI6ICJudWxsIgoicm91dGUiOiAKICAiZ3JvdXBfYnkiOiAKICAtICJqb2IiCiAgImdyb3VwX2ludGVydmFsIjogIjVtIgogICJncm91cF93YWl0IjogIjMwcyIKICAicmVjZWl2ZXIiOiAibnVsbCIKICAicmVwZWF0X2ludGVydmFsIjogIjEyaCIKICAicm91dGVzIjogCiAgLSAibWF0Y2giOiAKICAgICAgImFsZXJ0bmFtZSI6ICJEZWFkTWFuc1N3aXRjaCIKICAgICJyZWNlaXZlciI6ICJudWxsIg==\nkind: Secret\nmetadata:\n  name: alertmanager-main\n  namespace: monitoring\ntype: Opaque\n```\n\n可以将 alertmanager.yaml 对应的 value 值做一个 base64 解码：\n\n```yaml\n$ echo \"Imdsb2JhbCI6IAogICJyZXNvbHZlX3RpbWVvdXQiOiAiNW0iCiJyZWNlaXZlcnMiOiAKLSAibmFtZSI6ICJudWxsIgoicm91dGUiOiAKICAiZ3JvdXBfYnkiOiAKICAtICJqb2IiCiAgImdyb3VwX2ludGVydmFsIjogIjVtIgogICJncm91cF93YWl0IjogIjMwcyIKICAicmVjZWl2ZXIiOiAibnVsbCIKICAicmVwZWF0X2ludGVydmFsIjogIjEyaCIKICAicm91dGVzIjogCiAgLSAibWF0Y2giOiAKICAgICAgImFsZXJ0bmFtZSI6ICJEZWFkTWFuc1N3aXRjaCIKICAgICJyZWNlaXZlciI6ICJudWxsIg==\" | base64 -d\n\"global\":\n  \"resolve_timeout\": \"5m\"\n\"receivers\":\n- \"name\": \"null\"\n\"route\":\n  \"group_by\":\n  - \"job\"\n  \"group_interval\": \"5m\"\n  \"group_wait\": \"30s\"\n  \"receiver\": \"null\"\n  \"repeat_interval\": \"12h\"\n  \"routes\":\n  - \"match\":\n      \"alertname\": \"DeadMansSwitch\"\n    \"receiver\": \"null\"\n```\n\n我们可以看到内容和上面查看的配置信息是一致的，所以如果我们想要添加自己的接收器，或者模板消息，我们就可以更改这个文件：\n\n```yaml\ncat alertmanager.yaml \nglobal:\n  resolve_timeout: 5m\n  smtp_smarthost: 'smtp.163.com:25'\n  smtp_from: 'yanghongfei97@163.com'\n  smtp_auth_username: 'yanghongfei97@163.com'\n  smtp_auth_password: '\u003cemail_password\u003e'\n  smtp_hello: '163.com'\n  smtp_require_tls: false\nroute:\n  group_by: ['job', 'severity']\n  group_wait: 30s\n  group_interval: 5m\n  repeat_interval: 12h\n  receiver: default\n  routes:\n  - receiver: webhook\n    match:\n      alertname: CoreDNSDown\nreceivers:\n- name: 'default'\n  email_configs:\n  - to: '1923671815@qq.com'\n    send_resolved: true\n- name: 'webhook'\n  webhook_configs:\n  - url: 'https://oapi.dingtalk.com/robot/send?access_token=xxxxx'\n    send_resolved: true\n```\n\n将上面文件保存为 alertmanager.yaml，然后使用这个文件创建一个 Secret 对象：\n\n```shell\n# 先将之前的 secret 对象删除\n$ kubectl delete secret alertmanager-main -n monitoring\nsecret \"alertmanager-main\" deleted\n$ kubectl create secret generic alertmanager-main --from-file=alertmanager.yaml -n monitoring\nsecret \"alertmanager-main\" created\n```\n\n**这时候再次确认配置已经被更改，如下图**\n\n![](./images/altermanager_config.png)\n\n**以上是自带规则，当然读到这里你已经知道了他的rules逻辑，你可以自己进行自定义规则**\n\n这里是我的自定义规则面板，和测试结果，rules规则文件我也放到了`promethues-rules`目录中,[更多rules文件参考](https://github.com/yanghongfei/Kubernetes/tree/master/kube-prometheus/manifests/prometheus/prometheus_rules)\n\n![](./images/prometheus_alerts.png)\n\n![](./images/prometheus_rules.png)\n\n**CPU 15分钟Load\u003e2报警**\n\n![](./images/email.png)\n\n\n\n#### 13.4 使用Promethues监控外接ETCD集群 \n\n- Q：为什么要监控ETCD？\n- A： 因为K8S数据都是存在ETCD里面，ETCD非常重要！\n\n\u003e 因为默认Prometheus-operator是没有收集ETCD里面的数据的，因为ETCD比较重要，我们ETCD使用外接二进制形式部署高可用，因此这里ETCD需要单独配置serviceMonitor和services\n\n- 参考文档：https://www.qikqiak.com/post/prometheus-operator-monitor-etcd/\n\n**首先我们将需要使用到的证书通过 secret 对象保存到集群中去：(在 etcd 运行的节点)**\n\n我这里是外接ETCD，证书也是前面部署生成的，所以我知道证书位置，若你的etcd是Kubeadm默认部署的，你需要先查看信息`kubectl get pod \u003cetcd_pod_name\u003e -n kube-system -o yaml`详细请参考[Prometheus监控ETCD](https://www.qikqiak.com/post/prometheus-operator-monitor-etcd/)\n\n```shell\n$ kubectl -n monitoring create secret generic etcd-certs --from-file=/etc/etcd/ssl/etcd.pem --from-file=/etc/etcd/ssl/etcd-key.pem --from-file=/etc/etcd/ssl/ca.pem\n```\n\n然后将上面创建的 etcd-certs 对象配置到 prometheus 资源对象中，直接更新 prometheus 资源对象即可：\n\n```yaml\n#因为的promethues有2个节点，所以2台机器都进行操作修改\n$ kubectl edit prometheus k8s -n monitoring\n#添加如下secrets属性\nnodeSelector:\n  beta.kubernetes.io/os: linux\nreplicas: 2\nsecrets:\n- etcd-certs\n```\n\n更新完成后，我们就可以在 Prometheus 的 Pod 中获取到上面创建的 etcd 证书文件了，具体的路径我们可以进入 Pod 中查看\n\n```shell\n$ kubectl exec -it prometheus-k8s-0 /bin/sh -n monitoring\nDefaulting container name to prometheus.\nUse 'kubectl describe pod/prometheus-k8s-0 -n monitoring' to see all of the containers in this pod.\n/ $ ls /etc/prometheus/secrets/etcd-certs/\nca.pem      etcd.pem  etcd-key.pem\n```\n\n**创建ServiceMonitor**\n\n现在 Prometheus 访问 etcd 集群的证书已经准备好了，接下来创建 ServiceMonitor 对象即可（prometheus-serviceMonitorEtcd.yaml）\n\n```yaml\napiVersion: monitoring.coreos.com/v1\nkind: ServiceMonitor\nmetadata:\n  name: etcd-k8s\n  namespace: monitoring\n  labels:\n    k8s-app: etcd-k8s\nspec:\n  jobLabel: k8s-app\n  endpoints:\n  - port: port\n    interval: 30s\n    scheme: https\n    tlsConfig:\n      caFile: /etc/prometheus/secrets/etcd-certs/ca.pem\n      certFile: /etc/prometheus/secrets/etcd-certs/etcd.pem\n      keyFile: /etc/prometheus/secrets/etcd-certs/etcd-key.pem\n      insecureSkipVerify: true\n  selector:\n    matchLabels:\n      k8s-app: etcd\n  namespaceSelector:\n    matchNames:\n    - kube-system\n```\n\n上面我们在 monitoring 命名空间下面创建了名为 etcd-k8s 的 ServiceMonitor 对象，基本属性和前面章节中的一致，匹配 kube-system 这个命名空间下面的具有 k8s-app=etcd 这个 label 标签的 Service，jobLabel 表示用于检索 job 任务名称的标签，和前面不太一样的地方是 endpoints 属性的写法，配置上访问 etcd 的相关证书，endpoints 属性下面可以配置很多抓取的参数，比如 relabel、proxyUrl，tlsConfig 表示用于配置抓取监控数据端点的 tls 认证，由于证书 serverName 和 etcd 中签发的可能不匹配，所以加上了 insecureSkipVerify=true\n\n\n\n**创建Service**\n\nServiceMonitor 创建完成了，但是现在还没有关联的对应的 Service 对象，所以需要我们去手动创建一个 Service 对象（prometheus-etcdService.yaml）：\n\n```yaml\napiVersion: v1\nkind: Service\nmetadata:\n  name: etcd-k8s\n  namespace: kube-system\n  labels:\n    k8s-app: etcd\nspec:\n  type: ClusterIP\n  clusterIP: None\n  ports:\n  - name: port\n    port: 2379\n    protocol: TCP\n\n---\napiVersion: v1\nkind: Endpoints\nmetadata:\n  name: etcd-k8s\n  namespace: kube-system\n  labels:\n    k8s-app: etcd\nsubsets:\n- addresses:\n  - ip: 172.16.1.50\n   # nodeName: K8S01-Master01   集群有多个IP都要写进去\n  - ip: 172.16.1.51\n   # nodeName: K8S01-Master02\n  - ip: 172.16.1.52\n   # nodeName: K8S01-Node01\n  - ip: 172.16.1.53\n    #nodeName: K8S01-Node02\n  - ip: 172.16.1.54\n    #nodeName: K8S01-Node03\n  ports:\n  - name: port\n    port: 2379\n    protocol: TCP\n```\n\nEndpoints 的 subsets 中填写 etcd 集群的地址即可，我们这里是单节点的，填写一个即可，直接创建该 Service 资源：\n\n```shell\n$ kubectl create -f prometheus-etcdService.yaml\n```\n\n创建完成后，隔一会儿去 Prometheus 的 Dashboard 中查看 targets，便会有 etcd 的监控项了：\n\n![](./images/etcd_target.png)\n\n![](./images/etcd_graph.png)\n\n\n\n**配置ETCD监控报警规则**\n\n到了这一步集群里面已经能prometheus到ETCD了，并且可以拿到数据，接下来就根据数据配置prometheus-rules规则，更多规则请参考：https://github.com/yanghongfei/Kubernetes/tree/master/kube-prometheus/manifests/prometheus/prometheus_rules\n\n```yaml\napiVersion: monitoring.coreos.com/v1\nkind: PrometheusRule\nmetadata:\n  labels:\n    prometheus: k8s\n    role: alert-rules\n  name: prometheus-etcd-rules\n  namespace: monitoring\nspec:\n  groups:\n  - name: EtcdMonitoring\n    rules:\n    - alert: EtcdDown \n      annotations:\n        detail:  \"{{$labels.instance}}: etcd down (当前值: {{ $value }})\"\n        summary: \"{{$labels.instance}}: etcd 出现异常，请管理员尽快排查\"\n      expr: |\n        up{endpoint=\"port\",job=\"etcd\",namespace=\"kube-system\",service=\"etcd-k8s\"} == 0\n      for: 1m\n      labels:\n        severity: 严重\n```\n\n**另外附上自己整理的监控K8S集群规则**\n\n规则路径：https://github.com/yanghongfei/Kubernetes/tree/master/kube-prometheus/manifests/prometheus/prometheus_rules\n\n```shell\n.\n├── prometheus-altermanager-rules.yaml    #监控Altermanger存活\n├── prometheus-cpu-rules.yaml             #监控NodeCPU利用率和NodeCPULoad负载\n├── prometheus-deployment-rules.yaml      #监控K8S集群中是否有部署报错\n├── prometheus-disk-rules.yaml            #监控磁盘空间，磁盘IO后续更新\n├── prometheus-etcd-rules.yaml            #监控ETCD集群存活\n├── prometheus-grafana-rules.yaml         #监控Grafana是否存活\n├── prometheus-k8s-pod-rules.yaml         #监控K8S集群中POD重启和异常POD信息\n├── prometheus-k8s-rules.yaml             #监控K8SMaster核心组件，如:APIServer 调度器等\n├── prometheus-memory-rules.yaml          #监控Node主机内存使用情况\n├── prometheus-node_exporter-rules.yaml   #监控Node采集器存活情况\n└── prometheus-prometheus-rules.yaml      #监控Promethues服务存活\n```\n\n![](./images/promethues_alters.png)\n\n\n\n### 维护相关\n\n#### 01. 强制删除Pod\n\n```shell\n$ kubectl delete  pods calico-node-wkmfl   --grace-period=0 --force --namespace kube-system\n##强制重启pod\nkubectl get pods coredns-bb849946b-jflvr -n kube-system -o yaml | kubectl replace --force -f -\n\n##强制重新部署\nkubectl get deployment grafana -n monitoring -o yaml   | kubectl replace --force -f -\n```\n\n#### 02. 重新获取集群Token\n\n\u003e 集群初始化时如未设置tokenTTL: \"0s\" 那么默认生成的token的有效期为24小时，当过期之后，该token就不可用了。另如果丢失kubeadm join加入集群命令，也同样可以通过以下方法解决\n\n```shell\n#重新生成token\n$ kubeadm token create\n\n#查看\n$ kubeadm token list\n\n#获取ca证书sha256编码hash值\n$ openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2\u003e/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'\n\n#使用以上命令生成的token和hash值，操作节点加入集群\n$ kubeadm join --token qfk0zc.uobckpxnh54v5at3 --discovery-token-ca-cert-hash sha256:68efb4e280d110f3004a4e16ed09c5ded6c2421cb24a6077cf6171d5167b04d2  10.0.0.15:6443 --skip-preflight-checks\n```\n\n#### 03. Master节点也允许Pod容器\n\n\u003e 默认集群安装完成后，Master节点是不进行Pod分配的，资源不足/实验环境，让Master 节点也参与分配pod\n\n```shell\n$ kubectl taint nodes --all node-role.kubernetes.io/master-\n```\n\n\n\n#### 04. modprobe: FATAL: Module br_netfilter not found\n\n\u003e 有些系统内核里面没有br_netfilter，这时候需要我们更改系统内核。\n\n```shell\nmodprobe br_netfilter   //报错: modprobe: FATAL: Module br_netfilter not found\ncat \u003c\u003cEOF \u003e  /etc/sysctl.d/k8s.conf\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.bridge.bridge-nf-call-iptables = 1\nnet.ipv4.ip_forward = 1\nvm.swappiness=0\nEOF\nsysctl -p /etc/sysctl.d/k8s.conf\nls /proc/sys/net/bridge\n```\n\n- 解决办法：\n  - 参考文档：https://linux.cn/article-8310-1.html\n\n```shell\nrpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org\nrpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm\nyum --disablerepo=\"*\" --enablerepo=\"elrepo-kernel\" list available\nyum --enablerepo=elrepo-kernel install kernel-ml -y\n# 修改 GRUB 初始化页面的第一个内核将作为默认内核\nsed -i s#GRUB_DEFAULT=saved#GRUB_DEFAULT=0#g /etc/default/grub  \n# 重新创建内核配置\ngrub2-mkconfig -o /boot/grub2/grub.cfg\n# 重启加载新内核\nreboot\n```\n\n\n\n#### 05. Node配置Keepalived，避免整个Node挂掉，Tarefik解析不可用问题\n\n- 配置VIP漂移\n- Tarefik解析到VIP上(不再是NodeIP单点)\n\n```shell\n# 所有Node节点执行\nyum install -y keepalived\nsystemctl enable keepalived\n```\n\n- Node01配置\n\n```shell\ncat \u003c\u003cEOF \u003e/etc/keepalived/keepalived.conf\nglobal_defs {\n   router_id LVS_k8s\n}\n\n# vrrp_script CheckK8sMaster {\n#     script \"curl -k https://172.16.1.49:6443\"    #VIP Address\n#     interval 3\n#     timeout 9\n#     fall 2\n#     rise 2\n# }\n\nvrrp_instance VI_1 {\n    state MASTER\n    interface ens32       #Your Network Interface Name\n    virtual_router_id 61\n    priority 120          #权重，数字大的为主，数字一样则选择第一台为Master\n    advert_int 1\n    mcast_src_ip 172.16.1.52  #local IP\n    nopreempt\n    authentication {\n        auth_type PASS\n        auth_pass sqP05dQgMSlzrxHj\n    }\n    unicast_peer {\n        #172.16.1.52\n        172.16.1.53    #node02\n        172.16.1.54\n    }\n    virtual_ipaddress {\n        172.16.1.59/24    # VIP\n    }\n    # track_script {\n    #     CheckK8sMaster\n    # }\n\n}\nEOF\n```\n\n- Node02配置\n\n```shell\ncat \u003c\u003cEOF \u003e/etc/keepalived/keepalived.conf\nglobal_defs {\n   router_id LVS_k8s\n}\n\n# vrrp_script CheckK8sMaster {\n#     script \"curl -k https://172.16.1.49:6443\"\n#     interval 3\n#     timeout 9\n#     fall 2\n#     rise 2\n# }\n\nvrrp_instance VI_1 {\n    state MASTER\n    interface ens32\n    virtual_router_id 61\n    priority 110\n    advert_int 1\n    mcast_src_ip 172.16.1.53   #local_ip\n    nopreempt\n    authentication {\n        auth_type PASS\n        auth_pass sqP05dQgMSlzrxHj\n    }\n    unicast_peer {\n        172.16.1.52\n        #172.16.1.53\n        172.16.1.54\n    }\n    virtual_ipaddress {\n        172.16.1.59/24      #vip\n    }\n    # track_script {\n    #     CheckK8sMaster\n    # }\n\n}\nEOF\n```\n\n- Node03配置\n\n```shell\ncat \u003c\u003cEOF \u003e/etc/keepalived/keepalived.conf\nglobal_defs {\n   router_id LVS_k8s\n}\n\n# vrrp_script CheckK8sMaster {\n#     script \"curl -k https://172.16.1.49:6443\"\n#     interval 3\n#     timeout 9\n#     fall 2\n#     rise 2\n# }\n\nvrrp_instance VI_1 {\n    state MASTER\n    interface ens32\n    virtual_router_id 61\n    priority 100\n    advert_int 1\n    mcast_src_ip 172.16.1.54   #local_ip\n    nopreempt\n    authentication {\n        auth_type PASS\n        auth_pass sqP05dQgMSlzrxHj\n    }\n    unicast_peer {\n        172.16.1.52\n        172.16.1.53\n        #172.16.1.54\n    }\n    virtual_ipaddress {\n        172.16.1.59/24      #vip\n    }\n    # track_script {\n    #     CheckK8sMaster\n    # }\n\n}\nEOF\n```\n\n\n\n- 启动服务\n\n```shell\nsed s#'KEEPALIVED_OPTIONS=\"-D\"'#'KEEPALIVED_OPTIONS=\"-D -d -S 0\"'#g /etc/sysconfig/keepalived -i   //配置日志文件\necho \"local0.*    /var/log/keepalived.log\" \u003e\u003e /etc/rsyslog.conf\nservice rsyslog restart\nsystemctl start keepalived\nsystemctl status keepalived\n```\n\n- 测试参考**8.4测试Keepalived可用性**\n\n\n\n#### 06. POD容器不能访问外网问题\n\n\u003e 详细排查思路请参考**10.4测试Calico和DNS网络问题**\n\n```shell\n\ncat config.yaml   #你的SVC和POD地址段，添加2条规则\n$ ps uax |grep kube-proxy  //确保加载了--cluster-cidr\n$ iptables -nvL |grep FORWARD  //查看FORWARD 是否为ACCEPT \n$ iptables -P FORWARD ACCEPT  //开启IPtableS转发ACCEPT\n$ sysctl -a | grep ip_forward //确认系统ip_forward开启\n\n$ /sbin/iptables -t nat -I POSTROUTING -s  100.64.0.0/10 -j MASQUERADE\n$ /sbin/iptables -t nat -I POSTROUTING -s  10.96.0.0/12 -j MASQUERADE\n\n$ kubectl run -it --rm --image=infoblox/dnstools dns-client #测试\ndnstools# ping qq.com  #可以ping通\n```\n\n#### 07. prometheus-operator部署完成后Targets 部分显示0/0问题\n\n![](./images/targets_error.png)\n\n这里是因为Promethues-operator里面的serviceMonitor匹配不到Labels，详细请参考[阳明的博客](https://www.qikqiak.com/post/first-use-prometheus-operator/)\n\n\n\n### 参考文档\n\n部署参考文档：\n\n\n\nhttps://www.kubernetes.org.cn/4956.html\n\nhttps://www.qikqiak.com/post/how-to-use-ipvs-in-kubernetes/\n\nhttps://www.qikqiak.com/post/first-use-prometheus-operator/\n\nhttp://www.servicemesher.com/blog/prometheus-operator-manual/\n\nhttps://www.qikqiak.com/post/prometheus-operator-monitor-etcd/\n\nhttps://kubernetes.io/zh/docs/reference/setup-tools/kubeadm/kubeadm-init/\n\nhttps://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1\n\nhttps://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/README.md\n\nKubernetes二进制部署参考文档：https://github.com/yanghongfei/Kubernetes/tree/master/kubernetes-Binary，仅供参考，K8S版本迭代太快，这个二进制V1.6版本已经无法再完整部署起来了。\n\n#哇哈哈哈-test\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyanghongfei%2FKubernetes","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyanghongfei%2FKubernetes","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyanghongfei%2FKubernetes/lists"}