目录
文章目录
- 目录
- Kubernetes 在生产环境中架构
- 高可用集群部署拓扑
- 1、网络代理配置
- 2、Load Balancer 环境准备
- 3、Kubernetes Cluster 环境准备
- 安装 Container Runtime
- 安装 kubeadm、kubelet 和 kubectl
- 4、初始化 Master 主控制平面节点
- kubeadm init 的工作流
- 执行初始化
- （可选）清理或重新进行初始化
- 5、添加 Master 冗余控制平面节点
- 6、添加 Node 工作负载节点
- 7、安装 CNI 网络插件
- 8、安装 Metrics Server
- Kubernetes Metrics Server
- 启用 API Aggregator
- 安装 Metrics Server
- 9、安装 Dashboard GUI
- 10、访问 Dashboard UI
- 11、通过 NFS 实现持久化存储
- NFS Server 安装
- NFS Client 挂载
- Kubernetes 部署 nfs-client-provisioner
Kubernetes 在生产环境中架构
- Client 层：外部用户、客户端等；
- 服务访问层：Traefik、Kong APIGW 等 Ingress。外部客户端访问 Kubernetes Cluster 内部的 Service 时，使用 Ingress 支持的服务发现、负载均衡、路由规则定义来实现。此外，应当实现 Ingress 的 HA。
- 业务应用层：即基于 Kubernetes 之上构建和运行的企业业务应用。
- 镜像管理：使用 Harbor 私有镜像仓库服务；
- 日志管理：使用 ELK Stack；
- 监控告警管理：使用 Prometheus 和 Grafana；
- 微服务架构：使用 Istio 的 Service Mesh 方案，或者使用 APIGW；
- DevOps：使用 Gitlab、Jenkins 等 CI/CD 工具；
- 单体应用：无状态类服务使用 Deployment，有状态类服务则使用 Statefulset，如果关联的服务较多且复杂则使用 Helm。
- 规划好 Namespace：应当做到每个 Namespace 专属用于某类型的应用，例如：monitor namespace 用于统一管理监控告警、日志管理方面的 pod、service、pvc、ingress 等资源对象。
- 基础设施层：即由 Kubernetes、Calico SDN、Ceph SDS 或 NFS 等系统组成的基础设施服务。
高可用集群部署拓扑
官方文档：https://kubernetes.io/zh/docs/setup/production-environment/
- 基础设施：OpenStack
- 虚拟机集群：3 Master、2 Node、2 Load Balancer
- 计算资源：x86-64 processor、2CPU、2GB RAM、20GB free disk space
- 操作系统：CentOS 7.x+
- 版本：Kubernetes 1.18.14
- Container Runtime：Docker
1、网络代理配置
需要对 HTTP/S Proxy 和 No Proxy 进行精心的配置，否则要么下不下来软件，要么出现网络连通性的错误。
export https_proxy=http://{proxy_ip}:7890 http_proxy=http://{proxy_ip}:7890 all_proxy=socks5://{proxy_ip}:7890 no_proxy=localhost,127.0.0.1,{apiserver_endpoint_ip},{k8s_mgmt_network_ip_pool},{pod_network_ip_pool},{service_network_ip_pool}2、Load Balancer 环境准备
基于 OpenStack Octavia LBaaS 来提供 HA Load Balancer，也可以手动的配置 keepalived and haproxy（https://github.com/kubernetes/kubeadm/blob/master/docs/ha-considerations.md#options-for-software-load-balancing）。
- VIP 选择 kube-mgmt-subnet
- Listener 选择 TCP :6443 Socket（kube-apiserver 的监听端口）
- Members 选择 3 个 k8s-master
- Monitor 同样选择 TCP :6443 Socket
注意：创建好 Load Balancer 之后，首先要测试一下 TCP 反向代理运行正常。由于 apiserver 现在尚未运行，所以预期会出现一个连接拒绝错误。在我们初始化了第一个控制平面节点之后，要记得再次进行测试。
# nc -v LOAD_BALANCER_IP PORT
nc -v 192.168.0.100 64433、Kubernetes Cluster 环境准备
注意：在所有节点上执行以下操作。
- 添加全节点的 Hostname 解析。
# vi /etc/hosts
192.168.0.100 kube-apiserver-endpoint
192.168.0.148 k8s-master-1
192.168.0.112 k8s-master-2
192.168.0.193 k8s-master-3
192.168.0.208 k8s-node-1
192.168.0.174 k8s-node-2- 开启全节点之间的 SSH 免密登录。
- 禁用 Swap 交换分区，为了保证 kubelet 正常工作。
- 确保 iptables 工具不使用 nftables 后端，nftables 后端与当前的 kubeadm 软件包不兼容，它会导致重复的防火墙规则并破坏 kube-proxy。
- 确保节点之间的网络联通性。
- 关闭 SELinux，为了允许容器访问主机的文件系统。
# 将 SELinux 设置为 permissive 模式（相当于将其禁用）
setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config- 在 RHEL/CentOS 7 上为了保证 kube-proxy 控制的数据流量必须进过 iptables 的处理来进行本地路由，所以要确保 sysctl 配置中的 net.bridge.bridge-nf-call-iptables 被设置为 1。
# 确保加载了 br_netfilter 模块。
modprobe br_netfilter
lsmod | grep br_netfilter
# 确保 sysctl 配置，将 Bridge 的 IPv4 流量传递到 iptables 的 Chain（链）
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system- 安装基础依赖软件：
yum install ebtables ethtool ipvsadm -y安装 Container Runtime
注意：当 Linux 使用 systemd 时，会创建一个 cgroup，此时需要保证 Container Runtime、kubelet 和 systemd 使用的是同一个 cgroup，否则会出现不可预测的问题。为此，我们需要将 Container Runtime、kubelet 配置成使用 systemd 来作为 cgroup 驱动，以此使系统更为稳定。
对于 Docker 而言，设置 native.cgroupdriver=systemd 选项即可。
- 安装：
# 安装依赖包
sudo yum install -y yum-utils device-mapper-persistent-data lvm2
# 新增 Docker 仓库
sudo yum-config-manager --add-repo \
https://download.docker.com/linux/centos/docker-ce.repo
# 安装 Docker CE
sudo yum update -y && sudo yum install -y \
containerd.io-1.2.13 \
docker-ce-19.03.11 \
docker-ce-cli-19.03.11- 配置：
# 创建 /etc/docker 目录
sudo mkdir /etc/docker
# 设置 Docker daemon
cat <<EOF | sudo tee /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
]
}
EOF- 重启：
# Create /etc/systemd/system/docker.service.d
sudo mkdir -p /etc/systemd/system/docker.service.d
# 重启 Docker
sudo systemctl daemon-reload
sudo systemctl restart docker
sudo systemctl enable docker
sudo systemctl status docker安装 kubeadm、kubelet 和 kubectl
注意：kubeadm 是 Kubernetes Cluster 的部署工具，但 kubeadm 不能用于安装、管理 kubelet 或 kubectl，所以我们需要收到安装它们，并且确保三者的版本仓库是一致的。
- 更新 Kubernetes YUM 仓库：
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF- 安装：
# 查询版本
$ yum list kubelet kubeadm kubectl --showduplicates | grep 1.18.14 | sort -r
kubelet.x86_64 1.18.14-0 kubernetes
kubectl.x86_64 1.18.14-0 kubernetes
kubeadm.x86_64 1.18.14-0 kubernetes
# 安装指定版本
yum install -y kubelet-1.18.14 kubeadm-1.18.14 kubectl-1.18.14 --disableexcludes=kubernetes
# 确定版本一致
$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.14", GitCommit:"89182bdd065fbcaffefec691908a739d161efc03", GitTreeState:"clean", BuildDate:"2020-12-18T12:08:45Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
$ kubectl version --client
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.14", GitCommit:"89182bdd065fbcaffefec691908a739d161efc03", GitTreeState:"clean", BuildDate:"2020-12-18T12:11:25Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
$ kubelet --version
Kubernetes v1.18.14- 配置：上面我们提到过，需要将 Container Runtime、kubelet 配置成使用 systemd 来作为 cgroup 驱动，以此使系统更为稳定。
# vi /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS=--cgroup-driver=systemd- 启动：
$ systemctl daemon-reload
$ systemctl restart kubelet
$ systemctl enable --now kubelet
$ systemctl status kubelet注意：kubelet.sercice 每隔几秒就会重启一次，循环等待 kubeadm 的指令。
4、初始化 Master 主控制平面节点
kubeadm init 的工作流
kubeadm init 命令通过执行下列步骤来启动一个 Kubernetes Master：
- 预检测系统状态：当出现 ERROR 时就退出 kubeadm，除非问题得到解决或者显式指定了
--ignore-preflight-errors=<错误列表>参数。此外，也会出现 WARNING。 - 生成一个自签名的 CA 证书来为每个系统组件建立身份标识：可以显式指定
--cert-dirCA 中心目录（默认为 /etc/kubernetes/pki），在该目录下方式 CA 证书、密钥等文件。API Server 证书将为任何--apiserver-cert-extra-sans参数值提供附加的 SAN 条目，必要时将其小写。 - 将 kubeconfig 文件写入 /etc/kubernetes/ 目录：以便 kubelet、Controller Manager 和 Scheduler 用来连接到 API Server，它们都有自己的身份标识，同时生成一个名为 admin.conf 的独立的 kubeconfig 文件，用于管理操作。
- 为 API Server、Controller Manager 和 Scheduler 生成 static Pod 的清单文件：存放在 /etc/kubernetes/manifests 下，kubelet 会轮训监视这个目录，在启动 Kubernetes 时用于创建系统组件的 Pod。假使没有提供一个外部的 etcd 服务的话，也会为 etcd 生成一份额外的 static Pod 清单文件。
待 Master 的 static Pods 都运行正常后，kubeadm init 的工作流程才会继续往下执行。
- 对 Master 使用 Labels 和 Stain mark（污点标记）：以此隔离生产工作负载不会调度到 Master 上。
- 生成 Token：将来其他的 Node 可使用该 Token 向 Master 注册自己。也可以显式指定
--token提供 Token String。 - 为了使 Node 能够遵照启动引导令牌（Bootstrap Tokens）和 TLS 启动引导（TLS bootstrapping）这两份文档中描述的机制加入到 Cluster 中，kubeadm 会执行所有的必要配置：
- 创建一个 ConfigMap 提供添加 Node 到 Cluster 中所需的信息，并为该 ConfigMap 设置相关的 RBAC 访问规则。
- 允许启动引导令牌访问 CSR 签名 API。
- 配置自动签发新的 CSR 请求。
- 通过 API Server 安装一个 DNS 服务器（CoreDNS）和 kube-proxy：注意，尽管现在已经部署了 DNS 服务器，但直到安装 CNI 时才调度它。
执行初始化
注意 1：因为我们要部署高可用集群，所以必须使用选项 --control-plane-endpoint 指定 API Server 的 HA Endpoint。
注意 2：由于 kubeadm 默认从 k8s.grc.io 下载所需镜像，因此可以通过 --image-repository 指定阿里云的镜像仓库。
注意 3：如果显式指定 --upload-certs，则意味着在扩展冗余 Master 时，你必须要手动地将 CA 证书从主控制平面节点复制到将要加入的冗余控制平面节点上，推荐使用。
- 初始化：
kubeadm init \
--control-plane-endpoint "192.168.0.100" \
--kubernetes-version "1.18.14" \
--pod-network-cidr "10.0.0.0/8" \
--service-cidr "172.16.0.0/16" \
--token "abcdef.0123456789abcdef" \
--token-ttl "0" \
--image-repository registry.aliyuncs.com/google_containers \
--upload-certs
W1221 00:02:43.240309 10942 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.18.14
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master-1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [172.16.0.1 192.168.0.148 192.168.0.100]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master-1 localhost] and IPs [192.168.0.148 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master-1 localhost] and IPs [192.168.0.148 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W1221 00:02:47.773223 10942 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W1221 00:02:47.774303 10942 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 23.117265 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.18" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
463868e92236803eb8fdeaa3d7b0ada67cf0f882c45974682c6ac2f20be1d544
[mark-control-plane] Marking the node k8s-master-1 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master-1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of the control-plane node running the following command on each as root:
kubeadm join 192.168.0.100:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:88dc9773b5dfc0cde6082314a1a4a9bbdb6ddfd3f1f84a7113581a3b07e839e1 \
--control-plane --certificate-key 463868e92236803eb8fdeaa3d7b0ada67cf0f882c45974682c6ac2f20be1d544
Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.0.100:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:88dc9773b5dfc0cde6082314a1a4a9bbdb6ddfd3f1f84a7113581a3b07e839e1- 查看 Pods：检查 Master 的组件是否齐全。
# 配置 kubectl
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
$ kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-7ff77c879f-fh9vb 0/1 Pending 0 23m
coredns-7ff77c879f-qmk7z 0/1 Pending 0 23m
etcd-k8s-master-1 1/1 Running 0 24m
kube-apiserver-k8s-master-1 1/1 Running 0 24m
kube-controller-manager-k8s-master-1 1/1 Running 0 24m
kube-proxy-7hx55 1/1 Running 0 23m
kube-scheduler-k8s-master-1 1/1 Running 0 24m- 查看 Images：
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.aliyuncs.com/google_containers/kube-proxy v1.18.14 8e6bca1d4e68 2 days ago 117MB
registry.aliyuncs.com/google_containers/kube-apiserver v1.18.14 f17e261f4c8a 2 days ago 173MB
registry.aliyuncs.com/google_containers/kube-controller-manager v1.18.14 b734a959c6fb 2 days ago 162MB
registry.aliyuncs.com/google_containers/kube-scheduler v1.18.14 95660d582e82 2 days ago 95.3MB
registry.aliyuncs.com/google_containers/pause 3.2 80d28bedfe5d 10 months ago 683kB
registry.aliyuncs.com/google_containers/coredns 1.6.7 67da37a9a360 10 months ago 43.8MB
registry.aliyuncs.com/google_containers/etcd 3.4.3-0 303ce5db0e90 14 months ago 288MB- 查看 Containers：
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f9a068b890d7 8e6bca1d4e68 "/usr/local/bin/kube…" 2 minutes ago Up 2 minutes k8s_kube-proxy_kube-proxy-7hx55_kube-system_aacb0da3-16ec-414c-b138-856e2b470bb9_0
3b6adfa0b1a5 registry.aliyuncs.com/google_containers/pause:3.2 "/pause" 2 minutes ago Up 2 minutes k8s_POD_kube-proxy-7hx55_kube-system_aacb0da3-16ec-414c-b138-856e2b470bb9_0
dcc47de63e50 f17e261f4c8a "kube-apiserver --ad…" 3 minutes ago Up 3 minutes k8s_kube-apiserver_kube-apiserver-k8s-master-1_kube-system_c693bd1fadf036d8e2e4df0afd49f062_0
53afb7fbe8c0 b734a959c6fb "kube-controller-man…" 3 minutes ago Up 3 minutes k8s_kube-controller-manager_kube-controller-manager-k8s-master-1_kube-system_f75424d466cd7197fb8095b0f59ea8d9_0
a4101a231c1b 303ce5db0e90 "etcd --advertise-cl…" 3 minutes ago Up 3 minutes k8s_etcd_etcd-k8s-master-1_kube-system_f85e02734d6479f3bb3e468eea87fd3a_0
197f510ff6c5 95660d582e82 "kube-scheduler --au…" 3 minutes ago Up 3 minutes k8s_kube-scheduler_kube-scheduler-k8s-master-1_kube-system_0213a889f9350758ac9847629f75db19_0
3a4590590093 registry.aliyuncs.com/google_containers/pause:3.2 "/pause" 3 minutes ago Up 3 minutes k8s_POD_kube-controller-manager-k8s-master-1_kube-system_f75424d466cd7197fb8095b0f59ea8d9_0
4bbdc99a7a68 registry.aliyuncs.com/google_containers/pause:3.2 "/pause" 3 minutes ago Up 3 minutes k8s_POD_kube-apiserver-k8s-master-1_kube-system_c693bd1fadf036d8e2e4df0afd49f062_0
19488127c269 registry.aliyuncs.com/google_containers/pause:3.2 "/pause" 3 minutes ago Up 3 minutes k8s_POD_etcd-k8s-master-1_kube-system_f85e02734d6479f3bb3e468eea87fd3a_0
e67d2f7a27b0 registry.aliyuncs.com/google_containers/pause:3.2 "/pause" 3 minutes ago Up 3 minutes k8s_POD_kube-scheduler-k8s-master-1_kube-system_0213a889f9350758ac9847629f75db19_0- 测试 API Server LB 是否正常：
$ nc -v 192.168.0.100 6443
Connection to 192.168.0.100 port 6443 [tcp/sun-sr-https] succeeded!注意：上述 Token 的过期时间是 24 小时，如果希望在 24 小时之后继续添加不通的节点，则需要重新生产 Token：
# 新建 Token
kubeadm token create
# output: 5didvk.d09sbcov8ph2amjw
# 新建 --discovery-token-ca-cert-hash
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \
openssl dgst -sha256 -hex | sed 's/^.* //'
# output: 8cb2de97839780a412b93877f8507ad6c94f73add17d5d7058e91741c9d5ec78（可选）清理或重新进行初始化
要再次运行 kubeadm init，你必须首先卸载集群，可以在 Master 上触发尽力而为的清理：
kubeadm resetReset 过程不会重置或清除 iptables 规则或 IPVS 表。如果你希望重置 iptables 或 IPVS，则必须手动进行：
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
ipvsadm -C根据需求调整参数，重新进行初始化：
kubeadm init <args>或许，彻底删除节点：
kubectl delete node <node name>5、添加 Master 冗余控制平面节点
在第一个 Master 初始化完毕之后，我们就可以继续添加冗余 Master 节点了。
- 添加 k8s-master-2：
kubeadm join 192.168.0.100:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:88dc9773b5dfc0cde6082314a1a4a9bbdb6ddfd3f1f84a7113581a3b07e839e1 \
--control-plane --certificate-key 463868e92236803eb8fdeaa3d7b0ada67cf0f882c45974682c6ac2f20be1d544
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks before initializing the new control plane instance
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[download-certs] Downloading the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master-2 localhost] and IPs [192.168.0.112 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master-2 localhost] and IPs [192.168.0.112 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master-2 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [172.16.0.1 192.168.0.112 192.168.0.100]
[certs] Generating "front-proxy-client" certificate and key
[certs] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[certs] Using the existing "sa" key
[kubeconfig] Generating kubeconfig files
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
W1221 00:30:18.978564 27668 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W1221 00:30:18.986650 27668 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W1221 00:30:18.987613 27668 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[check-etcd] Checking that the etcd cluster is healthy
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
[etcd] Announced new etcd member joining to the existing etcd cluster
[etcd] Creating static Pod manifest for "etcd"
[etcd] Waiting for the new etcd member to join the cluster. This can take up to 40s
{"level":"warn","ts":"2020-12-21T00:30:34.018+0800","caller":"clientv3/retry_interceptor.go:61","msg":"retrying of unary invoker failed","target":"passthrough:///https://192.168.0.112:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[mark-control-plane] Marking the node k8s-master-2 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master-2 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
This node has joined the cluster and a new control plane instance was created:
* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Control plane (master) label and taint were applied to the new node.
* The Kubernetes control plane instances scaled up.
* A new etcd member was added to the local/stacked etcd cluster.
To start administering your cluster from this node, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Run 'kubectl get nodes' to see this node join the cluster.- 添加 k8s-master-3：
kubeadm join 192.168.0.100:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:88dc9773b5dfc0cde6082314a1a4a9bbdb6ddfd3f1f84a7113581a3b07e839e1 \
--control-plane --certificate-key 463868e92236803eb8fdeaa3d7b0ada67cf0f882c45974682c6ac2f20be1d544- 检查 Master 节点数：
# 配置 kubectl
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master-1 NotReady master 35m v1.18.14
k8s-master-2 NotReady master 8m14s v1.18.14
k8s-master-3 NotReady master 2m30s v1.18.146、添加 Node 工作负载节点
部署完高可用的 Master 控制平面之后，我们就可以注册任意个 Node 工作负载节点了。
- 添加 Node：
kubeadm join 192.168.0.100:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:88dc9773b5dfc0cde6082314a1a4a9bbdb6ddfd3f1f84a7113581a3b07e839e1
W1221 00:39:36.256784 29495 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.- 检查 Node：
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master-1 NotReady master 37m v1.18.14
k8s-master-2 NotReady master 10m v1.18.14
k8s-master-3 NotReady master 4m24s v1.18.14
k8s-node-1 NotReady <none> 51s v1.18.14
k8s-node-2 NotReady <none> 48s v1.18.147、安装 CNI 网络插件
我们选择使用 Calico SDN 方案。官方文档：https://docs.projectcalico.org/about/about-calico
注意：
- Pod 网络不得与任何主机网络重叠：所以我们在执行 kubeadm init 时显式指定了
--pod-network-cidr参数。 - 确保 CNI 网络插件支持 RBAC（基于角色的访问控制）。
- 确保 CNI 支持 IPv6 或 IPv4v6，当你需要使用的时候。
- 在 OpenStack 环境中要关闭 “端口安全” 功能，否则 Calico IPIP 隧道无法打通。
- 安装：
$ kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
poddisruptionbudget.policy/calico-kube-controllers created- 检查 Calico Pods：
$ watch kubectl get pod --all-namespaces
Every 2.0s: kubectl get pod --all-namespaces Mon Dec 21 13:12:30 2020
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-7dbc97f587-nqrxv 1/1 Running 0 9m34s
kube-system calico-node-47xmr 1/1 Running 0 9m34s
kube-system calico-node-8zwbg 1/1 Running 0 9m34s
kube-system calico-node-dj4qt 1/1 Running 0 9m34s
kube-system calico-node-glqqj 1/1 Running 0 9m34s
kube-system calico-node-jb4t4 1/1 Running 0 9m34s
kube-system coredns-7ff77c879f-fh9vb 1/1 Running 0 13h
kube-system coredns-7ff77c879f-qmk7z 1/1 Running 0 13h
kube-system etcd-k8s-master-1 1/1 Running 0 13h
kube-system etcd-k8s-master-2 1/1 Running 0 12h
kube-system etcd-k8s-master-3 1/1 Running 0 12h
kube-system kube-apiserver-k8s-master-1 1/1 Running 0 13h
kube-system kube-apiserver-k8s-master-2 1/1 Running 0 12h
kube-system kube-apiserver-k8s-master-3 1/1 Running 0 12h
kube-system kube-controller-manager-k8s-master-1 1/1 Running 1 13h
kube-system kube-controller-manager-k8s-master-2 1/1 Running 0 12h
kube-system kube-controller-manager-k8s-master-3 1/1 Running 0 12h
kube-system kube-proxy-7hx55 1/1 Running 0 13h
kube-system kube-proxy-8dmc4 1/1 Running 0 12h
kube-system kube-proxy-9clqs 1/1 Running 0 12h
kube-system kube-proxy-cq5tq 1/1 Running 0 12h
kube-system kube-proxy-pm79q 1/1 Running 0 12h
kube-system kube-scheduler-k8s-master-1 1/1 Running 1 13h
kube-system kube-scheduler-k8s-master-2 1/1 Running 0 12h
kube-system kube-scheduler-k8s-master-3 1/1 Running 0 12h- 检查 Cluster 节点的状态：安装了 CNI 之后节点的状态应该是 Ready 的。
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master-1 Ready master 13h v1.18.14
k8s-master-2 Ready master 12h v1.18.14
k8s-master-3 Ready master 12h v1.18.14
k8s-node-1 Ready <none> 12h v1.18.14
k8s-node-2 Ready <none> 12h v1.18.148、安装 Metrics Server
Kubernetes Metrics Server
Kubernetes Metrics Server 是 Cluster 的核心监控数据的聚合器，kubeadm 默认是不部署的。
Metrics Server 供 Dashboard 等其他组件使用，是一个扩展的 APIServer，依赖于 API Aggregator。所以，在安装 Metrics Server 之前需要先在 kube-apiserver 中开启 API Aggregator。
- Metrics API 只可以查询当前的度量数据，并不保存历史数据。
- Metrics API URI 为 /apis/metrics.k8s.io/，在 k8s.io/metrics 下维护。
- 必须部署 metrics-server 才能使用该 API，metrics-server 通过调用 kubelet Summary API 获取数据。
使用 Metrics Server 有必备两个条件：
- API Server 启用 Aggregator Routing 支持。否则 API Server 不识别请求：
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get pods.metrics.k8s.io)- API Server 能访问 Metrics Server Pod IP。否则 API Server 无法访问 Metrics Server：
E1223 07:23:04.330206 1 available_controller.go:420] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.171.248.214:4443/apis/metrics.k8s.io/v1beta1: Get https://10.171.248.214:4443/apis/metrics.k8s.io/v1beta1: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)启用 API Aggregator
API Aggregation 允许在不修改 Kubernetes 核心代码的同时扩展 Kubernetes API，即：将第三方服务注册到 Kubernetes API 中，这样就可以通过 Kubernetes API 来访问第三方服务了，例如：Metrics Server API。
注：另外一种扩展 Kubernetes API 的方法是使用 CRD（Custom Resource Definition，自定义资源定义）。
- 检查 API Server 是否开启了 Aggregator Routing：查看 API Server 是否具有
--enable-aggregator-routing=true选项。
$ ps -ef | grep apiserver
root 23896 29500 0 12:40 pts/0 00:00:00 grep --color=auto apiserver
root 28613 28551 1 12月21 ? 01:05:29 kube-apiserver --advertise-address=192.168.0.112 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port=0 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-cluster-ip-range=172.16.0.0/16 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key- 修改每个 API Server 的 kube-apiserver.yaml 配置开启 Aggregator Routing：修改 manifests 配置后会 API Server 会自动重启生效。
$ vi /etc/kubernetes/manifests/kube-apiserver.yaml
...
spec:
containers:
- command:
...
- --enable-aggregator-routing=true安装 Metrics Server
- 检查 Cluster 是否安装了 Metrics Server：
$ kubectl top pods
Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)- 部署 Metrics Server：
# 下载 YAML 文件
wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.4.1/components.yaml
# 编辑修改 metrics-server 的启动参数：
# --kubelet-insecure-tls 跳过 TLS 认证，否则会出现 x509 的认证问题，用于测试环境。
# --kubelet-preferred-address-types=InternalIP 使用 Node IP 进行通信。
- args:
- --cert-dir=/tmp
- --secure-port=4443
- --kubelet-preferred-address-types=InternalIP
- --kubelet-use-node-status-port
- --kubelet-insecure-tls
# 部署
$ kubectl apply -f components.yaml
serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created注意：如果出现了 ErrImagePull 的问题，那么意味着 k8s.gcr.io/metrics-server/metrics-server:v0.4.1 镜像下载失败了：
$ docker pull k8s.gcr.io/metrics-server/metrics-server:v0.4.1
Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)这时候就需要我们在每个节点上都手动的下载镜像了：
$ docker pull bitnami/metrics-server:0.4.1
$ docker tag bitnami/metrics-server:0.4.1 k8s.gcr.io/metrics-server/metrics-server:v0.4.1
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
bitnami/metrics-server 0.4.1 4fb6df85a88d 6 hours ago 171MB
k8s.gcr.io/metrics-server/metrics-server v0.4.1 4fb6df85a88d 6 hours ago 171MB然后再次执行 Metrics Server 的部署指令。
- 检查 Metrics Server Service：
$ kubectl get svc --all-namespaces | grep metrics-server
kube-system metrics-server ClusterIP 172.16.128.176 <none> 443/TCP 5h55m- 检查 API Server 是否可以连通 Metrics Server：
$ kubectl describe svc metrics-server -n kube-system
Name: metrics-server
Namespace: kube-system
Labels: k8s-app=metrics-server
Annotations: Selector: k8s-app=metrics-server
Type: ClusterIP
IP: 172.16.128.176
Port: https 443/TCP
TargetPort: https/TCP
Endpoints: 10.171.248.214:4443
Session Affinity: None
Events: <none>
# 在 Master Node 上 Ping。
$ ping 10.171.248.214
64 bytes from 10.171.248.214: icmp_seq=1 ttl=63 time=0.282 ms- 检查 Metrics Server：
$ kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
k8s-master-1 174m 8% 1156Mi 66%
k8s-master-2 123m 6% 1134Mi 65%
k8s-master-3 104m 5% 1075Mi 61%
k8s-node-1 78m 3% 853Mi 49%
k8s-node-2 78m 3% 824Mi 47%9、安装 Dashboard GUI
默认情况下不会部署 Dashboard，需要我们手动安装。在安装 Dashboard 之前我们还需要确保安装了 Metrics Server。否则，错误提示我们在 Cluster 中没有安装 Metrics Server。
$ kubectl logs -f -n kubernetes-dashboard dashboard-metrics-scraper-6b4884c9d5-qppkd
192.168.0.208 - - [21/Dec/2020:07:54:08 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.18"
{"level":"error","msg":"Error scraping node metrics: the server is currently unable to handle the request (get nodes.metrics.k8s.io)","time":"2020-12-21T07:54:09Z"}
$ kubectl logs -f -n kubernetes-dashboard kubernetes-dashboard-7b544877d5-p6g8t
2020/12/21 07:54:48 Metric client health check failed: the server is currently unable to handle the request (get services dashboard-metrics-scraper). Retrying in 30 seconds.另外，官方 YAML 部署的 Dashboard 的 Service 也不是 NodePort 类型的，我们需要手动修改。
- 编辑
$ wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0/aio/deploy/recommended.yaml
$ vi recommended.yaml
---
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
type: NodePort
ports:
- port: 443
targetPort: 8443
nodePort: 30000
selector:
k8s-app: kubernetes-dashboard
---- 部署：出于安全访问的考虑，默认情况下，Dashboard 会使用最基础的 RBAC 配置进行部署。
$ kubectl create -f recommended.yaml
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created- 检查 Dashboard Pods：
$ kubectl get pods --all-namespaces | grep dashboard
kubernetes-dashboard dashboard-metrics-scraper-6b4884c9d5-ctn6h 1/1 Running 0 79s
kubernetes-dashboard kubernetes-dashboard-7b544877d5-sghsf 1/1 Running 0 79s- 检查 Dashboard Services：检查是否与 YAML 描述的一致。
$ kubectl get svc -n kubernetes-dashboard
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
dashboard-metrics-scraper ClusterIP 172.16.165.243 <none> 8000/TCP 2m32s
kubernetes-dashboard NodePort 172.16.219.86 <none> 443:30000/TCP 2m33s10、访问 Dashboard UI
因为 kubernetes-dashboard Service 是 NodePod 类型，所以我们使用任意节点 IP + NodePortNum 就可以访问了。
注意：NodePort 并非 VIP HA LB 的方式，所以我们可以使用高可用部署环境中的 Load Balancer 进行代理。
由于 Dashboard 默认是自建的 HTTPS 证书，该证书是不受浏览器信任的，强制跳转即可。
登录 Dashboard 的时候支持 Kubeconfig 和 Token 两种认证方式，由于 Kubeconfig 中也依赖 Token 字段，所以生成 Token 这个步骤是必不可少的。
- 创建 Dashboard Admin 管理员：
$ cat dashboard-admin.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: kubernetes-dashboard
name: dashboard-admin
namespace: kubernetes-dashboard
$ kubectl create -f ./dashboard-admin.yaml
serviceaccount/dashboard-admin created
# CLI: kubectl create serviceaccount dashboard-admin -n kubernetes-dashboard- 为 Dashboard Admin 分配权限：
$ cat dashboard-admin-bind-cluster-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: dashboard-admin-bind-cluster-role
labels:
k8s-app: kubernetes-dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: dashboard-admin
namespace: kubernetes-dashboard
$ kubectl create -f ./dashboard-admin-bind-cluster-role.yaml
clusterrolebinding.rbac.authorization.k8s.io/dashboard-admin-bind-cluster-role created
# CLI: kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kubernetes-dashboard:dashboard-admin- 生成 Token：
$ kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep dashboard-admin | awk '{print $1}')
Name: dashboard-admin-token-bc2dh
Namespace: kubernetes-dashboard
Labels: <none>
Annotations: kubernetes.io/service-account.name: dashboard-admin
kubernetes.io/service-account.uid: 3c26e908-49b2-4de8-8f08-699dba736a56
Type: kubernetes.io/service-account-token
Data
====
ca.crt: 1025 bytes
namespace: 20 bytes
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IkV3a2JIWGhCdTFTSEFpd2hxam1WM29RTzQwcXdlT2dLYlRTRFU2TGotNHMifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4tYmMyZGgiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiM2MyNmU5MDgtNDliMi00ZGU4LThmMDgtNjk5ZGJhNzM2YTU2Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmVybmV0ZXMtZGFzaGJvYXJkOmRhc2hib2FyZC1hZG1pbiJ9.v24ad9kFp4_kEUo4pPVRxqHxe6rXCv4kZryR3QoWrCKTdHM2FmJPrU1q_w5wGyXVShQQGyBlQJpdAAX5RAIy4gVvLLbc7W9qE74jHVijb4pBp9j8r-lsYpGvpiXadVCqXYCXHBxQSeYKXNM6hwDGKFGxQZ2LDA8K_j590fVeozSxM1RBJxetM4KfF0KomDurOhjITu0rortufeKhkOvRcoB0EikPkwqbU0Q5Ip7m_YIYHZsEqo292GieTyfZIM0KRLGGn4GX53qTbgk1bhQYsequUjs3sKw5Ng8vSOq8NPX0vB88Mjl0fzPWtKvnvpWokg1JL_fW4-qKA7HbUcenXQ- 访问：
11、通过 NFS 实现持久化存储
NFS Server 安装
$ yum install -y nfs-utils
$ yum install -y rpcbind
$ mkdir /public
$ vi /etc/exports
/public 192.168.186.*
$ systemctl start rpcbind && systemctl enable rpcbind && systemctl status rpcbind
$ systemctl start nfs-server && systemctl enable nfs-server && systemctl status nfs-serverNFS Client 挂载
$ mkdir /mnt/public
$ vi /etc/fstab
192.168.186.198:/public /mnt/public nfs defaults 0 0
$ mount -aKubernetes 部署 nfs-client-provisioner
- 下载 nfs-client
git clone https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner.git- 创建 ServiceAccount
$ NS=$(kubectl config get-contexts|grep -e "^\*" |awk '{print $5}')
$ NAMESPACE=${NS:-default}
$ cd nfs-subdir-external-provisioner/deploy
$ sed -i'' "s/namespace:.*/namespace: $NAMESPACE/g" ./deploy/rbac.yaml ./deployment.yaml
$ kubectl create -f ./rbac.yaml- 部署 NFS Provisioner
$ vi deployment.yaml
...
spec:
serviceAccountName: nfs-client-provisioner
containers:
- name: nfs-client-provisioner
image: quay.io/external_storage/nfs-client-provisioner:latest
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: fuseim.pri/ifs
- name: NFS_SERVER
value: 192.168.186.198 # NFS Server IP
- name: NFS_PATH
value: /public # Share Dir Path
volumes:
- name: nfs-client-root
nfs:
server: 192.168.186.198 # NFS Server IP
path: /public # Share Dir Path
# 创建 NFS Provisioner
$ kubectl apply -f deployment.yaml- 创建 NFS Storage Class
kubectl apply -f class.yaml
本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。
