kvm证书过期 kubeadm证书过期

目录

• 一、查看证书过期时间

• 1.1 方式一
• 1.2 方式二

• 二、通过命令续期

• 2.1 修改集群内所有机器的时间，模拟证书在过期的边缘
• 2.2 查看证书有效期
• 2.3 备份原有数据
• 2.4 备份证书
• 2.5 更新证书
• 2.6 确认证书有效期
• 2.7 更新kubeconfig文件
• 2.8 更新客户端证书
• 2.9 重启相关的pod
• 2.10 查看pod是否运行正常
• 2.11 更新节点上kubelet证书有效期

• 三、编译源码kubeadm，证书时间自定义

• 3.1 备份集群配置
• 3.2 获取对应的kubeadm源码
• 3.3 修改CA证书有效期
• 3.4 修改其他证书有效期
• 3.5 安装go环境进行编译
• 3.6 go设置国内代理
• 3.7 编译kubeadm
• 3.8 替换kubeadm指令
• 3.9 更新集群证书
• 3.10 更新kubeconfig文件
• 3.11 重启相关pod
• 3.12 替换admin文件
• 3.13 确认指令正常
• 3.14 确认证书更新成功

K8S CA证书是10年，但是组件证书的日期只有1年，为了证书一直可用状态需要更新，目前主流的一共有3种：

1、版本升级，只要升级就会让各个证书延期1年，官方设置1年有效期的目的就是希望用户在一年内能升级1次；
2、通过命令续期 （这种只能延长一年）；
3、编译源码Kubeadm，证书有效期可自定义；

本实验环境是单master集群环境，如果是多master集群环境那么需要将master上更新的证书分发到各个节点上！

此文档采用K8s 1.18.3版本，不保证其他版本也适用，建议自行测试。

一、查看证书过期时间

1.1 方式一

$ kubeadm alpha certs check-expiration

个人搭建的集群不知道为什么，使用上述命令无法查看ca的证书有效期，所以记录上方式二！

1.2 方式二

$ for item in `find /etc/kubernetes/pki -maxdepth 2 -name "*.crt"`;
do openssl x509 -in $item -text -noout| grep Not;
echo ======================$item===============;
done

也可以一个一个的进行查看：

$ openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep ' Not '

二、通过命令续期

2.1 修改集群内所有机器的时间，模拟证书在过期的边缘

$ date -s "2022-3-1 12:00"

2.2 查看证书有效期

为了更直观的看到证书的有效期！

$ kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Mar 03, 2022 16:02 UTC   2d                                      no      
apiserver                  Mar 03, 2022 16:02 UTC   2d              ca                      no      
apiserver-etcd-client      Mar 03, 2022 16:02 UTC   2d              etcd-ca                 no      
apiserver-kubelet-client   Mar 03, 2022 16:02 UTC   2d              ca                      no      
controller-manager.conf    Mar 03, 2022 16:02 UTC   2d                                      no      
etcd-healthcheck-client    Mar 03, 2022 16:02 UTC   2d              etcd-ca                 no      
etcd-peer                  Mar 03, 2022 16:02 UTC   2d              etcd-ca                 no      
etcd-server                Mar 03, 2022 16:02 UTC   2d              etcd-ca                 no      
front-proxy-client         Mar 03, 2022 16:02 UTC   2d              front-proxy-ca          no      
scheduler.conf             Mar 03, 2022 16:02 UTC   2d                                      no      

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Mar 01, 2031 16:02 UTC   9y              no      
etcd-ca                 Mar 01, 2031 16:02 UTC   9y              no      
front-proxy-ca          Mar 01, 2031 16:02 UTC   9y              no

如果证书过期的话，就会出现以下情况：

$ kubectl get pod -n kube-system
Unable to connect to the server: x509: certificate has expired or is not yet valid

2.3 备份原有数据

$ kubeadm config view > /root/kubeadm.yaml 
$ cat /root/kubeadm.yaml
apiServer:
  extraArgs:
    authorization-mode: Node,RBAC
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.18.3
networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
scheduler: {}

2.4 备份证书

备份主要就是为了升级失败之后，便于回滚！

$ cp -rp /etc/kubernetes /etc/kubernetes_$(date +%F)
$ ls /etc/kubernetes_2022-03-01/
admin.conf  controller-manager.conf  kubelet.conf  manifests  pki  scheduler.conf

2.5 更新证书

$ kubeadm alpha certs renew all --config=/root/kubeadm.yaml

2.6 确认证书有效期

$ kubeadm alpha certs check-expiration
CERTIFICATE                EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
admin.conf                 Mar 01, 2023 04:02 UTC   364d            no      
apiserver                  Mar 01, 2023 04:02 UTC   364d            no      
apiserver-etcd-client      Mar 01, 2023 04:02 UTC   364d            no      
apiserver-kubelet-client   Mar 01, 2023 04:02 UTC   364d            no      
controller-manager.conf    Mar 01, 2023 04:02 UTC   364d            no      
etcd-healthcheck-client    Mar 01, 2023 04:02 UTC   364d            no      
etcd-peer                  Mar 01, 2023 04:02 UTC   364d            no      
etcd-server                Mar 01, 2023 04:02 UTC   364d            no      
front-proxy-client         Mar 01, 2023 04:02 UTC   364d            no      
scheduler.conf             Mar 01, 2023 04:02 UTC   364d            no

2.7 更新kubeconfig文件

$ rm -f /etc/kubernetes/*.conf
$ kubeadm init phase kubeconfig all --config /root/kubeadm.yaml

2.8 更新客户端证书

$ cp $HOME/.kube/config{,.default}
$ cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ chown $(id -u):$(id -g) $HOME/.kube/config

2.9 重启相关的pod

$ docker ps |egrep "k8s_kube-apiserver|k8s_kube-controller-manager|k8s_kube-scheduler|k8s_etcd_etcd" | awk '{print $1}' | xargs docker rm -f

也可以干脆一点，直接重启docker即可！

2.10 查看pod是否运行正常

$ kubectl get pod -A
NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE
kube-system   coredns-58cc8c89f4-8lq2k             1/1     Running   1          363d
kube-system   coredns-58cc8c89f4-hz774             1/1     Running   1          363d
kube-system   etcd-k8s-master                      1/1     Running   1          363d
kube-system   kube-apiserver-k8s-master            1/1     Running   1          363d
kube-system   kube-controller-manager-k8s-master   1/1     Running   1          363d
kube-system   kube-flannel-ds-amd64-fh9nx          1/1     Running   1          363d
kube-system   kube-flannel-ds-amd64-gmjth          1/1     Running   1          363d
kube-system   kube-flannel-ds-amd64-mvtdg          1/1     Running   1          363d
kube-system   kube-proxy-8dtfw                     1/1     Running   1          363d
kube-system   kube-proxy-9xwgb                     1/1     Running   1          363d
kube-system   kube-proxy-zcdvn                     1/1     Running   1          363d
kube-system   kube-scheduler-k8s-master            1/1     Running   1          363d

2.11 更新节点上kubelet证书有效期

$ cp /etc/kubernetes/kubelet.conf{,.default}
#kubeadm init phase kubeconfig kubelet --node-name <节点名称> --kubeconfig-dir /tmp/ --apiserver-advertise-address <集群VIP>，例如：
$ kubeadm init phase kubeconfig kubelet --node-name k8s-master --kubeconfig-dir /tmp/ --apiserver-advertise-address 10.4.7.10
$ \cp /tmp/kubelet.conf /etc/kubernetes/
$ systemctl restart kubelet

kubelet 的配置文件master节点可以和node节点共用！

三、编译源码kubeadm，证书时间自定义

3.1 备份集群配置

$ kubeadm config view > kubeadm-cluster.yaml    # 备份
$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"16", GitVersion:"v1.18.3", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T19:15:39Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
# 我这里的版本是1.18.3

3.2 获取对应的kubeadm源码

$ wget https://github.com/kubernetes/kubernetes/archive/v1.18.3.tar.gz
$ tar zxvf v1.18.3.tar.gz

3.3 修改CA证书有效期

$ vim kubernetes-1.18.3/staging/src/k8s.io/client-go/util/cert/cert.go
 65                 NotBefore:             now.UTC(),
 66                 NotAfter:              now.Add(duration365d * 100).UTC(),  # 默认是10，改成100
 67                 KeyUsage:              x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign,
 68                 BasicConstraintsValid: true,
 69                 IsCA:                  true,

3.4 修改其他证书有效期

$ vim kubernetes-1.18.3/cmd/kubeadm/app/constants/constants.go
# 跳转至46行，修改如下(追加 * 100):
 46         CertificateValidity = time.Hour * 24 * 365 * 100

3.5 安装go环境进行编译

$ wget https://dl.google.com/go/go1.13.9.linux-amd64.tar.gz
$ tar zxf go1.13.9.linux-amd64.tar.gz -C /usr/local/
$ echo 'export PATH=/usr/local/go/bin:$PATH' >> /etc/profile
$ source /etc/profile
$ go version
go version go1.13.9 linux/amd64

3.6 go设置国内代理

Golang V1.13之后支持通过设置变量GOPROXY来修改代理地址，默认的代理服务器，https://proxy.golang.org在国内访问经常出现timeout！详见：https://github.com/goproxy/goproxy.cn/blob/master/README.zh-CN.md 在终端执行即可！

$ go env -w GOPROXY=https://goproxy.cn,direct
$ go env -w GOSUMDB="sum.golang.google.cn"

3.7 编译kubeadm

$ cd kubernetes-1.18.3/        # 进入kubeadm源码目录
$ make all WHAT=cmd/kubeadm GOFLAGS=-v

3.8 替换kubeadm指令

$ cp /usr/bin/kubeadm{,.bak}
$ \cp _output/local/bin/linux/amd64/kubeadm /usr/bin

3.9 更新集群证书

$ kubeadm config view > kubeadm-cluster.yaml
# 如果有多个master节点，请将 kubeadm-cluster.yaml 文件和编译后的kubeadm指令发送至其他master节点

# 更新证书（若有多个master，则需要在所有master上执行）
$ kubeadm alpha certs renew all --config=kubeadm-cluster.yaml
W0904 07:23:15.938694   59308 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

3.10 更新kubeconfig文件

$ rm -f /etc/kubernetes/*.conf
$ kubeadm init phase kubeconfig all --config kubeadm-cluster.yaml 
W0904 07:25:41.882636   61426 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file

3.11 重启相关pod

在所有Master上执行重启kube-apiserver、kube-controller、kube-scheduler、etcd这4个容器，以便使证书生效。

$ docker ps |egrep "k8s_kube-apiserver|k8s_kube-controller-manager|k8s_kube-scheduler|k8s_etcd_etcd" | awk '{print $1}' | xargs docker restart

3.12 替换admin文件

$ cp ~/.kube/config{,.old}
$ \cp -i /etc/kubernetes/admin.conf ~/.kube/config
$ chown $(id -u):$(id -g) ~/.kube/config

3.13 确认指令正常

$ kubectl get pod -A
NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-5b8b769fcd-cpls6   1/1     Running   0          13h
kube-system   calico-node-2hk5w                          1/1     Running   0          13h
kube-system   calico-node-bwmmk                          1/1     Running   0          13h
kube-system   calico-node-gvldn                          1/1     Running   0          13h
kube-system   coredns-546565776c-g7j2f                   1/1     Running   0          13h
kube-system   coredns-546565776c-wtxt4                   1/1     Running   0          13h
kube-system   etcd-k8s-master                            1/1     Running   0          13h
kube-system   kube-apiserver-k8s-master                  1/1     Running   0          13h
kube-system   kube-controller-manager-k8s-master         1/1     Running   1          13h
kube-system   kube-proxy-bwkv6                           1/1     Running   0          13h
kube-system   kube-proxy-jdzps                           1/1     Running   0          13h
kube-system   kube-proxy-xjpxf                           1/1     Running   0          13h
kube-system   kube-scheduler-k8s-master                  1/1     Running   0          13h
kube-system   kuboard-7986796cf8-mk66v                   1/1     Running   0          12h
kube-system   metrics-server-7f96bbcc66-qldnm            1/1     Running   0          12h

3.14 确认证书更新成功

$ kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Mar 03, 2121 16:02 UTC   2d                                      no      
apiserver                  Mar 03, 2121 16:02 UTC   2d              ca                      no      
apiserver-etcd-client      Mar 03, 2121 16:02 UTC   2d              etcd-ca                 no      
apiserver-kubelet-client   Mar 03, 2121 16:02 UTC   2d              ca                      no      
controller-manager.conf    Mar 03, 2121 16:02 UTC   2d                                      no      
etcd-healthcheck-client    Mar 03, 2121 16:02 UTC   2d              etcd-ca                 no      
etcd-peer                  Mar 03, 2121 16:02 UTC   2d              etcd-ca                 no      
etcd-server                Mar 03, 2121 16:02 UTC   2d              etcd-ca                 no      
front-proxy-client         Mar 03, 2121 16:02 UTC   2d              front-proxy-ca          no      
scheduler.conf             Mar 03, 2121 16:02 UTC   2d                                      no      

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Mar 01, 2031 16:02 UTC   9y              no      
etcd-ca                 Mar 01, 2031 16:02 UTC   9y              no      
front-proxy-ca          Mar 01, 2031 16:02 UTC   9y              no

*************** 当你发现自己的才华撑不起野心时，就请安静下来学习吧！***************