主机重启后k8s kubelet无法自动启动问题解决梳理

原创

小黑要上天 2023-10-27 08:53:45 博主文章分类：k8s ©著作权

文章标签 k8s 文章分类 运维 yyds干货盘点

©著作权归作者所有：来自51CTO博客作者小黑要上天的原创作品，请联系作者获取转载授权，否则将追究法律责任

1.问题描述

OS Version:CentOS Linux release 7.9.2009 (Core)

K8S Version:Kubernetes v1.20.4

K8S安装配置完成后，重启服务器发现，kubelet没有正常启动(systemctl status kubelet)

命令：

systemctl status kubelet

[root@centos79-3 ~]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: activating (auto-restart) (Result: exit-code) since Wed 2023-10-11 10:08:43 CST; 1s ago
     Docs: https://kubernetes.io/docs/
  Process: 8583 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=255)
 Main PID: 8583 (code=exited, status=255)

Oct 11 10:08:43 centos79-3 kubelet[8583]: goroutine 148 [chan receive]:
Oct 11 10:08:43 centos79-3 kubelet[8583]: k8s.io/kubernetes/vendor/k8s.io/client-go...0)
Oct 11 10:08:43 centos79-3 kubelet[8583]: /workspace/src/k8s.io/kubernetes/_output/...aa
Oct 11 10:08:43 centos79-3 kubelet[8583]: created by k8s.io/kubernetes/vendor/k8s.i...er
Oct 11 10:08:43 centos79-3 kubelet[8583]: /workspace/src/k8s.io/kubernetes/_output/...6e
Oct 11 10:08:43 centos79-3 kubelet[8583]: goroutine 149 [chan receive]:
Oct 11 10:08:43 centos79-3 kubelet[8583]: k8s.io/kubernetes/vendor/k8s.io/client-go...0)
Oct 11 10:08:43 centos79-3 kubelet[8583]: /workspace/src/k8s.io/kubernetes/_output/...aa
Oct 11 10:08:43 centos79-3 kubelet[8583]: created by k8s.io/kubernetes/vendor/k8s.i...er
Oct 11 10:08:43 centos79-3 kubelet[8583]: /workspace/src/k8s.io/kubernetes/_output/...6e
Hint: Some lines were ellipsized, use -l to show in full.
[root@centos79-3 ~]#

输入kubectl -n kube-system get pods命令，有如下报错：

命令：

kubectl -n kube-system get pods

[root@centos79-3 ~]# kubectl -n kube-system get pods
The connection to the server 192.168.10.236:6443 was refused - did you specify the right host or port?
[root@centos79-3 ~]#

2.原因分析

由于K8S的要求，在部署K8S的服务器，必须全程关闭交换内存(swap)，之前安装K8S时，我只是使用swapoff -a命令暂时关闭swap，而机器重启后，swap还是会自动启动，从而导致kubelet无法启动。

3.解决方案

3.1.方案一

1.首先执行命令关闭swap

命令：

swapoff -a

[root@centos79-3 ~]# free -m        --关闭前
              total        used        free      shared  buff/cache   available
Mem:           3931         384        3121          27         425        3306
Swap:          3967           0        3967
[root@centos79-3 ~]# swapoff -a     --执行关闭
[root@centos79-3 ~]# free -m        --关闭后
              total        used        free      shared  buff/cache   available
Mem:           3931         420        3034          27         476        3269
Swap:             0           0           0
[root@centos79-3 ~]#

2.编辑/etc/fstab文件

命令：

vim /etc/fstab

3.将/dev/mapper/centos-swap swap swap default 0 0这一行注释掉(在其前面加个#)。

#
# /etc/fstab
# Created by anaconda on Tue Jun  6 13:46:52 2023
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root /                       xfs     defaults        0 0
UUID=56bcbfd5-823b-48ef-af0e-0d6269c5c353 /boot                   xfs     defaults        0 0
/dev/mapper/centos-home /home                   xfs     defaults        0 0
#/dev/mapper/centos-swap swap                    swap    defaults        0 0     --#号注释

4.编辑完成后保存退出，重启服务器后kubelet就可以正常自动启动了

命令：

cat /etc/fstab

[root@centos79-3 ~]# cat /etc/fstab 

#
# /etc/fstab
# Created by anaconda on Tue Jun  6 13:46:52 2023
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root /                       xfs     defaults        0 0
UUID=56bcbfd5-823b-48ef-af0e-0d6269c5c353 /boot                   xfs     defaults        0 0
/dev/mapper/centos-home /home                   xfs     defaults        0 0
#/dev/mapper/centos-swap swap                    swap    defaults        0 0
[root@centos79-3 ~]#

5.验证

命令：

kubectl -n kube-system get pods

[root@centos79-3 ~]# kubectl -n kube-system get pods
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-7854b85cf7-hn2zk   1/1     Running   2          14h
calico-node-6wt56                          1/1     Running   2          14h
calico-node-f2s8w                          1/1     Running   1          14h
coredns-7f89b7bc75-plrq9                   1/1     Running   2          17h
coredns-7f89b7bc75-xdx8j                   1/1     Running   2          17h
etcd-centos79-3                            1/1     Running   3          17h
kube-apiserver-centos79-3                  1/1     Running   5          17h
kube-controller-manager-centos79-3         1/1     Running   3          17h
kube-proxy-gbl8k                           1/1     Running   3          17h
kube-proxy-l8mm8                           1/1     Running   2          17h
kube-scheduler-centos79-3                  1/1     Running   3          17h
[root@centos79-3 ~]#

kubectl -n kube-system get nodes

[root@centos79-3 ~]# kubectl -n kube-system get nodes
NAME         STATUS   ROLES                  AGE   VERSION
cent79-2     Ready    <none>                 17h   v1.20.4
centos79-3   Ready    control-plane,master   17h   v1.20.4
[root@centos79-3 ~]#