- 对服务进行重启后发现pod状态一直是Terminating
- 查看pod详情
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedKillPod 3m24s (x81 over 5h23m) kubelet error killing pod: [failed to "KillContainer" for "xxxxx-xxx" with KillContainerError: "rpc error: code = Unknown desc = operation timeout: context deadline exceeded", failed to "KillPodSandbox" for "688342f1-f270-493e-ae3b-59bf0272a1df" with KillPodSandboxError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"]
- 处理过程
网上查了资料,说是查看docker和kubelet状态之类,回到环境查看发现kubelet状态无法查看,不多docker服务显示运行正常的。有文档说是要重启容器,无奈根据报错pod未找到容器。
Unit kubelet.service could not be found.
最后受到floud文章启发,对故障pod进行强制删除。成功解决问题。
- 强制删除Terminating状态的pod
删除前先将deployment调度为0。
kubectl -n <ns> delete pod foo --grace-period=0 --force