k8s中namespace有两种常见的状态,即Active和Terminating状态,其中Terminating状态一般会比较少见,当对应的命名空间下还存在运行的资源,但该命名空间被删除时才会出现所谓的Terminating状态,这种情况下只要等待k8s本身将命名空间下的资源回收后,该命名空间将会被系统自动删除。
但是在某些情况下,即使命名空间下没有运行的资源,但依然无法删除Terminating状态的命名空间的情况,它会一直卡在Terminating状态下,即使用--force --grace-period=0 也删除不了。
接下来的步骤将描述如何手动删除卡住的命名空间:
- Run the following command to view the namespaces that are stuck in the Terminating state:
kubectl get namespaces
- Select a terminating namespace and view the contents of the namespace to find out the
finalizer
. Run the following command:
kubectl get namespace <terminating-namespace> -o yaml
Your YAML contents might resemble the following output:
apiVersion: v1
kind: Namespace
metadata:
creationTimestamp: 2018-11-19T18:48:30Z
deletionTimestamp: 2018-11-19T18:59:36Z
name: <terminating-namespace>
resourceVersion: "1385077"
selfLink: /api/v1/namespaces/<terminating-namespace>
uid: b50c9ea4-ec2b-11e8-a0be-fa163eeb47a5
spec:
finalizers:
- kubernetes
status:
phase: Terminating
- Run the following command to create a temporary JSON file:
kubectl get namespace <terminating-namespace> -o json >tmp.json
- Edit your
tmp.json
file. Remove thekubernetes
value from thefinalizers
field and save the file.
Yourtmp.json
file might resemble the following output:
{
"apiVersion": "v1",
"kind": "Namespace",
"metadata": {
"creationTimestamp": "2018-11-19T18:48:30Z",
"deletionTimestamp": "2018-11-19T18:59:36Z",
"name": "<terminating-namespace>",
"resourceVersion": "1385077",
"selfLink": "/api/v1/namespaces/<terminating-namespace>",
"uid": "b50c9ea4-ec2b-11e8-a0be-fa163eeb47a5"
},
"spec": {
"finalizers":
},
"status": {
"phase": "Terminating"
}
}
- To set a temporary proxy IP and port, run the following command. Be sure to keep your terminal window open until you delete the stuck namespace:
kubectl proxy
Your proxy IP and port might resemble the following output:
Starting to serve on 127.0.0.1:8001
- From a new terminal window, make an API call with your temporary proxy IP and port:
curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/<terminating-namespace>/finalize
Your output might resemble the following content:
{
"kind": "Namespace",
"apiVersion": "v1",
"metadata": {
"name": "<terminating-namespace>",
"selfLink": "/api/v1/namespaces/<terminating-namespace>/finalize",
"uid": "b50c9ea4-ec2b-11e8-a0be-fa163eeb47a5",
"resourceVersion": "1602981",
"creationTimestamp": "2018-11-19T18:48:30Z",
"deletionTimestamp": "2018-11-19T18:59:36Z"
},
"spec": {
},
"status": {
"phase": "Terminating"
}
}
Note: The finalizer
parameter is removed.
- Verify that the terminating namespace is removed, run the following command:
kubectl get namespaces
Continue to follow the steps for other namespaces that are stuck in the Terminating state.
参考:https://www.ibm.com/support/knowledgecenter/en/SSBS6K_3.1.1/troubleshoot/ns_terminating.html
参考:https://support.huaweicloud.com/cce_faq/cce_faq_00277.html