Openstack 无法删除存储卷

报错日志

[root@st-controller03 ~]# tailf /var/log/cinder/volume.log

2021-05-31 16:39:31.710 22870 WARNING cinder.volume.drivers.rbd [req-902b0232-155f-4a65-9796-bcb20d70f716 083a5c802fc748658db1821f3f778ddf dd3bc80d13fb431bacc466cbaaf7dbd7 - - -] ImageBusy error raised while deleting rbd volume. This may have been caused by a connection from a client that has crashed and, if so, may be resolved by retrying the delete after 30 seconds has elapsed.
2021-05-31 16:39:31.715 22870 ERROR cinder.volume.manager [req-902b0232-155f-4a65-9796-bcb20d70f716 083a5c802fc748658db1821f3f778ddf dd3bc80d13fb431bacc466cbaaf7dbd7 - - -] Unable to delete busy volume.

找到没有连接的卷，闲置的表现故障是删除这个卷之后显示删除成功，但是之后还是显示可用，日志有上面的提示。

[root@st-controller01 ~]# cinder list|grep ava
| 1a318065-c649-4d6a-883a-5fe442e1fb79 | available |                                      |  22  |      -      |   true   |                                      |
| da605031-49d3-49c9-9b7d-968b100dcb87 | available |                                      |  54  |      -      |   true   |                                      |
| edf140c7-afdc-4e8f-a105-7881106d785a | available |                                      |  43  |      -      |   true   |                                      |

在存储集群中找打这个卷

[root@st-ceph01 ~]# rbd ls -p volumes |grep da605031-49d3-49c9-9b7d-968b100dcb87
volume-da605031-49d3-49c9-9b7d-968b100dcb87

查看卷的信息

[root@st-ceph01 ~]# rbd info volume-da605031-49d3-49c9-9b7d-968b100dcb87 -p volumes
rbd image 'volume-da605031-49d3-49c9-9b7d-968b100dcb87':
	size 54GiB in 13824 objects
	order 22 (4MiB objects)
	block_name_prefix: rbd_data.2be40e33483d1e
	format: 2
	features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
	flags: 
	create_timestamp: Wed Feb 10 23:51:03 2021
	# 有一个快照
	parent: volumes/0ad7ae89-8b34-4db4-a873-7904c46ce17b@snap
	overlap: 50GiB
	
# 查看这个卷有没有客户端，如果有可以吧客户端断开之后再删除，比如rbd map
[root@st-ceph01 ~]# rbd status volume-da605031-49d3-49c9-9b7d-968b100dcb87 -p volumes
Watchers: none

# 查看快照信息，可以看出这个快照，并且是受保护状态，如果要删除，确定对集群不会产生影响，删除快照再删除块设备即可。
[root@st-ceph01 ~]# rbd info volumes/0ad7ae89-8b34-4db4-a873-7904c46ce17b@snap
rbd image '0ad7ae89-8b34-4db4-a873-7904c46ce17b':
	size 50GiB in 6400 objects
	order 23 (8MiB objects)
	block_name_prefix: rbd_data.2b6dd7341ee655
	format: 2
	features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
	flags: 
	create_timestamp: Wed Feb 10 23:18:07 2021
	protected: True

处理方法二

# 查看快照的子信息，
[root@st-ceph01 ~]# rbd children volumes/0ad7ae89-8b34-4db4-a873-7904c46ce17b@snap
volumes/volume-da605031-49d3-49c9-9b7d-968b100dcb87

# 直接吧这个快照给独立出来成为一个单独的rbd
[root@st-ceph01 ~]# rbd flatten volumes/volume-da605031-49d3-49c9-9b7d-968b100dcb87
Image flatten: 34% complete...

# 然后，在删除镜像就可以了。
[root@st-ceph01 ~]# rbd info volume-da605031-49d3-49c9-9b7d-968b100dcb87 -p volumes
rbd image 'volume-da605031-49d3-49c9-9b7d-968b100dcb87':
	size 54GiB in 13824 objects
	order 22 (4MiB objects)
	block_name_prefix: rbd_data.2be40e33483d1e
	format: 2
	features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
	flags: 
	create_timestamp: Wed Feb 10 23:51:03 2021

查看块设备实际占用大小

rbd diff volumes/volume-14b85304-b5cf-4005-a846-2e1858368c28 | awk '{ SUM += $2 } END { print SUM/1024/1024 " MB" }'