Updated August 8, 2019By Bobbin ZachariahLINUX HOWTO, TROUBLESHOOTING
When you do an NFS mount, it sometimes shows "device is busy” status, in such case we need to perform force unmount in a graceful way. There are different ways and options we can try out if normal nfs unmount fails.
Scenario
In our scenario, we have created /var/linoxide directory for the mount. When we try to umount the remote partition, we have an error message. Good to read on NFS Mount Options in Linux.
You can all the mounted folders with the df command
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 20G 1.3G 18G 7% /
devtmpfs 236M 0 236M 0% /dev
tmpfs 245M 0 245M 0% /dev/shm
tmpfs 245M 8.4M 237M 4% /run
tmpfs 245M 0 245M 0% /sys/fs/cgroup
tmpfs 49M 0 49M 0% /run/user/0
10.128.20.241:/var/linoxide 20G 1.3G 18G 7% /mnt/nfs/linoxide_srv
10.128.20.241:/home 20G 1.3G 18G 7% /mnt/nfs/home_srv
In the last two lines, you can see mounted folders on the client. Below example shows the unmount fails because the device is busy
# umount /mnt/nfs/linoxide_srv/
umount.nfs4: /mnt/nfs/linoxide_srv: device is busy
1) With lsof
The lsof (list open files) command displays a list of all open files and the processes associated with them on a specific file system, directory, or device. By default, it lists all files, shared libraries, and directories that are currently open and provides as much information as possible about each of them. The output gives us some information such as the PID, USER so that we can use a pipe to filter its output.
# lsof /mnt/nfs/linoxide_srv/
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
bash 24098 root cwd DIR 253,1 4096 519062 /mnt/nfs/linoxide_srv
bash 24125 root cwd DIR 253,1 4096 519062 /mnt/nfs/linoxide_srv
vim 24144 linoxide cwd DIR 253,1 4096 519062 /mnt/nfs/linoxide_srv
You can see that we have the PID of process which uses the mounted folder, we see the commands in execution, the user who executes the command. It is possible to kill the busy process but take care of the executed command. You can see vim command which means that a file is being edited by the linoxide user. So if we kill the process, his progress will be lost. Suppose that we have informed him, let's see the result
# lsof /mnt/nfs/linoxide_srv/
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
bash 24098 root cwd DIR 253,1 4096 519062 /mnt/nfs/linoxide_srv
bash 24125 root cwd DIR 253,1 4096 519062 /mnt/nfs/linoxide_srv
You can look that our user has stopped his modification but we still have bash command in execution but we don't know why. We can now kill the two processes with kill command. Be sure to don't miss the pid of the process to kill.
Now we will kill the first bash process
# kill -9 24098
We can verify the result
# lsof /mnt/nfs/linoxide_srv/
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
bash 24125 root cwd DIR 253,1 4096 519062 /mnt/nfs/linoxide_srv
We can see that one process is killed
# kill -9 24125
Now let's verify for the second process
# lsof /mnt/nfs/linoxide_srv/
Now let's try to unmount the folder
# umount /mnt/nfs/linoxide_srv/
umount: /mnt/nfs/linoxide_srv/: not mounted
It seems that kill the process has automatically unmounted the folder but let's check with df command
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 20G 1.3G 18G 7% /
devtmpfs 236M 0 236M 0% /dev
tmpfs 245M 0 245M 0% /dev/shm
tmpfs 245M 8.3M 237M 4% /run
tmpfs 245M 0 245M 0% /sys/fs/cgroup
tmpfs 49M 0 49M 0% /run/user/0
10.128.20.241:/home 20G 1.3G 18G 7% /mnt/nfs/home_srv
The folder /mnt/nfs/linoxide_srv has been unmounted as we want.
2) With fuser
The fuser (find user processes) command helps to identify processes that are preventing you from unmounting file systems. It finds user processes that are associated with whatever files, directories, or file system mount points that you supply as command-line arguments.
# fuser /mnt/nfs/linoxide_srv/
/mnt/nfs/linoxide_srv: 24191c
We can use fuser command with -m option which lists all the processes accessing the files or mount point on the file system and the -v option which shows a result like ps command with PID, user and the executed command.
# fuser -mv /mnt/nfs/linoxide_srv/
USER PID ACCESS COMMAND
/mnt/nfs/linoxide_srv:
root kernel mount /mnt/nfs/home_srv
root 24191 ..c.. bash
root 24275 ..c.. bash
linoxide 24290 ..c.. vim
You can see the command in execution. We must prevent our linoxide user to save his work.
# fuser -mv /mnt/nfs/linoxide_srv/
USER PID ACCESS COMMAND
/mnt/nfs/linoxide_srv:
root kernel mount /mnt/nfs/home_srv
root 24191 ..c.. bash
root 24275 ..c.. bash
With fuser command, it is possible to directly kill the process in execution with -k option without kill command
# fuser -kmv /mnt/nfs/linoxide_srv/
USER PID ACCESS COMMAND
/mnt/nfs/linoxide_srv:
root kernel mount /mnt/nfs/home_srv
root 24191 ..c.. bash
root 24275 ..c.. bash
Check the result
# fuser -mv /mnt/nfs/linoxide_srv/
USER PID ACCESS COMMAND
/mnt/nfs/linoxide_srv:
root kernel mount /mnt/nfs/home_srv
It seems that only the mount is in execution. Let's try to unmount the folder
# umount /mnt/nfs/linoxide_srv/
We don't have error message. Check the mount point
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 20G 1.3G 18G 7% /
devtmpfs 236M 0 236M 0% /dev
tmpfs 245M 0 245M 0% /dev/shm
tmpfs 245M 8.3M 237M 4% /run
tmpfs 245M 0 245M 0% /sys/fs/cgroup
tmpfs 49M 0 49M 0% /run/user/0
10.128.20.241:/home 20G 1.3G 18G 7% /mnt/nfs/home_srv
We can see that the /mnt/nfs/linoxide_srv folder has been unmounted as we want.
3) Lazy unmount
umount command has an -l option to perform a lazy unmount. The mount will be removed from the filesystem namespace (so you won't see it under /mnt/nfs/linoxide anymore) but it stays mounted, so programs accessing it can continue to do so. When the last program accessing it exits, the unmount will actually occur.
# fuser -mv /mnt/nfs/linoxide_srv/
USER PID ACCESS COMMAND
/mnt/nfs/linoxide_srv:
root kernel mount /mnt/nfs/home_srv
root 24366 ..c.. bash
root 24381 ..c.. bash
linoxide 24398 ..c.. vim
We can see that the folder is busy. Now let's try to do a lazy unmount
# umount -l /mnt/nfs/linoxide_srv/
We don't have an error message. We will check if the command was being executed without error
# echo $?
0
Now let's check the mount point
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 20G 1.3G 18G 7% /
devtmpfs 236M 0 236M 0% /dev
tmpfs 245M 0 245M 0% /dev/shm
tmpfs 245M 8.4M 237M 4% /run
tmpfs 245M 0 245M 0% /sys/fs/cgroup
tmpfs 49M 0 49M 0% /run/user/0
10.128.20.241:/home 20G 1.3G 18G 7% /mnt/nfs/home_srv
We can see that the mount point /mnt/nfs/linoxide_srv doesn't appear again but as we said earlier, for example, our linoxide user is still modifying his file, can create new files, etc. On the server, we can see the file that the user is modifying.
We can need to unmount a partition because of an emergency or simply to remove a device but a problem can occur because that device is busy. It is important to examine every process on the system before taking a decision for the method to resolve the problem. The lsof and fuser commands make it easy to identify the processes that are preventing you from unmounting a file system.
