在日常运维中，经常遇到磁盘空间满，但是找不到相应文件的情况。
通常这种情况都是文件被删除，但是还被进程占用，造成du与df结果不一致。
处理办法通常是停止占用文件的进程。
但是如果进程不能被停止呢？
另一个处理办法就是通过清空文件释放空间。
处理办法如下：
- 通过lsof | grep deleted 找到未能删除掉的文件，确定占用的进程号；
- 通过 ls -l /proc/PID/fd/* | grep 文件名，找到相应文件句柄；
- 清除文件内容 echo > /proc/PID/fd/FD_NUM
这个操作被不会将文件删除，而是通过将文档内容清空的方法释放空间，文件还是存在的。
实验如下：
1.创造一个大文件
使用dd创建1个5000MB的文件，看df的输出，可用空间从13G降到了7.5G。
[root@test1 /]# df -TH
Filesystem Type Size Used Avail Use% Mounted on
devtmpfs devtmpfs 2.0G 0 2.0G 0% /dev
tmpfs tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs tmpfs 2.0G 30M 2.0G 2% /run
tmpfs tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
/dev/mapper/centos-root xfs 39G 27G 13G 68% /
/dev/sda1 xfs 1.1G 394M 671M 37% /boot
tmpfs tmpfs 396M 0 396M 0% /run/user/0
[root@test1 /]# dd if=/dev/zero of=/delete.tmp bs=1000MB count=5
5+0 records in
5+0 records out
5000000000 bytes (5.0 GB) copied, 5.35441 s, 934 MB/s
[root@test1 /]# df -TH
Filesystem Type Size Used Avail Use% Mounted on
devtmpfs devtmpfs 2.0G 0 2.0G 0% /dev
tmpfs tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs tmpfs 2.0G 30M 2.0G 2% /run
tmpfs tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
/dev/mapper/centos-root xfs 39G 32G 7.5G 81% /
/dev/sda1 xfs 1.1G 394M 671M 37% /boot
tmpfs tmpfs 396M 0 396M 0% /run/user/0
[root@test1 /]# du -sh /delete.tmp
4.7G /delete.tmp
2.使用tail 打开文件
用tail 打开文件，保证删除文件时，文件仍被占用
[root@test1 /]# tail -f /delete.tmp3.删除文件
使用rm 删除文件，在以下df输出中会发现，可用空间还是7.5G，没有变化，但是文件已经消失了。
[root@test1 /]# rm -f /delete.tmp
[root@test1 /]# df -TH
Filesystem Type Size Used Avail Use% Mounted on
devtmpfs devtmpfs 2.0G 0 2.0G 0% /dev
tmpfs tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs tmpfs 2.0G 30M 2.0G 2% /run
tmpfs tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
/dev/mapper/centos-root xfs 39G 32G 7.5G 81% /
/dev/sda1 xfs 1.1G 394M 671M 37% /boot
tmpfs tmpfs 396M 0 396M 0% /run/user/0
[root@test1 /]# du -sh /delete.tmp
du: cannot access ‘/delete.tmp’: No such file or directory
4.查找删除的文件
lsof 显示了deleted状态的文件名和大小（5000000000）。
[root@test1 ~]# lsof | grep deleted
tail 419 root 3r REG 253,0 5000000000 55981 /delete.tmp (deleted)
5.查找文件句柄
[root@test1 ~]# ll /proc/419/fd | grep delete.tmp
lr-x------ 1 root root 64 May 23 16:05 3 -> /delete.tmp (deleted)
6.清空文件
[root@test1 ~]# echo > /proc/419/fd/3
[root@test1 ~]# df -TH
Filesystem Type Size Used Avail Use% Mounted on
devtmpfs devtmpfs 2.0G 0 2.0G 0% /dev
tmpfs tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs tmpfs 2.0G 30M 2.0G 2% /run
tmpfs tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
/dev/mapper/centos-root xfs 39G 27G 13G 68% /
/dev/sda1 xfs 1.1G 394M 671M 37% /boot
tmpfs tmpfs 396M 0 396M 0% /run/user/0
那么/proc/PID/fd 是啥呢？
man proc
/proc/[pid]/fd/
This is a subdirectory containing one entry for each file
which the process has open, named by its file descriptor,
and which is a symbolic link to the actual file. Thus, 0
is standard input, 1 standard output, 2 standard error,
and so on.
For file descriptors for pipes and sockets, the entries
will be symbolic links whose content is the file type with
the inode. A readlink(2) call on this file returns a
string in the format:
type:[inode]
For example, socket:[2248868] will be a socket and its
inode is 2248868. For sockets, that inode can be used to
find more information in one of the files under
/proc/net/.
For file descriptors that have no corresponding inode
(e.g., file descriptors produced by bpf(2),
epoll_create(2), eventfd(2), inotify_init(2),
perf_event_open(2), signalfd(2), timerfd_create(2), and
userfaultfd(2)), the entry will be a symbolic link with
contents of the form
anon_inode:<file-type>
In many cases (but not all), the file-type is surrounded
by square brackets.
For example, an epoll file descriptor will have a symbolic
link whose content is the string anon_inode:[eventpoll].
In a multithreaded process, the contents of this directory
are not available if the main thread has already
terminated (typically by calling pthread_exit(3)).
Programs that take a filename as a command-line argument,
but don't take input from standard input if no argument is
supplied, and programs that write to a file named as a
command-line argument, but don't send their output to
standard output if no argument is supplied, can
nevertheless be made to use standard input or standard
output by using /proc/[pid]/fd files as command-line
arguments. For example, assuming that -i is the flag
designating an input file and -o is the flag designating
an output file:
$ foobar -i /proc/self/fd/0 -o /proc/self/fd/1 ...
and you have a working filter.
/proc/self/fd/N is approximately the same as /dev/fd/N in
some UNIX and UNIX-like systems. Most Linux MAKEDEV
scripts symbolically link /dev/fd to /proc/self/fd, in
fact.
Most systems provide symbolic links /dev/stdin,
/dev/stdout, and /dev/stderr, which respectively link to
the files 0, 1, and 2 in /proc/self/fd. Thus the example
command above could be written as:
$ foobar -i /dev/stdin -o /dev/stdout ...
Permission to dereference or read (readlink(2)) the
symbolic links in this directory is governed by a ptrace
access mode PTRACE_MODE_READ_FSCREDS check; see ptrace(2).
Note that for file descriptors referring to inodes (pipes
and sockets, see above), those inodes still have
permission bits and ownership information distinct from
those of the /proc/[pid]/fd entry, and that the owner may
differ from the user and group IDs of the process. An
unprivileged process may lack permissions to open them, as
in this example:
$ echo test | sudo -u nobody cat
test
$ echo test | sudo -u nobody cat /proc/self/fd/0
cat: /proc/self/fd/0: Permission denied
File descriptor 0 refers to the pipe created by the shell
and owned by that shell's user, which is not nobody, so
cat does not have permission to create a new file
descriptor to read from that inode, even though it can
still read from its existing file descriptor 0.
