在拷900G大数据时出现如下错误;
客户端产生的错误日志:文件写不进去;查看rsync问题,没有解决
day_140121/20140121_0c9cb0e43a3ee1cd57bd492YbauZrxQ8.jpg
32768 25% 15.62MB/s 0:00:00
rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32)
rsync: close failed on "/day_140121/.20140121_09389f3cdfa66f9a3a73rMgZU2MLNTo3.png.V65wiu": Input/output error (5)
rsync error: error in file IO (code 11) at receiver.c(730) [receiver=3.0.6]
rsync: connection unexpectedly closed (1434 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
tail: /tmp/a:文件已截断
day_131019/20131019_e6e35aa3326cdbb0bd19YDtGdzS8ay36_120x120.jpg
7843 100% 0.00kB/s 0:00:00 (xfer#1, to-check=1031/135274)
rsync: close failed on "/day_131019/.20131019_e6e35aa3326cdbb0bd19YDtGdzS8ay36_120x120.jpg.AKrGPn": Input/output error (5)
rsync error: error in file IO (code 11) at receiver.c(730) [receiver=3.0.6]
rsync: connection unexpectedly closed (1673 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
tail: /tmp/a:文件已截断
mfsmaster端错误日志,提示分配了元数据,但是块服务器没找到块的地址;
Jan 24 13:42:13 to8tobbs_master1 mfsmaster[7458]: (192.168.2.111:9422) chunk: 0000000000345B30 creation status: IO error
Jan 24 13:42:13 to8tobbs_master1 mfsmaster[7458]: (192.168.2.185:9422) chunk: 0000000000345B30 creation status: IO error
Jan 24 13:43:39 to8tobbs_master1 mfsmaster[7458]: chunk 0000000000345B30 has only invalid copies (2) - please repair it manually
Jan 24 13:43:39 to8tobbs_master1 mfsmaster[7458]: chunk 0000000000345B30_00000001 - invalid copy on (192.168.2.185 - ver:00000000)
Jan 24 13:43:39 to8tobbs_master1 mfsmaster[7458]: chunk 0000000000345B30_00000001 - invalid copy on (192.168.2.111 - ver:00000000)
Jan 24 13:43:42 to8tobbs_master1 mfsmaster[7458]: chunk 0000000000345B2F has only invalid copies (2) - please repair it manually
Jan 24 13:43:42 to8tobbs_master1 mfsmaster[7458]: chunk 0000000000345B2F_00000001 - invalid copy on (192.168.2.185 - ver:00000000)
在块服务器的日志里提示如下错误:
ERROR:No space left on device
到google.com里查了下,说是磁盘inode是文件的索引节点不够用了;
查看块服务器的磁盘inode,
df -i
全部用完了;
只好重新格式化磁盘,设置足够用的inode节点;重拷数据;
这里大家在部署mfs分布式文件系统时,块服务器的磁盘inode一定要设置清楚;
[root@web1 140123]# df -i
文件系统 Inode 已用(I) 可用(I) 已用(I)%% 挂载点
/dev/sda3 6766592 699208 6067384 11% /
tmpfs 6172064 1 6172063 1% /dev/shm
/dev/sda1 51200 44 51156 1% /boot
/dev/sdb1 122028032 15234160 106793872 13% /img
192.168.3.62:9421 1014750381 13744354 1001006027 2% /img_pic
这里我原来/img_pic的磁盘inode只有3456125一下子用完了,大于2T的磁盘分区大多数用于视频数据,所以默认的inode节点很小的,大家要注意下自己的数据是小文件多的情况,就要关注下inode;
[root@web1 140123]# df -h
文件系统 容量 已用 可用 已用%% 挂载点
/dev/sda3 102G 54G 43G 56% /
tmpfs 24G 0 24G 0% /dev/shm
/dev/sda1 194M 45M 140M 25% /boot
/dev/sdb1 1.8T 900G 841G 52% /img
192.168.3.62:9421 6.2T 1.7T 4.6T 27% /img_pic