Error: java.io.IOException: All datanodes 10.3.1.63:56010 are bad. Aborting...
13/08/27 15:47:09 INFO mapreduce.Job: Task Id : attempt_1377256988757_0077_r_000000_2, Status : FAILED
Error: java.io.IOException: All datanodes 10.3.1.63:56010 are bad. Aborting...
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:941)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:756)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:425)
经查明，问题原因是linux机器打开了过多的文件导致。用命令ulimit -n可以发现linux默认的文件打开数目为1024
用root帐户修改/etc/security/limits.conf 添加如下一行：* - nofile 102400
修改/etc/pam.d/login添加如下一行 session required pam_limits.so
Refs：
永久修改linux open file: http://www.5dlinux.com/article/1/2007/linux_2713.html
hadoop使用中的几个小细节：http://www.alidata.org/archives/245
