Hadoop之——Permission denied error 13 - Python on Hadoop

原创

冰河技术 2019-07-14 22:14:13 ©著作权

文章标签 Hadoop 文章分类 Hadoop 大数据

©著作权归作者所有：来自51CTO博客作者冰河技术的原创作品，请联系作者获取转载授权，否则将追究法律责任

转载请注明出处：https://blog.csdn.net/l1028386804/article/details/95932591

问题：

在Hadoop中运行Python程序，命令行输入如下命令：

-bash-4.1$ hadoop jar /usr/local/hadoop-3.2.0/share/hadoop/tools/lib/hadoop-streaming-3.2.0.jar -file /home/hadoop/python/wordcount/wordcount_mapper.py -mapper wordcount_mapper.py -file /home/hadoop/python/wordcount/wordcount_reducer.py -reducer wordcount_reducer.py -input /data/input/word.txt -output /data/output

运行后报错：Permission denied error 13 - Python on Hadoop

具体报错信息如下:

2019-07-14 14:39:16 INFO util.NativeCodeLoader: Loaded the native-hadoop library
2019-07-14 14:39:16 WARN snappy.LoadSnappy: Snappy native library not loaded
2019-07-14 14:39:16 INFO mapred.FileInputFormat: Total input paths to process : 1
2019-07-14 14:39:16 INFO streaming.StreamJob: getLocalDirs(): [/tmp/hadoop-hadoop/mapred/local]
2019-07-14 14:39:16 INFO streaming.StreamJob: Running job: job_201509231312_0003
2019-07-14 14:39:16 INFO streaming.StreamJob: To kill this job, run:
2019-07-14 14:39:16 INFO streaming.StreamJob: /home/hadoop/hadoop-1.2.1/libexec/../bin/hadoop job -Dmapred.job.tracker=192.168.56.102:9001 -kill job_201509231312_0003
2019-07-14 14:39:16 INFO streaming.StreamJob: Tracking URL: http://192.168.56.102:50030/jobdetails.jsp?jobid=job_201509231312_0003
2019-07-14 14:39:17 INFO streaming.StreamJob: map 0% reduce 0%
2019-07-14 14:39:41 INFO streaming.StreamJob: map 100% reduce 100%
2019-07-14 14:39:41 INFO streaming.StreamJob: To kill this job, run:
2019-07-14 14:39:41 INFO streaming.StreamJob: /home/hadoop/hadoop-1.2.1/libexec/../bin/hadoop job -Dmapred.job.tracker=192.168.56.102:9001 -kill job_201509231312_0003
2019-07-14 14:39:41 INFO streaming.StreamJob: Tracking URL: http://192.168.56.102:50030/jobdetails.jsp?jobid=job_201509231312_0003
2019-07-14 14:39:41 ERROR streaming.StreamJob: Job not successful. Error: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201509231312_0003_m_000000
2019-07-14 14:39:41 INFO streaming.StreamJob: killJob...
Streaming Command Failed!

================================================================
java.io.IOException: Cannot run program "/tmp/hadoop-hadoop/mapred/local/taskTracker/hadoop/jobcache/job_201509231312_0003/attempt_201509231312_0003_m_000001_3/work/./mapper.py": error=13, Permission denied
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:214)
at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:426)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.io.IOException: error=13, Permission denied
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:186)
at java.lang.ProcessImpl.start(ProcessImpl.java:130)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
... 24 more

解决方案：

将hadoop jar命令中的

-mapper wordcount_mapper.py 修改为 -mapper "python wordcount_mapper.py"；

-reducer wordcount_reducer.py 修改为-reducer "python wordcount_reducer.py"

修改后的命令为：

hadoop jar /usr/local/hadoop-3.2.0/share/hadoop/tools/lib/hadoop-streaming-3.2.0.jar -file /home/hadoop/python/wordcount/wordcount_mapper.py -mapper "python wordcount_mapper.py" -file /home/hadoop/python/wordcount/wordcount_reducer.py -reducer "python wordcount_reducer.py" -input /data/input/word.txt -output /data/output