任何程序只要可以从标准输入流中读取数据并且可以写入数据到标准输出流就可以通过hadoop流使用其他语言编写mapreduce程序的map函数和reduce函数。map的输出作为reduce的输入。
####使用shell的hadoop流测试:
1 本地新建的input目录中创建3个文件:
ashin@linux:~/test/hadoop/input$ echo "ashin hello blog hadoop" >> f1 ashin@linux:~/test/hadoop/input$ echo "milk test hi blog hadoop" >> f2 ashin@linux:~/test/hadoop/input$ echo "milk miss ashin blog hadoop" >> f3
2 将创建的文件传到hadoop的hdfs上:
ashin@linux:~/test/hadoop/input$ cd ../hadoop-1.1.2/bin/ ashin@linux:~/test/hadoop/hadoop-1.1.2/bin$ ./hadoop fs -put ../../input/ input1
input1是在hdfs上的目录,默认为user/ashin/input1
ashin@linux:~/test/hadoop/hadoop-1.1.2/bin$ ./hadoop fs -ls Found 5 items drwxr-xr-x - ashin supergroup 0 2013-04-22 19:30 /user/ashin/input drwxr-xr-x - ashin supergroup 0 2013-04-22 19:35 /user/ashin/input1 drwxr-xr-x - ashin supergroup 0 2013-04-22 19:21 /user/ashin/output drwxr-xr-x - ashin supergroup 0 2013-04-22 19:35 /user/ashin/output1 drwxr-xr-x - ashin supergroup 0 2013-04-22 19:44 /user/ashin/output2
删除hdfs上的目录:
ashin@linux:~/test/hadoop/hadoop-1.1.2/bin$ ./hadoop fs -rmr /user/ashin/input Deleted hdfs://192.168.2.180:9000/user/ashin/input
3 hadoop流使用shell命令统计文件内容信息:
ashin@linux:~/test/hadoop/hadoop-1.1.2/bin$ ./hadoop jar ../contrib/streaming/hadoop-streaming-1.1.2.jar -input input -output output1 -mapper /bin/cat -reducer /usr/bin/wc
4 查看结果(文件行数,单词数,字节数):
ashin@linux:~/test/hadoop/hadoop-1.1.2/bin$ ./hadoop fs -cat output1/* 3 14 80 cat: File does not exist: /user/ashin/output1/_logs
---------------------------------------------
hadoop流使用shell命令对文件排序:
ashin@linux:~/test/hadoop/hadoop-1.1.2/bin$ ./hadoop jar ../contrib/streaming/hadoop-streaming-1.1.2.jar -input input1 -output output3 -mapper /bin/cat -reducer "sort"
查看结果:
ashin@linux:~/test/hadoop/hadoop-1.1.2/bin$ ./hadoop fs -cat output3/* ashin hello blog hadoop milk miss ashin blog hadoop milk test hi blog hadoop cat: File does not exist: /user/ashin/output3/_logs
----------------------------------------------
hadoop流使用shell命令找出文件中含有ashin的文件(貌似含有空格的命令必须用引号引起来才会正常运行)
ashin@linux:~/test/hadoop/hadoop-1.1.2/bin$ ./hadoop jar ../contrib/streaming/hadoop-streaming-1.1.2.jar -input input1 -output output2 -mapper /bin/cat -reducer "/bin/grep ashin"
查看结果:
ashin@linux:~/test/hadoop/hadoop-1.1.2/bin$ ./hadoop fs -cat output2/* ashin hello blog hadoop milk miss ashin blog hadoop cat: File does not exist: /user/ashin/output2/_logs
------------------------------------------------------------------
####使用Python的hadoop流测试
由于shell命令是linux系统命令,所以无需额外参数,任何可执行文件都可以被指定为mapper/reducer。这些可执行文件不需要事先存放在集群上; 如果在集群上还没有,则需要用-file选项让framework把可执行文件作为作业的一部分。
map.py
#!/usr/bin/env python #-*- coding:utf-8 -*- __author__ = 'AshIn' import sys for line in sys.stdin: words = line.split() for word in words: print >> sys.stdout, '%s\t%s'%(word, 1)
reduce.py
#!/usr/bin/env python #-*- coding:utf-8 -*- __author__ = 'AshIn' import sys word_count = {} for line in sys.stdin: word, count = line.split() count = word_count.get(word, 0) + int(count) word_count[word] = count for key, value in word_count.items(): print >> sys.stdout, '%s\t%s'%(key, value)
本地测试python程序是否能够运行(注意先修改脚本权限):
ashin@linux:~/test/hadoop/input$ cat f1|'./map.py'|'./reduce.py' ashin1 blog1 hello1 hadoop1
使用hadoop streaming运行,使用-file参数可执行文件可以不传到hdfs上:
ashin@linux:~/test/hadoop/hadoop-1.1.2/bin$ ./hadoop jar ../contrib/streaming/hadoop-streaming-1.1.2.jar -input input -output output5 -mapper ../../map.py -reducer ../../reduce.py -file ../../map.py -file ../../reduce.py ashin@linux:~/test/hadoop/hadoop-1.1.2/bin$ ./hadoop fs -cat output5/* ashin2 hadoop3 hello1 blog3 hi1 test1 miss1 milk2
这篇文章讲hadoop streaming讲的挺详细:
http://blog.csdn.net/sunlylorn/article/details/8516808
文章为阿小信的个人笔记,转载请注明出处。