环境为window64系统
1、安装Hadoop开发插件
下载hadoop-eclipse-plugin-2.4.0.jar,拷贝到myeclipse根目录下/dropins目录下。
2、 启动myeclipse,打开Perspective:
【Window】->【Open Perspective】->【Other...】->【Map/Reduce】->【OK】
3、 打开一个View:
【Window】->【Show View】->【Other...】->【MapReduce Tools】->【Map/Reduce Locations】->【OK】
4、 添加Hadoop location:
此处添加自己的hadoop安装路径。
5、new Hadoop location
修改其中内容:
Map/Reduce Master 这个框里:这两个参数就是mapred-site.xml里面mapred.job.tracker里面的ip和port
DFS Master 这个框里:这两个参数就是core-site.xml里面fs.default.name里面的ip和port
user name:这个是连接hadoop的用户名
因为我是用root户安装的hadoop,而且没建立其他的用户,所以用root。下面的不用填写。然后点击finish按钮,此时,这个视图中就有多了一条记录。
修改后:
重启myeclipse并重新编辑刚才建立的那个连接记录,现在我们编辑advance parameters tab页
此页只需修改,其中箭头所示处,后面填core-site.xml里所对应的路径即可。
然后点击finish,然后就连接上了(先要启动sshd服务,启动hadoop进程),连接上的标志如图:
6、再跑wordcount例子
新建Map/Reduce Project:
【File】->【New】->【Project...】->【Map/Reduce】->【Map/Reduce Project】->【Project name: WordCount】->【Configure Hadoop install directory...】->【Hadoop installation directory: usr/local/hadoop/hadoop-1.2.1】->【Apply】->【OK】->【Next】->【Allow output folders for source folders】->【Finish】
新建WordCount类
添加/编写源代码:此代码是hadoop自带的,所以在hadoop安装目录下,如下图:(代码复制过来即可用)
package org.apache.hadoop.examples; import java.io.IOException; import java.util.StringTokenizer; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.util.GenericOptionsParser; public class WordCount { public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable>{ private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(Object key, Text value, Context context ) throws IOException, InterruptedException { StringTokenizer itr = new StringTokenizer(value.toString()); while (itr.hasMoreTokens()) { word.set(itr.nextToken()); context.write(word, one); } } } public static class IntSumReducer extends Reducer<Text,IntWritable,Text,IntWritable> { private IntWritable result = new IntWritable(); public void reduce(Text key, Iterable<IntWritable> values, Context context ) throws IOException, InterruptedException { int sum = 0; for (IntWritable val : values) { sum += val.get(); } result.set(sum); context.write(key, result); } } public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs(); if (otherArgs.length != 2) { System.err.println("Usage: wordcount <in> <out>"); System.exit(2); } Job job = new Job(conf, "word count"); job.setJarByClass(WordCount.class); job.setMapperClass(TokenizerMapper.class); job.setCombinerClass(IntSumReducer.class); job.setReducerClass(IntSumReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path(otherArgs[0])); FileOutputFormat.setOutputPath(job, new Path(otherArgs[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); } }
在新建的项目WordCount,点击WordCount.java,右键-->Run As-->Run Configurations
点击Run,运行程序(如有错误,看下面错误解决)
在此刻看到运行结果,如下图:
7.错误解决
下载如下两个文件:
DirectX_Repair-v3.3
hadoopbin
1.复制 winutils.exe winutils.pdb Hadoop.dll 到window系统中的
/hadoop/bin的目录下
2.复制 Hadoop.dll 到 window系统中的C:\window\system32\
3.点击winutils.exe看有没有提示错误,若有错误,运行DirectX_Repair-v3.3.zip进行修复
文件下载地址为:
http://down.51cto.com/data/2213607
hadoop-eclipse-plugin-2.4.0.jar
DirectX_Repair-v3.3.zip
hadoopbin.zip