ubuntu安装hadoop2.6

原创

wx6131ce061d3e2 2021-09-04 10:52:32 ©著作权

文章标签 hadoop java hdfs xml bash 文章分类 代码人生

©著作权归作者所有：来自51CTO博客作者wx6131ce061d3e2的原创作品，请联系作者获取转载授权，否则将追究法律责任

一：单机版

1.sudo gedit ~/.bashrc 加入JDK路径

#HADOOP VARIABLES START  
export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-amd64
export HADOOP_INSTALL=/home/sendi/hadoop-2.6.0 
export PATH=$PATH:$HADOOP_INSTALL/bin  
export PATH=$PATH:$HADOOP_INSTALL/sbin  
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL  
export HADOOP_COMMON_HOME=$HADOOP_INSTALL  
export HADOOP_HDFS_HOME=$HADOOP_INSTALL  
export YARN_HOME=$HADOOP_INSTALL  
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native  
export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib"  
#HADOOP VARIABLES END

ubantu下jdk路径可用如下命令取得：

update-alternatives --config java

2.执行下面命令使改动生效：

source ~/.bashrc

3.修改hadoop-env.sh的配置：

sudo gedit /usr/local/hadoop/etc/hadoop/hadoop-env.sh

找到JAVA_HOME改为：

/usr/lib/jvm/java-1.7.0-openjdk-amd64

再在下面加上一行代码：

export HADOOP_OPTS="-Djava.library.path=$HADOOP_PREFIX/lib:$HADOOP_PREFIX/lib/native"

二：伪分布式：

1.sudo gedit /usr/local/hadoop/etc/hadoop/core-site.xml

<configuration>
 <property>  
   <name>hadoop.tmp.dir</name>  
   <value>/home/sendi/hadoop-2.6.0/tmp</value>  
   <description>Abase for other temporary directories.</description>  
 </property>  
 <property>  
   <name>fs.defaultFS</name>  
   <value>hdfs://localhost:9000</value>
 </property>
</configuration>

2.sudo gedit /usr/local/hadoop/etc/hadoop/mapred-site.xm

<configuration>  
    <property>     
          <name>mapred.job.tracker</name>    
          <value>localhost:9001</value>     
    </property>   
</configuration>

3.sudo gedit /usr/local/hadoop/etc/hadoop/yarn-site.xml

<configuration>  
<property>  
 <name>mapreduce.framework.name</name>  
 <value>yarn</value>  
</property>  
<property>
 <name>yarn.nodemanager.aux-services</name>  
 <value>mapreduce_shuffle</value>  
</property>  
</configuration>

4.sudo gedit /usr/local/hadoop/etc/hadoop/hdfs-site.xml

<configuration>
   <property>  
        <name>dfs.replication</name>  
        <value>1</value>  
    </property>  
    <property>  
        <name>dfs.namenode.name.dir</name>  
        <value>/home/sendi/hadoop-2.6.0/dfs/name</value>
    </property>  
    <property>  
        <name>dfs.datanode.data.dir</name>  
        <value>/home/sendi/hadoop-2.6.0/dfs/data</value>  
    </property>  
    <property>                
        <name>dfs.permissions</name>  
        <value>false</value>  
   </property> 
</configuration>

5.sudo gedit /usr/local/hadoop/etc/hadoop/masters 添加：localhost

6.sudo gedit /usr/local/hadoop/etc/hadoop/slaves 添加：localhost

7.初始化文件系统HDFS

bin/hdfs namenode -format

8.启动,启动之后用下面链接查看结果

http://localhost:50070

sbin/start-dfs.sh  
sbin/start-yarn.sh 
-------------------------------------------------------------------------------------------------------------------------------------------
sendi@sendijia:~/hadoop-2.6.0$ sbin/start-dfs.sh
Starting namenodes on [localhost]
localhost: starting namenode, logging to /home/sendi/hadoop-2.6.0/logs/hadoop-sendi-namenode-sendijia.out
localhost: starting datanode, logging to /home/sendi/hadoop-2.6.0/logs/hadoop-sendi-datanode-sendijia.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /home/sendi/hadoop-2.6.0/logs/hadoop-sendi-secondarynamenode-sendijia.out
sendi@sendijia:~/hadoop-2.6.0$ sbin/start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /home/sendi/hadoop-2.6.0/logs/yarn-sendi-resourcemanager-sendijia.out
localhost: starting nodemanager, logging to /home/sendi/hadoop-2.6.0/logs/yarn-sendi-nodemanager-sendijia.out