CentOS 7 安装Hadoop

1、Hadoop安装前的准备

(1)复制文件到CentOS 7

1)建立共享文件夹,如下图所示:

centos7进入hadoop centos7 hadoop_hadoop

2)将文件复制到hadoop文件夹下

centos7进入hadoop centos7 hadoop_centos7进入hadoop_02

(2)时间同步,如下图所示:

centos7进入hadoop centos7 hadoop_xml_03


centos7进入hadoop centos7 hadoop_xml_04


(3)修改主机名,如下图所示:

centos7进入hadoop centos7 hadoop_centos7进入hadoop_05


centos7进入hadoop centos7 hadoop_centos7进入hadoop_06

(4)配置网络环境,如下图所示:

centos7进入hadoop centos7 hadoop_hadoop_07


centos7进入hadoop centos7 hadoop_hadoop_08


centos7进入hadoop centos7 hadoop_Hadoop_09


centos7进入hadoop centos7 hadoop_hadoop_10


(5)关闭防火墙,如下图所示:

centos7进入hadoop centos7 hadoop_hadoop_11

(6)配置hosts列表,如下图所示:

centos7进入hadoop centos7 hadoop_xml_12

(7)测试主节点和从节点是否能够相互ping通,如下图所示:

centos7进入hadoop centos7 hadoop_centos_13


centos7进入hadoop centos7 hadoop_Hadoop_14

(8) 安装jdk

1)新建目录java,并将jdk复制到java下,如下图所示:

centos7进入hadoop centos7 hadoop_xml_15

2)解压jdk的压缩包,如下图所示:

centos7进入hadoop centos7 hadoop_Hadoop_16


centos7进入hadoop centos7 hadoop_centos_17


3)添加jdk的路径,如下图所示:

centos7进入hadoop centos7 hadoop_hadoop_18


centos7进入hadoop centos7 hadoop_Hadoop_19

centos7进入hadoop centos7 hadoop_centos_20


4)查看jdk版本,如下图所示: 从节点也安装好了

centos7进入hadoop centos7 hadoop_centos_21


centos7进入hadoop centos7 hadoop_centos_22

(9)免密钥登陆

1)切换为普通用户,如下图所示:

centos7进入hadoop centos7 hadoop_xml_23

2)在master节点终端生成密钥,如下图所示:

centos7进入hadoop centos7 hadoop_Hadoop_24


centos7进入hadoop centos7 hadoop_centos_25


3)复制公钥文件,如下图所示:

centos7进入hadoop centos7 hadoop_centos7进入hadoop_26

4)修改文件权限,如下图所示:

centos7进入hadoop centos7 hadoop_Hadoop_27

5)将文件复制到slave节点

centos7进入hadoop centos7 hadoop_xml_28

6)在slave节点终端生成密钥,如下图所示:

centos7进入hadoop centos7 hadoop_centos7进入hadoop_29

7)将密钥文件移动到.ssh目录,如下图所示:

centos7进入hadoop centos7 hadoop_centos7进入hadoop_30

8)修改文件权限,如下图所示:

centos7进入hadoop centos7 hadoop_xml_31

9)验证,如下图所示:

centos7进入hadoop centos7 hadoop_hadoop_32

2、Hadoop安装

(1)复制Hadoop安装包,如下图所示:

centos7进入hadoop centos7 hadoop_centos7进入hadoop_33


centos7进入hadoop centos7 hadoop_centos7进入hadoop_34


(2)解压Hadoop安装包,如下图所示:

centos7进入hadoop centos7 hadoop_centos7进入hadoop_35


centos7进入hadoop centos7 hadoop_xml_36


(3)修改hadoop-env.sh文件,如下图所示:

centos7进入hadoop centos7 hadoop_xml_37


centos7进入hadoop centos7 hadoop_Hadoop_38


centos7进入hadoop centos7 hadoop_xml_39

(4)修改core-site.xml文件,如下图所示:

centos7进入hadoop centos7 hadoop_Hadoop_40


centos7进入hadoop centos7 hadoop_centos_41

(5)修改hdfs-site.xml文件,如下图所示:

centos7进入hadoop centos7 hadoop_centos7进入hadoop_42


centos7进入hadoop centos7 hadoop_hadoop_43

(6)修改mapred-site.xml文件,如下图所示:

centos7进入hadoop centos7 hadoop_hadoop_44


centos7进入hadoop centos7 hadoop_centos7进入hadoop_45

(7)修改workers文件,如下图所示:

centos7进入hadoop centos7 hadoop_Hadoop_46


centos7进入hadoop centos7 hadoop_xml_47


(8)修改yarn-site.xml文件,如下图所示:

centos7进入hadoop centos7 hadoop_centos7进入hadoop_48


centos7进入hadoop centos7 hadoop_xml_49


(9)复制到从节点,如下图所示:

centos7进入hadoop centos7 hadoop_xml_50


centos7进入hadoop centos7 hadoop_centos_51

(10)修改bash_profile文件,并使其生效,主从节点都要做,如下图所示:

centos7进入hadoop centos7 hadoop_centos_52


centos7进入hadoop centos7 hadoop_hadoop_53


centos7进入hadoop centos7 hadoop_Hadoop_54

centos7进入hadoop centos7 hadoop_Hadoop_55


centos7进入hadoop centos7 hadoop_centos_56

(11)新建文件夹hadoopdata,主从节点都要做,如下图所示:

centos7进入hadoop centos7 hadoop_Hadoop_57


centos7进入hadoop centos7 hadoop_centos_58

(12)格式化文件系统,如下图所示:

centos7进入hadoop centos7 hadoop_hadoop_59

(13)启动hadoop,并jps验证,如下图所示:

centos7进入hadoop centos7 hadoop_centos7进入hadoop_60

(14)运行pi实例,如下图所示:

centos7进入hadoop centos7 hadoop_xml_61


centos7进入hadoop centos7 hadoop_centos_62

3、遇到的问题

1、第一次配置的时候,从节点配置到了works文件中,配置错了,于是又重新格式化了一遍,启动没问题了;

2、在运行pi实例的时候,出现了错误,不能找到或加载主类,找了半天也不知道是为什么,就去问了老师,发现是yarn-site.xml这个配置文件写错了。

3、第一个错误解决了以后,发现又出现了新的错误,时间不同步,原来是配置了静态IP后虚拟机连不上网了,于是去修改IP地址,如下图所示,第一个图是自己电脑的IP地址

centos7进入hadoop centos7 hadoop_Hadoop_63


centos7进入hadoop centos7 hadoop_hadoop_64

使IP地址生效

centos7进入hadoop centos7 hadoop_centos_65

centos7进入hadoop centos7 hadoop_centos_66


centos7进入hadoop centos7 hadoop_Hadoop_67

使IP地址生效

centos7进入hadoop centos7 hadoop_xml_68


centos7进入hadoop centos7 hadoop_xml_69

能连上网了,如下图所示:

centos7进入hadoop centos7 hadoop_Hadoop_70

修改hosts文件,主从节点都要修改,并进行ping测试,如下图所示:

centos7进入hadoop centos7 hadoop_centos_71


centos7进入hadoop centos7 hadoop_centos_72


centos7进入hadoop centos7 hadoop_xml_73

4、上一个问题解决之后,又出现了一个新的问题,如下图所示,于是按照网上的方法修改了mapred-site.xml文件,重新格式化,启动Hadoop;

centos7进入hadoop centos7 hadoop_Hadoop_74


centos7进入hadoop centos7 hadoop_hadoop_75


centos7进入hadoop centos7 hadoop_Hadoop_76

5、上一个问题解决之后,又出现了新的问题,虚拟内存不够,如下图所示在网上找解决方案,修改yarn-site.xml和mapred-site.xml

centos7进入hadoop centos7 hadoop_xml_77


在yarn-site.xml中添加下图所示内容

centos7进入hadoop centos7 hadoop_hadoop_78


在mapred-site.xml中添加下图所示内容

centos7进入hadoop centos7 hadoop_Hadoop_79


centos7进入hadoop centos7 hadoop_xml_80

6、实验过程中还出现了如下图所示的错误,原因是登录的时候使用root用户登录的,级视切换到了yss,也不能使用gedit,可以重启虚拟机登录一下,也可以用vi这个命令代替gedit。

centos7进入hadoop centos7 hadoop_Hadoop_81