Hadoop Database安装手册

精选原创

candon123 2017-12-04 20:07:56 博主文章分类：大数据 ©著作权

文章标签 Hadoop 文章分类 Hadoop 大数据

©著作权归作者所有：来自51CTO博客作者candon123的原创作品，谢绝转载，否则将追究法律责任

本文描述在Ubuntu 16.04.3 LTS 安装配置hadoop以及hbase。 1、环境准备 1.1 创建用户

root@bdi:/u01# adduser hadoop
Adding user `hadoop' ...
Adding new group `hadoop' (1002) ...
Adding new user `hadoop' (1002) with group `hadoop' ...
Creating home directory `/home/hadoop' ...
Copying files from `/etc/skel' ...
Enter new UNIX password: 
Retype new UNIX password: 
passwd: password updated successfully

1.2 配置SSH密钥为了避免后续启动程序的时候提示输入密码，必须设置localhost的ssh互访。

hadoop@bdi:~$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): 
Created directory '/home/hadoop/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:8a7/pCryrmEaTS0Gy/Vm0IdJjPt2DJGlXsmBRz82jK8 hadoop@bdi
The key's randomart image is:
+---[RSA 2048]----+
|     o.=+        |
|    .o==.*       |
|  . o.=oB *      |
| . +.=.o = o     |
|  o +.*oS o      |
|   + +o oo       |
|  . +. .E . .    |
|   +...  . o     |
|  . .=o.ooo..    |
+----[SHA256]-----+
hadoop@bdi:~$ cd .ssh/
hadoop@bdi:~/.ssh$ ssh-copy-id -i id_rsa.pub localhost
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "id_rsa.pub"
The authenticity of host 'localhost (127.0.0.1)' can't be established.
ECDSA key fingerprint is SHA256:1tE3j//p5LSl88OgjJBzZQHvOaHayRDoRk0qp3J3ABk.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
hadoop@localhost's password: 
Number of key(s) added: 1
Now try logging into the machine, with:   "ssh 'localhost'"
and check to make sure that only the key(s) you wanted were added.

1.3 安装JDK 这里从官方下载的JDK版本为1.8，直接解压即可。

root@bdi:~#  mkdir /usr/java
root@bdi:~#  cd /usr/java
root@bdi:~#  tar -xzf /root/jdk-8u152-linux-x64.tar.gz 
编辑/etc/profile，加入以下内容：
export JAVA_HOME=/usr/java/jdk1.8.0_152
export JRE_HOME=${JAVA_HOME}/jre  
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib  
export PATH=${JAVA_HOME}/bin:$PATH
root@bdi:~# java -version
java version "1.8.0_152"
Java(TM) SE Runtime Environment (build 1.8.0_152-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.152-b16, mixed mode)
root@bdi:~# alternatives --install /usr/bin/java java usr/local/java/bin/java 2
root@bdi:~# alternatives --install /usr/bin/javac javac usr/local/java/bin/javac 2
root@bdi:~# alternatives --install /usr/bin/jar jar usr/local/java/bin/jar 2
root@bdi:~# alternatives --set java usr/local/java/bin/java
root@bdi:~# alternatives --set javac usr/local/java/bin/javac
root@bdi:~# alternatives --set jar usr/local/java/bin/jar

2、Hadoop安装配置 Hadoop当前的稳定版为2.9，下载链接：Hadoop。 2.1 安装

hadoop@bdi:~$ wget https://mirrors.cnnic.cn/apache/hadoop/common/hadoop-2.9.0/hadoop-2.9.0.tar.gz
hadoop@bdi:~$ cd /u01
hadoop@bdi:~$ tar -xzf /home/hadoop/hadoop-2.9.0.tar.gz

Hadoop安装已完成，设置hadoop用户环境变量，加入以下内容：

hadoop@bdi:~$ vi .bashrc
export HADOOP_HOME=/u01/hadoop
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib:$HADOOP_COMMON_LIB_NATIVE_DIR"
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
hadoop@bdi:~$ source .bashrc
hadoop@bdi:~$ hadoop version
Hadoop 2.9.0
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r 756ebc8394e473ac25feac05fa493f6d612e6c50
Compiled by arsuresh on 2017-11-13T23:15Z
Compiled with protoc 2.5.0
From source with checksum 0a76a9a32a5257331741f8d5932f183
This command was run using /u01/hadoop/share/hadoop/common/hadoop-common-2.9.0.jar
如果能显示出版本信息，则说明安装成功。

2.2 Hadoop配置 Hadoop的配置文件位于$HADOOP_HOME/etc目录下，下面对各种配置文件进行修改。

JAVA_HOME环境变量设置编辑hadoop-env.sh文件，设置JAVA_HOME环境变量，如下：

export JAVA_HOME=/usr/java/jdk1.8.0_152

修改core-site.xml文件此文件包含了hadoop实例的端口号，分配给文件系统的内存大小，读写缓存等信息。加入以下内容：
修改 hdfs-site.xml文件此文件包含了本地文件系统的同步数据、namenode路径和datanode路径信息。加入以下内容：
修改yarn-site.xml文件此文件主要针对hadoop的yarn配置，加入以下内容：
修改mapred-site.xml文件 3、验证hadoop安装 3.1 配置Name Node 执行hdfs进行格式化操作。

hadoop@bdi:~$ hdfs namenode -format
......
/************************************************************
STARTUP_MSG: Starting NodeManager
STARTUP_MSG:   host = bdi/192.168.120.95
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 2.9.0
......
/************************************************************ 
SHUTDOWN_MSG: Shutting down NameNode at bdi/192.168.120.95 
************************************************************/

3.2 验证hadoop的dfs

hadoop@bdi:~$ start-dfs.sh

3.3 验证hadoop的yarn

hadoop@bdi:~$ start-yarn.sh

3.4 使用浏览器访问hadoop 使用浏览器访问地址：http://127.0.0.1:50070 3.5 验证集群的所有应用浏览器访问：http://localhost:8088 4 安装Hbase 这里使用的版本是hbase-1.2.6。

hadoop@bdi:~$ wget https://mirrors.cnnic.cn/apache/hbase/stable/hbase-1.2.6-bin.tar.gz
hadoop@bdi:~$ cd /u01
hadoop@bdi:~$ tar -xzf /home/hadoop/hbase-1.2.6-bin.tar.gz

4.1 配置Hbase为单机版模式单机版模式配置很简单，首先编辑hbase-env.sh文件，配置JDK的JAVA_HOME环境变量，其次是编辑hbase-site.xml文件，加入以下内容：最后，执行start-hbase.sh启动即可。 4.2 配置Hbase为伪分布模式修改hbase-site.xml文件，加入以下内容：注意：这里定义的hdfs的端口号要和$HADOOP_HOME/etc/hadoop/core-site.xml文件里的一致。最后执行start-hbase.sh启动即可。 4.3 检查HDFS的HBASE目录信息执行图中的命令，如下： 4.4 Hbase的web interface 浏览器访问：http://localhost:16010 ，如图所示：