0024-CENTOS6.5安装CDH5.12.1(一)

原创

Hadoop实操 2018-11-20 11:03:35 ©著作权

文章标签 cdh 安装 文章分类 开源

©著作权归作者所有：来自51CTO博客作者Hadoop实操的原创作品，如需转载，请与作者联系，否则将追究法律责任

温馨提示：要看高清无码套图，请使用手机打开并单击图片放大查看。

1.概述

本文档主要描述离线环境下，CentOS6.5操作系统部署CDH5.12.1企业版的过程。本安装文档主要分为4个步骤：

1.前期准备（包括hostname、关闭防火墙、关闭SELinux、时钟同步等）

2.安装Cloudera Manager Server

3.安装CDH集群

4.集群完整性检查（包括HDFS文件系统、MapReduce、Hive等服务是否可以正常运行）

这篇文档将着重介绍Cloudera管理器与CDH的安装，并基于以下假设：

1.操作系统版本：CentOS6.5

2.MySQL版本：5.1.73

3.CM版本：CM 5.12.1

4.CDH版本：CDH 5.12.1

5.采用root对集群进行部署

6.您已经下载CDH和CM的安装包（CDH使用parcels包方式安装）

2.前期准备

2.1hostname及hosts配置

集群中各个节点之间能互相通信使用静态IP地址。IP地址和主机名通过/etc/hosts配置，主机名/etc/hostname进行配置（推荐hostname使用FQDN完全域名的方式配置）。

以服务器（172.31.6.148）为例：

hostname配置

修改/etc/sysconfig/network文件：

[root@ip-172-31-6-148~]# vim /etc/sysconfig/network

NETWORKING=yes

NETWORKING_IPV6=no

HOSTNAME=ip-172-31-6-148.fayson.com

注意：修改hostname后需要重启服务器。

检查hostname是否修改成功：

[root@ip-172-31-6-148~]# hostname

ip-172-31-6-148.fayson.com

[root@ip-172-31-6-148 ~]#

hosts配置

/etc/hosts文件如下:

[root@ip-172-31-6-148~]# more /etc/hosts

127.0.0.1 localhost.localdomain localhost

::1 localhost6.localdomain6 localhost6

172.31.6.148 ip-172-31-6-148.fayson.com

172.31.5.190 ip-172-31-5-190.fayson.com

172.31.10.118 ip-172-31-10-118.fayson.com

172.31.9.33 ip-172-31-9-33.fayson.com

[root@ip-172-31-6-148 ~]#

以上两步操作，在集群中其它节点做相应配置。

2.2禁用SELinux

在所有节点执行sudo setenforce 0 命令

[root@ip-172-31-6-148~]# sudo setenforce 0

修改集群所有节点的/etc/selinux/config文件，内容如下：

[root@ip-172-31-6-148~]# vim /etc/selinux/config

This file controls the state ofSELinux on the system.

SELINUX= can take one of these three values:

enforcing - SELinux security policyis enforced.

permissive - SELinux printswarnings instead of enforcing.

disabled - No SELinux policy isloaded.

SELINUX=disabled

SELINUXTYPE= can take one of these twovalues:

targeted - Targeted processes areprotected,

mls - Multi Level Securityprotection.

SELINUXTYPE=targeted

2.3关闭防火墙

在集群所有节点执行如下操作，并永久关闭防火墙

[root@ip-172-31-6-148~]# service iptables stop

iptables: Setting chains to policy ACCEPT: filter [ OK ]

iptables: Flushing firewall rules: [ OK ]

iptables: Unloading modules: [ OK ]

[root@ip-172-31-6-148 ~]# chkconfig iptables off

[root@ip-172-31-6-148 ~]#

2.4配置操作系统本地yum源

挂载操作系统iso文件

[root@ip-172-31-6-148~]# mkdir /mnt/iso

[root@ip-172-31-6-148 ~]# mount -o loop CentOS-6.5-x86_64-minimal.iso/mnt/iso/

[root@ip-172-31-6-148 ~]# df -h

Filesystem Size UsedAvail Use% Mounted on

/dev/xvde 99G 3.4G 91G 4% /

tmpfs 7.4G 0 7.4G 0%/dev/shm

/root/CentOS-6.5-x86_64-minimal.iso 398M 398M 0 100% /mnt/iso

[root@ip-172-31-6-148 ~]#

配置操作系统repo

[root@ip-172-31-6-148~]# vim /etc/yum.repos.d/os.repo

localrepo

name=CentOS6.5

baseurl=file:///mnt/iso

gpgcheck=false

enabled=true

查看yum源列表是否有新建的localrepo

[root@ip-172-31-6-148~]# yum repolist

Loaded plugins: fastestmirror, presto

Loading mirror speeds from cached hostfile

* base: mirror.0x.sg

* extras: mirror.0x.sg

* updates: mirror.0x.sg

localrepo | 3.6 kB 00:00...

localrepo/primary_db | 492 kB 00:00 ...

repo id reponame status

base CentOS-6 - Base 6,706

cloudera-manager Cloudera Manager, Version 5.12.0 7

cmrepo cm_repo 7

extras CentOS-6 - Extras 45

localrepo CentOS6.5 248

updates CentOS-6 - Updates 589

repolist: 7,602

[root@ip-172-31-6-148 ~]#

2.5安装http服务

使用yum安装http服务

在集群其中一节点上安装http服务，执行如下命令：

[root@ip-172-31-6-148~]# yum -y install httpd

如上图所示则表明安装成功。

将httpd服务加入系统自启动服务并设置开机启动

[root@ip-172-31-6-148~]# chkconfig --add httpd

[root@ip-172-31-6-148 ~]# chkconfig httpd on

[root@ip-172-31-6-148 ~]# chkconfig --list |grep httpd

httpd 0:off 1:off 2:on 3:on 4:on 5:on 6:off

[root@ip-172-31-6-148 ~]#

启动http服务

[root@ip-172-31-6-148~]# service httpd start

Starting httpd: [ OK ]

2.6重做操作系统yum源

在2.5安装了httpd服务后，重新制作操作系统的yum源，使用http的方式，操作如下：

在/var/www/html目录下创建centos6.5目录

_root@ip-172-31-6-148_html# mkdir centos6.5

[root@ip-172-31-6-148 html]# ll

total 4

drwxr-xr-x 2 rootroot 4096 Sep 4 16:04 centos6.5

[root@ip-172-31-6-148 html]#

将/mnt/iso目录下的所有文件拷贝至/var/www/html/centos6.5目录下

_root@ip-172-31-6-148_html# scp -r /mnt/iso/*/var/www/html/centos6.5/

修改/etc/yum.repo.d/os.repo配置

[root@ip-172-31-6-148~]# vim /etc/yum.repos.d/os.repo

localrepo

name=CentOS6.5

baseurl=http://ip-172-31-6-148.fayson.com/centos6.5

gpgcheck=false

enabled=true

检查yum源是否配置正常

将/etc/yum.repo.d/os.repo配置文件同步到集群所有节点的/etc/yum.repo.d/目录下

2.7集群时钟同步

在集群的所有服务器上安装ntp服务，用于集群时钟同步，此处选择ip-172-31-6-148.fayson.com服务器作为本地NTP服务器，集群其它节点与其保持同步，配置如下：

在集群所有节点安装ntp服务，命令如下

[root@ip-172-31-6-148~]# yum -y install ntp

如上图显示，则说明安装成功。

将ntpd加入系统自启动服务并设置开机启动

在所有节点执行如下操作：

[root@ip-172-31-6-148~]# chkconfig --add ntpd

[root@ip-172-31-6-148 ~]# chkconfig ntpd on

在ip-172-31-6-148.fayson.com上配置与自己同步

[root@ip-172-31-6-148~]# vim /etc/ntp.conf

...

Use public servers from thepool.ntp.org project.

Please consider joining the pool (http://www.pool.ntp.org/join.html).

#server 0.centos.pool.ntp.org iburst

#server 1.centos.pool.ntp.org iburst

#server 2.centos.pool.ntp.org iburst

#server 3.centos.pool.ntp.org iburst

server 127.127.1.0 #local clock

fudge 127.127.1.0 stratum 10

集群其它节点配置如下

[root@ip-172-31-5-190~]# vim /etc/ntp.conf

...

Use public servers from thepool.ntp.org project.

Please consider joining the pool (http://www.pool.ntp.org/join.html).

#server 0.centos.pool.ntp.org iburst

#server 1.centos.pool.ntp.org iburst

#server 2.centos.pool.ntp.org iburst

#server 3.centos.pool.ntp.org iburst

server 172.31.6.148 iburst

重启集群所有节点的ntpd服务

[root@ip-172-31-6-148~]# service ntpd restart

Shutting down ntpd: [ OK ]

Starting ntpd: [ OK ]

[root@ip-172-31-6-148 ~]#

验证时钟同步，在所有节点执行ntpq -p命令

[root@ip-172-31-6-148~]# ntpq -p

remote refid stt when poll reach delay offset jitter

==============================================================================

*LOCAL(0) .LOCL. 10 l 15 64 7 0.000 0.000 0.001

[root@ip-172-31-6-148 ~]# ssh -i fayson.pem.txtip-172-31-5-190.fayson.com "ntpq-p"

remote refid st t when pollreach delay offset jitter

==============================================================================

*ip-172-31-6-148 LOCAL(0) 11 u 68 128 377 0.274 1.005 0.243

[root@ip-172-31-6-148 ~]# ssh -i fayson.pem.txtip-172-31-10-118.fayson.com "ntpq-p"

remote refid stt when poll reach delay offset jitter

==============================================================================

*ip-172-31-6-148 LOCAL(0) 11 u 75 128 377 0.285 1.214 0.213

[root@ip-172-31-6-148 ~]# ssh -i fayson.pem.txt ip-172-31-9-33.fayson.com"ntpq -p"

remote refid stt when poll reach delay offset jitter

==============================================================================

*ip-172-31-6-148 LOCAL(0) 11 u 1 64 367 0.294 -9.249 1.511

[root@ip-172-31-6-148 ~]#

注：标记部分，有“*”显示则表示同步成功。

2.8安装MySQL

安装MySQL

[root@ip-172-31-6-148~]# yum -y install mysql mysql-server

如上截图则表示MySQL安装成功。

将mysqld加入系统自启动服务并设置开机启动

[root@ip-172-31-6-148~]# chkconfig --add mysqld

[root@ip-172-31-6-148 ~]# chkconfig mysqld on

启动并配置Mysql

[root@ip-172-31-6-148~]# service mysqld start

...

[ OK ]

Starting mysqld: [ OK ]

初始化MySQL

[root@ip-172-31-6-148~]# mysql_secure_installation

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MySQL

SERVERS IN PRODUCTIONUSE! PLEASE READ EACH STEP CAREFULLY!

In order to log into MySQL to secure it, we'll need the current

password for the root user. If you've just installed MySQL, and

you haven't set the root password yet, the passwordwill be blank,

so you should just press enter here.

Enter current password for root (enter for none):

OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MySQL

root user without the proper authorisation.

Set root password? Y/n y

New password:

Re-enter new password:

Password updated successfully!

Reloading privilege tables..

... Success!

By default, a MySQL installation has an anonymous user, allowing anyone

to log into MySQL without having to have a user account created for

them. This is intended only for testing,and to make the installation

go a bit smoother. You should removethem before moving into a

production environment.

Remove anonymous users? Y/n y

... Success!

Normally, root should only be allowed to connect from 'localhost'. This

ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? Y/n n

... skipping.

By default, MySQL comes with a database named 'test' that anyone can

access. This is also intended only fortesting, and should be removed

before moving into a production environment.

Remove test database and access to it? Y/n y

Dropping test database... ... Success!
Removing privileges on test database... ... Success!

Reloading the privilege tables will ensure that all changes made so far

will take effect immediately.

Reload privilege tables now? Y/n y

... Success!

Cleaning up...

All done! If you've completed all of the above steps, your MySQL

installation should now be secure.

Thanks for usingMySQL!

[root@ip-172-31-6-148 ~]#

注意：标红部分为输入项，出现“Thanks for using MySQL!”则表示初始化成功。

创建CM及CDH服务的数据库

建库语句：

create database metastore default character set utf8;

CREATE USER 'hive'@'%' IDENTIFIED BY 'password';

GRANT ALL PRIVILEGES ON metastore. * TO 'hive'@'%';

FLUSH PRIVILEGES;

create database cm default character set utf8;

CREATE USER 'cm'@'%' IDENTIFIED BY 'password';

GRANT ALL PRIVILEGES ON cm. * TO 'cm'@'%';

FLUSH PRIVILEGES;

create database am default character set utf8;

CREATE USER 'am'@'%' IDENTIFIED BY 'password';

GRANT ALL PRIVILEGES ON am. * TO 'am'@'%';

FLUSH PRIVILEGES;

create database rm default character set utf8;

CREATE USER 'rm'@'%' IDENTIFIED BY 'password';

GRANT ALL PRIVILEGES ON rm. * TO 'rm'@'%';

FLUSH PRIVILEGES;

create database hue default character set utf8;

CREATE USER 'hue'@'%' IDENTIFIED BY 'password';

GRANT ALL PRIVILEGES ON hue. * TO 'hue'@'%';

FLUSH PRIVILEGES;

create database oozie default character set utf8;

CREATE USER 'oozie'@'%' IDENTIFIED BY 'password';

GRANT ALL PRIVILEGES ON oozie. * TO 'oozie'@'%';

FLUSH PRIVILEGES;

命令行操作如下：

[root@ip-172-31-6-148~]# mysql -uroot -p

Enter password:

...

mysql> create database metastore default character set utf8;

Query OK, 1 row affected (0.00 sec)

mysql_> CREATE USER_ 'hive'@'%' IDENTIFIEDBY 'password';