前置条件

  1. ntp服务安装
  2. 防火墙关闭

执行以下优化代码

systemctl stop firewalld.service 
echo  "* soft    nofile  128000" >>/etc/security/limits.conf 
echo  "* hard    nofile  128000" >>/etc/security/limits.conf 
echo  "* soft    nproc  128000" >>/etc/security/limits.conf 
echo  "* hard    nproc  128000" >>/etc/security/limits.conf 
cat /etc/security/limits.conf
sed -i 's/4096/unlimited/' /etc/security/limits.d/20-nproc.conf
cat /etc/security/limits.d/20-nproc.conf
sed -i 's/4096/unlimited/' /etc/security/limits.d/90-nproc.conf
cat /etc/security/limits.d/90-nproc.conf
ulimit -SHn 128000
ulimit -SHu 128000

echo  "vm.swappiness=10" >>/etc/sysctl.conf
sysctl -p
sysctl -a|grep swappiness
echo  "net.core.somaxconn = 32768 " >>/etc/sysctl.conf
sysctl -p
sysctl -a|grep somaxconn
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled

CDH6.2 下载信息

https://www.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_cdh_62_download.html
yum安装：https://archive.cloudera.com/cdh6/6.2.0/redhat7/yum/
parcel安装：https://archive.cloudera.com/cdh6/6.2.0/parcels/

 

这里我们讲解使用 parcel 方式安装

 

ClouderaManager 安装包下载

https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/cloudera-manager-agent-6.2.0-968826.el7.x86_64.rpm
https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/cloudera-manager-daemons-6.2.0-968826.el7.x86_64.rpm
https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/cloudera-manager-server-6.2.0-968826.el7.x86_64.rpm
https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/cloudera-manager-server-db-2-6.2.0-968826.el7.x86_64.rpm
https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/enterprise-debuginfo-6.2.0-968826.el7.x86_64.rpm
https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
https://archive.cloudera.com/cm6/6.2.0/allkeys.asc

CDH 安装包下载

https://archive.cloudera.com/cdh6/6.2.0/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel
https://archive.cloudera.com/cdh6/6.2.0/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel.sha1
https://archive.cloudera.com/cdh6/6.2.0/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel.sha256
https://archive.cloudera.com/cdh6/6.2.0/parcels/manifest.json

安装 createrepo

作用:createrepo 命令用于创建yum源（软件仓库），即为存放于本地特定位置的众多rpm包建立索引，描述各包所需依赖信息，并形成元数据。

yum install -y createrepo

在CM目录下执行 createrepo . 生成rpm源

[root@cdh01 CM]# createrepo .
Spawning worker 0 with 1 pkgs
Spawning worker 1 with 1 pkgs
Spawning worker 2 with 1 pkgs
Spawning worker 3 with 1 pkgs
Spawning worker 4 with 1 pkgs
Spawning worker 5 with 1 pkgs
Spawning worker 6 with 0 pkgs
Spawning worker 7 with 0 pkgs
Spawning worker 8 with 0 pkgs
Spawning worker 9 with 0 pkgs
Spawning worker 10 with 0 pkgs
Spawning worker 11 with 0 pkgs
Spawning worker 12 with 0 pkgs
Spawning worker 13 with 0 pkgs
Spawning worker 14 with 0 pkgs
Spawning worker 15 with 0 pkgs
Spawning worker 16 with 0 pkgs
Spawning worker 17 with 0 pkgs
Spawning worker 18 with 0 pkgs
Spawning worker 19 with 0 pkgs
Spawning worker 20 with 0 pkgs
Spawning worker 21 with 0 pkgs
Spawning worker 22 with 0 pkgs
Spawning worker 23 with 0 pkgs
Workers Finished
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete

安装 httpd 服务

yum install -y httpd

启动服务

systemctl start httpd
 systemctl status httpd

将CM 与 CDH 目录移动到目录 /var/www/html/ 下

mv CM/ CDH/ /var/www/html/

验证是否安装成功，访问 http://ip/CM ，http://ip/CDH

显示如下即可

Centos7.x 安装 CDH 6.x_hive

提示以下内容说明没有成功，原因是 SELinux 未禁用成功，在所有服务器执行 setenforce 0  即可

Centos7.x 安装 CDH 6.x_cloudera_02

配置安装源

vim /etc/yum.repos.d/cm.repo
[cmrepo]
name = cm_repo
baseurl = http://ip/CM
enable = true
gpgcheck = false

安装JDK

yum install oracle-j2sdk1.8-1.8.0+update181-1.x86_64

安装 Cloudera Manager Server

yum install -y cloudera-manager-server

Centos7.x 安装 CDH 6.x_cloudera_03

到这里CM服务端安装完成，先不要启动，安装数据库

安装MariaDB

        MariaDB数据库管理系统是MySQL的一个分支，主要由开源社区在维护，采用GPL授权许可 MariaDB的目的是完全兼容MySQL，包括API和命令行，使之能轻松成为MySQL的代替品。在存储引擎方面，使用XtraDB（英语：XtraDB）来代替MySQL的InnoDB。 MariaDB由MySQL的创始人Michael Widenius（英语：Michael Widenius）主导开发，他早前曾以10亿美元的价格，将自己创建的公司MySQL AB卖给了SUN，此后，随着SUN被甲骨文收购，MySQL的所有权也落入Oracle的手中。MariaDB名称来自Michael Widenius的女儿Maria的名字。

       MariaDB基于事务的Maria存储引擎，替换了MySQL的MyISAM存储引擎，它使用了Percona的 XtraDB，InnoDB的变体，分支的开发者希望提供访问即将到来的MySQL 5.4 InnoDB性能。这个版本还包括了 PrimeBase XT (PBXT) 和 FederatedX存储引擎。

yum -y install mariadb
yum -y install mariadb-server

启动、配置MariaDB

systemctl start mariadb
[root@cdh01 ~]# /usr/bin/mysql_secure_installation

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
      SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we'll need the current
password for the root user.  If you've just installed MariaDB, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none): 
OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MariaDB
root user without the proper authorisation.

Set root password? [Y/n] y #设置root密码
New password: 
Re-enter new password: 
Password updated successfully!
Reloading privilege tables..
 ... Success!


By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them.  This is intended only for testing, and to make the installation
go a bit smoother.  You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n] y #是否禁止匿名登录
 ... Success!

Normally, root should only be allowed to connect from 'localhost'.  This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] n #是否禁止远程登录
 ... skipping.

By default, MariaDB comes with a database named 'test' that anyone can
access.  This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n] y #删除测试数据库
 - Dropping test database...
 ... Success!
 - Removing privileges on test database...
 ... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n] y #加载最新设置
 ... Success!

Cleaning up...

All done!  If you've completed all of the above steps, your MariaDB
installation should now be secure.

Thanks for using MariaDB!

登录mysql 创建数据库

mysql -uroot -p123456 

create database metastore default character set utf8;  #创建数据库名称为metastore，默认编码utf-8
CREATE USER 'hive'@'%' IDENTIFIED BY '123456';    #创建用户名为hive的用户，密码为123456，可以在任何机器上访问
GRANT ALL PRIVILEGES ON metastore. * TO 'hive'@'%';   #授权hive用户metastore所有权限
FLUSH PRIVILEGES;   #刷新配置

create database cm default character set utf8;  
CREATE USER 'cm'@'%' IDENTIFIED BY '123456';   
GRANT ALL PRIVILEGES ON cm. * TO 'cm'@'%';   
FLUSH PRIVILEGES;  


create database hue default character set utf8;   
CREATE USER 'hue'@'%' IDENTIFIED BY '123456';    
GRANT ALL PRIVILEGES ON hue. * TO 'hue'@'%';    
FLUSH PRIVILEGES;

create database oozie default character set utf8;   
CREATE USER 'oozie'@'%' IDENTIFIED BY '123456';    
GRANT ALL PRIVILEGES ON oozie. * TO 'oozie'@'%';    
FLUSH PRIVILEGES;

create database sentry default character set utf8;   
CREATE USER 'sentry'@'%' IDENTIFIED BY '123456';    
GRANT ALL PRIVILEGES ON sentry. * TO 'sentry'@'%';    
FLUSH PRIVILEGES;

安装Mysql驱动

此处不安装，服务无法找到mysql驱动，会导致服务失败，集群所有服务都安装，避免以后遇到问题

[root@cdh01 cdh]# mkdir  /usr/share/java 
[root@cdh01 cdh]# mv mysql-connector-java-5.1.30.jar /usr/share/java/
[root@cdh01 cdh]# chmod 777 /usr/share/java/mysql-connector-java-5.1.30.jar 
[root@cdh01 cdh]# ln -s /usr/share/java/mysql-connector-java-5.1.30.jar /usr/share/java/mysql-connector-java.jar 
[root@cdh01 cdh]# ll /usr/share/java/
总用量 932
-rwxrwxrwx. 1 root root 954038 6月  19 11:28 mysql-connector-java-5.1.30.jar
lrwxrwxrwx. 1 root root     47 9月   1 13:59 mysql-connector-java.jar -> /usr/share/java/mysql-connector-java-5.1.30.jar

Cloudera Manager Server 初始化数据库

注意这里 123456 是我们上一步创建的数据库密码

[root@cdh01 ~]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql cm cm 123456
JAVA_HOME=/usr/local/java/jdk1.8.0_191
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing:  /usr/local/java/jdk1.8.0_191/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
[                          main] DbCommandExecutor              INFO  Successfully connected to database.
All done, your SCM database is configured correctly!

启动Cloudera Manager Server 服务

systemctl start cloudera-scm-server
systemctl status cloudera-scm-server

启动可能需要半分钟左右，可以查看日志启动状态，提示如下内容就启动成功了

日志目录：/var/log/cloudera-scm-server

2019-09-01 14:05:02,994 INFO WebServerImpl:org.eclipse.jetty.server.AbstractConnector: Started ServerConnector@534fca49{HTTP/1.1,[http/1.1]}{0.0.0.0:7180}
2019-09-01 14:05:02,995 INFO WebServerImpl:org.eclipse.jetty.server.Server: Started @95217ms
2019-09-01 14:05:02,995 INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.

浏览器访问，提示登录，默认账号密码均为 admin

http://ip:7180/cmf/login

Centos7.x 安装 CDH 6.x_java_04

Centos7.x 安装 CDH 6.x_cloudera_05

接受继续

Centos7.x 安装 CDH 6.x_hive_06

这里选择免费版本(也可以选择试用版，到期后自动恢复成免费版本)

Centos7.x 安装 CDH 6.x_cloudera_07

开始安装集群

Centos7.x 安装 CDH 6.x_hive_08

Centos7.x 安装 CDH 6.x_hive_09

 

Centos7.x 安装 CDH 6.x_hive_10

选择我们刚刚安装的httpd服务链接

Centos7.x 安装 CDH 6.x_cloudera_11

Centos7.x 安装 CDH 6.x_cloudera_12

Centos7.x 安装 CDH 6.x_java_13

Centos7.x 安装 CDH 6.x_cloudera_14

Centos7.x 安装 CDH 6.x_hive_15

Centos7.x 安装 CDH 6.x_hive_16

Centos7.x 安装 CDH 6.x_hive_17

Centos7.x 安装 CDH 6.x_java_18

agent 安装成功，下一步

Centos7.x 安装 CDH 6.x_hive_19

安装 parcel ，这个过程可能要等一段时间

Centos7.x 安装 CDH 6.x_hive_20

 

Centos7.x 安装 CDH 6.x_java_21

 

Centos7.x 安装 CDH 6.x_hive_22

这里我们看下检查结果

Centos7.x 安装 CDH 6.x_hive_23

有两处异常，根据提示修改配置：

 sysctl vm.swappiness=10 修改CDH建议值，重新检测。

解决了第一个问题

Centos7.x 安装 CDH 6.x_hive_24

设置一下两个配置

echo never > /sys/kernel/mm/transparent_hugepage/defrag echo never > /sys/kernel/mm/transparent_hugepage/enabled

再次查看，全部正常

Centos7.x 安装 CDH 6.x_cloudera_25

Centos7.x 安装 CDH 6.x_cloudera_26

 

Centos7.x 安装 CDH 6.x_cloudera_27

Centos7.x 安装 CDH 6.x_hive_28

根据需求自己分配角色

Centos7.x 安装 CDH 6.x_java_29

Centos7.x 安装 CDH 6.x_cloudera_30

Centos7.x 安装 CDH 6.x_hive_31

Centos7.x 安装 CDH 6.x_cloudera_32

Centos7.x 安装 CDH 6.x_java_33

安装成功

Centos7.x 安装 CDH 6.x_hive_34