Ambari-2.7.5+HDP-3.1.5安装手册 部署1

一、服务器集群规划与安装

建议Ambari-Server单独安装在一台配置较低服务器，方便用于管理，且他不作为hadoop集群中的一台，这样有利用管理，是否配置FQDN等信息。如果FQDN域名信息为bigdata.local则各个机器名为hadoop79.bigdata.local，可以不配置，如果配置在/etc/hosts中配置。

服务器IP	功能规划（）	描述说明
10.99.67.79	（hadoop79）Ambari-Server,Mysql-5.7	如果可以考虑这台服务器单独出hadoop集群
10.99.67.83	（hadoop83）Hadoop,Yarn,Zookeeper
10.99.67.84	（hadoop84）Hadoop,Yarn,Zookeeper
10.99.67.85	（hadoop85）Hadoop,Yarn,Zookeeper

 

二、环境准备与安装

相关软件

1.mysql-5.7.20-1.el7.x86_64.rpm-bundle.tar
2.mysql-connector-java-5.1.47.jar
3.jdk1.8.0_231.tar.gz
4.ambari-2.7.5.0-centos7.tar.gz
5.HDP-3.1.5.0-centos7-rpm.tar.gz
6.HDP-GPL-3.1.5.0-centos7-gpl.tar.gz
7.HDP-UTILS-1.1.0.22-centos7.tar.gz
8.libtirpc-0.2.4-0.16.el7.x86_64.rpm
9.libtirpc-devel-0.2.4-0.16.el7.x86_64.rpm

安装一些系统所有必要的工具（所有节点）

yum -y install gcc gcc-c++ kernel-devel libtirpc-devel 
yum -y install rpm-build yum -y install net-tools 
yum -y install openssl openssl-devel patch 
yum -y install wget curl 
yum -y install bash-completion 
yum -y install unzip zip 
yum -y install libtirpc libtirpc-devel （如果他安装没有，可以是提供的rpm来安装） 
rpm -ivh libtirpc-0.2.4-0.16.el7.x86_64.rpm 
rpm -ivh libtirpc-devel-0.2.4-0.16.el7.x86_64.rpm

关闭selinux（需重启）（所有节点）

​​vim /etc/sysconfig/selinux​​

修改SELINUX=disabled

查看状态：getenforce，返回Disable为关闭。

 

关闭防火墙（所有节点）

systemctl stop firewalld 
systemctl status firewalld 
systemctl disable firewalld

关闭防火墙：systemctl stop firewalld

查看防火墙的状态：systemctl status firewalld

设置开机不启动：systemctl disable firewalld

查看防火墙服务是否设置开机启动：systemctl is-enabled firewalld

设置最大文件打开数最大进程数（所有节点）

# 编辑配置文件：/etc/security/limits.conf，

​​vim /etc/security/limits.conf​​

添加如下配置：

* soft nproc 204800
* hard nproc 204800
* soft nofile 655350
* hard nofile 655350

修改Linux系统内核内存参数（属于调优，可以不用调整）（所有节点）

# 编辑配置文件：/etc/sysctl.conf，

​​vim /etc/sysctl.conf​​

添加如下配置：

vm.swappiness=1
vm.min_free_kbytes=4194304
vm.zone_reclaim_mode=0

# 使修改生效：sysctl -p

关闭透明大页THP（需重启）（所有节点）

# 查看透明大页情况：

cat /sys/kernel/mm/transparent_hugepage/defrag 
cat /sys/kernel/mm/transparent_hugepage/enabled

# 关闭透明大页：vim /etc/rc.d/rc.local，添加如下配置：

if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
  echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
  echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi

# 保存退出，然后赋予rc.local文件执行权限：chmod +x /etc/rc.d/rc.local

安装时间同步工具（所有节点）

#安装时间同步工具（所有节点）

#ntp或者chrony

yum install -y ntp 
systemctl enable ntpd

java安装 （所有节点）

1.java解压安装

cd /opt tar -zxvf jdk1.8.0_231.tar.gz

2.编辑环境变量配置

​​vim /etc/profile ​​

export JAVA_HOME=/opt/jdk1.8.0_231 
export PATH=$JAVA_HOME/bin:$PATH 
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

3.生效环境变量配置

​​source /etc/profile​​

4.检验JDK配置结果

​​jave -version​​

配置hosts（所有节点）

配置hosts

​​vim /etc/hosts​​

添加如下配置

10.99.67.79 hadoop79 
10.99.67.83 hadoop83 
10.99.67.84 hadoop84 
10.99.67.85 hadoop85

配置SSH无密码登陆（ambari-server节点或主节点，建议集群所有集群都做SSH无密码登陆）

ssh-keygen -t rsa 
ssh-copy-id hadoop79 
ssh-copy-id hadoop83 
ssh-copy-id hadoop84 
ssh-copy-id hadoop85

验证：

ssh hadoop79 
ssh hadoop83 
ssh hadoop84 
ssh hadoop85

 

三、安装mysql

解压mysql-5.7的RPM安装包

​​tar -xvf mysql-5.7.20-1.el7.x86_64.rpm-bundle.tar​​

查找系统自带mysql或者maradb

rpm -qa |grep mysql 
rpm -qa |grep mariadb

如果找到按下面命令删除对应包

​​rpm -e mariadb-libs --nodeps​​

先yum安装部分依赖包

yum -y install perl perl-devel autoconf libaio libaio-devel perl-Time-HiRes perl-JSON perl-Data-Dumper net-tools numactl

安装rpm包

rpm -ivh mysql-community-common-5.7.20-1.el7.x86_64.rpm 
rpm -ivh mysql-community-libs-5.7.20-1.el7.x86_64.rpm 
rpm -ivh mysql-community-libs-compat-5.7.20-1.el7.x86_64.rpm 
rpm -ivh mysql-community-embedded-5.7.20-1.el7.x86_64.rpm 
rpm -ivh mysql-community-client-5.7.20-1.el7.x86_64.rpm 
rpm -ivh mysql-community-devel-5.7.20-1.el7.x86_64.rpm 
rpm -ivh mysql-community-embedded-compat-5.7.20-1.el7.x86_64.rpm 
rpm -ivh mysql-community-embedded-devel-5.7.20-1.el7.x86_64.rpm 
rpm -ivh mysql-community-server-5.7.20-1.el7.x86_64.rpm 
rpm -ivh mysql-community-test-5.7.20-1.el7.x86_64.rpm

查看状态并启动mysqld

# 查看mysql服务

service mysqld status 
#systemctl status mysqld.service

# 启动mysql服务

service mysqld start 
#systemctl start mysqld.service

# 查看密码

cat /var/log/mysqld.log | grep password

登陆修改且密码

mysql -u root -p

# 首次登录必须先修改密码（强密码）

alter user 'root'@'localhost' identified by 'root__123AbCdHjKl';

设置可以配置弱密码，（这步可以不用配置，如果使用弱密码则需要配置）

SHOW variables LIKE 'validate_password%'; 
set global validate_password_policy=0; 
set global validate_password_length=4;

# 可以修改为弱密码啦 
#ALTER USER 'root'@'localhost' IDENTIFIED BY "bigdata123"; 
#flush privileges; 
# 更新权限

配置root可以远程访问

use mysql; 
update user set host = '%' where user ='root'; 
# 配置可远程登录 
flush privileges; 
# 更新权限 
exit; 
# 退出

配置mysql服务器编码

# 查看mysql服务的编码

show variables like '%char%';

# 修改MySQL配置文件：/etc/my.cnf

​​vim /etc/my.cnf​​

# 在[mysqld]下添加

[mysqld] 
character-set-client-handshake = FALSE 
character-set-server = utf8mb4 
collation-server = utf8mb4_unicode_ci 
default-storage-engine=INNODB 
#init_connect='SET NAMES utf8mb4'

# 在[client]下添加（如果没有[client]，则创建）

[client] 
default-character-set = utf8mb4 
[mysql] 
default-character-set = utf8mb4

重启mysql服务

service mysqld restart 
#systemctl restart mysqld

设置开机启动

chkconfig mysqld on 
#systemctl enable mysqld

msyql服务器安装完成。

 

四、安装Ambari-Server

安装一些yum工具命令，一般系统已经安装

yum install yum-utils createrepo yum-plugin-priorities

安装httpd

# 安装服务

yum install httpd -y

# 启动httpd服务：systemctl start httpd

# 查看httpd状态：systemctl status httpd

# 设置开启启动：systemctl enable httpd

# 查看是否开机启动：systemctl is-enabled httpd

把HDP文件放到httpd去

# 解压Ambari压缩包
tar -zxvf /root/ambari-2.7.5.0-centos7.tar.gz -C /var/www/html/
# 解压HDP压缩包
tar -zxvf /root/HDP-3.1.5.0-centos7-rpm.tar.gz -C /var/www/html/
# 解压HDP-UTILS压缩包
tar -zxvf /root/HDP-UTILS-1.1.0.22-centos7.tar.gz -C /var/www/html/
# 解压HDP-GPL压缩包
tar -zxvf /root/HDP-GPL-3.1.5.0-centos7-gpl.tar.gz -C /var/www/html/

创建对应的HDP源

在/etc/yum.repo.d/下创建4个文件，请按照实际的ambari-server服务器IP来修改下面文件内容

# ambari库源
# cat ambari.repo 
[ambari-2.7.5.0-72]
name=ambari-2.7.5.0-72
baseurl=http://10.99.67.79/ambari/centos7/2.7.5.0-72/
path=/
enabled=1
gpgkey= http://10.99.67.79/ambari/centos7/2.7.5.0-72/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
gpgcheck=1
priority=1
# HDP库源
# cat HDP.repo 
[HDP-3.1.5]
name=HDP-3.1.5
baseurl=http://10.99.67.79/HDP/centos7/3.1.5.0-152/
path=/
enabled=1
gpgkey=http://10.99.67.70/HDP/centos7/3.1.5.0-152/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
gpgcheck=1
priority=1
# HDP-UTILS库源
# cat HDP-UTILS.repo 
[HDP-UTILS-1.1.0.22]
name=HDP-UTILS-1.1.0.22
baseurl=http://10.99.67.79/HDP-UTILS/centos7/1.1.0.22/
path=/
enabled=1
gpgkey=http://10.99.67.79/HDP-UTILS/centos7/1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
gpgcheck=1
priority=1
# HDP-GPL库源
# cat HDP-GPL.repo 
[HDP-3.1.5-GPL]
name=HDP-3.1.5-GPL
baseurl=http://10.99.67.79/HDP-GPL/centos7/3.1.5.0-152/
path=/
enabled=1
gpgkey=http://10.99.67.79/HDP-GPL/centos7/3.1.5.0-152/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
gpgcheck=1
priority=1

拷贝HDP库源文件到所有服务器

scp *.repo root@hadoop83:/etc/yum.repos.d/ 
scp *.repo root@hadoop84:/etc/yum.repos.d/ 
scp *.repo root@hadoop85:/etc/yum.repos.d/

测试HDP源（所有服务器）

yum clean all 
yum repolist 
yum makecache

安装ambari-server

yum -y install ambari-server

安装ambari-server的mysql的驱动

mkdir /usr/share/java 
cp mysql-connector-java-5.1.47.jar /usr/share/java/mysql-connector-java.jar 
cp /usr/share/java/mysql-connector-java.jar /var/lib/ambari-server/resources/

配置ambari-server的mysql的驱动

配置/etc/ambari-server/conf/ambari.properties，添加如下行 server.jdbc.driver.path=/usr/share/java/mysql-connector-java.jar

执行ambari-server的mysql的驱动注册

ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar

# ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar
Using python  /usr/bin/python
Setup ambari-server
Copying /usr/share/java/mysql-connector-java.jar to /var/lib/ambari-server/resources/mysql-connector-java.jar
If you are updating existing jdbc driver jar for mysql with mysql-connector-java.jar. Please remove the old driver jar, from all hosts. Restarting services that need the driver, will automatically copy the new jar to the hosts.
JDBC driver was successfully initialized.
Ambari Server 'setup' completed successfully.

初始化ambari-server

ambari-server setup

# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK
==============================================================================
Enter choice (1): 2
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /opt/jdk1.8.0_231
Validating JDK on Ambari Server...done.
Check JDK version for Ambari Server...
JDK version found: 8
Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.
Checking GPL software agreement...
GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (y)? 
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 3
Hostname (localhost): 
Port (3306): 
Database name (ambari): 
Username (ambari): 
Enter Database Password (bigdata): ambari123
Re-enter password: 
Configuring ambari database...
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL directly from the database shell to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views...
ambari-admin-2.7.5.0.72.jar
....
Ambari repo file doesn't contain latest json url, skipping repoinfos modification
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.

ambari数据库的导入

-- 创建ambari数据库
CREATE DATABASE ambari; 
-- 创建ambari用户
CREATE USER 'ambari'@'%' IDENTIFIED BY 'ambari123'; 
-- 给创建的ambari赋权
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%'; 

#CREATE USER 'ambari'@'localhost' IDENTIFIED BY 'ambari123'; 
#GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'localhost'; 
#CREATE USER 'ambari'@'hadoop79' IDENTIFIED BY 'ambari123'; 
#GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'hadoop79';
-- 刷新权限
flush privileges;

-- 导入ambari的数据库
use ambari;
source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql

-- 验证数据表
show tables;
-- 验证用户
use mysql; 
select host,user from user where user='ambari';

-- #创建其它大数据组件的库
CREATE DATABASE hive; 
use hive; 
CREATE USER 'hive'@'%' IDENTIFIED BY 'hive123'; 
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%'; 
#CREATE USER 'hive'@'localhost' IDENTIFIED BY 'hive123'; 
#GRANT ALL PRIVILEGES ON *.* TO 'hive'@'localhost'; 
#CREATE USER 'hive'@'hadoop79' IDENTIFIED BY 'hive123'; 
#GRANT ALL PRIVILEGES ON *.* TO 'hive'@'hadoop79'; 
FLUSH PRIVILEGES;

CREATE DATABASE oozie; 
use oozie; 
CREATE USER 'oozie'@'%' IDENTIFIED BY 'oozie123'; 
GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'%'; 
#CREATE USER 'oozie'@'localhost' IDENTIFIED BY 'oozie123'; 
#GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'localhost'; 
#CREATE USER 'oozie'@'hadoop79' IDENTIFIED BY 'oozie123'; 
#GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'hadoop79'; 
FLUSH PRIVILEGES;

安装ambari-agent(所有节点执行)

yum -y install ambari-agent 
systemctl start ambari-agent

启动ambari服务

启动ambari服务
#启动
ambari-server start
#查看状态
#ambari-server status
#停止
ambari-server stop

页面创建集群

打开http://10.99.67.79:8080 创建集群。

 

 

五、通过页面创建hadoop集群

打开http://10.99.67.79:8080 ，输入admin/admin登陆。

点击向导开始

输入要创建的集群名称

 

删除其它系统，只保留可以系统的，并填写对应的URL。

填写好hosts和AmbariServer的私有key。

开始安装连接注册集群机器

注册集群机器成功

机器检测

组件选择

组件分配到各个实例

其它的组件分配选择

配置账号

hive的数据库配置

各个组件的配置

 

对应的账户，这里默认。

Rewiew信息

开始安装

继续安装

安装完成

 

安装完成的Summary

Dashboard页面

HDFS页面