如何在Redhat7.7中安装CDP DC7.0.3

标签(空格分隔): 大数据平台构建


  • 一:关于CDP DC 7.0.3 概述
  • 二: 系统环境初始化
  • 三: 构建CDP DC 7.0.3

一: 关于CDP DC 7.0.3 概述

1.1 CDP dc 7.0.3

CDP DC7.0.3是Cloudera与Hortonworks合并后,第一个融合CDH和HDP所有组件的on-premise版本,CDP Data Center主要由Cloudera Runtime构成,Cloudera Runtime由超过35个开源项目组成,当然CDP Data Center还包括其它功能如管理功能Cloudera Manager,Key Management,专业支持等,如下图所示:

image_1e3mfkh9g13vi13c8a0a5prrmc9.png-748.6kB

1.2 Cloudera Runtime的主要组件版本如下图所示:

image_1e3mfuobeatb1935jb99io17orm.png-146.7kB

二: 系统环境初始化

2.1 系统环境介绍

OS: rhel7.7x64 

cat /etc/hosts 
---
192.168.100.21  rhel01.flyfish
192.168.100.22  rhel02.flyfish
192.168.100.23  rhel03.flyfish
192.168.100.24  rhel04.flyfish
192.168.100.25  rhel05.flyfish
192.168.100.26  rhel06.flyfish
---

本次采用 前四台 机器

image_1e3mg7g6gtf41ieafiuh61qut1g.png-39.7kB

image_1e3mg5npfrblloc1ja4acn19u813.png-40.8kB

2.2 无密钥登录配置

做root用户无密钥认证

 ssh-keygen ---一直敲回车到最后

 cat id_rsa.pub >> authorized_keys 

 将所有的公钥导入authorized_keys  分发到 所有的 机器的.ssh/ 下面

 然后测试

image_1e3mgsn6s213cl4175l1odjnjj1t.png-115.9kB


2.3关闭 firewalld,iptables 与 selinux

所有节点都要配置:

yum install -y firewall* iptable* 

  (1) 关闭firewalld

 systemctl stop firewalld.service
 systemctl disable firewalld.service
 firewall-cmd --state

 (2) 关闭iptables 
 iptables -F
 systemctl stop iptables.service 
 service iptables save
 systemctl disable iptables.service 

 (3) 禁用 selinux 

  sed -i '/SELINUX/s/enforcing/disabled/' /etc/selinux/config  
  setenforce 0
  getenforce 0
  sestatus

  重启机器

  reboot

image_1e3mh5q0f1qgi1vjm8c61604sng2q.png-216.5kB


2.4 所有机器更改RHEL7.7X64的文件句柄数

vim /etc/security/limits.conf
---

*               soft    nofile          65535
*               hard    nofile          1029345
*               soft    nproc           unlimited
*               hard    nproc           unlimited
*               soft    memlock         unlimited
*               hard    memlock         unlimited

---

image_1e3mhnvi91l92no1kogsbh1er837.png-98.3kB

ansible cdh7 -m copy -a "src=/etc/security/limits.conf dest=/etc/security/" 

image_1e3muuec7ivqaad1oftl0hif893.png-133.2kB

image_1e3muv02o12601fu51c5i4tc1b459g.png-135.9kB


2.5 时间同步:

所有机器 yum install chrony*
1. rhel01.flyfish:

vim /etc/chrony.conf 增加
----
注释掉相关默认的server
server ntp1.aliyun.com iburst

allow 192.168.100.0/24

service chronyd start
chkconfig chronyd on 

2. rhel02.flyfish----rhel04.flyfish 同步rhel01.flyfish

vim /etc/chrony.conf
----
注释掉默认的server
server 192.168.100.21 iburst
----

service chronyd stop 
service chronyd start 
chkconfig chronyd on 

image_1e3mjvngm1i9c1vgg256o9q19ac4r.png-193.9kB

image_1e3mk79o01b3610gjie1ul14p66s.png-113.5kB

image_1e3mk162i1up21fru9r1ng81nhj58.png-92kB

image_1e3mk5gevaql1tq912dmulalnf5l.png-147.8kB

image_1e3mk5tt319m39mg1eq0hkq1r0t62.png-93.4kB

image_1e3mk6g5250316dqderc35u1m6f.png-103.8kB


2.7 cdp dc 7.0.3 安装所需文件下载

CM 的rpm包:

---
cloudera-manager-agent-7.0.3-1635136.el7.x86_64.rpm
cloudera-manager-daemons-7.0.3-1635136.el7.x86_64.rpm
cloudera-manager-server-7.0.3-1635136.el7.x86_64.rpm
cloudera-manager-server-db-2-7.0.3-1635136.el7.x86_64.rpm
enterprise-debuginfo-7.0.3-1635136.el7.x86_64.rpm
openjdk8-8.0+232_9-cloudera.x86_64.rpm
allkeys.asc

下载地址:
http://archive.cloudera.com/cm7/7.0.3/redhat7/yum/RPMS/x86_64/
http://archive.cloudera.com/cm7/7.0.3/allkeys.asc

----

cdh7.0.3 的parcel:

--
CDH-7.0.3-1.cdh7.0.3.p0.1635019-el7.parcel
CDH-7.0.3-1.cdh7.0.3.p0.1635019-el7.parcel.sha256
manifest.json

下载地址:
http://archive.cloudera.com/cdh7/7.0.3.0/parcels/

---

image_1e3mpiorp1js68b15m61ti3a8c7c.png-171.6kB

2.8 设置swap

将所有的swap为1,并即时生效

ansible cdh7 -m shell -a "echo '1' > /proc/sys/vm/swappiness"

ansible cdh7 -m shell -a "sysctl -a | grep vm.swappiness"

image_1e3muoi8h3l61t758mr7rq1r0289.png-54.1kB

image_1e3mupfkd1bvr1epk1m61oms1nh98m.png-256.7kB

vim /etc/sysctl.conf 
----
最后加上:
vm.swappiness = 1 
----
sysctl -p

image_1e3mv57c91gu4bl6199g9rl7toad.png-145.1kB

image_1e3mv60io1a0ecu01qlrko312cba.png-167.9kB

image_1e3o6k4ij1jkk10l1o02dpp1calbn.png-177.2kB

2.8 设置最大透明数

ansible cdh7 -m shell -a "echo 'never' > /sys/kernel/mm/transparent_hugepage/defrag"

ansible cdh7 -m shell -a "echo 'never' > /sys/kernel/mm/transparent_hugepage/enabled"

ansible cdh7 -m shell -a "cat /sys/kernel/mm/transparent_hugepage/enabled"
ansible cdh7 -m shell -a "cat /sys/kernel/mm/transparent_hugepage/defrag"

image_1e3o6s5g7oc01qn11kvdp72nrac4.png-221.7kB


修改所有节点的/etc/rc.d/rc.local文件的权限以实现开机执行
ansible cdh7 -m shell -a "chmod +x /etc/rc.d/rc.local"

vim /etc/rc.d/rc.local
---
最后加上:

if test -f /sys/kernel/mm/transparent_hugepage/enabled; 

then

echo never > /sys/kernel/mm/transparent_hugepage/enabled

fi

if test -f /sys/kernel/mm/transparent_hugepage/defrag; 
then

echo never > /sys/kernel/mm/transparent_hugepage/defrag

fi

---

ansible cdh7 -m copy -a "src=/etc/rc.d/rc.local dest=/etc/rc.d/"

ansible cdh7 -m shell -a "tail -f /etc/rc.d/rc.local"

image_1e3o7hup21an49u315meark13akch.png-178.6kB

image_1e3o7if7prj49jr13t6rio18alcu.png-155.1kB

image_1e3o7ivnhv13ha51gudgp31pqldb.png-125.8kB


2.9 配置httpd 服务器

ansible rhel01.flyfish -m yum -a "state=present name=httpd"
ansible rhel01.flyfish -m service -a "name=httpd state=started"
ansible rhel01.flyfish -m shell -a "ps -ef |grep httpd"

image_1e3o87uq31vpp1iie11tr1b0216hido.png-53.9kB

image_1e3o88s3jbs91svfvhjro6uqbe5.png-158kB

image_1e3o89gtkskqq1etqb1o721l2rei.png-138.6kB


修改 /etc/httpd/conf/httpd.conf 在<IfModule mime_module>中修改以下内容

vim /etc/httpd/conf/httpd.conf

---
AddType application/x-compress .Z
AddType application/x-gzip .gz .tgz .parcel
---

service httpd restart 

image_1e3o8ddst58bcds1j4h1o3a18ejev.png-121.6kB

image_1e3o8fu1tek114sm1h7pa06365fc.png-50.2kB

2.10 安装maridb 数据库

yum install mariadb* 

service mariadb start 

chkonfig mariadb on 

mysql_secure_installation

image_1e3o8md7cuo8je115pl13191nt4fp.png-182kB

image_1e3o8niqoudh516beejjt1t2fgm.png-195.9kB

2.11 创建CM,Hive等需要的表


vim cm.sql
----
create database metastore default character set utf8;
CREATE USER 'hive'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON metastore. * TO 'hive'@'%';
FLUSH PRIVILEGES;
create database cm default character set utf8;
CREATE USER 'cm'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON cm. * TO 'cm'@'%';
FLUSH PRIVILEGES;
create database rm default character set utf8;
CREATE USER 'rm'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON rm. * TO 'rm'@'%';
FLUSH PRIVILEGES;
create database hue default character set utf8;
CREATE USER 'hue'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON hue. * TO 'hue'@'%';
FLUSH PRIVILEGES;
create database oozie default character set utf8;
CREATE USER 'oozie'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON oozie. * TO 'oozie'@'%';
FLUSH PRIVILEGES;
create database ranger default character set utf8;
CREATE USER 'rangeradmin'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON ranger. * TO 'rangeradmin'@'%';
FLUSH PRIVILEGES;
----

mysql -uroot -pflyfish225

source cm.sql

image_1e3o93pdi52pa29inm1ipms21h3.png-198.6kB

image_1e3o948evds31h9e18rm1vb014l5hg.png-148.6kB

image_1e3o94qbsrri12onb4bivu9r5ht.png-55.1kB

2.12 安装JDBC驱动

mkdir -p /usr/share/java/
tar -zxvf mysql-connector-java-5.1.44.tar.gz

mv mysql-connector-java-5.1.44-bin.jar /usr/share/java

cd /usr/share/java/

mv mysql-connector-java-5.1.44-bin.jar mysql-connector-java.jar 

chmod 777 mysql-connector-java.jar

image_1e3o99mbfv3e1beh1u08edf1preia.png-74.6kB

image_1e3o9dt0f1ais1khb11e61m75et0j7.png-248.6kB

image_1e3o9eak130phkg1d9j9u1368jk.png-45.8kB

三: 构建CDP DC 7.0.3

3.1 安装 jdk

所有节点 卸载系统自带jdk

rpm -qa |grep jdk 
rpm -e java-1.8.0-openjdk-1.8.0.222.b03-1.el7.x86_64 copy-jdk-configs-3.3-10.el7_5.noarch java-1.8.0-openjdk-headless-1.8.0.222.b03-1.el7.x86_64 --nodeps 

yum -y install openjdk8-8.0+232_9-cloudera

vim /etc/profile
---
到最后加上jdk 的环境变量

export JAVA_HOME=/usr/java/jdk1.8.0_232-cloudera
export CLASSPATH=.:$JAVA_HOME/jre/lib:$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin

----
source /etc/profile

java -version 

image_1e3ob7je7194mb9v1m2p17ot1jj8m2.png-53.3kB

image_1e3ob8a74mt21bnham31ofao5tmf.png-59kB

image_1e3ob8od91e2l1rdk1l3rtld11mbms.png-52.3kB

image_1e3ob9607hp3kh91nct113u17osn9.png-55.8kB

3.2 配置cdh7.0.3 的源

mv cdh7.0.3 cdh7
cd cdh7 

createrepo .

cd ..

mv cdh7 /var/www/html/

image_1e3o9mqtmsiokhepp3op16gek1.png-166.6kB

image_1e3o9nb8q30pgfkm4pg2h1o9eke.png-221.8kB

image_1e3o9pagm18bu7c51002157q1i8gkr.png-349.9kB


制作Cloudera Manager的repo源

cd /etc/yum.reos.d/

vim cdh.repo
---
[cmrepo]
name = cm_repo
baseurl = http://192.168.100.21/cdh7/
enable = true
gpgcheck = false

---

ansible cdh7 -m copy -a "src=/etc/repos.d/cdh7.repo dest=/etc/repos.d/"
yum repolist

image_1e3oai2k41afg1qvl1ljj7m1visll.png-167.9kB

image_1e3oah4kck128publgeai6gkl8.png-98kB

3.3 安装Cloudera Manager Server

yum -y install cloudera-manager-server

image_1e3obm7sqkk141m1kf6ntnp31nm.png-173.6kB

初始化数据库

/opt/cloudera/cm/schema/scm_prepare_database.sh mysql cm cm password

image_1e3obprog1f3l5o1kn8ligc35o3.png-134.8kB

启动Cloudera Manager Server

systemctl start cloudera-scm-server
systemctl status cloudera-scm-server

cd /var/log/cloudera-scm-server
tail -100f cloudera-scm-server.log

netstat -nultp |grep 7180

image_1e3oc1q9b71u6a4uns3d9djkp0.png-164.9kB

image_1e3oc2qkn1dil18jq19821pfsbgrpd.png-55.7kB

image_1e3oc5r9me94r901uf5omo4vtpq.png-587.8kB

3.4 通过http://192.168.100.21:7180/cmf/login访问CM

用户名:admin
密码: admin

image_1e3occj24ior1osr12dh1jlb1uu8q7.png-336.7kB

登录到CM,选择60天试用,点击继续

image_1e3oce7te1qlo1bjhocfdbg1c7mqk.png-818kB

集群安装:

image_1e3ocfhr71bevcs81ilkns51879r1.png-884.8kB

集群名字

image_1e3ocgp70v31ho5ad5ed7upnre.png-762kB

识别集群的主机

image_1e3ochvp31855qma14bb8e918ivrr.png-534kB

image_1e3ocjj4a15uf65k1vorj6j1umos8.png-838.4kB

image_1e3ocomcq146duc11qdhdv91v38sl.png-1020kB

Parcel Repository & Network Settings

image_1e3oct5ou1hprg2d1pf51fk4cvnt2.png-410.8kB

image_1e3ocv1el1akvvnr8i4a1jfgtf.png-515.9kB

image_1e3od0mnrrqhcm0r901cdmfhpts.png-785.1kB


image_1e3od1qrq1jk354e1oekj9d1dbmu9.png-848kB

image_1e3od3kbisa51j7soevg1n1f4eum.png-684.3kB

安装 cloudera-scm-agent

image_1e3odh4it1loffv19ep1aib8i8v3.png-793.5kB

image_1e3oe7dn77e117q0f0t1ri515ksvg.png-715.8kB

image_1e3oe935jgc0imhnei5pvk72vt.png-527kB

分配parcel包

image_1e3oefr61n333ct1o70ajc2610a.png-734.4kB

image_1e3ogbr8s4nehsj1u2u1hk171g10n.png-752.1kB

点击Inspect Network Performance和Inspect Hosts,检查主机

image_1e3oghhnc1mo3t8c1jr81tag1hus114.png-786.2kB

定义服务:

image_1e3ogj69kmlkefa1trvq7drir11h.png-1023.6kB

image_1e3ogucel1qcu2statsdfp16bk11u.png-950.9kB

image_1e3ogvc6dhf0uss7lfapk1k6812b.png-855.1kB

点击“继续”,进入下一步,测试数据库连接

image_1e3ogvtt9fkb54b1pne16in18an12o.png-874kB

image_1e3oh0riqrntgm3pm01vqb15nc13h.png-864.6kB

image_1e3oh5pvfh8m1pt81ksu1a0c1ved13u.png-848.9kB

首先输入CM的用户名和密码admin/admin

以下4个密码,都需要包含至少一个字母与一个数字,为了安装方便都输入admin@123

image_1e3oha6t51cke9v1t1h16ej3en9.png-977.8kB

image_1e3oialnj1tok1tun1km91j4c1b4jm.png-195.8kB

image_1e3oib7rv1501mh018jo1e2qhec13.png-405.9kB

image_1e3oic57d4o8nfe1evd1rgq1pac1g.png-339.4kB

安装服务初始化

image_1e3ohkldq13s61fec1gsha7jg1o9.png-973.4kB

image_1e3ojrrg9vk41roe1t9v8q4lh1t.png-1082.2kB