大数据平台CDH5.16.2 安装配置

标签(空格分隔): 大数据平台构建


  • 一: 环境初始化
  • 二: 安装CDH5.16.2

1.1: 系统环境介绍

系统: RHEL7.7X64

cat /etc/hosts 
---
192.168.100.21   rhel01.flyfish
192.168.100.22   rhel02.flyfish
192.168.100.23   rhel03.flyfish
192.168.100.24   rhel04.flyfish

-----

本次 安装只配置其中三台机器: 192.168.100.21到 192.168.100.23 

1.2: 关闭 firewalld,iptables 与 selinux

 yum install -y firewall* iptable* 
  (1) 关闭firewalld
 systemctl stop firewalld.service
 systemctl disable firewalld.service
 firewall-cmd --state
 (2) 关闭iptables 
 iptables -F
 systemctl stop iptables.service 
 service iptables save
 systemctl disable iptables.service 
 (3) 禁用 selinux 
  sed -i '/SELINUX/s/enforcing/disabled/' /etc/selinux/config  
  setenforce 0
  getenforce 0
  sestatus
  重启机器
   reboot

image_1e6o0p2nf8dv1sdj750jpvjnh9.png-97.2kB

image_1e6o0prpa25g1ni113lu1j9mnn7m.png-78.1kB

image_1e6o0qg5c1gjm1vkc90l118d2fh13.png-80.7kB

1.3:无密钥登录配置

做root用户无密钥认证
 ssh-keygen ---一直敲回车到最后
 cat id_rsa.pub >> authorized_keys 
 将所有的公钥导入authorized_keys  分发到 所有的 机器的.ssh/ 下面
 然后测试

image_1e6o0rid71psohm31uc013n4i8a1g.png-209.6kB

1.4 配置jdk1.8环境 (所有机器都配置)

1): 卸载原来的jdk
rpm -e java-1.8.0-openjdk-headless-1.8.0.242.b08-0.el7_7.x86_64 javapackages-tools-3.4.1-11.el7.noarch tzdata-java-2019c-1.el7.noarch python-javapackages-3.4.1-11.el7.noarch postgresql-jdbc-javadoc-9.2.1002-6.el7_5.noarch --nodeps

2) 安装 jdk:

rpm -ivh jdk-8u162-linux-x64.rpm

vim /etc/profile 

最后增加:

### JDK ####

export JAVA_HOME=/usr/java/jdk1.8.0_162
export CLASSPATH=.:$JAVA_HOME/jre/lib:$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin
---

java -version 

image_1e6o1h4fe12frh8f1r375tfi1s2a.png-35.6kB

image_1e6o1gip6blbds01pem16t5114e1t.png-49.6kB

image_1e6o1hjiu12r0h2u109a3mrpt32n.png-54.2kB

1.5:chrony 时间同步服务器

所有机器 yum install chrony*
1. rhel01.flyfish:
vim /etc/chrony.conf 增加
----
注释掉相关默认的server
server ntp1.aliyun.com iburst
allow 192.168.100.0.0/16
service chronyd start
chkconfig chronyd on 
2. rhel02.flyfish---rhel03.flyfish 同步rhel01.flyfish
vim /etc/chrony.conf
----
注释掉默认的server
server 192.168.100.21 iburst
----
service chronyd start 
chkconfig chronyd on 

image_1e6o1la6so8t1ajc8sdrp17e534.png-165.4kB

image_1e6oijgh91cb5no71ruf1c41e5h3h.png-93kB

image_1e6oik50f1qp133c9mr1els1bro3u.png-91.9kB

1.6: 安装maridb-server 数据库

1)安装mariadb-server
yum install -y mariadb*

2) 启动 mariadb-server

service mariadb start 
chkcofig mariadb on 

3) 配置数据库密码:

mysql_secure_installation

4) 登录mariadb-server 数据库

   mysql -uroot -pflyfish225

image_1e6oj1gqr1l521upuqchhrd16v14b.png-111.7kB


1.8 关于系统参数配置

vim /etc/sysctl.conf

---
执行以下命令使参数生效:
##内核默认参数 
kernel.sysrq = 0 
kernel.core_uses_pid = 1 
kernel.msgmnb = 65536 
kernel.msgmax = 65536 
kernel.shmmax = 68719476736 
kernel.shmall = 4294967296 
##打开文件数参数(20*1024*1024) 
fs.file-max= 20971520 
##WEB Server参数 
net.ipv4.tcp_tw_reuse=1 
net.ipv4.tcp_tw_recycle=1 
net.ipv4.tcp_fin_timeout=30 
net.ipv4.tcp_keepalive_time=1200 
net.ipv4.ip_local_port_range = 1024 65535 
net.ipv4.tcp_rmem=4096 87380 8388608 
net.ipv4.tcp_wmem=4096 87380 8388608 
net.ipv4.tcp_max_syn_backlog=8192 
net.ipv4.tcp_max_tw_buckets = 5000 
##TCP补充参数 
net.ipv4.ip_forward = 0 
net.ipv4.conf.default.rp_filter = 1 
net.ipv4.conf.default.accept_source_route = 0 
net.ipv4.tcp_syncookies = 1 
net.ipv4.tcp_sack = 1 
net.ipv4.tcp_window_scaling = 1 
net.core.wmem_default = 8388608 
net.core.rmem_default = 8388608 
net.core.rmem_max = 16777216 
net.core.wmem_max = 16777216 
net.core.netdev_max_backlog = 262144 
net.ipv4.tcp_max_orphans = 3276800 
net.ipv4.tcp_timestamps = 0 
net.ipv4.tcp_synack_retries = 1 
net.ipv4.tcp_syn_retries = 1 
net.ipv4.tcp_mem = 94500000 915000000 927000000 
##禁用ipv6 
net.ipv6.conf.all.disable_ipv6 =1 
net.ipv6.conf.default.disable_ipv6 =1 
##swap使用率优化 
vm.swappiness=0 
# sysctl -p 
同步到所有机器

image_1e6oj4r479igs2n6fc1iif1lc04o.png-178.4kB

image_1e6oj56nqbbq13v1d8l13i2nhd55.png-59.4kB

1.7 修改文件句柄数

vim /etc/security/limits.conf
---

*               soft    nofile          65535
*               hard    nofile          1029345
*               soft    nproc           unlimited
*               hard    nproc           unlimited
*               soft    memlock         unlimited
*               hard    memlock         unlimited

---

image_1e6oj6anb1s8duvl10sm1tnrb3r5i.png-57.5kB


二:安装CDH5.16.2 平台

2.1: 安装配置依赖包(全部节点都安装)

yum -y install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb postgresql* portmap mod_ssl openssl-devel python-psycopg2 MySQL-python

image_1e6ojc74f18gof98ou81862109s9.png-183.1kB

2.2:安装cloudera-server 与更改配置 (节点全部安装)

rpm -ivh cloudera-manager*
rpm -ivh enterprise-debuginfo-5.14.3-1.cm5143.p0.4.el7.x86_64.rpm

image_1e6ojips2nis6cd18kb9pf1n7am.png-144.8kB

2.3:在mariadb-server 当中创建CM 的库

mysql -uroot -pflyfish225.com
mysql> create database cmf character set utf8;
mysql> grant all privileges on cmf.* to 'cmf'@'%' identified by 'cmf_1234' with grant option;
mysql> grant all privileges on cmf.* to 'cmf'@'node-01.flyfish' identified by 'cmf_1234' with grant option;
mysql> flush privileges;

image_1e6ojkfv01emm1sdfb85foh4gv13.png-186.3kB

2.4:安装离线cdh parcel-repo

#cd cdh5.16.2
#cp -p CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel* /opt/cloudera/parcel-repo/
#cp -p manifest.json /opt/cloudera/parcel-repo/
#cd /opt/cloudera/parcel-repo
#mv CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel.sha1 CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel.sha   
chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/ (所有主机都要执行这条命令)

image_1e6ok287912agpag1l4kcf5i721g.png-203.1kB

image_1e6ok7a621dmq12rbn27vaasct1t.png-78.8kB

2.5 配置jdbc的驱动

tar -zxvf mysql-connector-java-5.1.44.tar.gz
cd mysql-connector-java-5.1.44
mv mysql-connector-java-5.1.44-bin.jar /usr/share/java
cd /usr/share/java
mv mysql-connector-java-5.1.44-bin.jar mysql-connector-java.jar
chmod 777 mysql-connector-java.jar

image_1e6okd8jdko31adf83mpb21arn2a.png-227.7kB

image_1e6okeimvdja161r1o5vf311dc22n.png-221.4kB

2.6 配置数据的注册生成表

1)备份db生成表的文件
cp /etc/cloudera-scm-server/db.properties /etc/cloudera-scm-server/db.propertiesbak 

2)生成表

/usr/share/cmf/schema/scm_prepare_database.sh mysql cmf cmf cmf_1234

image_1e6okio6ajj1dfo5h0k4pvej4h.png-169kB

2.7:修改config.ini 文件

vim /etc/cloudera-scm-agent/config.ini
---
server_host=192.168.100.21
---

scp /etc/cloudera-scm-agent/config.ini root@rhel02.flyfish:/etc/cloudera-scm-agent/

scp /etc/cloudera-scm-agent/config.ini root@rhel03:/etc/cloudera-scm-agent/

image_1e6okljd617ug1cnqrr37u7c175b.png-77.7kB

image_1e6okkst0o7co7s1tav1ifd6u54u.png-68.1kB

2.8: 启动CM-SERVER

cd /etc/init.d/
./cloudera-scm-server start

image_1e6oknugc7t61vjkn7q15bd73a5o.png-108.9kB

查看日志:
cd /var/log/cloudera-scm-server

tail -100f cloudera-scm-server.log

查看有7180 端口启动表示CM管理server启动

netstat -nultp |grep 7180

image_1e6okpakk4gc8fs11pvo3b1l9e65.png-171.4kB

image_1e6okptql1hifdkdd95n81fie6i.png-56.3kB

使用浏览器web打开:

http://192.168.100.21:7180

image_1e6okrcrkmp14ks1rhtunpbk86v.png-147.9kB

image_1e6okrt6k18ssuje8ll12bk5uh7c.png-464.5kB

image_1e6okshf6tsdo36s5o1mclmo87p.png-393.5kB

image_1e6okvc44mqibgi1k6r1spj1sec9d.png-434.4kB

启动agent 端:

cd /etc/init.d/
./cloudera-scm-agent start

image_1e6oku1k3u1j55l116oak21t3786.png-67.6kB

image_1e6okudfuuhg19gts0j11us1rea8j.png-28.8kB

image_1e6okuq7fvbpra18711h5sg2190.png-39.4kB

三: 分配主机与分配角色

image_1e6ol0acq1rpu3hul3l14ku14ce9q.png-209.1kB

image_1e6ol0vpk1jdhko01okm1fdm1j7na7.png-286.1kB

image_1e6ol1jpi185h1u2eu1713oh11m9ak.png-186.8kB

image_1e6ols0eu1m8a5onscnpdaiicd2.png-247.3kB

image_1e6olt02q190a1e5h1rpe1ko0chrdf.png-346.7kB

所有节点执行:
#echo never > /sys/kernel/mm/transparent_hugepage/defrag
#echo never > /sys/kernel/mm/transparent_hugepage/enabled
vim /etc/rc.local 
在最下面增加:
---
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
---
同步所有机器

scp /etc/rc.d/rc.local root@rhel02.flyfish:/etc
scp /etc/rc.d/rc.local root@rhel03.flyfish:/etc

image_1e6ol2tlq177vic582n1gsr1ffcb1.png-28kB

image_1e6ol5nph1971pv31vb11qv61j7mbe.png-44.3kB

image_1e6ol6pvp5qov1s708h81f1ubr.png-95.8kB

image_1e6olv06et32570m251kb3qa0ds.png-399.2kB

image_1e6om2q4i1f561otk12dcrkcd3mec.png-400.5kB

mysql 授权的配置

数据库授权:
mysql -uroot -pflyfish225
---
创建hive 所需要的库并授权:
mysql >create database cdh_hive default character set latin1; 
字符必须为拉丁(latin1),如果uft8可能会报错Specified key was too long; max key length is 767 bytes 
mysql >grant all privileges on cdh_hive.* to 'hive'@'%' identified by 'Hive_123' with grant option;
mysql >grant all privileges on cdh_hive.* to 'hive'@'rhel01.flyfish' identified by 'Hive_123' with grant option;
mysql >flush privileges; 

创建oozie 所需要的库
mysql >create database cdh_oozie default character set utf8; 
mysql >grant all privileges on cdh_oozie.* to 'oozie'@'%' identified by 'Oozie_123' with grant option;
mysql >grant all privileges on cdh_oozie.* to 'oozie'@'rhel01.flyfish' identified by 'Oozie_123' with grant option;
mysql >flush privileges; 

创建hue所需要的库:
mysql >create database cdh_hue default character set utf8; 
mysql >grant all privileges on cdh_hue.* to 'hue'@'%' identified by 'Hue_123' with grant option;
mysql >grant all privileges on cdh_hue.* to 'hue'@'rhel01.flyfish' identified by 'Hue_123' with grant option;
mysql >flush privileges; 

创建CM 的 监控端Activity Monitor 的库
Activity Monitor:
mysql >create database cdh_am default character set utf8; 
mysql >grant all privileges on cdh_am.* to 'am'@'%' identified by 'Am_123' with grant option;
mysql >grant all privileges on cdh_am.* to 'am'@'rhel01.flyfish' identified by 'Am_123' with grant option;
mysql >flush privileges;

创建RM 的 监控端 Reports Manager 的库
Reports Manager
mysql >create database cdh_rm default character set utf8; 
mysql >grant all privileges on cdh_rm.* to 'rm'@'%' identified by 'Rm_123' with grant option;
mysql >grant all privileges on cdh_rm.* to 'rm'@'rhel01.flyfish' identified by 'Rm_123' with grant option;
mysql >flush privileges;

image_1e6olqpoh16d91ajneem136pcclc8.png-219.2kB

image_1e6olr6u8kpveb61qp8kb81ar1cl.png-233.1kB

image_1e6omcuh41m2a1btf14liapv1trdep.png-387.4kB

image_1e6ome46j1ih31qt22d61ih71junf6.png-377.8kB

image_1e6omeq7v17ngcgb1t714n017svfj.png-338.5kB

image_1e6omg565acudvpbjh1qfjistg0.png-287.8kB

image_1e6ompmo3aaml9bpepc5c3fdgg.png-424.6kB

image_1e6omq9g81p771mb919k59nmbs4gt.png-206.6kB

image_1e6onmsl5pcb1nchf5b43fuj3ha.png-464.8kB