CDH集群的spark 搭建cdh集群需要多少内存

转载

JAVA小侠影 2024-03-13 21:32:49

文章标签 CDH集群的spark mysql java centos 文章分类 Spark 大数据

1、主机准备

这里以三节点为例；

准备三台环境纯净的centos7主机，要可以正常连接yum源, root用户操作；

随便安装一个包，测试一下yum:

yum -y install lrzsz

主机情况：2核 4G，硬盘40G；这个配置其实太小，单节点内存尽量要给到8G以上比较好；

主机资源看个人情况，我这里是因为我的电脑一共才16G内存，用的VMware虚拟机；

2、更改三台主机的hostname，

这里分别为node1、node2、node3

写入三台主机的/etc/hosts :

192.168.3.31 node1

192.168.3.32 node2

192.168.3.33 node3

3、关闭防火墙、selinux;

4、免密登录

node1可以免密登录到集群的三台台主机上；其实不做也可以；

这里是为了方便操作；而且也可以统一执行命令；

(1)三台节点都执行：

vim /etc/ssh/sshd_config   #修改以下两行
PermitRootLogin yes                         
PubkeyAuthentication yes
 
systemctl restart sshd    #重启sshd
 
ssh-keygen -t rsa -P ""       #三台主机都生成密钥对，一路回车即可

(2)node1执行：

ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.3.31
ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.3.32
ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.3.33

在node1上测试,是否还需要密码：

ssh node1

ssh node2

ssh node3

5、远程执行命令的脚本

此脚本在node1上运行，可同时操作node1 node2 node3;

[root@node1 ~]# cat rcmd.sh
#!/bin/bash
CMD1="$1"
 
echo -e "\033[34m---------node1-----------\033[0m"
ssh node1 "${CMD1}"
echo -e "\033[34m---------node2-----------\033[0m"
ssh node2 "${CMD1}"
echo -e "\033[34m---------node3-----------\033[0m"
ssh node3 "${CMD1}"

6、主机时钟同步

(1)node1执行：

[root@node1 ~]# sh rcmd.sh "yum -y remove chrony"    #这里已经开始使用rcmd.sh脚本批量操作了
[root@node1 ~]# sh rcmd.sh "yum -y install ntp"
 
vim /etc/ntp.conf
#注释掉下面四行
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst
 
#添加下面两行，这两行是让node1和node1自己做时间同步
server  127.127.1.0     # local clock
fudge   127.127.1.0 stratum 10

(2)node2、node3执行(node2和node3去与node1同步):

vim /etc/ntp.conf
#注释掉下面四行
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst
 
#添加一行
server 192.168.3.31

(3)所有主机重启ntp服务并开始时间同步：

[root@node1 ~]# sh rcmd.sh "systemctl restart ntpd"
[root@node1 ~]# sh rcmd.sh "ntpq -p"    #执行同步

时间同步说明：这里是自己搭建的测试集群，如果是生产中，应该会有专门的时间服务器，使用之即可；

7、设置swap、设置透明大页，下列echo命令可以写进开机自启动文件

#设置swap
[root@node1 ~]# sh rcmd.sh "echo vm.swappiness = 10 >> /etc/sysctl.conf"
[root@node1 ~]# sh rcmd.sh "sysctl vm.swappiness=10"
 
#设置透明大页
[root@node1 ~]# sh rcmd.sh "echo never > /sys/kernel/mm/transparent_hugepage/defrag"
[root@node1 ~]# sh rcmd.sh "echo never > /sys/kernel/mm/transparent_hugepage/enabled"

8、所有节点安装依赖

[root@node1 ~]# sh rcmd.sh "yum install -y bind-utils libxslt cyrus-sasl-plain cyrus-sasl-gssapi portmap"
[root@node1 ~]# sh rcmd.sh "yum install -y fuse-libs /lib/lsb/init-functions httpd mod_ssl"
[root@node1 ~]# sh rcmd.sh "yum install -y openssl-devel python-psycopg2 Mysql-python fuse"

9、安装http服务，用于提供CDH的yum源服务

(1)node1执行(我这里选择安装到node1上)：

[root@node1 ~]# yum -y install httpd
[root@node1 ~]# systemctl  start|stop|restart  httpd
 
[root@node1 ~]# cd /var/www/html
[root@node1 ~]# mkdir -p CDH5.16.2/{CDH,CM5}

把CDH5.16.2目录下的文件全都上传到/var/www/html/CDH5.16.2/CDH/

把CM5.16.2目录下的文件全都上传到/var/www/html/CDH5.16.2/CM5/

#安装createrepo命令，然后进入到CM5目录创建yum源
[root@node1 ~]# yum install -y createrepo
[root@node1 ~]# cd /var/www/html/CDH5.16.2/CM5
[root@node1 ~]# createrepo .     #创建yum源的描述meta

CDH集群的spark 搭建cdh集群需要多少内存_java

配置yum源文件，内容如下：

[root@node1 ~]# cat /etc/yum.repos.d/cloudera-manager.repo
[cloudera-manager]
name = Cloudera Manager, Version 5.16.2
baseurl = http://192.168.3.31/CDH5.16.2/CM5
enabled = 1
gpgcheck = 0
 
[root@node1 ~]# systemctl  start  httpd       #启动httpd
[root@node1 ~]# yum clean all
[root@node1 ~]# yum repolist      #查看可用源

CDH集群的spark 搭建cdh集群需要多少内存_java_02

在浏览器中，同样可以查看：

http://192.168.3.31/CDH5.16.2

CDH集群的spark 搭建cdh集群需要多少内存_mysql_03

10、node1安装mysql，用于存储一些元数据

说明：这里因为是测试集群，自己安装了mysql，生产中应该会有专门的MySQL来用，最好是高可用的mysql，

(1)#先看一下是否已经安装了mysql或者mariadb

[root@node1 ~]# rpm -qa |grep mysql
[root@node1 ~]# rpm -qa |grep mariadb  #发现有一个mariadb包
mariadb-libs-5.5.60-1.el7_5.x86_64

#把mariadb包卸载掉，若有多个，要全部卸载
[root@node1 ~]# rpm -e --nodeps mariadb-libs-5.5.60-1.el7_5.x86_64
 
#上传mysql包
[root@node1 mysql-pkg]# mkdir /usr/local/src/mysql-pkg && cd /usr/local/src/mysql-pkg

CDH集群的spark 搭建cdh集群需要多少内存_java_04

#安装命令
rpm -ivh mysql-community-common-5.7.17-1.el7.x86_64.rpm
rpm -ivh mysql-community-libs-5.7.17-1.el7.x86_64.rpm
rpm -ivh mysql-community-client-5.7.17-1.el7.x86_64.rpm
rpm -ivh mysql-community-devel-5.7.17-1.el7.x86_64.rpm
rpm -ivh mysql-community-libs-compat-5.7.17-1.el7.x86_64.rpm
rpm -ivh mysql-community-server-5.7.17-1.el7.x86_64.rpm

(2)#初始化mysql

[root@node1 mysql-pkg]# mysqld --initialize --user=mysql
 
#查看mysql初始密码,在文件末尾
tail -n 3 /var/log/mysqld.log
 
#启动MySQL
systemctl start mysqld
systemctl enable mysqld
echo "systemctl start mysqld" >> /etc/rc.local
 
#mysql设置
mysql -u root -p
mysql> set password=password('123456');             #进去以后改root密码
mysql> grant all privileges on *.* to root@'%' identified by '123456';  #解决客户端root用户远程连接服务器的问题
mysql> grant all privileges on *.* to 'root'@'node1' identified by '123456' with grant option;  #解决root权限访问所有库的问题
mysql> flush privileges;

(3)#创建集群所需要的库，在mysql中创建，SQL如下：

create database metastore default character set utf8; 
CREATE USER 'hive'@'%' IDENTIFIED BY '123456';  
GRANT ALL PRIVILEGES ON metastore. * TO 'hive'@'%';  
FLUSH PRIVILEGES; 
 
create database cm default character set utf8; 
CREATE USER 'cm'@'%' IDENTIFIED BY '123456';  
GRANT ALL PRIVILEGES ON cm. * TO 'cm'@'%';  
FLUSH PRIVILEGES; 
 
create database am default character set utf8;  
CREATE USER 'am'@'%' IDENTIFIED BY '123456';   
GRANT ALL PRIVILEGES ON am. * TO 'am'@'%';   
FLUSH PRIVILEGES;     
 
create database rm default character set utf8;  
CREATE USER 'rm'@'%' IDENTIFIED BY '123456';   
GRANT ALL PRIVILEGES ON rm. * TO 'rm'@'%';   
FLUSH PRIVILEGES;
 
create database hue default character set utf8;  
CREATE USER 'hue'@'%' IDENTIFIED BY '123456';   
GRANT ALL PRIVILEGES ON hue. * TO 'hue'@'%';   
FLUSH PRIVILEGES;
 
create database oozie default character set utf8;  
CREATE USER 'oozie'@'%' IDENTIFIED BY '123456';   
GRANT ALL PRIVILEGES ON oozie. * TO 'oozie'@'%';   
FLUSH PRIVILEGES;
 
create database sentry default character set utf8;  
CREATE USER 'sentry'@'%' IDENTIFIED BY '123456';   
GRANT ALL PRIVILEGES ON sentry. * TO 'sentry'@'%';   
FLUSH PRIVILEGES;
 
create database nav_ms default character set utf8;  
CREATE USER 'nav_ms'@'%' IDENTIFIED BY '123456';   
GRANT ALL PRIVILEGES ON nav_ms. * TO 'nav_ms'@'%';   
FLUSH PRIVILEGES;
 
create database nav_as default character set utf8; 
CREATE USER 'nav_as'@'%' IDENTIFIED BY '123456';  
GRANT ALL PRIVILEGES ON nav_as. * TO 'nav_as'@'%';  
FLUSH PRIVILEGES;

(4)#node1安装jdbc驱动

[root@node1 ~]# mkdir -p /usr/share/java/
[root@node1 ~]# ll mysql-connector-java-5.1.34.jar
-rw-r--r--. 1 root root 960372 11月 27 13:13 mysql-connector-java-5.1.34.jar
 
[root@node1 ~]# mv mysql-connector-java-5.1.34.jar /usr/share/java/
[root@node1 ~]# cd /usr/share/java
[root@node1 java]# ln -s mysql-connector-java-5.1.34.jar mysql-connector-java.jar

mysql-connector-java各种版本下载地址：

地址：http://mvnrepository.com/artifact/mysql/mysql-connector-java

选择对应版本号-->点击jar进行下载：

CDH集群的spark 搭建cdh集群需要多少内存_mysql_05

11、安装JDK

(1)node1解压、安装

[root@node1 ~]# cd /usr/local/src/

CDH集群的spark 搭建cdh集群需要多少内存_CDH集群的spark_06

[root@node1 src]# tar zxf jdk-8u171-linux-x64.tar.gz -C /usr/local/

CDH集群的spark 搭建cdh集群需要多少内存_CDH集群的spark_07

(2)#node1添加JDK环境变量

[root@node1 src]# vim /etc/profile   #写入以下内容
#JDK1.8
export JAVA_HOME="/usr/local/jdk1.8.0_171"
export PATH=${PATH}:${JAVA_HOME}/bin
 
[root@node1 src]# source /etc/profile
[root@node1 src]# which java
/usr/local/jdk1.8.0_171/bin/java

(2)#node1上将目录/usr/local/jdk1.8.0_171 和/etc/profile 远程scp到node2和node3

[root@node1 src]# scp -pr /usr/local/jdk1.8.0_171 root@node2:/usr/local/
[root@node1 src]# scp -pr /usr/local/jdk1.8.0_171 root@node3:/usr/local/
[root@node1 src]# scp -pr /etc/profile root@node2:/etc/profile
[root@node1 src]# scp -pr /etc/profile root@node3:/etc/profile
 
[root@node1 ~]# sh rcmd.sh "source /etc/profile"   #source

(4)#程序默认会去使用/usr/java下的jdk，所以解决办法执行以下两条命令即可：

[root@node1 ~]# sh rcmd.sh "mkdir -p /usr/java"
[root@node1 ~]# sh rcmd.sh "ln -s /usr/local/jdk1.8.0_171 /usr/java/default"

CDH集群的spark 搭建cdh集群需要多少内存_java_08

前面的一系列工作，如果都没问题...

接下来开始正式安装CDH...

12、先安装cloudera manager

(1)#node1上安装cloudera manager

[root@node1 ~]# yum -y install cloudera-manager-server

(2)#初始化数据库，

[root@node1 ~]# /usr/share/cmf/schema/scm_prepare_database.sh mysql cm cm 123456

CDH集群的spark 搭建cdh集群需要多少内存_mysql_09

说明：cm cm 123456 分别指：库名用户名密码 #就是上面安装MySQL时创建的

(3)#启动cloudera manager

[root@node1 ~]# systemctl start cloudera-scm-server
[root@node1 ~]# systemctl enable cloudera-scm-server 
 
#7180是cloudera manager的端口，启动成功, 有时候需要稍等一下，端口才会出现
[root@node1 ~]# netstat -ntlp |grep 7180
tcp     0    0 0.0.0.0:7180      0.0.0.0:*      LISTEN   46014/java

13、通过浏览器操作接下来的集群安装工作

http://192.168.3.31:7180

默认账号、密码：admin admin

CDH集群的spark 搭建cdh集群需要多少内存_centos_10