cdh集群集成spark 3 cdh集群搭建

首先声明，这篇博文相对比较长，大约需要10 min

1、阿里云官网登录，购买ecs云服务器     www.aliyun.com/

    如果你是新用户，需要注册，之后充值100

购买ecs服务器

我们可以看到公网和内网IP，将公网IP配置笔记本上的hosts文件，地址自己去找吧，这里不说了，我配置后的结果：

这里顺便说下停止ecs服务器

 

2、配置远程连接

    我这里使用的远程连接工具是xshell,你们也可以使用crt或者MobaXterm工具

输入用户root和自己设置的密码就OK了

然后以同样的方式进行后两台的连接

至此，3台机器已能够实现远程连接。

 

3、安装包准备

1）JDK

2）MySQL

3）MySQL jdbc  jar包

4）cm tar包   

  下载地址： http://archive.cloudera.com/cm5/cm/5/

5）parcel文件  3个

   下载地址：http://archive.cloudera.com/cdh5/parcels/5.16.2/

cm和parcel官网下载会非常慢，可以考虑使用迅雷

最终结果:

上传安装包到/root/cdh5.16.2，上传前增加滴1台机器的带宽

然后上传，之后再将带宽修改为1M即可。

 

 

 

4、集群节点初始化

1）hosts文件准备（3台都需要配置）

[root@mycluster001 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

172.16.170.107    mycluster001
172.16.170.106    mycluster002
172.16.170.105    mycluster003

2）防火墙关闭

    自己的机房或者IDC托管的机器需要关闭，如果是云主机可以忽略关闭操作，但是云主机有web防火墙

systemctl stop firewalld
systemctl disable firewalld
iptables -F

3）关闭selinux,云主机是关闭状态的，如果不是disabled,可以使用下列命令进行关闭

sed -i 's#SELINUX=enable#SELINUX=disabled#g' /etc/selinux/config

4）时区/时钟同步(如果机器为云主机，下面操作可忽略)

[root@mycluster001 ~]# timedatectl
      Local time: Sat 2020-04-18 11:22:06 CST
  Universal time: Sat 2020-04-18 03:22:06 UTC
        RTC time: Sat 2020-04-18 11:22:06
       Time zone: Asia/Shanghai (CST, +0800)
     NTP enabled: yes
NTP synchronized: yes
 RTC in local TZ: yes
      DST active: n/a

ntp选择第一台节点作为ntp的主节点，其他节点为从节点

yum -y install ntp（3台都需要操作）
vi /etc/ntp.conf（只在第一台机器进行配置）
    #time
    server 0.asia.pool.ntp.org
    server 1.asia.pool.ntp.org
    server 2.asia.pool.ntp.org
    server 3.asia.pool.ntp.org

    #当外部时间不可用，可以使用当前本地硬件时间
    server 127.127.1.0 iburst local clock

    #允许哪些网段的机器来同步时间
    restrict 172.16.170.0 mask 255.255.255.0 nomodify notrap

除了001机器，其他都关闭
systemctl  stop ntpd
systemctl disable ntpd
[root@mycluster002 ~]# ntpdate mycluster001
18 Apr 11:54:58 ntpdate[2394]: adjust time server 172.16.170.107 offset -0.004965 sec
[root@mycluster003 ~]# ntpdate mycluster001
18 Apr 11:55:23 ntpdate[2383]: adjust time server 172.16.170.107 offset 0.002791 sec

在002和003机器配置定时任务
[root@mycluster002 ~]# crontab -e
no crontab for root - using an empty one

00 00 * * * ntpdate mycluster001

5）jdk tar包解压部署

（3台均需操作） mkdir /usr/java/ tar -zxvf jdk-8u45-linux-x64.gz -C /usr/java/ cd /usr/java/ [root@mycluster001 java]# ll total 4 drwxr-xr-x 8 10 143 4096 Apr 11 2015 jdk1.8.0_45 [root@mycluster001 java]# chown -R root:root jdk1.8.0_45/ [root@mycluster001 java]# ll total 4 drwxr-xr-x 8 root root 4096 Apr 11 2015 jdk1.8.0_45

坑： 上面解压后的jdk属主和属组改为root

vi /etc/profile

source /etc/profile

java -version    #查看是否安装成功

 

6）离线部署MySQL

1.解压及创建目录
[root@mycluster001 cdh5.16.2]# mv mysql-5.7.11-linux-glibc2.5-x86_64.tar.gz.zip.001 /usr/local/
[root@mycluster001 local]# tar xzvf mysql-5.7.11-linux-glibc2.5-x86_64.tar.gz
[root@mycluster001 local]# mv mysql-5.7.11-linux-glibc2.5-x86_64 mysql
[root@mycluster001 local]# mkdir mysql/arch mysql/data mysql/tmp


2.创建my.cnf
[root@mycluster001 local]# vi /etc/my.cnf
[client]
port            = 3306
socket          = /usr/local/mysql/data/mysql.sock
default-character-set=utf8mb4

[mysqld]
port            = 3306
socket          = /usr/local/mysql/data/mysql.sock

skip-slave-start

skip-external-locking
key_buffer_size = 256M
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 4M
query_cache_size= 32M
max_allowed_packet = 16M
myisam_sort_buffer_size=128M
tmp_table_size=32M

table_open_cache = 512
thread_cache_size = 8
wait_timeout = 86400
interactive_timeout = 86400
max_connections = 600

# Try number of CPU's*2 for thread_concurrency
#thread_concurrency = 32 

#isolation level and default engine 
default-storage-engine = INNODB
transaction-isolation = READ-COMMITTED

server-id  = 1739
basedir     = /usr/local/mysql
datadir     = /usr/local/mysql/data
pid-file     = /usr/local/mysql/data/hostname.pid

#open performance schema
log-warnings
sysdate-is-now

binlog_format = ROW
log_bin_trust_function_creators=1
log-error  = /usr/local/mysql/data/hostname.err
log-bin = /usr/local/mysql/arch/mysql-bin
expire_logs_days = 7

innodb_write_io_threads=16

relay-log  = /usr/local/mysql/relay_log/relay-log
relay-log-index = /usr/local/mysql/relay_log/relay-log.index
relay_log_info_file= /usr/local/mysql/relay_log/relay-log.info

log_slave_updates=1
gtid_mode=OFF
enforce_gtid_consistency=OFF

# slave
slave-parallel-type=LOGICAL_CLOCK
slave-parallel-workers=4
master_info_repository=TABLE
relay_log_info_repository=TABLE
relay_log_recovery=ON

#other logs
#general_log =1
#general_log_file  = /usr/local/mysql/data/general_log.err
#slow_query_log=1
#slow_query_log_file=/usr/local/mysql/data/slow_log.err

#for replication slave
sync_binlog = 500


#for innodb options 
innodb_data_home_dir = /usr/local/mysql/data/
innodb_data_file_path = ibdata1:1G;ibdata2:1G:autoextend

innodb_log_group_home_dir = /usr/local/mysql/arch
innodb_log_files_in_group = 4
innodb_log_file_size = 1G
innodb_log_buffer_size = 200M

#根据生产需要，调整pool size 
innodb_buffer_pool_size = 2G
#innodb_additional_mem_pool_size = 50M #deprecated in 5.6
tmpdir = /usr/local/mysql/tmp

innodb_lock_wait_timeout = 1000
#innodb_thread_concurrency = 0
innodb_flush_log_at_trx_commit = 2

innodb_locks_unsafe_for_binlog=1

#innodb io features: add for mysql5.5.8
performance_schema
innodb_read_io_threads=4
innodb-write-io-threads=4
innodb-io-capacity=200
#purge threads change default(0) to 1 for purge
innodb_purge_threads=1
innodb_use_native_aio=on

#case-sensitive file names and separate tablespace
innodb_file_per_table = 1
lower_case_table_names=1

[mysqldump]
quick
max_allowed_packet = 128M

[mysql]
no-auto-rehash
default-character-set=utf8mb4

[mysqlhotcopy]
interactive-timeout

[myisamchk]
key_buffer_size = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M

3.创建用户组及用户
[root@mycluster001 local]# groupadd -g 101 dba
[root@mycluster001 local]# useradd -u 514 -g dba -G root -d /usr/local/mysql mysqladmin
useradd: warning: the home directory already exists.
Not copying any file from skel directory into it.
根据命令操作结果的提示：
4.copy 环境变量配置文件至mysqladmin用户的home目录中,为了以下步骤配置个人环境变量
[root@mycluster001 local]# cp /etc/skel/.* /usr/local/mysql  ###important

5.配置环境变量
[root@mycluster001 local]# vi mysql/.bash_profile

# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs
export MYSQL_BASE=/usr/local/mysql
export PATH=${MYSQL_BASE}/bin:$PATH


unset USERNAME

#stty erase ^H
set umask to 022
umask 022
PS1=`uname -n`":"'$USER'":"'$PWD'":>"; export PS1

## end


6.赋权限和用户组，切换用户mysqladmin，安装
[root@mycluster001 local]# chown  mysqladmin:dba /etc/my.cnf 
[root@mycluster001 local]# chmod  640 /etc/my.cnf  
[root@mycluster001 local]# chown -R mysqladmin:dba /usr/local/mysql
[root@mycluster001 local]# chmod -R 755 /usr/local/mysql 

7.配置服务及开机自启动
[root@mycluster001 local]# cd /usr/local/mysql
[root@mycluster001 mysql]# cp support-files/mysql.server /etc/rc.d/init.d/mysql 
[root@mycluster001 mysql]#  chmod +x /etc/rc.d/init.d/mysql
[root@mycluster001 mysql]# chkconfig --add mysql
[root@mycluster001 mysql]#  chkconfig --level 345 mysql on

8.安装libaio及安装mysql的初始db
[root@mycluster001 mysql]# yum -y install libaio
[root@mycluster001 mysql]# sudo su - mysqladmin
Last login: Sat Apr 18 12:51:42 CST 2020 on pts/0
mycluster001:mysqladmin:/usr/local/mysql:>bin/mysqld \
> --defaults-file=/etc/my.cnf \
> --user=mysqladmin \
> --basedir=/usr/local/mysql/ \
> --datadir=/usr/local/mysql/data/ \
> --initialize
在初始化时如果加上 –initial-insecure，则会创建空密码的 root@localhost 账号，否则会创建带密码的 root@localhost 账号，密码直接写在 log-error 日志文件中
（在5.6版本中是放在 ~/.mysql_secret 文件里，更加隐蔽，不熟悉的话可能会无所适从）

9.查看临时密码
mycluster001:mysqladmin:/usr/local/mysql:>cd data/
mycluster001:mysqladmin:/usr/local/mysql/data:>cat hostname.err |grep password 
2020-04-18T05:01:49.024813Z 1 [Note] A temporary password is generated for root@localhost: (Oz5geZ31RaX

10.启动
mycluster001:mysqladmin:/usr/local/mysql/data:>/usr/local/mysql/bin/mysqld_safe --defaults-file=/etc/my.cnf &

11.登录及修改用户密码
mycluster001:mysqladmin:/usr/local/mysql/data:>mysql -uroot -p'Oz5geZ31RaX'
mysql> alter user root@localhost identified by '000000';
Query OK, 0 rows affected (0.00 sec)
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '000000' ;
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql>  flush privileges;
Query OK, 0 rows affected (0.00 sec)
mysql> exit
Bye

12.重启
mycluster001:mysqladmin:/usr/local/mysql/data:>service mysql restart
Shutting down MySQL..2020-04-18T05:08:53.985098Z mysqld_safe mysqld from pid file /usr/local/mysql/data/hostname.pid ended
[  OK  ]
Starting MySQL.[  OK  ]
[1]+  Done                    /usr/local/mysql/bin/mysqld_safe --defaults-file=/etc/my.cnf
mycluster001:mysqladmin:/usr/local/mysql/data:>mysql -uroot -p000000
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.11-log MySQL Community Server (GPL)

Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

 

7）添加database并授权

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
4 rows in set (0.00 sec)

mysql> create database cmf default character set utf8;
Query OK, 1 row affected (0.00 sec)

mysql> grant all privileges on cmf.* to cmf@'%' identified by '000000';
Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> 
mysql> create database amon default character set utf8;
Query OK, 1 row affected (0.00 sec)

mysql> grant all privileges on amon.* to amon@'%' identified by '000000';
Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> 
mysql> flush privileges
    -> ;
Query OK, 0 rows affected (0.00 sec)
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| amon               |
| cmf                |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
6 rows in set (0.00 sec)

 

8)移动并重命名jdbc连接包

[root@mycluster001 cdh5.16.2]# ls
CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel           manifest.json
CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel.sha1      mysql-connector-java-5.1.47.jar
cloudera-manager-centos7-cm5.16.2_x86_64.tar.gz
[root@mycluster001 cdh5.16.2]# mkdir -p  /usr/share/java
[root@mycluster001 cdh5.16.2]# mv mysql-connector-java-5.1.47.jar /usr/share/java/mysql-connector-java.jar
[root@mycluster001 cdh5.16.2]# ll /usr/share/java/ |grep mysql-con
-rw-r--r-- 1 root root 1007502 Mar  2 21:13 mysql-connector-java.jar