CDH6.0.1详细安装步骤

原创

sunfect 2018-12-29 17:21:46 ©著作权

文章标签 Cloudera Manager 文章分类 Hadoop 大数据

©著作权归作者所有：来自51CTO博客作者sunfect的原创作品，请联系作者获取转载授权，否则将追究法律责任

CDH6.0.1详细安装步骤

系统环境 • 操作系统：3台 CentOS 7.3 x64 • Cloudera Manager：6.0.1 • CDH: 6.0.1

| VM Name | VM IP Adress | Centos_HostName |Role | -------- | -------- | -------- | | Centos_wjg1 | 192.168.1.101 | hadoop01 |Cloudera Manager Server + Cloudera Manager Agent | Centos_wjg2 | 192.168.1.102 | hadoop02 |Cloudera Manager Agent | Centos_wjg6 | 192.168.1.106 | hadoop06 |Cloudera Manager Agent

一、配置网络环境

编辑/etc/sysconfig/network-scripts/ifcfg-eth0配置文件，修改其中以下红色颜色标示的项目： vi /etc/sysconfig/network-scripts/ifcfg-eth0 TYPE=Ethernet BOOTPROTO=static DEFROUTE=yes PEERDNS=yes PEERROUTES=yes IPV4_FAILURE_FATAL=no IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_PEERDNS=yes IPV6_PEERROUTES=yes IPV6_FAILURE_FATAL=no NAME=eth0 UUID=3719cf03-5d05-454b-903f-e1e26ad34fd6 DEVICE=eth0 ONBOOT=yes IPADDR=192.168.1.101 NETWORK=192.168.1.1 NETSTAT=255.255.255.0 GATEWAY=192.168.1.1 DNS1=222.186.160.70 DNS2=114.114.114.114 2.编辑/etc/default/grub文件，在GRUB_CMD_LINE_LINUX=""项中，插入"net.ifnames=0 biosdevname=0"（与原有项之间用空格隔开），保存退出。 3.调用命令grub2-mkconfig -o /boot/grub2/grub.cfg更新配置，然后直接reboot重启。这样之后再用ifconfig -a命令查看时，就有eth0网卡存在了。
更改hostname hostnamectl set-hostname hadoop01
修改hosts文件 vi /etc/hosts

192.168.1.101 hadoop01 192.168.1.102 hadoop02 192.168.1.106 hadoop06 6. 关闭防火墙 sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config setenforce 0 systemctl stop firewalld.service systemctl disable firewalld.service

配置免密登录：在每台主机上生成rsa公钥和私钥 ssh-keygen -t rsa cd .ssh cat id_rsa.pub >> authorized_keys 登录其他主机，将其他主机的公钥文件内容都拷贝到hadoop01主机上的authorized_keys文件中，命令如下： [root@hadoop06 .ssh]# ssh-copy-id -i hadoop01 /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys wjg@hadoop01's password:

Number of key(s) added: 1

Now try logging into the machine, with: "ssh 'hadoop01'" and check to make sure that only the key(s) you wanted were added. 在每台主机上重复以上步骤，使任意两台主机间都能相互连通。

至此，免密码登录已经设定完成。

更改系统时间：进入home目录，修改.bash_profile #setup time region TZ='Asia/Shanghai' export TZ 或者: tzselect

将当前时间和日期写入BIOS，避免重启后失效 hwclock –w

配置NTP同步各台服务器的时间 9.1主节点配置

在配置之前，先使用ntpdate手动同步一下时间，免得本机与对时中心时间差距太大，使得ntpd不能正常同步。这里选用65.55.56.206作为对时中心,ntpdate -u ntp1.aliyun.com vi /etc/ntp.conf #Hosts on local network are less restricted. 取消下面一行的注释，并将其中IP和掩码修改为真实环境IP和掩码，此行配置为允许ntp客户端连接的配置 restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap 找到server 0.centos.pool.ntp.org iburst，并将所有server配置进行注释添加下面两行内容 server 127.127.1.0 # local clock fudge 127.127.1.0 stratum 10

启动NTP服务设置后，重启ntpd服务，用ntpstat来检查效果【大概几秒钟就可以看到效果】 #systemctl restart ntpd #ntpstat

vi /etc/sysconfig/ntpdate #Set to 'yes' to sync hw clock after successful ntpdate SYNC_HWCLOCK=yes

9.2 Agent节点配置修改/etc/ntp.conf 对该文件进行以下内容的修改： 1、注释所有restrict和server配置 2、添加下面注释，需要修改以下的IP为NTP服务器的IP（在本案例中就是server主机的IP） server 192.168.1.100

将所有主机的ntp服务设置为开机自动启动 chkconfig ntpd on systemctl enable ntpd.service

二、配置163 yum源 1）下载repo文件 cd /etc/yum.repos.d/ wget http://mirrors.163.com/.help/CentOS7-Base-163.repo

2）备份并替换系统的repo文件 mv CentOS-Base.repo CentOS-Base.repo.bak cp CentOS7-Base-163.repo CentOS-Base.repo

3）执行yum源更新命令 yum clean all yum makecache yum update

yum install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb –y

yum install httpd mod_ssl openssl-devel python-psycopg2 –y yum install MySQL-python yum install perl yum install autoconf

三、安装MySQL:

查看并卸载CentOS系统自带的MySQL和Mariadb rpm -qa | grep MySQL rpm -qa | grep mariadb

rpm -e --nodeps MySQL-python-1.2.5-1.el7.x86_64 rpm -e --nodeps mariadb-libs-5.5.60-1.el7_5.x86_64 下载并上传以下rpm包到主节点 MySQL-client-5.6.42-1.el7.x86_64.rpm MySQL-devel-5.6.42-1.el7.x86_64.rpm MySQL-server-5.6.42-1.el7.x86_64.rpm 在主节点上执行以下命令安装MySQL rpm -ivh MySQL*

安装完成后启动MySQL: service mysql start systemctl start mysql.service

修改MySQL的配置文件 vi /etc/my.cnf [mysqld] datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock transaction-isolation = READ-COMMITTED #Disabling symbolic-links is recommended to prevent assorted security risks; #to do so, uncomment this line: symbolic-links = 0 key_buffer_size = 32M max_allowed_packet = 32M thread_stack = 256K thread_cache_size = 64 query_cache_limit = 8M query_cache_size = 64M query_cache_type = 1 max_connections = 550 #expire_logs_days = 10 #max_binlog_size = 100M #log_bin should be on a disk with enough free space. #Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your #system and chown the specified folder to the mysql user. log_bin=/var/lib/mysql/mysql_binary_log #In later versions of MySQL, if you enable the binary log and do not set #a server_id, MySQL will not start. The server_id must be unique within #the replicating group. server_id=1 binlog_format = mixed read_buffer_size = 2M read_rnd_buffer_size = 16M sort_buffer_size = 8M join_buffer_size = 8M #InnoDB settings innodb_file_per_table = 1 innodb_flush_log_at_trx_commit = 2 innodb_log_buffer_size = 64M innodb_buffer_pool_size = 4G innodb_thread_concurrency = 8 innodb_flush_method = O_DIRECT innodb_log_file_size = 512M [mysqld_safe] log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid sql_mode=STRICT_ALL_TABLES

获取系统初始密码： [root@hadoop01 wjg]# cat /root/.mysql_secret #The random password set for the root user at Tue Dec 18 20:37:59 2018 (local time): pP4xR06U9nPyWrhA

登录MySQL: [root@hadoop01 wjg]# mysql -uroot -ppP4xR06U9nPyWrhA Warning: Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 2 Server version: 5.6.42

SET PASSWORD FOR 'root'@'localhost' = PASSWORD('xxxxxxx')

grant all on . to root@"%" identified by " xxxxxxx "

设置MySQL开机自启动 systemctl enable mysql.service 初始化MySQL /usr/bin/mysql_secure_installation

下载mysql-connector-java-5.1.46.tar.gz并解压在解压目录下找到mysql-connector-java.jar, 复制到所有主机的/usr/share/java目录下 cd mysql-connector-java-5.1.46 cp mysql-connector-java-5.1.46.jar /usr/share/java/mysql-connector-java.jar

[root@hadoop01 usr]# ls -l /usr/share/java/mysql-connector-java.jar -rw-r--r-- 1 root root 1004838 Dec 19 20:45 /usr/share/java/mysql-connector-java.jar

下载Oracle JDK包oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm上传到所有主机并安装下载地址：https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/RPMS/x86_64/ rpm -ivh oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm

Creating Databases for Cloudera Software 需要建的库有 scm、amon、rman、hue、metastore、sentry、nav、navms、oozie CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'scm@123'; CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'rman@123'; CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'hue@123'; CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON metastore.* TO 'metastore'@'%' IDENTIFIED BY 'metastore@123'; CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry@123'; CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'nav@123'; CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'navms@123'; CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie@123';

初始化cloudera manager数据库 /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm

四、下载并安装Cloudera Manager https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/RPMS/x86_64/

cloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpm
cloudera-manager-server-6.0.1-610811.el7.x86_64.rpm
cloudera-manager-agent-6.0.1-610811.el7.x86_64.rpm

在主节点hadoop01服务器上安装1,2,3；在Agent服务器上安装1,3；

在主节点hadoop01上执行： systemctl start cloudera-scm-server systemctl restart cloudera-scm-server

查看日志： tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log 显示如下表示服务已经正常起来 INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server. Cloudera Manager将自动下载文件到/opt/cloudera/parcel-repo目录下 CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.torrent manifest.json 在所有Agent节点hadoop01, hadoop02, hadoop06上执行：

配置Agent的Server指向 vi /etc/cloudera-scm-agent/config.ini [General] #Hostname of the CM server. server_host=hadoop01

[Security] #Use TLS and certificate validation when connecting to the CM server. use_tls=1

启动cloudera-scm-agent服务： systemctl status cloudera-scm-agent systemctl start cloudera-scm-agent systemctl restart cloudera-scm-agent

Agent Log文件位置：/var/log/cloudera-scm-agent cloudera-scm-agent.log

tail -f /var/log/cloudera-scm-agent/cloudera-scm-agent.log

登录Cloudera Manager管理界面

http://192.168.1.101:7180

用户名和密码都是admin Cloudera 安装Parcel提示主机运行状况不良解决方法删除agent目录下面的cm_guid文件，并重启失败节点的agent服务。 [root@hadoop06 ~]# cd /var/lib/cloudera-scm-agent/ [root@hadoop06 cloudera-scm-agent]# ll total 32 -rw------- 1 root root 2 Dec 28 14:27 active_parcels.json drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Dec 28 23:01 agent-cert -rw-r--r-- 1 root root 36 Dec 28 14:25 cm_guid -rw------- 1 root root 13813 Dec 29 07:10 response.avro -rw-r--r-- 1 root root 36 Dec 28 14:25 uuid [root@hadoop06 cloudera-scm-agent]# rm -rf cm_guid [root@hadoop06 cloudera-scm-agent]# systemctl restart cloudera-scm-agent

在所有节点上删除cm_guid以后，cloudera manager会自动继续进行安装。稍微等一段时间后就会看到parcel已经安装到所有主机上并全部激活。点击继续就会进入检查主机正确性的界面第1个警告：Cloudera 建议将 /proc/sys/vm/swappiness 设置为最大值 10。当前设置为 30。使用 sysctl 命令在运行时更改该设置并编辑 /etc/sysctl.conf，以在重启后保存该设置。您可以继续进行安装，但 Cloudera Manager 可能会报告您的主机由于交换而运行状况不良。以下主机将受到影响：查看详细信息 hadoop[01-02, 06] 解决办法：临时调整： sysctl vm.swappiness=10 永久调整： vi /etc/sysctl.conf 增加下面一行 vm.swappiness=10

第2个警告：已启用透明大页面压缩，可能会导致重大性能问题。请运行“echo never > /sys/kernel/mm/transparent_hugepage/defrag”和“echo never > /sys/kernel/mm/transparent_hugepage/enabled”以禁用此设置，然后将同一命令添加到 /etc/rc.local 等初始化脚本中，以便在系统重启时予以设置。以下主机将受到影响: 查看详细信息 hadoop[01-02, 06] 解决办法： echo never > /sys/kernel/mm/transparent_hugepage/defrag echo never > /sys/kernel/mm/transparent_hugepage/enabled 修改/etc/rc.local将上面两行添加进去 vi /etc/rc.local echo never > /sys/kernel/mm/transparent_hugepage/defrag echo never > /sys/kernel/mm/transparent_hugepage/enabled

第3个警告：psycopg2版本太低的处理安装pip yum -y install epel-release yum -y install python-pip pip install --upgrade pip yum install postgresql-devel

安装psycopg2 下载psycopg2-2.7.6.1.tar.gz并解压 [root@hadoop01 psycopg2-2.7.6.1]# python setup.py build [root@hadoop01 psycopg2-2.7.6.1]# python setup.py install

重新运行检查主机，所有项目检查通过点击继续进入选择安装产品界面选择所有服务然后点击继续，进入主机角色分配界面点击按主机查看对主机分配的角色会看得更清楚一些点击继续进入数据库设置界面，填写先前创建的数据库名称、用户名和密码点击继续进入审核更改界面点击继续开始进行安装安装完成后点击继续，进入Summary界面