• 一: CentOS6.8x64 环境初始化
  • 二: cdh5.3.x 安装使软件配置

一: CentOS6.8x64 环境初始化

1.1 系统主机名配置

192.168.3.1     master.hadoop.com       master
192.168.3.2     slave1.hadoop.com       slave1
192.168.3.3     slave2.hadoop.com       slave2

1.2 关闭iptables 与selinux 与禁用IPv6

1. 关闭iptables 

iptables -F
service iptables save 
service iptables stop 
chkconfig iptables off 

2. 关闭selinux 
vim /etc/selinx/config

SELINUX=disabled

3. 禁用IPv6 (从新启动系统验证)
# echo "alias net-pf-10 off" >> /etc/modprobe.d/dist.conf
# echo "alias ipv6 off" >> /etc/modprobe.d/dist.conf    

1.3 做无密钥认证

2. 做hadoop用户无密钥认证

 ssh-keygen ---一直敲回车到最后

 cat id_rsa.pub >> authorized_keys 

 将所有的公钥导入authorized_keys  分发到 所有的 机器的.ssh/ 下面

 然后测试

1.4 卸载openjdk 的rpm 包

 rpm -e --nodeps xxx yyy zzz (包含jdk 的rpm 包)

1.5 安装jdk

rpm -ivh jdk-7u79-linux-x64.rpm

vim /etc/profile

增加环境变量

export JAVA_HOME=/usr/java/jdk1.7.0_79

export CLASSPATH=.:$JAVA_HOME/jre/lib:$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar

export HADOOP_HOME=/opt/cloudera/parcels/CDH

PATH=$PATH:$HOME/bin:$JAVA_HOME/bin:${HADOOP_HOME}/bin

1.6 设置进程的参数:(所有节点都配置)

设置文件打开数量和用户最大进程数
        >> 文件打开数量
        $ ulimit -a 
        >> 用户最大进程数
        $ ulimit -u
        >> 设置
        $ sudo vi /etc/security/limits.conf 
            内容:
*               soft    nofile          32728
*               hard    nofile          1029345
*               soft    nproc           unlimited
*               hard    nproc           unlimited
*               soft    memlock         unlimited
*               hard    memlock         unlimited 

image_1alk447nuokgei41f2m107qgd96p.png-6.5kB

1.7 配置时间同步服务器

1. master:
vim /etc/ntp.conf 增加
restrict 192.168.3.0 mask 255.255.255.0 nomodify notrap

vim /etc/sysconfig/ntpd
echo "SYSNC_HWCLOCK=yes" >> /etc/sysconfig/ntpd

2. slave1,slave2 同步master

crontab -l 

*/5 * * * *     /usr/sbin/ntpdate 192.168.3.1

3. 同步到blos 时间
 hwclock --localtime -w 

image_1alk468gq154h1vgq1ejouv6l1076.png-9.2kB

image_1alk46o1g1msc1v7k1qtb14buvr87j.png-7.2kB

1.8 安装mysql 数据库

1. 卸载原有数据库
rpm -qa |grep mysql 
rpm -e mysql-libs-* --nodeps 

2. 安装新版本的数据库:

rpm -ivh MySQL-server-5.6.31-1.el6.x86_64.rpm
rpm -ivh MySQL-client-5.6.31-1.el6.x86_64.rpm
rpm -ivh MySQL-devel-5.6.31-1.el6.x86_64.rpm
rpm -ivh MySQL-embedded-5.6.31-1.el6.x86_64.rpm
rpm -ivh MySQL-shared-5.6.31-1.el6.x86_64.rpm
rpm -ivh MySQL-shared-compat-5.6.31-1.el6.x86_64.rpm
rpm -ivh MySQL-test-5.6.31-1.el6.x86_64.rpm

---启动mysql---

service mysql start
mysql 的 初次使用密码在 : cat .mysql_secret 
----更改mysql 密码:
mysql -uroot -p7OR2voJyB72iT4Px

mysql> set password = password('123456');
mysql> flush privileges;

二: cdh5.3.x 安装使软件配置

2.1 安装CM 所依赖的包:(全部节点都安装)

yum -y install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb

2.2. 创建安装CM 的目录 所有节点全部配置:(三台机器全部配置)

1.mkdir /opt/cloudera-manager

上传已经下载好的CM 5.3.6 软件
cloudera-manager-el6-cm5.3.6_x86_64.tar.gz
去哪里下载:
http://archive.cloudera.com/cm5/cm/5/

2. 解压到/opt/cloudera-manager (所有节点全部配置)
tar -zxvf cloudera-manager-el6-cm5.3.6_x86_64.tar.gz -C /opt/cloudera-manager
----

3. 修改agent 的配置文件config.ini (所有节点全部更改)
cd /opt/cloudera-manager/cm-5.3.6/etc/cloudera-scm-agent
vim config.ini
server_host=master.hadoop.com

4. 创建cloudera-scm 用户用作安装(节点全部配置)
#useradd --system --home=/opt/cloudera-manager/cm-5.3.6/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

image_1alk5grthvmtpgudua15qi104k80.png-17.2kB


2.3 初始化cloudema-manager表的设置

mysql 授权用户处理:
mysql -uroot -p123456

---- 先在mysql创建用户名(temp)、密码(temp)
grant all privileges on *.* to 'temp'@'%' identified by 'temp' with grant option;

grant all privileges on *.* to 'scm'@'%' identified by 'scm' with grant option;

grant all privileges on *.* to 'temp'@'master.hadoop.com' identified by 'temp' with grant option;

flush privileges;

2.4 执行cloudera-manager创表脚本

  拷贝jdbc 到 java的目录
  tar -zxvf mysql-connector-java-5.1.42.tar.gz
  cd mysql-connector-java-5.1.42
  cp mysql-connector-java-5.1.42-bin.jar  /usr/share/java/
  cd /usr/share/java/
  mv mysql-connector-java-5.1.42-bin.jar mysql-connector-java.jar 

  脚本:scm_prepare_database.sh

 /opt/cloudera-manager/cm-5.3.6/share/cmf/schema/scm_prepare_database.sh mysql -h node01.yangyang.com -utemp -ptemp --scm-host node01.yangyang.com scm scm scm
 在mysql 中生成数据表scm 

image_1allpte1q1njml161ic71nsljf59.png-29.9kB

2.5 制作cdh 的本地源

1. cloudera-manager-server 节点:

mkdir -p /opt/cloudera/parcel-repo
chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/
mkdir -p /opt/cloudera/parcels
chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcels/

2. cloudera-manager-agent 节点:(所有节点)

mkdir -p /opt/cloudera/parcels

chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcels/

3. 上传文件
CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.parcel 与 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha1 (把这个末尾的1去掉) 文件到 /opt/cloudera/parcel-repo

mv CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha1 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha
chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/

2.6 启动CM-server与agent-server

master.hadoop.com 主机

cd /opt/cloudera-manager/cm-5.3.6/etc/init.d
./cloudera-scm-server start 

image_1alm1l70k1674rdckfn9ls8qlm.png-8.5kB

查看启动日志

查看启动结果:
        需要通过查看server的启动日志文件
        /opt/cloudera-manager/cm-5.3.6/log/cloudera-scm-server
    tail -f cloudera-scm-server.log

    看到jetty 表示启动成功

image_1alm1orrm1tn2a2e1qbf1u9l6bc13.png-18.4kB

查看端口号:
netstat -nultp 

看到 7180 表示启动成功

image_1alm1t3461kov110jslc1e5b10bk1g.png-53.4kB


2.7 打开界面进行安装CM 组建

http://master.hadoop.com:7180

用户名: admin 密码:admin

image_1alm23nb11lkn1tdq1sr21qh31djc1t.png-48.8kB

选择免费版本

image_1alm25fi0ncml3l1a3j1ql5pdb2a.png-72.2kB

下一步

image_1alm26o6q1rcn1an5je31qmdjf82n.png-78.3kB

启动master.slave1与slave2 节点的agent

master:

cd /opt/cloudera-manager/cm-5.3.6/etc/init.d
./cloudera-scm-agent start

image_1alm2rqhh18ic1lrh1qt11hn0es13u.png-10.6kB

slave1:
cd /opt/cloudera-manager/cm-5.3.6/etc/init.d
./cloudera-scm-agent start

image_1alm2svmvcpcih81mq31ostfhm4b.png-7.9kB

slave2:
cd /opt/cloudera-manager/cm-5.3.6/etc/init.d
./cloudera-scm-agent start

image_1alm2u5qd1gbhop11kr51l8o18754o.png-12.6kB

点击:管理的主机

image_1alm38t8n14ks1p6qo3f2581gln55.png-86.2kB

下一步:

image_1alobj8vd1s96rintdp142612749.png-69.5kB

下一步:

image_1aloc91ji12i1v1o1lgo1tn9jrim.png-75.4kB

下一步:

image_1alocau50f7kbij1aje18jf16o213.png-60.8kB

下一步:

image_1alod6icdfi4m7v0bkv411go1g.png-93.5kB

在三台机上面执行:
echo "0" > /proc/sys/vm/swappiness  
cat /proc/sys/vm/swappiness 
sysctl -p 
echo "never" > /sys/kernel/mm/redhat_transparent_hugepage/defrag
vim /etc/rc.d/rc.local
在最后加上:
echo "never" > /sys/kernel/mm/redhat_transparent_hugepage/defrag

---

然后从新运行处理

image_1alodiq191f5f3d1hjll75jnv1t.png-78.3kB

下一步:

image_1alodlpmjl1chm8c8o1eu81cuu2a.png-92.8kB

下一步:选择自定义安装:
安装一个hdfs

image_1aloe1426h3ufml15mb197h1ro934.png-73.6kB

分配角色

image_1aloe01t014ut4618cr7d417dm2n.png-89.2kB

下一步:

image_1aloebqc98cr1njmaj01cjjsbv3h.png-85.7kB

image_1aloeg8151c0s1je7j33797he63u.png-75kB

image_1aloeh2b3148ohip1vkm1ghs8ui4b.png-50kB

点击完成即可。
配置服务器组建,根据配置增加服务

image_1alp2qq5aj721uvs7p18p7ai39.png-136.4kB