大数据平台HDP-2.6.4 安装 与配置

标签(空格分隔): 大数据平台构建


  • 一: 关于HDP的介绍
  • 二: 系统环境介绍
  • 三: 系统环境初始化
  • 四: 安装HDP-2.6.4平台
  • 五: 启用NameNode HA 与 YARN HA

一:关于HDP的介绍

1.1:hdp 是什么

HDP全称叫做Hortonworks Data Platform。

  Hortonworks数据平台是一款基于Apache Hadoop的是开源数据平台,提供大数据云存储,大数据处理和分析等服务。该平台是专门用来应对多来源和多格式的数据,并使其处理起来能变成简单、更有成本效益。HDP还提供了一个开放,稳定和高度可扩展的平台,使得更容易地集成Apache Hadoop的数据流业务与现有的数据架构。该平台包括各种的Apache Hadoop项目以及Hadoop分布式文件系统(HDFS)、MapReduce、Pig、Hive、HBase、Zookeeper和其他各种组件,使Hadoop的平台更易于管理,更加具有开放性以及可扩展性。

  官网地址为:http://zh.hortonworks.com/

1.2 hdp 的封装的大数据平台组件

image_1ce8kudcqlq91sqph5e18156b10k.png-219.2kB

二: 系统环境介绍

2.1:系统主机名

系统: RHEL7.5 X64
---
172.17.100.11   ZY-001.flyfish
172.17.100.12   ZY-002.flyfish
172.17.100.13   ZY-003.flyfish
---

2.2: 安装整体步骤

1.集群服务器配置,包括安装操作系统、关闭防火墙、同步服务器时钟等; 
2.外部数据库安装 
3.安装Ambari管理器; 
4.安装HDP集群; 
5.集群完整性检查,包括 HDFS 文件系统、MapReduce、Hive 等是否可以正常运行。

三:系统环境初始化

3.1:无密钥登录配置

做root用户无密钥认证

 ssh-keygen ---一直敲回车到最后

 cat id_rsa.pub >> authorized_keys 

 将所有的公钥导入authorized_keys  分发到 所有的 机器的.ssh/ 下面

 然后测试

3.2: 关闭 firewalld,iptables 与 selinux

 yum install -y firewall* iptable* 

  (1) 关闭firewalld

 systemctl stop firewalld.service
 systemctl disable firewalld.service
 firewall-cmd --state

 (2) 关闭iptables 
 iptables -F
 systemctl stop iptables.service 
 service iptables save
 systemctl disable iptables.service 

 (3) 禁用 selinux 

  sed -i '/SELINUX/s/enforcing/disabled/' /etc/selinux/config  
  setenforce 0
  getenforce 0
  sestatus

  重启机器

  reboot

image_1ce817k4j1tch11s81tjc1lgu1fos9.png-107.7kB

3.3 所有机器更改RHEL7.5x64的文件句柄数

vim /etc/security/limits.conf
---
*               soft    nofile          65535
*               hard    nofile          1029345
*               soft    nproc           unlimited
*               hard    nproc           unlimited
*               soft    memlock         unlimited
*               hard    memlock         unlimited
---

image_1ce15l67nua61gl5a5h5sr14eb13.png-163.2kB

image_1ce15ms34s8s1cc01akb1g7sid13g.png-102.3kB

3.4 生产环境配置chrony 时间同步服务器

所有机器 yum install chrony*
1. ZY-001.flyfish:

vim /etc/chrony.conf 增加
----
注释掉相关默认的server
server ntp1.aliyun.com iburst

allow 172.17.100.0.0/16

service chronyd start
chkconfig chronyd on 

2. ZY-002.flyfish---ZY-003.flyfish 同步zy-001.flyfish

vim /etc/chrony.conf
----
注释掉默认的server
server 172.17.100.21 iburst
----

service chronyd stop 
service chronyd start 
chkconfig chronyd on 

image_1ce84a0mc1geu12391jn11ll317k8i.png-177.4kB

image_1ce8der7m1n041rcq1dp01tsb1m48pl.png-141.1kB

image_1ce8dg4uetd9ecgf1vb771j22qi.png-184.5kB

image_1ce8dgq4415i813ahjkkojpussqv.png-123.9kB


3.5: hdp 所需软件下载

ambari-2.6.1.5-centos7.tar.gz 

下载地址:
http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.1.5/ambari-2.6.1.5-centos7.tar.gz

HDP-2.6.4.0-centos7-rpm.tar.gz  
下载地址:
http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.3.0/HDP-2.6.4.0-centos7-rpm.tar.gz

HDP-UTILS-1.1.0.22-centos7.tar.gz

下载地址:

http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz

关于 ambari-2.6.1的docs

https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.1.5/bk_ambari-installation/content/hdp_26_repositories.html

3.6 安装jdk1.8.162

tar -zxvf jdk-8u162-linux-x64.tar.gz 

mkdir /training

mv jdk1.8.0_162 /training
cd /training 
ln -s jdk1.8.0_162 jdk
vim /etc/profile 
---
export JAVA_HOME=/training/jdk
export CLASSPATH=.:$JAVA_HOME/jre/lib:$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin
---
source /etc/profile 

java -version

image_1ce821tht1hojhl115fi1sungm.png-62.1kB

四: 安装 hdp2.6.4

4.1 : 配置ambari所需安装源

安装httpd 

yum install -y httpd

service httpd start

chkconfig httpd on

cd /var/www/html/

mkdir ambari 

1. ambari 的 yum源

cd /root/hdp
tar -zxvf ambari-2.6.1.5-centos7.tar.gz
mv /root/hdp/ambari /var/www/html/ambari

image_1ce829dkp169prej36h1hg4ttb1g.png-184kB

image_1ce82g7fmsmd11jns5n17d016e21t.png-212.8kB

cd /var/www/html/ambari/ambari/centos7/2.6.1.5-3
cp -p ambari.repo /etc/yum.repos.d/

cd /etc/yum.repos.d

vim ambari.repo

----
#VERSION_NUMBER=2.6.1.5-3
[ambari-2.6.1.5]
name=ambari Version - ambari-2.6.1.5
baseurl=http://172.17.100.21/ambari/ambari/centos7/2.6.1.5-3
gpgcheck=1
gpgkey=http://172.17.100.21/ambari/ambari/centos7/2.6.1.5-3/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
----

image_1ce83o0hc133t2lr1p801br185e2n.png-86.9kB

4.2 配置hdp和hdp-utils 的源

cd /root/hdp

tar -zxvf HDP-2.6.4.0-centos7-rpm.tar.gz

mv HDP /var/www/html/ambari

cd /var/www/html/ambari/HDP/centos7/2.6.4.0-91/

cp -p hdp.repo /etc/yum.repos.d/

cd /root/hdp

mkdir HDP-UTILS

tar -zxvf HDP-UTILS-1.1.0.22-centos7.tar.gz -C HDP-UTILS

mv HDP-UTILS /var/www/html/ambari/ 

cd /etc/yum.repos.d/

vim hdp.repo

---
#VERSION_NUMBER=2.6.4.0-91
[HDP-2.6.4.0]
name=HDP Version - HDP-2.6.4.0
baseurl=http://172.17.100.21/ambari/HDP/centos7/2.6.4.0-91/
gpgcheck=1
gpgkey=http://172.17.100.21/ambari/HDP/centos7/2.6.4.0-91/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=http://172.17.100.21/ambari/HDP-UTILS/
gpgcheck=1
gpgkey=http://172.17.100.21/ambari/HDP-UTILS/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
---

tar -zxvf HDP-GPL-2.6.4.0-centos7-rpm.tar.gz

mv HDP-GPL /var/www/html/ambari

cd /var/www/html/ambari/HDP-GPL/

mv hdp-gpl.repo /etc/yum.repos.d/

cd /etc/yum.repos.d/

vim hdp-gpl.repo

---
#VERSION_NUMBER=2.6.4.0-91
[HDP-GPL-2.6.4.0]
name=HDP-GPL Version - HDP-GPL-2.6.4.0
baseurl=http://172.17.100.21/ambari/HDP-GPL/centos7/2.6.4.0-91/
gpgcheck=1
gpgkey=http://172.17.100.21/ambari/HDP-GPL/centos7/2.6.4.0-91/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
---

cd /etc/yum.repos.d/
scp ambari.repo hdp.repo hdp-gpl.repo zy-002.flyfish:/etc/yum.repos.d/
scp ambari.repo hdp.repo hdp-gpl.repo zy-003.flyfish:/etc/yum.repos.d/

yum clean all

yum makecache 

image_1ce83ubq9178717rcug94dq16ru34.png-203kB

image_1ce83uuc0q8evjd111a1bq2vlo3h.png-43.6kB

image_1ce840tb88e91bu8g9rmdloh54b.png-42.6kB

image_1ce840dub1hpq18cvqce18f8p983u.png-225.8kB

image_1ce8427j312ld8es1mle1mk11b055o.png-149.2kB

image_1ce8ak32i1uf8mb1cjf7i91pu58.png-98.8kB

image_1ce8alkp9pia1emo1er110s211um5l.png-285.8kB


4.3 安装mariadb-server 数据库

yum install -y mariadb-server 

systemctl start mariadb

systemctl enable mariadb

mysql_secure_installation

image_1ce84rv73nbl1bem15gnfh2rikc3.png-247.6kB

image_1ce84sn7p1gdsb5mbs11etr1mbacg.png-178.5kB

image_1ce84tb631ve8fj4r692vm11d2ct.png-167.2kB

image_1ce84u1g619su11c5cdh1p69163nda.png-171.1kB

mysql -uroot -pflyfish225.com

ambari 数据库:

create database ambari character set utf8 ;  
CREATE USER 'ambari'@'%'IDENTIFIED BY 'Ambari-123';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%';
FLUSH PRIVILEGES;

hive 数据库:

create database hive character set utf8 ;  
CREATE USER 'hive'@'%'IDENTIFIED BY 'Hive-123';
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%';
FLUSH PRIVILEGES;

oozie 数据库:

create database oozie character set utf8 ;  
CREATE USER 'oozie'@'%'IDENTIFIED BY 'Oozie-123';
GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'%';
FLUSH PRIVILEGES;

image_1ce84vbgsa3b23612nia5oetndn.png-89.3kB

image_1ce84vutc1j8u1n2713qu12ntl4le4.png-93.9kB

image_1ce850j1m4n14dpct7up2rkf1.png-99.2kB

4.3 安装jdbc

  拷贝jdbc 到 java的目录
  tar -zxvf mysql-connector-java-5.1.44.tar.gz
  cd mysql-connector-java-5.1.44
  cp mysql-connector-java-5.1.44-bin.jar  /usr/share/java/
  cd /usr/share/java/
  mv mysql-connector-java-5.1.44-bin.jar mysql-connector-java.jar 
  chmod 777 mysql-connector-java.jar

image_1ce85cejb15321r8h1skdjhri43fe.png-151.1kB

image_1ce85d9t1enp12qn9181k3iqfr.png-141kB

4.4 安装ambari

yum install -y ambari-server 

image_1ce86i0tumqo135l3771mvnm0sgo.png-184.8kB

vim /etc/ambari-server/conf/ambari.properties

----
增加:
  server.jdbc.driver.path=/usr/share/java/mysql-connector-java.jar

---

ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar

image_1ce8o35idd1k1oke10edm6no45111.png-276.1kB

image_1ce8pq7r91u7itpk1uq2qclo9511e.png-107.3kB

ambari-server setup 

image_1ce890ved1dalvjv13721fim1eh313.png-181kB

image_1ce898j33cjqe7i1cevtrss1n1t.png-255.4kB

mysql -uroot -pflyfish225.com
use ambari
source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql

image_1ce875m9g1f4q1129a1t2i1bg1hv.png-138.9kB

4.5 启动ambari

ambari-server start

image_1ce88tt8a10ksija6br1aaiulb9.png-123.8kB

image_1ce88ugqk1cog11ppa1fgr61p71m.png-38.9kB

4.6 打开ambari的web

http://172.17.100.21:8080

用户名:admin
密码: admin

image_1ce89a8mp1fjj1i9d1stq4tipca2a.png-139.5kB

image_1ce89b09f10oofhhgovojesdr2n.png-256.8kB

image_1ce89c95h1e7r13b91b0116u3ros34.png-223.6kB

image_1ce89d64bir81u1e14nf1r7105e41.png-277.1kB

image_1ce8aakg21n6m1267r231v3a1le14e.png-152.7kB

image_1ce8adhpb1gqt11nhndr13bqg1l4r.png-284.8kB

image_1ce8amcuo46u1tsn1cc51qa21fd062.png-305kB

这里需要注意,主机注册成功后会对主机进行前置检查,如果有警告,需要进行检查并按要求进行相应的修改。

4.7 安装相关组件

image_1ce8csf5h4vo11sbh9ocqq1ftmkd.png-351.5kB

image_1ce8csrumpva19gn2n61g45o4ikq.png-170.9kB

image_1ce8ct8hh9h7jmogdtn7uqnl7.png-185kB

image_1ce8ctp9skn31s721aimfd7lkvlk.png-387.1kB

image_1ce8cuvmbede13bmoo01c7aohmm1.png-341.1kB

image_1ce8d16kl18tcueidthtch100dmr.png-210.1kB

image_1ce8d0ir616br16cf2173eiu8me.png-262.2kB

image_1ce8d20ev14ij1krjtda12t511djn8.png-367kB

报错:
请求 libtirpc-lib 

image_1ce8casrddl0n95cnd1ibl1ollij.png-244.4kB

wget http://mirror.centos.org/centos/7/os/x86_64/Packages/libtirpc-0.2.4-0.10.el7.x86_64.rpm

wget http://mirror.centos.org/centos/7/os/x86_64/Packages/libtirpc-devel-0.2.4-0.10.el7.x86_64.rpm

rpm -ivh libtirpc-* 

image_1ce8cc6tkg33essllv143n1urcj0.png-261.1kB

Retry 

image_1ce8j21gahm5sct6asb6a10e5rc.png-309.4kB

image_1ce8j2pcd1a4j1i57lkf1i8j1tj7s9.png-289.3kB

image_1ce8j7424h1h1irh1adu6sr1ctqtm.png-457.9kB

可以按照前面的步骤,增加其它组件,hive/oozie/pig/sqoop/hbase/spark/zeppelin

image_1ce8rpjfd1kea1aeoh4712ls5t2128.png-490.4kB

4.8 执行一个wordcount

su - hdfs 

hdfs dfs -mkdir /input 

vim wc.input 
----
zhangyy hadoop
hadoop  spark
yarn    spark
----

hdfs dfs -put wc.input /input 

hadoop jar /usr/hdp/2.6.4.0-91/hadoop-mapreduce/hadoop-mapreduce-examples.jar wordcount  /input/ /output

hdfs dfs -get /output

image_1ce8kbcv216b6g6r1qid4qrssfu3.png-81.2kB

image_1ce8kc55l9ju1lp110l106vvh7ug.png-111.2kB

image_1ce8kdcd61k1j1cdq1ed01ugi1m9qut.png-272.2kB

image_1ce8kgtfus60dgl6gbgcr16e3107.png-188kB

image_1ce8kevd01r9p1jpd1mo6g01lgva.png-151.9kB

五:启用NameNode HA 与 YARN HA

5.1 开启HDFS 的 HA 策略

image_1ce8runlnervqsai0o19v575912l.png-531.5kB

image_1ce8s0civ16kj7p91h14rjnvc6132.png-225.6kB

先停掉hbase 

image_1ce8s4ccu1vj31n5cpii17vih213f.png-457.3kB

image_1ce8s852fo7p9qo4jus7n69o13s.png-252.7kB

image_1ce8s9e351srv1ijssljsku1mev159.png-300.3kB

image_1ce8sa6cnjgn1vnr13e310fg170v15m.png-241.2kB

image_1ce8samtcuso1h2fn309q2u6163.png-167.1kB

image_1ce8sdqbr1kmtpse138617ruqau16g.png-262.5kB

sudo su hdfs -l -c 'hdfs dfsadmin -safemode enter'

sudo su hdfs -l -c 'hdfs dfsadmin -saveNamespace'

image_1ce8sevb81ooa1fq5j0s14561plh16t.png-53.8kB

image_1ce8sfvmrisf7egn91jae1p2p17a.png-258.6kB

image_1ce8snqbd1bg8l1qf6dha11nol17n.png-258.4kB

sudo su hdfs -l -c 'hdfs namenode -initializeSharedEdits'

image_1ce8spd6h8ve1i6cnvngue147a184.png-280.3kB

image_1ce8sr3q4s4ann21c0h1belpj318h.png-258.9kB

image_1ce8sst781ds4vr819sq12d71vet18u.png-244.8kB

image_1ce8stmqh116hms81rr8puu15q119b.png-250.7kB

ZY-001.flyfish:

   sudo su hdfs -l -c 'hdfs zkfc -formatZK'

ZY-002.flyfish

   sudo su hdfs -l -c 'hdfs namenode -bootstrapStandby'

image_1ce8t193hld11p0j141514oi12ui19o.png-158.6kB

image_1ce8t2e091qlh18c614kjmcq1h3u1a5.png-262.3kB

image_1ce8t32veeak1lh5u0p1at71kpp1ai.png-238kB

image_1ce8t3nll1s871herrmn186hnic1av.png-292.3kB

image_1ce8u7lmp10fs1ir61d031adcpur1bc.png-282.9kB

image_1ce8u8unl18423ge2hd1c9knp41bp.png-479.9kB

5.2 启用 YARN 的HA

image_1ce8ujrsclkg1nbu1vj1loa1blm1c6.png-486.2kB

image_1ce8ukk6c13v91ugf1ujmrfdjlt1cj.png-339.6kB

image_1ce8ulhgkubof6i1bt7d3t1r081d0.png-348.4kB

image_1ce8um4pkhdc6i9l481k0olvu1dd.png-290.5kB

image_1ce8umlri1u1f3mg6ms4ltia1dq.png-256.4kB

image_1ce901bgf1kr61lbv1tp7nfc9r71e7.png-313.2kB

image_1ce90354117b9nrb1ctl5jk13q91ek.png-461.3kB

image_1ce907vgf1hv9op4bm1dj7v1f1.png-413.1kB

image_1ce90ag0dibp1mf1vqhn9duit1fe.png-475kB