hadoop 集群架构
1.启动3个linux服务器
IP
192.168.108.135 192.168.108.136 192.168.108.137
主机名
www.hadoop1.org www.hadoop2.org www.hadoop3.org
硬件信息
2G 内存 1CPU 1G 内存 1CPU 1G 内存 1CPU
HDFS:
NameNode
DataNode DataNode DataNode
SecondaryNameNode
YARN:
ResourceManager
NodeManager NodeManager NodeManager
MapReduce:
JobHistoryServer
2.配置映射(假设修改IP不用全部改变 只需修改映射)
/etc/hosts
192.168.108.135 www.hadoop1.org www.hadoop1
192.168.108.136 www.hadoop2.org www.hadoop2
192.168.108.137 www.hadoop3.org www.hadoop3
测试能否ping通
ping www.hadoop1.org
3.配置JAVA环境
可以参考
http://12275610.blog.51cto.com/12265610/1917942
4. 配置hadoop
hdfs
hadoop-env.sh
修改文件的JAVA_HOME环境变量
export JAVA_HOME=/opt/app/jdk1.7.0_79
core-site.xml
<configuration>
<!--指定NameNode地址 -->
<property>
<name>fs.defaultFS</name>
<value>hdfs://www.hadoop1.org:8020</value>
</property>
<!-- 指定临时数据目录-->
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/app/hadoop-2.5.2/data/tmp</value>
</property>
<!-- 指定垃圾回收-->
<property>
<name>fs.trash.interval</name>
<value>420</value>
</property>
</configuration>
hdfs-site.xml
<configuration>
<!-- 指定保存的副本数 这里三台机器所以保存3份 -->
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<!-- 指定SecondaryNameNode -->
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>www.hadoop3.org:50090</value>
</property>
</configuration>
slaves
<!-- 配置dataNode 和 NodeManager机器 -->
www.hadoop1.org
www.hadoop2.org
www.hadoop3.org
yarn
yarn-env.sh
修改文件的JAVA_HOME环境变量
export JAVA_HOME=/opt/app/jdk1.7.0_79
yarn-site.xml
<configuration>
<!-- 配置yarn中可以运行MapReduce程序-->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<!-- 配置ResourceManager节点的地址-->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>www.hadoop2.org</value>
</property>
<!-- 配置每个NodeManager节点 为4G内存 -->
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>4096</value>
</property>
<!-- 配置每个NodeManager节点 CPU为4核 -->
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>4</value>
</property>
<!-- 配置开启日志 -->
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<!-- 日志保留时间-->
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>640080</value>
</property>
</configuration>
slaves
<!-- 配置dataNode 和 NodeManager机器 -->
www.hadoop1.org
www.hadoop2.org
www.hadoop3.org
mapduce
mapred-env.sh
mapred-site.xml
<configuration>
<!-- 配置mapreduce在yarn上运行-->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<!-- 配置JobHistory历史服务监控 rpc地址-->
<property>
<name>mapreduce.jobhistory.address</name>
<value>www.hadoop1.org:10020</value>
</property>
<!-- 配置JobHistory历史服务监控 web地址-->
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>www.hadoop1.org:19888</value>
</property>
</configuration>
5. 准备分发到各个服务器上
使用Scp命令
1.事先要做ssh无秘钥登录 （三台服务器都要做）
会在当前用户家目录下.ssh目录生成几个文件
2. ssh-keygen -t rsa （生成秘钥）
ssh-copy-id www.hadoop1.org
ssh-copy-id www.hadoop2.org
ssh-copy-id www.hadoop3.org
3.测试远程无秘钥登录
ssh www.hadoop1.org
ssh www.hadoop1.org
ssh www.hadoop1.org
4.分发到另外两台服务器
scp -r ./hadoop-2.5.2/ root@www.hadoop2.org:/opt/app
scp -r ./hadoop-2.5.2/ root@www.hadoop3.org:/opt/app
6.启动整个hadoop集群
1.格式化hdfs
bin/hdfs namenode -format
2.启动hdfs
sbin/start-dfs.sh
root@www hadoop-2.5.2]# sbin/start-dfs.sh
17/06/23 23:38:25 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [www.hadoop1.org]
www.hadoop1.org: starting namenode, logging to /opt/app/hadoop-2.5.2/logs/hadoop-root-namenode-www.hadoop1.org.out
www.hadoop3.org: starting datanode, logging to /opt/app/hadoop-2.5.2/logs/hadoop-root-datanode-www.hadoop3.org.out
www.hadoop1.org: starting datanode, logging to /opt/app/hadoop-2.5.2/logs/hadoop-root-datanode-www.hadoop1.org.out
www.hadoop2.org: starting datanode, logging to /opt/app/hadoop-2.5.2/logs/hadoop-root-datanode-www.hadoop2.org.out
Starting secondary namenodes [www.hadoop3.org]
www.hadoop3.org: starting secondarynamenode, logging to /opt/app/hadoop-2.5.2/logs/hadoop-root-secondarynamenode-www.hadoop3.org.out
17/06/23 23:38:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
3.使用jps命令查看进程
jps
4.关闭防火墙
service iptables stop
5.访问http://www.hadoop1.org:50070
（查看hdfs）
(记得在windows配置映射 可以使用switchHost这个工具 比较方便
6 对hadoop hdfs的基本测试
bin/hdfs dfs -mkdir -p /user/root/tmp
bin/hdfs dfs -put ./etc/hadoop/core-site.xml ./etc/hadoop/hdfs-site.xml /user/root/tmp/
bin/hdfs dfs -text /user/root/tmp/core*
7.启动yarn
sbin/start-yarn.sh
sbin/mr-jobhistory-daemon.sh start historyserver
[root@www hadoop-2.5.2]# sbin/start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /opt/app/hadoop-2.5.2/logs/yarn-root-resourcemanager-www.hadoop2.org.out
www.hadoop3.org: starting nodemanager, logging to /opt/app/hadoop-2.5.2/logs/yarn-root-nodemanager-www.hadoop3.org.out
www.hadoop1.org: starting nodemanager, logging to /opt/app/hadoop-2.5.2/logs/yarn-root-nodemanager-www.hadoop1.org.out
www.hadoop2.org: starting nodemanager, logging to /opt/app/hadoop-2.5.2/logs/yarn-root-nodemanager-www.hadoop2.org.out
8.使用jsp查看
jps
9.访问 http://www.hadoop2.org:8088/cluster
（查看yarn）
10.对yarn和mapreduce进行测试
bin/yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.2.jar wordcount /user/root/tmp/* /user/root/tmp-out
17/06/24 00:04:02 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/06/24 00:04:03 INFO client.RMProxy: Connecting to ResourceManager at www.hadoop2.org/192.168.108.136:8032
17/06/24 00:04:04 INFO input.FileInputFormat: Total input paths to process : 2
17/06/24 00:04:04 INFO mapreduce.JobSubmitter: number of splits:2
17/06/24 00:04:05 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1498287508587_0001
17/06/24 00:04:05 INFO impl.YarnClientImpl: Submitted application application_1498287508587_0001
17/06/24 00:04:05 INFO mapreduce.Job: The url to track the job: http://www.hadoop2.org:8088/proxy/application_1498287508587_0001/
17/06/24 00:04:05 INFO mapreduce.Job: Running job: job_1498287508587_0001
17/06/24 00:04:19 INFO mapreduce.Job: Job job_1498287508587_0001 running in uber mode : false
17/06/24 00:04:19 INFO mapreduce.Job: map 0% reduce 0%
17/06/24 00:04:46 INFO mapreduce.Job: map 100% reduce 0%
17/06/24 00:05:04 INFO mapreduce.Job: map 100% reduce 100%
17/06/24 00:05:04 INFO mapreduce.Job: Job job_1498287508587_0001 completed successfully
17/06/24 00:05:04 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=2896
FILE: Number of bytes written=297217
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=2671
HDFS: Number of bytes written=1344
HDFS: Number of read operations=9
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=2
Launched reduce tasks=1
Data-local map tasks=2
Total time spent by all maps in occupied slots (ms)=48458
Total time spent by all reduces in occupied slots (ms)=14830
Total time spent by all map tasks (ms)=48458
Total time spent by all reduce tasks (ms)=14830
Total vcore-seconds taken by all map tasks=48458
Total vcore-seconds taken by all reduce tasks=14830
Total megabyte-seconds taken by all map tasks=49620992
Total megabyte-seconds taken by all reduce tasks=15185920
Map-Reduce Framework
Map input records=68
Map output records=235
Map output bytes=3071
Map output materialized bytes=2902
Input split bytes=240
Combine input records=235
Combine output records=180
Reduce input groups=98
Reduce shuffle bytes=2902
Reduce input records=180
Reduce output records=98
Spilled Records=360
Shuffled Maps =2
Failed Shuffles=0
Merged Map outputs=2
GC time elapsed (ms)=665
CPU time spent (ms)=2820
Physical memory (bytes) snapshot=487112704
Virtual memory (bytes) snapshot=2516992000
Total committed heap usage (bytes)=257171456
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=2431
File Output Format Counters
Bytes Written=1344
11.查看统计结果
bin/hdfs dfs -text /user/root/tmp-out/part*
"AS 2
"License"); 2
(the 2
--> 7
2.0 2
<!-- 8
<!--指定NameNode地址 1
</configuration> 2
</property> 5
<?xml 2
<?xml-stylesheet 2
<configuration> 2
<name>dfs.namenode.secondary.http-address</name> 1
<name>dfs.replication</name> 1
<name>fs.defaultFS</name> 1
<name>fs.trash.interval</name> 1
<name>hadoop.tmp.dir</name> 1
<property> 5
<value>/opt/app/hadoop-2.5.2/data/tmp</value> 1
<value>3</value> 1
<value>420</value> 1
<value>hdfs://www.hadoop1.org:8020</value> 1
<value>www.hadoop3.org:50090</value> 1
ANY 2
Apache 2
BASIS, 2
CONDITIONS 2
IS" 2
KIND, 2
LICENSE 2
License 6
License, 2
License. 4
Licensed 2
OF 2
OR 2
Put 2
See 4
Unless 2
Version 2
WARRANTIES 2
WITHOUT 2
You 2
a 2
accompanying 2
agreed 2
an 2
and 2
applicable 2
at 2
by 2
compliance 2
copy 2
distributed 4
either 2
encoding="UTF-8"?> 2
except 2
express 2
file 2
file. 4
for 2
governing 2
href="configuration.xsl"?> 2
http://www.apache.org/licenses/LICENSE-2.0 2
implied. 2
in 6
is 2
language 2
law 2
limitations 2
may 4
not 2
obtain 2
of 2
on 2
or 4
overrides 2
permissions 2
property 2
required 2
site-specific 2
software 2
specific 2
the 14
this 4
to 2
type="text/xsl" 2
under 6
use 2
version="1.0" 2
with 2
writing, 2
you 2
指定SecondaryNameNode 1
指定临时数据目录--> 1
指定保存的副本数 1
指定垃圾回收--> 1
这里三台机器所以保存3份 1
7.集群的时间同步
* 找一台机器
时间服务器
* 所有的机器与这台机器时间进行定时的同步
比如，每日十分钟，同步一次时间
以www.hadoop1.org为时间服务器
修改/etc/ntp.conf
打开注释
restrict 192.168.108.0 mask 255.255.255.0 nomodify notrap
注释掉
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst
增加
server 127.127.1.0 #local clock
fudge 127.127.1.0 stratum 10
修改 /etc/sysconfig/ntpd
SYNC_HWCLOCK=yes
启动ntpd服务 开机自动启动ntpd服务
service ntpd start
chkconfig ntpd on
在另外两台服务器
crontab -e （必须在root用户）
0-59/10 * * * * /usr/sbin/ntpdate www.hadoop1.org
/usr/sbin/ntpdate www.hadoop1.org(代表手动同步)
###############################################################################################################
HA QJM架构
配置QJM要点
share edits
JournalNode
NameNode
Active Standby
Client
Proxy
Fence
同一个时刻只能有一台NameNode起作用
IP
192.168.108.135 192.168.108.136 192.168.108.137
主机名
www.hadoop1.org www.hadoop2.org www.hadoop3.org
硬件信息
2G 内存 1CPU 1G 内存 1CPU 1G 内存 1CPU
HDFS:
NameNode NameNode
DataNode DataNode DataNode
JournalNode JournalNode JournalNode
配置文件
hdfs
hadoop-env.sh
修改文件的JAVA_HOME环境变量
export JAVA_HOME=/opt/app/jdk1.7.0_79
core-site.xml
<configuration>
<!-- 指定nameservice-->
<property>
<name>fs.defaultFS</name>
<value>hdfs://ns1</value>
</property>
<!-- 指定临时数据目录-->
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/ha/hadoop-2.5.2/data/tmp</value>
</property>
<!-- 指定垃圾回收-->
<property>
<name>fs.trash.interval</name>
<value>420</value>
</property>
</configuration>
hdfs-site.xml
<configuration>
<!-- 指定保存的副本数 这里三台机器所以保存3份 -->
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<!-- nameservice-->
<property>
<name>dfs.nameservices</name>
<value>ns1</value>
</property>
<!-- nameservice namenodes-->
<property>
<name>dfs.ha.namenodes.ns1</name>
<value>nn1,nn2</value>
</property>
<!-- NameNode rpc address-->
<property>
<name>dfs.namenode.rpc-address.ns1.nn1</name>
<value>www.hadoop1.org:8020</value>
</property>
<property>
<name>dfs.namenode.rpc-address.ns1.nn2</name>
<value>www.hadoop3.org:8020</value>
</property>
<!-- NameNode web address-->
<property>
<name>dfs.namenode.http-address.ns1.nn1</name>
<value>www.hadoop1.org:50070</value>
</property>
<property>
<name>dfs.namenode.http-address.ns1.nn2</name>
<value>www.hadoop3.org:50070</value>
</property>
<!-- share edits JournalNode address -->
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://www.hadoop1.org:8485;www.hadoop2.org:8485;www.hadoop3.org:8485/ns1</value>
</property>
<!-- journalnode edits dir -->
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/opt/ha/hadoop-2.5.2/data/jn_data</value>
</property>
<!-- client proxy -->
<property>
<name>dfs.client.failover.proxy.provider.ns1</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<!-- namenode fence method ssh -->
<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
</property>
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/root/.ssh/id_rsa</value>
</property>
</configuration>
slaves
<!-- 配置dataNode 和 NodeManager机器 -->
www.hadoop1.org
www.hadoop2.org
www.hadoop3.org
启动
1.各个节点启动journalnode
sbin/hadoop-daemon.sh start journalnode
2. 格式化NameNode
bin/hdfs namenode -format
3.启动一台NameNode
sbin/hadoop-daemon.sh start namenode
4.同步编辑日志文件
bin/hdfs namenode -bootstrapStandby
5.启动另外一台NameNode
sbin/hadoop-daemon.sh start namenode
6.启动所有dataNode
sbin/hadoop-daemon.sh start datanode
7.将nn1变成激活状态
bin/hdfs haadmin -transitionToActive nn1
8.查看nn1状态
bin/hdfs haadmin -getServiceState nn1
9 配置故障转移
hdfs-site.xml 增加
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
core-site.xml增加
<property>
<name>ha.zookeeper.quorum</name>
<value>www.hadoop1.org:2181,www.hadoop2.org:2181,www.hadoop3.org:2181</value>
</property>
10
sbin/stop-dfs.sh
bin/zkServer.sh satrt
bin/hdfs zkfc -formatZK
sbin/start-dfs.sh (sbin/hadoop-daemon.sh start zkfc 这里不用启动)
11.验证
kill -9 （namenode pid）
hadoop集群搭建
原创
©著作权归作者所有：来自51CTO博客作者黄信程的原创作品，谢绝转载，否则将追究法律责任
提问和评论都可以，用心的回复会被更多人看到
评论
发布评论
相关文章
-
第二十四节 docker构建hadoop集群
大数据集群测试 hadoop实验
hadoop xml docker -
Hadoop集群搭建
Hadoop官方网站：http://hadoop.apache.org/Hadoop运行模式包括：本地
hadoop linux java xml 配置文件 -
hadoop搭建集群
hadoop集群搭建；SecondaryNameNode；firewalld；ntpdate；ssh无密码访问
hadoop集群搭建 SecondaryNameNode firewalld ntpdate ssh无密码访问 -
搭建hadoop集群
今天学习了Hadoop集群搭建的方法： 目的 本文描述了如何安装、配置和管理有实际意义的Hadoop集群
hadoop 守护进程 hdfs
