hadoop的集群先搞二台机器，一台管理机，一台node机，为什么呢。因为钱，机子也要钱。数据量是逐步增长起来的。如果一台node不能满足需求了，在增加node节点到集群。
在开始安装配置前，最好把该篇文章看上几遍，理顺了，在开始。特别是我踩过的坑。
一，服务器说明
1. 10.0.0.237 bigserver1 //master
2. 10.0.0.236 bigserver2 //datanode二，修改主机名，并配置hosts
1，修改主机名
1. [root@localhost ~]# hostname
2. localhost.localdomain
3. [root@localhost ~]# hostname bigserver1
4. [root@localhost ~]# hostname
5. bigserver12， 在/etc/hosts文件中增加，所有节点一样
1. 10.0.0.236 bigserver2
2. 10.0.0.237 bigserver1三，关闭防火墙和selinux
1. # systemctl stop firewalld //停止
2. # systemctl disable firewalld //取消启动
3.
4. # cat /etc/sysconfig/selinux
5. SELINUX=disabled //关闭改完重启一下电脑。hadoop安装配置好了以后，防火墙可以打开，开放端口。
四，ssh免密码登录
1. # ssh-keygen -t rsa
2.
3. # ssh-copy-id -i ~/.ssh/id_rsa.pub root@10.0.0.236 -p 22
4. # ssh-copy-id -i ~/.ssh/id_rsa.pub root@10.0.0.237 -p 22
5.
6. # scp ~/.ssh/id_rsa root@10.0.0.236:/root/.ssh/
7. # scp ~/.ssh/id_rsa root@10.0.0.237:/root/.ssh/
8.
9. 登录到236和237后
10. # cd ~/.ssh/
11. # chmod 600 id_rsa我是在本机生成了公私钥，分别传到了236，237机器。
五，安装java1.8
1. # yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel六，下载hadoop
https://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.7.7/hadoop-2.7.7.tar.gz
大家根据自己的需求去下载。
1. # tar zxvf hadoop-2.7.7.tar.gz
2. # mkdir /bigdata
3. # mv hadoop-2.2.7 /bigdata/hadoop
4.
5. # mkdir -pv /bigdata/hadoop/{tmp,var,dfs}
6. # mkdir -pv /bigdata/hadoop/dfs/{name,data}七，配置hadoop
1，备份
1. # cd /bigdata/hadoop/etc
2. # cp -r hadoop hadoop_bak这一步很重要，养成一个良好的习惯会事半功倍。
2，配置core-site.xml
1. <property>
2. <name>hadoop.tmp.dir</name>
3. <value>/bigdata/hadoop/tmp</value>
4. </property>
5. <property>
6. default.name</name>
7. //bigserver1:9000</value>
8. </property>在<configuration></configuration>中添加
3，修改 hadoop-env.sh
1. # whereis javac
2. javac: /usr/bin/javac /usr/share/man/man1/javac.1.gz
3.
4. # ll /usr/bin/javac
5. lrwxrwxrwx. 1 root root 23 Dec 27 00:08 /usr/bin/javac -> /etc/alternatives/javac
6.
7. # ll /etc/alternatives/javac
8. lrwxrwxrwx. 1 root root 70 Dec 27 00:08 /etc/alternatives/javac -> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64/bin/javac
9. //以上是查找环境变量
10.
11. # echo "export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64" >> ~/.bashrc
12. # source ~/.bashrc
13. # vim hadoop-env.sh
14. 将export JAVA_HOME=${JAVA_HOME}替换成
15. export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64如果不是管理工具包安装，填解压目录，即可
4，配置hdfs-site.xml
1. <property>
2. <name>dfs.name.dir</name>
3. <value>/bigdata/hadoop/dfs/name</value>
4. </property>
5. <property>
6. <name>dfs.data.dir</name>
7. <value>/bigdata/hadoop/dfs/data</value>
8. </property>
9. <property>
10. <name>dfs.replication</name>
11. <value>2</value>
12. </property>
13. <property>
14. <name>dfs.permissions</name>
15. <value>false</value>
16. </property>5，配置mapred-site.xml
1. # cp mapred-site.xml.template mapred-site.xml
2. # vim mapred-site.xml
3. <property>
4. <name>mapred.job.tracker</name>
5. <value>bigserver1:49001</value>
6. </property>
7.
8. <property>
9. <name>mapred.local.dir</name>
10. var</value>
11. </property>
12.
13. <property>
14. <name>mapreduce.framework.name</name>
15. <value>yarn</value>
16. </property>6，修改slaves
1. # cat slaves
2. bigserver27，配置yarn-site.xml
1. <property>
2. <name>yarn.resourcemanager.hostname</name>
3. <value>bigserver1</value>
4. </property>
5. <property>
6. <name>yarn.resourcemanager.address</name>
7. <value>${yarn.resourcemanager.hostname}:8032</value>
8. </property>
9. <property>
10. <name>yarn.resourcemanager.scheduler.address</name>
11. <value>${yarn.resourcemanager.hostname}:8030</value>
12. </property>
13. <property>
14. <name>yarn.resourcemanager.webapp.address</name>
15. <value>${yarn.resourcemanager.hostname}:8088</value>
16. </property>
17. <property>
18. <name>yarn.resourcemanager.webapp.https.address</name>
19. <value>${yarn.resourcemanager.hostname}:8090</value>
20. </property>
21. <property>
22. <name>yarn.resourcemanager.resource-tracker.address</name>
23. <value>${yarn.resourcemanager.hostname}:8031</value>
24. </property>
25. <property>
26. <name>yarn.resourcemanager.admin.address</name>
27. <value>${yarn.resourcemanager.hostname}:8033</value>
28. </property>
29. <property>
30. <name>yarn.nodemanager.aux-services</name>
31. <value>mapreduce_shuffle</value>
32. </property>
33. <property>
34. <name>yarn.nodemanager.vmem-check-enabled</name>
35. <value>false</value>
36. </property>不要轻易的去配置cpu，内存等。不然会影响mapredure。例如：
yarn.nodemanager.resource.memory-mb
yarn.scheduler.maximum-allocation-mb
yarn.nodemanager.vmem-pmem-ratio等以下是我配置不全导致的错误 ：
2018-12-27 09:11:54,178 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1545833322243_0001_m_000007_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2018-12-27 09:11:54,178 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1545833322243_0001_m_000008_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2018-12-27 09:11:54,178 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1545833322243_0001_r_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2018-12-27 09:11:54,179 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:1024, vCores:1>
2018-12-27 09:11:54,185 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: reduceResourceRequest:<memory:1024, vCores:1>
2018-12-27 09:11:54,196 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer setup for JobId: job_1545833322243_0001, File: hdfs://bigserver1:9000/tmp/hadoop-yarn/staging/root/.staging/job_1545833322243_0001/job_1545833322243_0001_1.jhist
2018-12-27 09:11:55,138 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:9 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0
2018-12-27 09:11:55,194 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1545833322243_0001: ask=3 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:0, vCores:0> knownNMs=1
2018-12-27 09:11:55,195 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to preempt 1 due to lack of space for maps
2018-12-27 09:11:56,198 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to preempt 1 due to lack of space for maps
2018-12-27 09:11:57,202 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to preempt 1 due to lack of space for maps
到这儿，hadoop就配置完成了，master节点和datanode节点配置一样。
八，初始化hadoop，并运行hadoop
1，只需要在master节点初始化，node节点不需要
1. # cd /bigdata/hadoop/bin/
2. ./hadoop namenode -format初始化成功后，会/bigdata/hadoop/dfs/name多出一个current文件夹。初始化一次后，最好不要在重新初始化，最好不要在重新初始化，最好不要在重新初始化。重要的事情说三遍。会导致master节点和datanode节点对不上。后面会具体说明。
2，只需要在master启动hadoop
1. # cd /bigdata/hadoop/sbin/
2. # ./start-all.sh
3. This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
4. Starting namenodes on [bigserver1]
5. bigserver1: starting namenode, logging to /home/bigdata/hadoop/logs/hadoop-root-namenode-bigserver1.out
6. bigserver2: starting datanode, logging to /home/bigdata/hadoop/logs/hadoop-root-datanode-bigserver2.out
7. Starting secondary namenodes [0.0.0.0]
8. 0.0.0.0: starting secondarynamenode, logging to /home/bigdata/hadoop/logs/hadoop-root-secondarynamenode-bigserver1.out
9. starting yarn daemons
10. starting resourcemanager, logging to /home/bigdata/hadoop/logs/yarn-root-resourcemanager-bigserver1.out
11. bigserver2: starting nodemanager, logging to /home/bigdata/hadoop/logs/yarn-root-nodemanager-bigserver2.out3，检查hadoop集群各节点是否正常启动
1. //master节点
2. [root@bigserver1 name]# netstat -tpnl |grep java
3. tcp 0 0 0.0.0.0:50070 0.0.0.0:* LISTEN 5573/java
4. tcp 0 0 10.0.0.237:9000 0.0.0.0:* LISTEN 5573/java
5. tcp 0 0 0.0.0.0:50090 0.0.0.0:* LISTEN 5768/java
6. tcp6 0 0 10.0.0.237:8088 :::* LISTEN 5930/java
7. tcp6 0 0 10.0.0.237:8030 :::* LISTEN 5930/java
8. tcp6 0 0 10.0.0.237:8031 :::* LISTEN 5930/java
9. tcp6 0 0 10.0.0.237:8032 :::* LISTEN 5930/java
10. tcp6 0 0 10.0.0.237:8033 :::* LISTEN 5930/java
11. [root@bigserver1 name]# jps
12. 3457 RunJar
13. 6851 Jps
14. 5573 NameNode
15. 5768 SecondaryNameNode
16. 5930 ResourceManager
17.
18. //datanode节点
19. [root@bigserver2 sbin]# netstat -tpnl |grep java
20. tcp 0 0 127.0.0.1:44205 0.0.0.0:* LISTEN 3405/java
21. tcp 0 0 0.0.0.0:50010 0.0.0.0:* LISTEN 3405/java
22. tcp 0 0 0.0.0.0:50075 0.0.0.0:* LISTEN 3405/java
23. tcp 0 0 0.0.0.0:50020 0.0.0.0:* LISTEN 3405/java
24. tcp6 0 0 :::43959 :::* LISTEN 3520/java
25. tcp6 0 0 :::13562 :::* LISTEN 3520/java
26. tcp6 0 0 :::8040 :::* LISTEN 3520/java
27. tcp6 0 0 :::8042 :::* LISTEN 3520/java
28. [root@bigserver2 sbin]# jps
29. 3520 NodeManager
30. 5761 Jps
31. 3405 DataNodejps显示的内容，如果少了一个说明没有配置成功。如果master和datanode节点进程缺少也说明没有成功。
如果都没有什么问题的话，可以通过url来访问了。
http://10.0.0.237:50070，节点健康检查工具
http://10.0.0.237:8088，集群各节点任务分析工具
如下图
hadoop 健康检查
hadoop集群
4，只需要在master停止hadoop
1. root@localhost sbin]# ./stop-all.sh //停止
2. This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh
3. Stopping namenodes on [bigserver1]
4. bigserver1: stopping namenode
5. bigserver2: no datanode to stop //刚开始配置时，datanode没有启动成功报的错
6. Stopping secondary namenodes [0.0.0.0]
7. 0.0.0.0: stopping secondarynamenode
8. stopping yarn daemons
9. stopping resourcemanager
10. bigserver2: stopping nodemanager
11. no proxyserver to stop通过jps查看，缺少了DataNode。但是nodemanager是起来的。问题出在master和datanode节点，集群点没有对上，这让我想起了mysql replication position，对不上也会出现无法同步的问题。导致no datanode to stop这个原因的产生，猜测是master节点，进行了多次的初始化。hadoop namenode -format。
解决办法如下：
master点，打开/bigdata/hadoop/dfs/name/current/VERSION，
datanode点，打开/bigdata/hadoop/dfs/data/current/VERSION，
将master节点的clusterID，copy到datanode中，重启就好。
网上查了一下，有人说同步namespaceID也可以，但是我用hadoop2.7.7版本中，datanode节点，/bigdata/hadoop/dfs/data/current/VERSION文件中根本没有namespaceID，我又不想加。哈哈。也不确定这样行不行。
九，测试hadoop集群
1，master节点hdfs创建测试目录
1. # ./bin/hdfs dfs -mkdir /test
2. # ./bin/hdfs dfs -ls /
3. Found 3 items
4. drwxr-xr-x - root supergroup 0 2018-12-26 20:42 /test
5. drwx------ - root supergroup 0 2018-12-26 21:27 /tmp
6. drwxr-xr-x - root supergroup 0 2018-12-26 20:20 /user2，master节点上传测试文件到hdfs
1. # ./bin/hdfs dfs -put ./etc/hadoop/*.xml /test/3，master节点测试mapredure
1. # ./bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar grep /test/ ./output 'dfs[a-z.]+'
2. 18/12/27 09:11:46 INFO client.RMProxy: Connecting to ResourceManager at bigserver1/10.0.0.237:8032
3. 18/12/27 09:11:48 INFO input.FileInputFormat: Total input paths to process : 9
4. 18/12/27 09:11:48 INFO mapreduce.JobSubmitter: number of splits:9
5. 18/12/27 09:11:48 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1545833322243_0001
6. 18/12/27 09:11:49 INFO impl.YarnClientImpl: Submitted application application_1545833322243_0001 //datanode端userlogs中有日志
7. 18/12/27 09:11:49 INFO mapreduce.Job: The url to track the job: http://bigserver1:8088/proxy/application_1545833322243_0001/
8. 18/12/27 09:11:49 INFO mapreduce.Job: Running job: job_1545833322243_0001
9. 18/12/27 09:11:55 INFO mapreduce.Job: Job job_1545833322243_0001 running in uber mode : false
10. 18/12/27 09:11:55 INFO mapreduce.Job: map 0% reduce 0% //这块卡死，map reduce都是0到datanode节点查看日志：
1. # cd /bigdata/hadoop/logs/userlogs
2.
3. [root@bigserver2 userlogs]# ls
4. application_1545825824765_0001 application_1545827800765_0001 application_1545828806710_0001 application_1545829094007_0001 application_1545833322243_0001
5.
6. [root@bigserver2 userlogs]# ll |grep application_1545833322243_0001
7. drwx--x--- 3 root root 52 12月 27 09:11 application_1545833322243_0001
8.
9. [root@bigserver2 userlogs]# cd application_1545833322243_0001
10. [root@bigserver2 application_1545833322243_0001]# cd container_1545833322243_0001_01_000001/
11.
12. [root@bigserver2 container_1545833322243_0001_01_000001]# ls
13. stderr stdout syslog
14.
15. [root@bigserver2 container_1545833322243_0001_01_000001]# tail -f syslog
16. 2018-12-27 09:14:12,652 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to preempt 1 due to lack of space for maps
17. 2018-12-27 09:14:13,654 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to preempt 1 due to lack of space for maps
18. 2018-12-27 09:14:14,657 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to preempt 1 due to lack of space for maps
19. 2018-12-27 09:14:15,659 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to preempt 1 due to lack of space for maps
20. 2018-12-27 09:14:16,662 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to preempt 1 due to lack of space for maps
21. 2018-12-27 09:14:17,665 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to preempt 1 due to lack of space for maps
22. 。。。。。。。。。。。。。。。。。。。。忽略。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。解决办法：上面也提到了，就是yarn-site.xml，配置内存和cpu相关的去掉。重启hadoop就好
看一下成功后
mapredure测试成功
十，查日志
hadoop的日志，还是很多的，还没有装hbase,hive,spark等。除了进入服务器查看外，还可以通过网页查看。
datanode logs 页面访问
master log页面
不怕问题，就怕出了问题，不知道错在哪里。随便点点网页log就发现个问题
2018-12-27 10:23:04,569 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint
java.io.IOException: Inconsistent checkpoint fields.
LV = -63 namespaceID = 1131284630 cTime = 0 ; clusterId = CID-ea915c79-c5cb-4d23-bc55-e4530f999cb0 ; blockpoolId = BP-508509447-10.0.0.237-1545809802003.
Expecting respectively: -63; 839710719; 0; CID-66e894a8-1cb1-4b8e-bac4-2bc3b526e062; BP-262790598-10.0.0.237-1545803654780.
at org.apache.hadoop.hdfs.server.namenode.CheckpointSignature.validateStorageInfo(CheckpointSignature.java:134)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:531)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:395)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:361)
at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:415)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:357)
at java.lang.Thread.run(Thread.java:748)
解决办法：
mater节点，删除该目录/bigdata/hadoop/tmp/dfs/namesecondary/current下的所有文件，重启hadoop即可。
