源码见：https://github.com/hiszm/hadoop-train

YARN产生背景


​Apache YARN​ (Yet Another Resource Negotiator) 是 hadoop 2.0 引入的集群资源管理系统。用户可以将各种服务框架部署在 YARN 上，由 YARN 进行统一地管理和资源分配。



The fundamental idea of MRv2 is to split up the two major functionalities of the JobTracker, ​resource management and job scheduling​ /monitoring, into separate daemons. The idea is to have a global ResourceManager (RM) and per-application ApplicationMaster (AM). An application is either a single job in the classical sense of Map-Reduce jobs or a DAG of jobs.


Hadoop基础-10-YARN_mapreduce

YARN架构详解

Hadoop基础-10-YARN_hadoop_02

  1. ​Client​

  • 向​​RM​​提交任务
  • 杀死任务

  1. ​ResourceManager​

  • ​​ResourceManager​​ 通常在独立的机器上以后台进程的形式运行，它是整个 ​集群资源的主要协调者和管理者​ 。
  • ​负责给用户提交的所有应用程序分配资源​ ，它根据应用程序优先级、队列容量、ACLs、数据位置等信息，做出决策，然后以共享的、安全的、多租户的方式制定分配策略，调度集群资源。

  1. ​NodeManager​

  • ​​NodeManager​​ 是 YARN 集群中的每个具体 ​节点的管理者​ 。
  • 主要 ​负责该节点内所有容器的生命周期的管理，监视资源和跟踪节点健康​ 。具体如下：

  • 启动时向 ​​ResourceManager​​​ 注册并定时发送心跳消息，等待 ​​ResourceManager​​ 的指令；
  • 维护 ​​Container​​​ 的生命周期，监控 ​​Container​​ 的资源使用情况；
  • 管理任务运行时的相关依赖，根据 ​​ApplicationMaster​​​ 的需要，在启动 ​​Container​​ 之前将需要的程序及其依赖拷贝到本地。


  1. ​ApplicationMaster​

  • 在用户提交一个应用程序时，YARN 会启动一个轻量级的 ​进程​ ​​ApplicationMaster​​。
  • ​​ApplicationMaster​​ 负责协调来自 ​​ResourceManager​​ 的资源，并通过 ​​NodeManager​​ 监视容器内资源的使用情况，同时还负责任务的监控与容错。具体如下：

  • 根据应用的运行状态来决定动态计算资源需求；
  • 向 ​​ResourceManager​​ 申请资源，监控申请的资源的使用情况；
  • 跟踪任务状态和进度，报告资源的使用情况和应用的进度信息；
  • 负责任务的容错。


  1. ​Container​

  • ​​Container​​ 是 YARN 中的 ​资源抽象​ ，它封装了某个节点上的多维度资源，如内存、CPU、磁盘、网络等。
  • 当 AM 向 RM 申请资源时，RM 为 AM 返回的资源是用 ​​Container​​ 表示的。
  • YARN 会为每个任务分配一个 ​​Container​​​，该任务只能使用该 ​​Container​​​ 中描述的资源。​​ApplicationMaster​​​ 可在 ​​Container​​​ 内运行任何类型的任务。例如，​​MapReduce ApplicationMaster​​ 请求一个容器来启动 map ​或​ reduce 任务

YARN执行流程

Hadoop基础-10-YARN_mapreduce_03


  2. 客户端​​client​​向​​yarn集群​​提交作业 , 首先①向​​ResourceManager​​申请分配资源
  3. ​​Resource Manager​​会为作业分配一个​​Container(Application manager)​​,​​Container​​里面运行这(Application Manager)
  4. ​​Resource Manager​​会找一个对应的​​NodeManager​​通信②，要求​​NodeManager​​在这个​​container​​上启动应用程序​​Application Master​​ ③
  5. ​​Application Master​​向​​Resource Manager​​申请资源④(采用轮询的方式通过​​RPC​​协议)，​​Resource scheduler​​将资源封装发给​​Application master​​④,
  6. ​​Application Master​​将获取到的资源分配给各个​​Node Manager​​，并监控运行情况⑤
  7. ​​Node Manage​​得到任务和资源开始执行作业⑥
  8. 再细分作业的话可以分为 先执行​​Map Task​​,结束后在执行​​Reduce Task​​ 最后再将结果返回給​​Application Master​​等依次往上层递交⑦

YARN环境部署

​​

​​

​​
  • YARN on Single Node


You can run a MapReduce job on YARN in a pseudo-distributed mode by setting a few parameters and running ResourceManager daemon and NodeManager daemon in addition.
The following instructions assume that 1. ~ 4. steps of the above instructions are already executed.


Configure parameters as follows:

etc/hadoop/mapred-site.xml:

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

etc/hadoop/yarn-site.xml:

<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
</configuration>

Start ResourceManager daemon and NodeManager daemon:

​​$ sbin/start-yarn.sh​​

Browse the web interface for the ResourceManager; by default it is available at:

ResourceManager - http://localhost:8088/

Run a MapReduce job.

When you’re done, stop the daemons with:

​​$ sbin/stop-yarn.sh​​

[hadoop@hadoop000 hadoop]$ pwd
/home/hadoop/app/hadoop-2.6.0-cdh5.15.1/etc/hadoop
[hadoop@hadoop000 hadoop]$ vi mapred-site.xml
[hadoop@hadoop000 hadoop]$ vi yarn-site.xml

[hadoop@hadoop000 sbin]$ jps
7234 NodeManager
8131 Jps
7588 NameNode
7962 SecondaryNameNode
7116 ResourceManager
7791 DataNode

​​

​​

​​

Hadoop基础-10-YARN_java_04

[hadoop@hadoop000 hadoop]$ pwd
/home/hadoop/app/hadoop-2.6.0-cdh5.15.1/share/hadoop
[hadoop@hadoop000 hadoop]$ ls
common  httpfs  mapreduce   mapreduce2  yarn
hdfs    kms     mapreduce1  tools
[hadoop@hadoop000 hadoop]$ pwd
/home/hadoop/app/hadoop-2.6.0-cdh5.15.1/share/hadoop
[hadoop@hadoop000 hadoop]$ cd mapreduce
[hadoop@hadoop000 mapreduce]$ ls
hadoop-mapreduce-client-app-2.6.0-cdh5.15.1.jar
hadoop-mapreduce-client-common-2.6.0-cdh5.15.1.jar
hadoop-mapreduce-client-core-2.6.0-cdh5.15.1.jar
hadoop-mapreduce-client-hs-2.6.0-cdh5.15.1.jar
hadoop-mapreduce-client-hs-plugins-2.6.0-cdh5.15.1.jar
hadoop-mapreduce-client-jobclient-2.6.0-cdh5.15.1.jar
hadoop-mapreduce-client-jobclient-2.6.0-cdh5.15.1-tests.jar
hadoop-mapreduce-client-nativetask-2.6.0-cdh5.15.1.jar
hadoop-mapreduce-client-shuffle-2.6.0-cdh5.15.1.jar
hadoop-mapreduce-examples-2.6.0-cdh5.15.1.jar
lib
lib-examples
sources

提交example案例到YARN上运行

hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.15.1.jar pi 2 3

Hadoop基础-10-YARN_大数据_05

Hadoop基础-10-YARN_mapreduce_06

[hadoop@hadoop000 ~]$ hadoop dfs -cat /wc/input/1.txt
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

hello world hello
hello
hello world
[hadoop@hadoop000 ~]$ 
hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.15.1.jar wordcount /wc/input /wc/output

[hadoop@hadoop000 ~]$ hadoop dfs -cat /wc/output/part-r-00000
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

hello 4
world 2

Hadoop基础-10-YARN_java_07

提交流量统计案例到YARN上运行

  • ​​mvn clean package -DskipTests​​ 注意在当前的项目环境
(base) locahost:untitled5 jacksun$ mvn clean package -DskipTests
[INFO] Scanning for projects...
[INFO] 
[INFO] -----------------------< org.example:untitled5 >------------------------
[INFO] Building untitled5 1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ untitled5 ---
[INFO] Deleting /Users/jacksun/IdeaProjects/untitled5/target
[INFO] 
[INFO] --- maven-resources-plugin:3.0.2:resources (default-resources) @ untitled5 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ untitled5 ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 15 source files to /Users/jacksun/IdeaProjects/untitled5/target/classes
[INFO] 
[INFO] --- maven-resources-plugin:3.0.2:testResources (default-testResources) @ untitled5 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /Users/jacksun/IdeaProjects/untitled5/src/test/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.8.0:testCompile (default-testCompile) @ untitled5 ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 2 source files to /Users/jacksun/IdeaProjects/untitled5/target/test-classes
[INFO] 
[INFO] --- maven-surefire-plugin:2.22.1:test (default-test) @ untitled5 ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ untitled5 ---
[INFO] Building jar: /Users/jacksun/IdeaProjects/untitled5/target/untitled5-1.0-SNAPSHOT.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  43.078 s
[INFO] Finished at: 2020-09-02T10:04:51+08:00
[INFO] ------------------------------------------------------------------------
(base) locahost:untitled5 jacksun$ ls
D:    access    output    src
Hadoop.iml  input   pom.xml   target
(base) locahost:untitled5 jacksun$ cd target/
(base) locahost:target jacksun$ ls
classes       maven-status
generated-sources   test-classes
generated-test-sources    untitled5-1.0-SNAPSHOT.jar
maven-archiver
(base) locahost:target jacksun$ 

(base) locahost:target jacksun$ scp untitled5-1.0-SNAPSHOT.jar hadoop@192.168.43.200:~/lib/
hadoop@192.168.43.200's password: 
untitled5-1.0-SNAPSHOT.jar          100%   18KB 750.6KB/s   00:00    
(base) locahost:target jacksun$

  • 到编译后的​​/target/​​目录jar包和相关的数据上传到服务器​​scp xxx hadoop@localhost:dir​​
  • 再上传到​​hdfs​​用​​Hadoop fs -put /dir​​ 

hadoop jar untitled5-1.0-SNAPSHOT.jar com.bigdata.hadoop.mr.access.AccessYARNApp /access/input/access.log /access/ouput/
  • 执行作业​​hadoop jar xx.jar​​​完整的类名和包名​​args参数​​ 
20/09/02 10:13:22 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
20/09/02 10:13:22 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
20/09/02 10:13:23 INFO input.FileInputFormat: Total input paths to process : 1
20/09/02 10:13:24 INFO mapreduce.JobSubmitter: number of splits:1
20/09/02 10:13:24 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1598998523059_0003
20/09/02 10:13:25 INFO impl.YarnClientImpl: Submitted application application_1598998523059_0003
20/09/02 10:13:25 INFO mapreduce.Job: The url to track the job: http://hadoop000:8088/proxy/application_1598998523059_0003/
20/09/02 10:13:25 INFO mapreduce.Job: Running job: job_1598998523059_0003
20/09/02 10:13:35 INFO mapreduce.Job: Job job_1598998523059_0003 running in uber mode : false
20/09/02 10:13:35 INFO mapreduce.Job:  map 0% reduce 0%
20/09/02 10:13:42 INFO mapreduce.Job:  map 100% reduce 0%
20/09/02 10:13:51 INFO mapreduce.Job:  map 100% reduce 33%
20/09/02 10:13:53 INFO mapreduce.Job:  map 100% reduce 67%
20/09/02 10:14:01 INFO mapreduce.Job:  map 100% reduce 100%
20/09/02 10:14:03 INFO mapreduce.Job: Job job_1598998523059_0003 completed successfully
20/09/02 10:14:03 INFO mapreduce.Job: Counters: 50
  File System Counters
    FILE: Number of bytes read=1185
    FILE: Number of bytes written=575593
    FILE: Number of read operations=0
    FILE: Number of large read operations=0
    FILE: Number of write operations=0
    HDFS: Number of bytes read=2444
    HDFS: Number of bytes written=552
    HDFS: Number of read operations=12
    HDFS: Number of large read operations=0
    HDFS: Number of write operations=6
  Job Counters 
    Killed reduce tasks=1
    Launched map tasks=1
    Launched reduce tasks=3
    Data-local map tasks=1
    Total time spent by all maps in occupied slots (ms)=13914
    Total time spent by all reduces in occupied slots (ms)=71064
    Total time spent by all map tasks (ms)=4638
    Total time spent by all reduce tasks (ms)=23688
    Total vcore-milliseconds taken by all map tasks=4638
    Total vcore-milliseconds taken by all reduce tasks=23688
    Total megabyte-milliseconds taken by all map tasks=14247936
    Total megabyte-milliseconds taken by all reduce tasks=72769536
  Map-Reduce Framework
    Map input records=23
    Map output records=23
    Map output bytes=1121
    Map output materialized bytes=1185
    Input split bytes=110
    Combine input records=0
    Combine output records=0
    Reduce input groups=21
    Reduce shuffle bytes=1185
    Reduce input records=23
    Reduce output records=21
    Spilled Records=46
    Shuffled Maps =3
    Failed Shuffles=0
    Merged Map outputs=3
    GC time elapsed (ms)=696
    CPU time spent (ms)=8510
    Physical memory (bytes) snapshot=783241216
    Virtual memory (bytes) snapshot=16559239168
    Total committed heap usage (bytes)=674758656
  Shuffle Errors
    BAD_ID=0
    CONNECTION=0
    IO_ERROR=0
    WRONG_LENGTH=0
    WRONG_MAP=0
    WRONG_REDUCE=0
  File Input Format Counters 
    Bytes Read=2334
  File Output Format Counters 
    Bytes Written=552


[hadoop@hadoop000 lib]$ hadoop fs -ls /access/ouput/
Found 4 items
-rw-r--r--   1 hadoop supergroup          0 2020-09-02 10:14 /access/ouput/_SUCCESS
-rw-r--r--   1 hadoop supergroup        393 2020-09-02 10:13 /access/ouput/part-r-00000
-rw-r--r--   1 hadoop supergroup         80 2020-09-02 10:13 /access/ouput/part-r-00001
-rw-r--r--   1 hadoop supergroup         79 2020-09-02 10:13 /access/ouput/part-r-00002


[hadoop@hadoop000 lib]$ hadoop fs -cat /access/ouput/part-r-00000
13480253104,180,180,360
13502468823,7335,110349,117684
13560436666,1116,954,2070
13560439658,2034,5892,7926
13602846565,1938,2910,4848
13660577991,6960,690,7650
13719199419,240,0,240
13726230503,2481,24681,27162
13726238888,12481,44681,57162
13760778710,120,120,240
13826544101,264,0,264
13922314466,3008,3720,6728
13925057413,11058,48243,59301
13926251106,240,0,240
13926435656,132,1512,1644
[hadoop@hadoop000 lib]$

Hadoop基础-10-YARN_大数据_08

  • 到​​http://192.168.43.200:8088/cluster/​​观察结果

Hadoop基础-10-YARN_java_09