Yarn 弹性队列测试

转载

mob604756f89f2f 2020-11-02 20:05:00

文章标签 yarn 人工智能大数据 mapreduce jar 文章分类 代码人生

Yarn 调度器中的 Capacity Scheduler 和 Fair Scheduler 都支持弹性扩容, 现对 Fari Scheduler 进行弹性扩容测试。

Yarn 动态资源池配置配置如下：

一、不限制最大资源数

1、yarn配置

不限制最大资源数

2、测试

$ sudo -u hdfs  hadoop jar hadoop-mapreduce-examples.jar pi -Dmapreduce.job.queuename=test 1000 100
WARNING: Use "yarn jar" to launch YARN applications.
Number of Maps  = 1000
Samples per Map = 100
Wrote input for Map #0
......
Wrote input for Map #999
Starting Job
20/11/02 18:57:41 INFO mapreduce.Job: Running job: job_1604307167678_0003
20/11/02 18:57:49 INFO mapreduce.Job: Job job_1604307167678_0003 running in uber mode : false
20/11/02 18:57:49 INFO mapreduce.Job:  map 0% reduce 0%
20/11/02 18:57:59 INFO mapreduce.Job:  map 5% reduce 0%
......
20/11/02 18:59:07 INFO mapreduce.Job:  map 100% reduce 0%
20/11/02 18:59:09 INFO mapreduce.Job:  map 100% reduce 100%
20/11/02 18:59:09 INFO mapreduce.Job: Job job_1604307167678_0003 completed successfully
20/11/02 18:59:10 INFO mapreduce.Job: Counters: 54
Job Finished in 89.816 seconds
Estimated value of Pi is 3.14120000000000000000

3、结果

从队列test可以看出，在不限制队列最大资源的情况下，虚线框可以看出队列test可以弹性扩容到最大100%与队列root相同。绿色部分表示队列实际使用的资源。

二、限制最大资源数为50%

1、yarn配置

2、测试

$ sudo -u hdfs  hadoop jar hadoop-mapreduce-examples.jar pi -Dmapreduce.job.queuename=test 1000 100
WARNING: Use "yarn jar" to launch YARN applications.
Number of Maps  = 1000
Samples per Map = 100
Wrote input for Map #0
......
Wrote input for Map #999
Starting Job
20/11/02 18:57:41 INFO mapreduce.Job: Running job: job_1604307167678_0005
20/11/02 18:57:49 INFO mapreduce.Job: Job job_1604307167678_0005 running in uber mode : false
20/11/02 18:57:49 INFO mapreduce.Job:  map 0% reduce 0%
20/11/02 18:57:59 INFO mapreduce.Job:  map 5% reduce 0%
......
20/11/02 18:59:07 INFO mapreduce.Job:  map 100% reduce 0%
20/11/02 18:59:09 INFO mapreduce.Job:  map 100% reduce 100%
20/11/02 18:59:09 INFO mapreduce.Job: Job job_1604307167678_0003 completed successfully
20/11/02 18:59:10 INFO mapreduce.Job: Counters: 54
Job Finished in 107.728 seconds
Estimated value of Pi is 3.14120000000000000000

从队列test可以看出，在限制队列最大资源为50%的情况下，虚线框可以看出队列test可以弹性扩容到总容量的50%。绿色部分表示队列实际使用的资源。

三、Fair Share 与最大资源数相同为10%

1、配置

2、测试

$ sudo -u hdfs  hadoop jar hadoop-mapreduce-examples.jar pi -Dmapreduce.job.queuename=test 1000 100
WARNING: Use "yarn jar" to launch YARN applications.
Number of Maps  = 1000
Samples per Map = 100
Wrote input for Map #0
......
Wrote input for Map #999
Starting Job
20/11/02 18:57:41 INFO mapreduce.Job: Running job: job_1604307167678_0004
20/11/02 18:57:49 INFO mapreduce.Job: Job job_1604307167678_0004 running in uber mode : false
20/11/02 18:57:49 INFO mapreduce.Job:  map 0% reduce 0%
20/11/02 18:57:59 INFO mapreduce.Job:  map 5% reduce 0%
......
20/11/02 18:59:07 INFO mapreduce.Job:  map 100% reduce 0%
20/11/02 18:59:09 INFO mapreduce.Job:  map 100% reduce 100%
20/11/02 18:59:09 INFO mapreduce.Job: Job job_1604307167678_0003 completed successfully
20/11/02 18:59:10 INFO mapreduce.Job: Counters: 54
Job Finished in 456.952 seconds
Estimated value of Pi is 3.14120000000000000000