Ambari2.7.5+HDP3.1.5中Yarn配置fair-scheduler

 

将Yarn的调度策略修改成Fair Scheduler的
A:找到YARN列表，然后找到yarn.resourcemanager.scheduler.class，然后将它的值进行修改，即：

<property>
  <name>yarn.resourcemanager.scheduler.class</name>
  <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
</property>

对应的Ambari中的界面：

 

还需要配置其它参数：

yarn.scheduler.capacity.ordering-policy.priority-utilization.underutilized-preemption.enabled=false
yarn.scheduler.fair.assignmultiple=false
yarn.scheduler.fair.sizebasedweight=false
yarn.scheduler.fair.user-as-default-queue=true
yarn.scheduler.fair.preemption=false

yarn.scheduler.fair.allocation.file=fair-scheduler.xml
yarn.resourcemanager.reservation-system.enable=true

yarn.scheduler.fair.sizebasedweight=true
yarn.scheduler.capacity.ordering-policy.priority-utilization.underutilized-preemption.enabled=true

再次还需要在页面上关闭Pre-emption，如图：

最后还需要去配置文件夹（/etc/hadoop/conf, /usr/hdp/3.1.5.0-152/hadoop/conf/这个和前面是同一个目录）

新建fair-scheduler.xml，修改对应权限。

touch fair-scheduler.xml
chown yarn:hadoop fair-scheduler.xml
#chown hdfs:hadoop fair-scheduler.xml
chmod 644 fair-scheduler.xml

vim fair-scheduler.xml (该文件来源https://hadoop.apache.org/docs/r3.1.1/hadoop-yarn/hadoop-yarn-site/FairScheduler.html,修改了默认queue)

<?xml version="1.0"?>
<allocations>
  <queue name="sample_queue">
    <minResources>1024 mb,0vcores</minResources>
    <maxResources>15368 mb,9vcores</maxResources>
    <maxRunningApps>50</maxRunningApps>
    <maxAMShare>0.1</maxAMShare>
    <weight>2.0</weight>
    <schedulingPolicy>fair</schedulingPolicy>
    <queue name="sample_sub_queue">
      <aclSubmitApps>charlie</aclSubmitApps>
      <minResources>5000 mb,0vcores</minResources>
    </queue>
    <queue name="sample_reservable_queue">
      <reservation></reservation>
    </queue>
  </queue>

  <queueMaxAMShareDefault>0.5</queueMaxAMShareDefault>
  <queueMaxResourcesDefault>40000 mb,9vcores</queueMaxResourcesDefault>

  <!-- Queue 'secondary_group_queue' is a parent queue and may have
              user queues under it -->
  <queue name="secondary_group_queue" type="parent">
  <weight>3.0</weight>
  <maxChildResources>4096 mb,4vcores</maxChildResources>
  </queue>

  <user name="sample_user">
    <maxRunningApps>30</maxRunningApps>
  </user>
  <userMaxAppsDefault>5</userMaxAppsDefault>

  <queuePlacementPolicy>
    <rule name="specified" />
    <rule name="primaryGroup" create="false" />
    <rule name="nestedUserQueue">
        <rule name="secondaryGroupExistingQueue" create="false" />
    </rule>
    <rule name="default" queue="root"/>
  </queuePlacementPolicy>
</allocations>

把对应 fair-scheduler.xml scp到其它机器上去。

scp fair-scheduler.xml hadoop166:/etc/hadoop/conf/
scp fair-scheduler.xml hadoop168:/etc/hadoop/conf/
scp fair-scheduler.xml hadoop171:/etc/hadoop/conf/

 

 

其它重点情况

一：重启中有加载已有的应用状态异常的。导致不能正常启动ResourceManager的。需要修改

yarn.resourcemanager.recovery.enabled 修改成false

#cat /etc/hadoop/conf/yarn-site.xml |grep yarn.resourcemanager.recovery.enabled -C 3
    
    <property>
      <name>yarn.resourcemanager.recovery.enabled</name>
      <value>false</value>
    </property>

二：修改 fair-scheduler.xml请注意

反正我是一不小心就各种报错。

三：yarn刷新Queues

yarn rmadmin -refreshQueues