美文网首页
Yarn Scheduler

Yarn Scheduler

作者: 踏雪寻梅4149 | 来源:发表于2017-09-06 20:34 被阅读0次

Yarn Scheduler

https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html
https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html

How to find the scheduler in used on the existing cluster?

There are 2 ways:

  1. Through Resource Manager Web UI ->Scheduler, e.g. http://vm-slaver1:8088/cluster/scheduler
  2. Check the configuration yarn.resourcemanager.scheduler.class in yarn-site.xml. If not found, go to check the yarn-default.xml corresponding to the version.
    Example for Capacity Scheduler
    capacity-scheduler.xml put the same directory to yarn-site.xml
 
<!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. -->
<configuration>

  <property>
    <name>yarn.scheduler.capacity.maximum-applications</name>
    <value>10000</value>
    <description> Maximum number of applications that can be pending and running. </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
    <value>0.1</value>
    <description> Maximum percent of resources in the cluster which can be used to run application masters i.e. controls number of concurrent running applications. </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.resource-calculator</name>
    <value>org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator</value>
    <description> The ResourceCalculator implementation to be used to compare Resources in the scheduler. The default i.e. DefaultResourceCalculator only uses Memory while DominantResourceCalculator uses dominant-resource to compare multi-dimensional resources such as Memory, CPU etc. </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.queues</name>
    <value>q0</value>
    <description> The queues at the this level (root is the root queue). </description>
  </property>

   <property>
     <name>yarn.scheduler.capacity.root.q0.queues</name>
     <value>q01,q02</value>
   </property>      

  <property>
    <name>yarn.scheduler.capacity.root.q0.capacity</name>
    <value>100</value>
    <description>Default queue target capacity.</description>
  </property>

 <property>
    <name>yarn.scheduler.capacity.root.q0.q01.capacity</name>
    <value>60</value>
    <description>Default queue target capacity.</description>
  </property>

 <property>
    <name>yarn.scheduler.capacity.root.q0.q02.capacity</name>
    <value>40</value>
    <description>Default queue target capacity.</description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.q0.user-limit-factor</name>
    <value>1</value>
    <description> Default queue user limit a percentage from 0.0 to 1.0. </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.q0.maximum-capacity</name>
    <value>100</value>
    <description> The maximum capacity of the default queue. </description>
  </property>

 <property>
    <name>yarn.scheduler.capacity.root.q0.q01.maximum-capacity</name>
    <value>80</value>
    <description> The maximum capacity of the default queue. </description>
  </property>

<property>
    <name>yarn.scheduler.capacity.root.q0.q02.maximum-capacity</name>
    <value>80</value>
    <description> The maximum capacity of the default queue. </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.q0.state</name>
    <value>RUNNING</value>
    <description> The state of the default queue. State can be one of RUNNING or STOPPED. </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.q0.q01.state</name>
    <value>RUNNING</value>
    <description> The state of the default queue. State can be one of RUNNING or STOPPED. </description>
  </property>

 <property>
    <name>yarn.scheduler.capacity.root.q0.q02.state</name>
    <value>STOPPED</value>
    <description> The state of the default queue. State can be one of RUNNING or STOPPED. </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.q0.acl_submit_applications</name>
    <value>*</value>
    <description> The ACL of who can submit jobs to the default queue. </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.q0.acl_administer_queue</name>
    <value>*</value>
    <description> The ACL of who can administer jobs on the default queue. </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.node-locality-delay</name>
    <value>40</value>
    <description> Number of missed scheduling opportunities after which the CapacityScheduler attempts to schedule rack-local containers. Typically this should be set to number of nodes in the cluster, By default is setting approximately number of nodes in one rack which is 40. </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.queue-mappings</name>
    <value></value>
    <description> A list of mappings that will be used to assign jobs to queues The syntax for this list is [u|g]:[name]:[queue_name][,next mapping]* Typically this list will be used to map users to queues, for example, u:%user:%user maps all users to queues with the same name as the user. </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.queue-mappings-override.enable</name>
    <value>false</value>
    <description> If a queue mapping is present, will it override the value specified by the user? This can be used by administrators to place jobs in queues that are different than the one specified by the user. The default is false. </description>
  </property>

</configuration>

Tips:
The sum of capacities for all queues, at each level, must be equal to 100. e.g. q0.q01+q0.q02=100

Example for Fair Scheduler

The default configuration is under the same directory to yarn-site.xml with name fair-scheduler.xml. You can specify it by yarn.scheduler.fair.allocation.file in yarn-site.xml.

examples: fair-scheduler-allocations.xml

<?xml version="1.0"?>
<allocations>
  <queue name="q0">
    <minResources>10000 mb,0vcores</minResources>
    <maxResources>90000 mb,1vcores</maxResources>
    <maxRunningApps>50</maxRunningApps>
    <maxAMShare>0.1</maxAMShare>
    <weight>2.0</weight>
    <schedulingPolicy>fair</schedulingPolicy>
    <queue name="q01">
      <aclSubmitApps>hadoop</aclSubmitApps>
      <minResources>5000 mb,0vcores</minResources>
    </queue>
  </queue>
 
  <queueMaxAMShareDefault>0.5</queueMaxAMShareDefault>
 
  <queue name="q1" type="parent">
  <weight>3.0</weight>
  </queue>
 
  <user name="hadoop">
    <maxRunningApps>30</maxRunningApps>
  </user>
  <userMaxAppsDefault>5</userMaxAppsDefault>
 
  <queuePlacementPolicy>
    <rule name="specified" />
    <rule name="primaryGroup" create="false" />
    <rule name="nestedUserQueue">
        <rule name="secondaryGroupExistingQueue" create="false" />
    </rule>
    <rule name="default" queue="q0"/>
  </queuePlacementPolicy>
</allocations>

Tips:
The name of queue you set in your applciation should be a leaf queue with path name ( with or without root is ok) e.g. q0.q01 or root.q0.q1
![yarn-img-1](/Users/canhuamei/Desktop/screamshot/yarn-1.png

)

(注:但是后来在spark research的时候,发现spark里面只要直接设置叶子节点名即可,sparkConf.set("spark.yarn.queue","q01");

HelloWorld for Yarn Application

相关文章

  • Yarn调度器

    在Yarn中有三种调度器可以选择:FIFO Scheduler,Capacity Scheduler,FairS ...

  • Yarn Scheduler

    Yarn Scheduler https://hadoop.apache.org/docs/current/had...

  • YARN的三种调度器

    在Yarn中有三种调度器可以选择:FIFO Scheduler ,Capacity Scheduler,FairS...

  • YARN调度器的选择

    在Yarn中有三种调度器可以选择:FIFO Scheduler ,Capacity Scheduler,FairS...

  • Yarn 队列设置

    设置:yarn.resourcemanager.scheduler.class Capacity Schedule...

  • Yarn中调度器介绍

    Yarn中主要包含3中调度器,FIFO Scheduler(先进先出调度器),Capacity Scheduler...

  • Yarn ACL(fair-scheduler)

    yarn-site.xml 添加如下内容 fair-scheduler.xml 完整的demo 参考资料YARN ...

  • Yarn Capacity Scheduler

    在Yarn中使用scheduler为不同的application分配资源,hadoop yarn的调用策略以可插拔...

  • YARN基础二:资源调度器

    1、Yarn调度器 在Yarn基础一的最后,我们谈到Yarn有三种调度器。分别是 FIFO Scheduler:F...

  • Yarn Capacity Scheduler

    Yarn有FIFO Capacity FairS三种调度器 Capacity是Yarn的默认调度器 Capacit...

网友评论

      本文标题:Yarn Scheduler

      本文链接:https://www.haomeiwen.com/subject/hmwvjxtx.html