通过调用yarn资源调度管理器提供的一些服务，感知集群即时的状态

目录
1.获取集群的状态信息
2.获取集群任务的整体状态
3.获取提交到集群所有任务的运行信息
4.获取提交到集群的单个任务的运行信息
5.获取当前资源调度的分配信息

image.png

1.获取集群的状态信息

http://192.168.10.22:8088/ws/v1/cluster/info

<clusterInfo>
<id>1593249986529</id>
#表示集群最近启动时间
<startedOn>1593249986529</startedOn>
#当前集群状态,STARTED表示当前集群为启动状态
<state>STARTED</state>
<haState>ACTIVE</haState>
<rmStateStoreName>org.apache.hadoop.yarn.server.resourcemanager.recovery.NullRMStateStore</rmStateStoreName>
#ResourceManager所用的源码包的版本
<resourceManagerVersion>2.7.2</resourceManagerVersion>
<resourceManagerBuildVersion>2.7.2 from Unknown by root source checksum c63f7cc71b8f63249e35126f0f7492d</resourceManagerBuildVersion>
<resourceManagerVersionBuiltOn>2017-05-22T11:00Z</resourceManagerVersionBuiltOn>
#当前集群所用的hadoop版本
<hadoopVersion>2.7.2</hadoopVersion>
<hadoopBuildVersion>2.7.2 from Unknown by root source checksum d0fda26633fa762bff87ec759ebe689c</hadoopBuildVersion>
<hadoopVersionBuiltOn>2017-05-22T10:49Z</hadoopVersionBuiltOn>
#另外一种状态是结合zk做高可用，显示为CONNECTED，现在这种状态是没有做高可用
<haZooKeeperConnectionState>ResourceManager HA is not enabled.</haZooKeeperConnectionState>
</clusterInfo>

上面的的监控很难做到自动化监控

既要让程序访问链接并做自动化解析监控

使用Python获取集群状态的代码案例

#需要安装request包，安装方式pip install requests
import requests
import json
cluster_status_url = "http://192.168.10.22:8088/ws/v1/cluster"
#通过HTTP请求方式获取链接的位置
res=request.get(clustr_status_url)
print res.content
#loads方法将字符串转化为JSON对象
ct=json.loads(res.content)
#获取集群的状态
print ct.get('clusterInfo').get('state')

2.获取集群任务的整体状态

http://192.168.10.22:8088/ws/v1/cluster/metrics

<clusterMetrics>
<appsSubmitted>0</appsSubmitted>
<appsCompleted>0</appsCompleted>
#当前等待运行的作业
<appsPending>0</appsPending>
#当前正在运行的作业
<appsRunning>0</appsRunning>
<appsFailed>0</appsFailed>
<appsKilled>0</appsKilled>
<reservedMB>0</reservedMB>
#当前可用内存
<availableMB>24576</availableMB>
<allocatedMB>0</allocatedMB>
<reservedVirtualCores>0</reservedVirtualCores>
#当前可用的虚拟核心
<availableVirtualCores>24</availableVirtualCores>
<allocatedVirtualCores>0</allocatedVirtualCores>
<containersAllocated>0</containersAllocated>
<containersReserved>0</containersReserved>
<containersPending>0</containersPending>
#总内存
<totalMB>24576</totalMB>
#总的虚拟核心数
<totalVirtualCores>24</totalVirtualCores>
<totalNodes>3</totalNodes>
<lostNodes>0</lostNodes>
<unhealthyNodes>0</unhealthyNodes>
<decommissionedNodes>0</decommissionedNodes>
<rebootedNodes>0</rebootedNodes>
<activeNodes>3</activeNodes>
</clusterMetrics>

3.获取提交到集群所有任务的运行信息

http://192.168.10.22:8088/ws/v1/cluster/apps
● startedTimeBegin：查询从指定时间开始运行的所有任务，格式为毫秒时间戳格式。
● startedTimeEnd：查询指定时间结束之前开始运行的所有任务，格式为毫秒时间戳格式。
● finishedTimeBegin：查询从指定时间开始结束运行的所有任务，格式为毫秒时间戳格式。
● finishedTimeEnd：查询指定时间结束运行的所有任务，格式为毫秒时间戳格式。
如果指定startedTimeBegin、startedTimeEnd、finishedTimeBegin和finishedTimeEnd，那么结果会返回开始时间介于startedTimeBegin和startedTimeEnd之间，或结束时间介于finishedTimeBegin和finishedTimeEnd之间的作业信息。

<apps>
<app>
<id>application_1593249986529_0001</id>
#● user：查询指定用户提交的所有作业。
<user>root</user>
<name>WordCountMain</name>
#● queue：查询提交到指定队列的所有任务。
<queue>default</queue>
#● states：查询处于特定状态的所有任务。
<state>FINISHED</state>
#● finalStatus：查询处于final状态的任务。
<finalStatus>SUCCEEDED</finalStatus>
<progress>100.0</progress>
<trackingUI>History</trackingUI>
<trackingUrl>http://node02:8088/proxy/application_1593249986529_0001/</trackingUrl>
<diagnostics/>
<clusterId>1593249986529</clusterId>
<applicationType>MAPREDUCE</applicationType>
<applicationTags/>
<startedTime>1593413473740</startedTime>
<finishedTime>1593413519074</finishedTime>
<elapsedTime>45334</elapsedTime>
<amContainerLogs>http://node03:8042/node/containerlogs/container_1593249986529_0001_01_000001/root</amContainerLogs>
<amHostHttpAddress>node03:8042</amHostHttpAddress>
<allocatedMB>-1</allocatedMB>
<allocatedVCores>-1</allocatedVCores>
<runningContainers>-1</runningContainers>
<memorySeconds>132403</memorySeconds>
<vcoreSeconds>75</vcoreSeconds>
<preemptedResourceMB>0</preemptedResourceMB>
<preemptedResourceVCores>0</preemptedResourceVCores>
<numNonAMContainerPreempted>0</numNonAMContainerPreempted>
<numAMContainerPreempted>0</numAMContainerPreempted>
</app>
</apps>

通过定时增量拉取这个服务的信息，可以获得作业提交在一天内或者指定周期内的分布和资源需求的分布，有助于判断指定时间内提交作业集群是否有足够的资源去支撑新增作业的提交。如果不足，可以考虑更改提交的时间段或者增加集群资源。

4.获取提交到集群的单个任务的运行信息

http://192.168.10.22:8088/ws/v1/cluster/apps/任务ID

<app>
<id>application_1593249986529_0001</id>
<user>root</user>
<name>WordCountMain</name>
<queue>default</queue>
<state>FINISHED</state>
<finalStatus>SUCCEEDED</finalStatus>
<progress>100.0</progress>
<trackingUI>History</trackingUI>
<trackingUrl>http://node02:8088/proxy/application_1593249986529_0001/</trackingUrl>
<diagnostics/>
<clusterId>1593249986529</clusterId>
<applicationType>MAPREDUCE</applicationType>
<applicationTags/>
<startedTime>1593413473740</startedTime>
<finishedTime>1593413519074</finishedTime>
<elapsedTime>45334</elapsedTime>
<amContainerLogs>http://node03:8042/node/containerlogs/container_1593249986529_0001_01_000001/root</amContainerLogs>
<amHostHttpAddress>node03:8042</amHostHttpAddress>
<allocatedMB>-1</allocatedMB>
<allocatedVCores>-1</allocatedVCores>
<runningContainers>-1</runningContainers>
<memorySeconds>132403</memorySeconds>
<vcoreSeconds>75</vcoreSeconds>
<preemptedResourceMB>0</preemptedResourceMB>
<preemptedResourceVCores>0</preemptedResourceVCores>
<numNonAMContainerPreempted>0</numNonAMContainerPreempted>
<numAMContainerPreempted>0</numAMContainerPreempted>
</app>

5.获取当前资源调度的分配信息

http://192.168.10.22:8088/ws/v1/cluster/scheduler
可以获取到当前层级调度的所有信息，包括当前队列可用虚拟CPU个数、可用内存、最大CPU个数，最大内存等信息。
不同的调度器 fair/FIFO/Scheduler显示内容可能不会一样。

<scheduler>
<schedulerInfo xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="capacityScheduler">
<capacity>100.0</capacity>
<usedCapacity>0.0</usedCapacity>
<maxCapacity>100.0</maxCapacity>
<queueName>root</queueName>
<queues>
<queue xsi:type="capacitySchedulerLeafQueueInfo">
<capacity>100.0</capacity>
<usedCapacity>0.0</usedCapacity>
<maxCapacity>100.0</maxCapacity>
<absoluteCapacity>100.0</absoluteCapacity>
<absoluteMaxCapacity>100.0</absoluteMaxCapacity>
<absoluteUsedCapacity>0.0</absoluteUsedCapacity>
<numApplications>0</numApplications>
<queueName>default</queueName>
<state>RUNNING</state>
<resourcesUsed>
<memory>0</memory>
<vCores>0</vCores>
</resourcesUsed>
<hideReservationQueues>false</hideReservationQueues>
<nodeLabels>*</nodeLabels>
<numActiveApplications>0</numActiveApplications>
<numPendingApplications>0</numPendingApplications>
<numContainers>0</numContainers>
<maxApplications>10000</maxApplications>
<maxApplicationsPerUser>10000</maxApplicationsPerUser>
<userLimit>100</userLimit>
<users/>
<userLimitFactor>1.0</userLimitFactor>
<AMResourceLimit>
<memory>3072</memory>
<vCores>1</vCores>
</AMResourceLimit>
<usedAMResource>
<memory>0</memory>
<vCores>0</vCores>
</usedAMResource>
<userAMResourceLimit>
<memory>3072</memory>
<vCores>1</vCores>
</userAMResourceLimit>
<preemptionDisabled>true</preemptionDisabled>
</queue>
</queues>
</schedulerInfo>
</scheduler>