基于Yarn的MapReduce集群搭建可参照官方文档 https://hadoop.apache.org/docs/r2.6.5/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html

基础知识

MapReduce
关于MapReduce基础知识，可查阅此篇文章：https://blog.csdn.net/luzhensmart/article/details/90202313
Yarn
关于Yarn基础知识，可查阅此篇文章：https://www.jianshu.com/p/3f406cf438be

服务器准备

本文搭建所用服务器环境是在上篇文章【Hadoop学习笔记三：高可用集群搭建（Hadoop2.x）】https://www.jianshu.com/p/666ff9bbf784 基础上进行的，基于Yarn的MapReduce集群服务器规划方案如下图。

基于Yarn的MapReduce集群.png

一、免密登录

两个ResourceManager节点类似于高可用集群NameNode一个级别，主备之间可能需要进行切换，所以主备节点需要免秘钥登录。按上图所示，需配置Node03和Node04之间免密钥登录。

rm-ha-overview.png

03节点 .ssh 目录下：

ssh-keygen -t dsa -P '' -f ./id_dsa
cat id_dsa.pub >> authorized_keys
scp id_dsa.pub node04:`pwd`/node03.pub

04节点 .ssh 目录下 ：

cat node03.pub >> authorized_keys
ssh-keygen -t dsa -P '' -f ./id_dsa
cat id_dsa.pub >> authorized_keys
scp id_dsa.pub node03:`pwd`/node04.pub

03节点 .ssh 目录下：

cat node04.pub >> authorized_keys

二、配置项

1.mapred-site.xml

#重命名
mv mapred-site.xml.template mapred-site.xml 

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>
#参照 官网Single Node Cluster配置Yarn

2.yarn-site.xml

#配置数据洗牌阶段归于yarn管理
 <property>
   <name>yarn.nodemanager.aux-services</name>
   <value>mapreduce_shuffle</value>
 </property>
#参照 官网Single Node Cluster配置Yarn

#官网给出的最简配置
 <property>
   <name>yarn.resourcemanager.ha.enabled</name>
   <value>true</value>
 </property>
 <property>
   <name>yarn.resourcemanager.cluster-id</name>
   <value>cluster1</value>
 </property>
 <property>
   <name>yarn.resourcemanager.ha.rm-ids</name>
   <value>rm1,rm2</value>
 </property>
 <property>
   <name>yarn.resourcemanager.hostname.rm1</name>
   <value>node03</value>
 </property>
 <property>
   <name>yarn.resourcemanager.hostname.rm2</name>
   <value>node04</value>
 </property>
 <property>
   <name>yarn.resourcemanager.zk-address</name>
   <value>node02:2181,node03:2181,node04:2181</value>
 </property>

3.分发两个文件到：02 03 04节点

scp mapred-site.xml yarn-site.xml node02:`pwd`
scp mapred-site.xml yarn-site.xml node03:`pwd`
scp mapred-site.xml yarn-site.xml node04:`pwd`

三、启动集群

1.启动zookeeper

#02/03/04节点同时执行
zkServer.sh start

2.启动hdfs

#node01 节点启动
start-dfs.sh
#注意，有一个脚本不要用，start-all.sh
#如果nn1和 nn2没有启动，需要在node01，node02分别手动启动：
hadoop-daemon.sh start namenode 

3.启动yarn

#启动NodeManager
 start-yarn.sh
#在03,04节点 启动ResourceManager
yarn-daemon.sh start resourcemanager

浏览器访问： node03:8088 node04:8088

四、停止集群

#node01上执行
stop-dfs.sh 

#node01上停止NodeManager
stop-yarn.sh

#node03，node04停止ResourceManager
yarn-daemon.sh stop resourcemanager 

#node02，node03，node04上停止zk
zkServer.sh stop

五、测试计算能力

#test.txt 是 【Hadoop学习笔记二：全分布式搭建（Hadoop1.x）】 时上传
hadoop jar hadoop-mapreduce-examples-2.6.5.jar wordcount test.txt /wordcount

计算过程可在浏览器查看进度
http://node04:8088/cluster

19/12/01 04:40:46 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
19/12/01 04:40:47 INFO input.FileInputFormat: Total input paths to process : 1
19/12/01 04:40:47 INFO mapreduce.JobSubmitter: number of splits:2
19/12/01 04:40:49 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1575143975107_0004
19/12/01 04:40:49 INFO impl.YarnClientImpl: Submitted application application_1575143975107_0004
19/12/01 04:40:50 INFO mapreduce.Job: The url to track the job: http://node04:8088/proxy/application_1575143975107_0004/
19/12/01 04:40:50 INFO mapreduce.Job: Running job: job_1575143975107_0004
19/12/01 04:41:49 INFO mapreduce.Job: Job job_1575143975107_0004 running in uber mode : false
19/12/01 04:41:49 INFO mapreduce.Job:  map 0% reduce 0%
19/12/01 04:43:30 INFO mapreduce.Job:  map 33% reduce 0%
19/12/01 04:43:31 INFO mapreduce.Job:  map 50% reduce 0%
19/12/01 04:46:44 INFO mapreduce.Job:  map 50% reduce 17%
19/12/01 04:46:46 INFO mapreduce.Job:  map 100% reduce 17%
19/12/01 04:46:50 INFO mapreduce.Job:  map 100% reduce 43%
19/12/01 04:46:53 INFO mapreduce.Job:  map 100% reduce 100%
19/12/01 04:46:55 INFO mapreduce.Job: Job job_1575143975107_0004 completed successfully

结果查看

hdfs dfs -cat /wordcount/part-r-00000