Hadoop-3.2.2(HA)完全分布式部署

1、环境准备

• 安装jdk配置环境变量 /etc/hosts等信息
• ssh免密登录
• 关闭防火墙
  检查防火墙状态(如果active(running)那么关闭)
  systemctl status firewalld
  service iptables status
  临时关闭防火墙
  systemctl stop firewalld
  service iptables stop
  永久关闭防火墙
  systemctl disable firewalld
  chkconfig iptables off

2、集群规划

目前使用5台机器、相关的角色分配如下

hadoop1101	hadoop1102	hadoop1103	hadoop1104	hadoop1105
zk	zk	zk	-	-
NameNode	NameNode	NameNode	-	-
zkfc	zkfc	zkfc	-	-
-	-	JournalNode	JournalNode	JournalNode
DataNode	DataNode	DataNode	DataNode	DataNode
-	-	-	ResourceManager	ResourceManager
NodeManager	NodeManager	NodeManager	NodeManager	NodeManager

3 配置Zookeeper集群

• 下载zookeeper
• tar -xvf apache-zookeeper-3.6.2-bin.tar.gz -C /opt/(可换成安装目录我安装在/opt下面)
• 进入安装目录(apache-zookeeper-3.6.2-bin/conf) cp zoo_sample.cfg zoo.cfg 编辑zoo.cfg

image.png

YARN-HA配置

1)、配置 yarn-site.xml,并分发到各个节点上

<configuration>
<property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>

    <!-- 启用resourcemanager ha -->
    <property>
        <name>yarn.resourcemanager.ha.enabled</name>
        <value>true</value>
    </property>

    <!-- 声明2台resourcemanager的地址 -->
    <property>
        <name>yarn.resourcemanager.cluster-id</name>
        <value>cluster-yarn1</value>
    </property>
    <!--指定resourcemanager的逻辑列表-->
    <property>
        <name>yarn.resourcemanager.ha.rm-ids</name>
        <value>rm1,rm2</value>
    </property>

<!-- ========== rm1的配置 ========== -->
    <!-- 指定rm1的主机名 -->
        <property>
            <name>yarn.resourcemanager.hostname.rm1</name>
            <value>hadoop1104</value>
    </property>
    <!-- 指定rm1的web端地址 -->
    <property>
         <name>yarn.resourcemanager.webapp.address.rm1</name>
         <value>hadoop1104:8088</value>
    </property>
    <!-- 指定rm1的内部通信地址 -->
    <property>
         <name>yarn.resourcemanager.address.rm1</name>
         <value>hadoop1104:8032</value>
    </property>
    <!-- 指定AM向rm1申请资源的地址 -->
    <property>
         <name>yarn.resourcemanager.scheduler.address.rm1</name>
         <value>hadoop1104:8030</value>
    </property>
    <!-- 指定供NM连接的地址 -->
    <property>
         <name>yarn.resourcemanager.resource-tracker.address.rm1</name>
         <value>hadoop1104:8031</value>
    </property>
<!-- ========== rm2的配置 ========== -->
    <!-- 指定rm2的主机名 -->
    <property>
        <name>yarn.resourcemanager.hostname.rm2</name>
        <value>hadoop1105</value>
    </property>
    <property>
         <name>yarn.resourcemanager.webapp.address.rm2</name>
         <value>hadoop1105:8088</value>
    </property>
    <property>
         <name>yarn.resourcemanager.address.rm2</name>
         <value>hadoop1105:8032</value>
    </property>
    <property>
         <name>yarn.resourcemanager.scheduler.address.rm2</name>
         <value>hadoop1105:8030</value>
    </property>
    <property>
         <name>yarn.resourcemanager.resource-tracker.address.rm2</name>
         <value>hadoop1105:8031</value>
    </property>

    <!-- 指定zookeeper集群的地址 -->
    <property>
        <name>yarn.resourcemanager.zk-address</name>
        <value>hadoop1101:2181,hadoop1102:2181,hadoop1103:2181</value>
    </property>

    <!-- 启用自动恢复 -->
    <property>
        <name>yarn.resourcemanager.recovery.enabled</name>
        <value>true</value>
    </property>

    <!-- 指定resourcemanager的状态信息存储在zookeeper集群 -->
    <property>
        <name>yarn.resourcemanager.store.class</name>
        <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
    </property>
<!-- 环境变量的继承 -->
 <property>
        <name>yarn.nodemanager.env-whitelist</name>
 <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
 </property>
<!-- Site specific YARN configuration properties -->

</configuration>

2. 、开启yarn
   start-yarn.sh

8088端口查看服务

image.png