Hadoop-配置历史服务器和日志聚集

配置历史服务器

历史服务器可以查看程序运行的历史情况

[server@hadoop102 hadoop]$ pwd
/opt/module/hadoop-3.1.3/etc/hadoop
[server@hadoop102 hadoop]$ vim mapred-site.xml

<configuration>
        <property>
              <name>mapreduce.framework.name</name>
              <value>yarn</value>
       </property>
        <property>
              <name>mapreduce.jobhistory.address</name>
              <value>hadoop102:10020</value>
       </property>
        <property>
              <name>mapreduce.jobhistory.webapp.address</name>
              <value>hadoop102:19888</value>
       </property>
</configuration>

[server@hadoop102 hadoop]$ xsync mapred-site.xml 

[server@hadoop102 hadoop-3.1.3]$ bin/mapred --daemon start historyserver

配置日志的聚集

日志聚集:应用运行完成以后，将程序运行日志信息上传到HDFS系统上。开启日志聚集可以查看程序运行详情，方便开发调试。

[server@hadoop102 hadoop]$ pwd
/opt/module/hadoop-3.1.3/etc/hadoop
[server@hadoop102 hadoop]$ vim yarn-site.xml 

<configuration>
    <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
    </property>
    <property>
            <name>yarn.resourcemanager.hostname</name>
            <value>hadoop103</value>
    </property>
    <property>
             <name>yarn.nodemanager.vmem-check-enabled</name>
             <value>false</value>
    </property>
    <property>
            <name>yarn.nodemanager.env-whitelist</name>
            <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
    </property>
    <property>
             <name>yarn.log-aggregation-enable</name>
             <value>true</value>
    </property>
    <property>
             <name>yarn.log.server.url</name>
             <value>http://hadoop102:19888/jobhistory/logs</value>
    </property>
    <property>
             <name>yarn.log-aggreation.retain-seconds</name>
             <value>604800</value>
    </property>
</configuration>

[server@hadoop102 hadoop]$ xsync yarn-site.xml 

[server@hadoop103 hadoop-3.1.3]$ pwd
/opt/module/hadoop-3.1.3
[server@hadoop102 hadoop-3.1.3]$ bin/mapred --daemon stop historyserver
[server@hadoop103 hadoop-3.1.3]$ sbin/stop-yarn.sh
[server@hadoop103 hadoop-3.1.3]$ sbin/start-yarn.sh 

[server@hadoop102 hadoop-3.1.3]$ bin/mapred --daemon start historyserver

hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /wcinput /wcoutput

http://hadoop103:8088/cluster