美文网首页
hadoop 3.x 案例1:job history显示问题

hadoop 3.x 案例1:job history显示问题

作者: 只是甲 | 来源:发表于2022-12-08 16:56 被阅读0次

    一. 问题描述

    访问 job history的时候,发现一长串描述,但是没有地方查看对应的日志。

    Logs not available for container_1668064306391_0004_01_000001. Aggregation may not be complete, Check back later or try to find the container logs in the local directory of nodemanager hp6:43050
    
    image.png

    二. 解决方案

    2.1 mapred-site.xml已添加如下配置:

    <property>
            <name>mapreduce.jobhistory.address</name>
            <value>hadoop000:10020</value>
            <description>MapReduce JobHistory Server IPC host:port</description>
    </property>
    
    <property>
            <name>mapreduce.jobhistory.webapp.address</name>
            <value>hadoop000:19888</value>
            <description>MapReduce JobHistory Server Web UI host:port</description>
    </property>
    
    <property>
            <name>mapreduce.jobhistory.done-dir</name>
            <value>/history/done</value>
    </property>
    
    <property>
            <name>mapreduce.jobhistory.intermediate-done-dir</name>
            <value>/history/done_intermediate</value>
    </property>
    

    2.2 日志聚合功能已开启:

    yarn-site.xml

    <property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
    </property>
    

    2.3 job historyserver 已经配置完毕

    [hadoop@hadoop000 hadoop]$ jps | grep -v Jps
    37602 ResourceManager
    4558 SecondaryNameNode
    4271 NameNode
    4372 DataNode
    36310 JobHistoryServer
    37699 NodeManager
    

    参考:

    1. https://coding.imooc.com/learn/questiondetail/88186.html

    相关文章

      网友评论

          本文标题:hadoop 3.x 案例1:job history显示问题

          本文链接:https://www.haomeiwen.com/subject/xymdfdtx.html