Ranger-Yarn插件安装

作者: 木木与呆呆 | 来源:发表于2018-04-26 00:11 被阅读205次

    Ranger-Yarn插件ranger-0.6.0-yarn-plugin安装到Yarn的所有ResourceManager节点,
    其他的NodeManager节点不需要安装。

    登陆hdfs安装的用户,garrison/zdh1234(用户组hadoop),获取安装包解压安装
    scp /home/backup/ranger/ranger-0.6.0-yarn-plugin.tar.gz .
    tar –zxvf ranger-0.6.0-yarn-plugin.tar.gz
    vi install.properties
    修改的参数如下:

    POLICY_MGR_URL=http://10.43.159.245:6080
    SQL_CONNECTOR_JAR=/usr/share/java/mysql-connector-java.jar
    REPOSITORY_NAME=yarndev
    CUSTOM_USER=garrison
    CUSTOM_GROUP=hadoop
    

    安装Ranger Yarn Plugin,注意: ./enable-yarn-plugin.sh script should be run as root.:
    ./enable-yarn-plugin.sh
    创建完成后,需要重新启动yarn.

    将zdh-245的包考到zdh-240的garrison里面
    scp -r garrison@zdh-245:/home/garrison/ranger-0.6.0-yarn-plugin .
    用root执行安装脚本,并且重新启动yarn.

    Ranger-Admin里注册hdfs plugin的服务
    YARN新建Service,修改如下

    Service Name = yarnpdev
    UserName = garrison
    Password = zdh1234
    YARN REST URL = http://10.43.159.240:8188
    

    然后点击TestConnection,成功即可保存。

    关闭all-queue策略,
    新建root.default策略,给mysql用户提交队列的权限。
    使用mysql用户执行mapreduce任务,给mysql访问hdfs相应目录的权限:

    export JAVA_HOME=/usr/share/java/jdk1.7.0_80
    /home/garrison/hadoop-2.7.1/bin/hadoop jar /home/garrison/hadoop-2.7.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar wordcount -Dmapreduce.job.queuename=default hdfs://gagcluster/usr/wordcout.txt /usr/wordresult_002
    /home/garrison/hadoop-2.7.1/bin/hadoop fs -text /usr/wordresult_002/part-r-00000
    

    没有权限提交作业会报错:
    User usersync cannot submit applications to queue root.default

    Yarn队列权限支持通过capacity schedule queues实现
    yarn-site.xml配置文件中加入配置项:

    <property>
        <name>yarn.resourcemanager.scheduler.class</name>    
        <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
    </property>
    

    capacity-scheduler.xml配置如下,只允许garrison用户提交作业:

    <property>
      <name>yarn.scheduler.capacity.root.acl_submit_applications</name>
       <value>garrison</value>
       <description>
         The ACL of who can submit jobs to the root queue.
       </description>
     </property>
     <property>
      <name>yarn.scheduler.capacity.root.acl_administer_queue</name>
      <value>garrison</value>
      <description>
        The ACL of who can administer jobs on the default queue.
      </description>
    </property>
    <property>
      <name>yarn.scheduler.capacity.root.default.acl_submit_applications</name>
      <value>garrison</value>
      <description>
        The ACL of who can submit jobs to the default queue.
      </description>
    </property>
    <property>
      <name>yarn.scheduler.capacity.root.default.acl_administer_queue</name>
      <value>garrison</value>
      <description>
        The ACL of who can administer jobs on the default queue.
      </description>
    </property>
    

    相关文章

      网友评论

        本文标题:Ranger-Yarn插件安装

        本文链接:https://www.haomeiwen.com/subject/mliulftx.html