美文网首页
hive on tez 步骤

hive on tez 步骤

作者: James0824 | 来源:发表于2018-11-02 17:58 被阅读0次

    1.下载tez src

    解压,修改pom.xml,将hadoop.version改为2.7.2,最好用非root用户编译

    mvn clean package -DskipTests=true -Dmaven.javadoc.skip=true

    2.编译成功后,在tez-dist/target目录下,能够发现如下文件

    archive-tmp maven-archiver tez-0.8.4 tez-0.8.4-minimal tez-0.8.4-minimal.tar.gz tez-0.8.4.tar.gz tez-dist-0.8.4-tests.jar

    3.将tez-0.8.4-minimal上传到hdfs上,本例中上传到/tez目录下

    Hadoop fs –put tez-0.8.4-minimal.tar.gz /tez/

    4, 将tez-0.8.4-minimal.tar.gz考到/usr/hive下,并解压到./tez下

    taz –zxvf tez-0.8.4-minimal.tar.gz –C ./tez

    拷贝tez的lib下或者主目录下jar包===》到hive主目录下lib中去

    5.在客户端安装tez-0.8.4-minimal,并且在conf目录下建立tez-site.xml并正确配置

     <property>
       <name>tez.lib.uris</name>
       <value>${fs.defaultFS}/user/tez/tez-0.8.5-minimal.tar.gz</value>
     </property>
    <property>
      <name>tez.use.cluster.hadoop-libs</name>
      <value>true</value>
    </property>
    

    6.在hive的客户端配置环境变量 (可以配置到/etc/profile.d/Hadoop.sh

    export TEZ_HOME=/usr/hive/tez
    
    for jar in `ls ${TEZ_HOME}|grep jar`;do
      export HADOOP_CLASSPATH=${TEZ_HOME}/$jar:${HADOOP_CLASSPATH}
    done
    for jar in `ls ${TEZ_HOME}/lib/|grep jar`;do
     export HADOOP_CLASSPATH=${TEZ_HOME}/lib/$jar:${HADOOP_CLASSPATH}
    done
    

    7.在hive-site.xml中配置参数

     <property>
       <name>hive.user.install.directory</name>
       <value>/user/</value>
       <description>
         If hive (in tez mode only) cannot find a usable hive jar in "hive.jar.directory",
         it will upload the hive jar to "hive.user.install.directory/user.name"
         and use it to run queries.
       </description>
     </property>
     <property>
       <name>hive.execution.engine</name>
       <value>tez</value>
       <description>
         Expects one of [mr, tez, spark].
         Chooses execution engine. Options are: mr (Map reduce, default), tez,     spark.       While MR
       remains the default engine for historical reasons, it is itself a historical engine
       and is deprecated in Hive 2 line. It may be removed without further warning.
       </description>
     </property>
    

    8.hive启动后执行

      set hive.execution.engine=tez;
    

     即可运行hive on tez任务。

    补充:
     hive默认计算引擎为tez
     编辑/usr/hive/conf/hive-site.xml

    相关文章

      网友评论

          本文标题:hive on tez 步骤

          本文链接:https://www.haomeiwen.com/subject/jkicxqtx.html