美文网首页大数据
Hive 3.1.x 安装

Hive 3.1.x 安装

作者: 陈sir的知识图谱 | 来源:发表于2020-02-15 11:52 被阅读0次

    Hive 安装主要有4个事情

    1. 安装meta数据存储,mysql
    2. 解压缩安装包配置环境变量
    3. 初始化 schema
    4. 配置hive并启动

    安装步骤

    安装环境

    hive 安装在hdfs的namenode节点中,这里,安装在hadoop301上。
    具体信息,参看hadoop 3.2.x 高可用集群搭建

    创建相关的HDFS 文件夹

    $HADOOP_HOME/bin/hadoop fs -mkdir /tmp
    $HADOOP_HOME/bin/hadoop fs -mkdir -p /user/hive/warehouse
    $HADOOP_HOME/bin/hadoop fs -chmod g+w /tmp
    $HADOOP_HOME/bin/hadoop fs -chmod g+w /user/hive/warehouse
    

    安装meta数据存储,mysql

    yum install -y mysql-server.x86_64
    # mysql 命令进入mysql数据库, 注意,因为是centos8 所以装出来的mysql 是8.0的版本,grant语句授权与5.7 有区别。
    mysql>use mysql;
    mysql>create user 'root'@‘%’ identified by 'openstack';
    mysql>grant all privileges on *.* to 'root'@'%' with grant option;
    mysql>flush privileges;
    mysql>exit;
    systemctl start mysqld
    

    点单说明一下 . 表示对所有数据库,所有数据库中的所有表授权。
    mysql 5.7 的授权语句是 grant all privileges on *.* to 'root'@'%' identified by 'openstack'; 语法是username@hostname, %代表任何主机

    解压安装包并配置环境变量

    tar -zxf hive-3.1.2.tar.gz
    mv hive-3.1.2 /opt/hive
    # 在/etc/profile 中添加或者在~/.bashrc中添加
    export HIVE_HOME=/opt/hive
    export HIVE_CONF_DIR=/opt/hive/conf
    export PATH=$PATH:$HIVE_HOME/bin
    

    初始化meta数据库

    vim /opt/hive/conf/hive-site
    

    配置内容如下:

    <?xml version="1.0" encoding="UTF-8" standalone="no"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    <configuration>
      <property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>openstack</value>
        <description>password to use against metastore database</description>
      </property>
    <property>  
      <name>hive.metastore.local</name> 
      <value>true</value>  
    </property>  
      <property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://192.168.142.101:3306/hive?createDatabaseIfNotExsit=true&amp;useSSL=false&amp;serverTimezone=UTC</value>
        <description>
          JDBC connect string for a JDBC metastore.
          To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL.
          For example, jdbc:postgresql://myhost/db?ssl=true for postgres database.
        </description>
      </property>
      <property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.cj.jdbc.Driver</value>
        <description>Driver class name for a JDBC metastore</description>
      </property>
      <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>root</value>
        <description>Username to use against metastore database</description>
      </property>
      <property>
        <name>hive.metastore.schema.verification</name>
        <value>true</value>
        <description>
          Enforce metastore schema version consistency.
          True: Verify that version information stored in is compatible with one from Hive jars.  Also disable automatic
                schema migration attempt. Users are required to manually migrate schema after Hive upgrade which ensures
                proper metastore schema migration. (Default)
          False: Warn if the version information stored in metastore doesn't match with one from in Hive jars.
        </description>
      </property>
      <property>
        <name>hive.metastore.schema.verification.record.version</name>
        <value>true</value>
        <description>
          When true the current MS version is recorded in the VERSION table. If this is disabled and verification is
           enabled the MS will be unusable.
        </description>
      </property>
    </configuration>
    

    注意这里有3个坑
    1 是hive.metastore.schema.verification 配置为true 有可能报错 unknow version,如果遇到改为false即可。
    2 是hive 文档说默认会读取hive-default.xml的配置,这个默认配置,在hive-default.xml.template 中有完整定义,当你吧hive-default.xml.template 复制一份并改名为 hive-default.xml时。会发现hive-default.xml的配置根本不生效。生效的配置文件名必须时hive-site.xml
    3 hive 和 hadoop 的slf4j 包冲突,错误信息如下,此时删除hive的包 mv /opt/hive/lib/log4j-slf4j-impl-2.10.0.jar /opt/hive/lib/log4j-slf4j-impl-2.10.0.jar.bak
    4 hive 和 hadoop 使用的guava冲突,删除低版本的
    cp /opt/hadoop-3.2.1/share/hadoop/common/lib/guava-27.0-jre.jar /opt/hive/lib
    mv /opt/hive/lib/guava-19.0.jar /opt/hive/lib/guava-19.0.jar.bak

    # 坑3的错误信息
    SLF4J: Class path contains multiple SLF4J bindings.
    SLF4J: Found binding in [jar:file:/opt/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/opt/hadoop-3.2.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
    SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
    Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
            at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357)
            at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338)
            at org.apache.hadoop.mapred.JobConf.setJar(JobConf.java:536)
            at org.apache.hadoop.mapred.JobConf.setJarByClass(JobConf.java:554)
            at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:448)
            at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5141)
            at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:5104)
            at org.apache.hive.beeline.HiveSchemaTool.<init>(HiveSchemaTool.java:96)
            at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1473)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:498)
            at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
            at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
    
    #坑4 的错误信息
    Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
            at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357)
            at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338)
            at org.apache.hadoop.mapred.JobConf.setJar(JobConf.java:536)
            at org.apache.hadoop.mapred.JobConf.setJarByClass(JobConf.java:554)
            at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:448)
            at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5141)
            at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:5104)
            at org.apache.hive.beeline.HiveSchemaTool.<init>(HiveSchemaTool.java:96)
            at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1473)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:498)
            at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
            at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
    
    

    拷贝mysql-connector-java-8.0.18.jar 到 /opt/hive/lib
    配置完成后初始化数据库

    /opt/hive/bin/schematool -dbType mysql -initSchema
    # 在/opt/hive/scripts/metastore/upgrade/mysql 中可以找到对应的初始化脚本,也就意味着可以手工初始化
    

    启动hive

    #方法一
    /opt/hive/bin/hive
    #方法二 
    /opt/hive/bin/beeline -u jdbc:hive2://
    #启动完成后,测试一下
    show tables;
    

    通过如下方式启动metastore 服务,对外开放metaservice,比如给presto,默认端口9083,

    hive --service metastore
    

    以上是metastore 在本地的模式。另外两种metastore 的部署方式见。
    metastore的三种方式

    相关文章

      网友评论

        本文标题:Hive 3.1.x 安装

        本文链接:https://www.haomeiwen.com/subject/ucsefhtx.html