美文网首页
10.Hadoop:Hive环境搭建

10.Hadoop:Hive环境搭建

作者: 負笈在线 | 来源:发表于2020-07-03 05:58 被阅读0次

本节主要内容:

Hive环境搭建

1.系统环境:

OS:CentOS Linux release 7.5.1804 (Core)

CPU:2核心

Memory:1GB

运行用户:root

JDK版本:1.8.0_252

Hadoop版本:cdh5.16.2

2.集群各节点角色规划为:

172.26.37.245 node1.hadoop.com---->namenode,zookeeper,journalnode,hadoop-hdfs-zkfc,resourcenode,historyserver,hbase,hbase-master,hive,hive-metastore,hive-server2,hive-hbase

172.26.37.246 node2.hadoop.com---->datanode,zookeeper,journalnode,nodemanager,hadoop-client,mapreduce,hbase-regionserver

172.26.37.247  node3.hadoop.com---->datanode,nodemanager,hadoop-client,mapreduce,hive,mysql-server

172.26.37.248  node4.hadoop.com---->namenode,zookeeper,journalnode,hadoop-hdfs-zkfc,hive,hive-server2

3.环境说明:

本次追加部署

172.26.37.245 node1.hadoop.com---->hive,hive-metastore,hive-server2,hive-hbase,mysql-connector-java

172.26.37.246 node2.hadoop.com---->

172.26.37.246 node2.hadoop.com---->hive,mysql-server

172.26.37.248  node4.hadoop.com---->hive,hive-server2

一.安装

node1节点:

         # yum install -y hive hive-metastore hive-server2 hive-hbase

node3节点:

         # yum install -y hive mysql-server

node4节点:

         # yum install -y hive hive-server2

二.数据库初始配置

启动mysql-server

         # service mysqld start

mysql-server开机自启动

         # chkconfig mysqld on

初始mysql-server配置(密码:123456)

         # /usr/bin/mysql_secure_installation

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MySQL

      SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!

In order to log into MySQL to secure it, we'll need the current

password for the root user.  If you've just installed MySQL, and

you haven't set the root password yet, the password will be blank,

so you should just press enter here.

Enter current password for root (enter for none):

OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MySQL

root user without the proper authorisation.

Set root password? [Y/n] Y

New password:

Re-enter new password:

Password updated successfully!

Reloading privilege tables..

... Success!

By default, a MySQL installation has an anonymous user, allowing anyone

to log into MySQL without having to have a user account created for

them.  This is intended only for testing, and to make the installation

go a bit smoother.  You should remove them before moving into a

production environment.

Remove anonymous users? [Y/n] Y

... Success!

Normally, root should only be allowed to connect from 'localhost'.  This

ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] N

... skipping.

By default, MySQL comes with a database named 'test' that anyone can

access.  This is also intended only for testing, and should be removed

before moving into a production environment.

Remove test database and access to it? [Y/n] Y

- Dropping test database...

ERROR 1008 (HY000) at line 1: Can't drop database 'test'; database doesn't exist

... Failed!  Not critical, keep moving...

- Removing privileges on test database...

... Success!

Reloading the privilege tables will ensure that all changes made so far

will take effect immediately.

Reload privilege tables now? [Y/n] Y

... Success!

All done!  If you've completed all of the above steps, your MySQL

installation should now be secure.

Thanks for using MySQL!

Cleaning up...

三.安装mysql JDBC驱动

node1、node3、node4节点,metastore要连接数据库,所以要安装jdbc驱动

         # yum -y install mysql-connector-java

将驱动链接到hive的lib库里

         # ln -s /usr/share/java/mysql-connector-java.jar /usr/lib/hive/lib/mysql-connector-java.jar

创建hive所用文件夹(node1节点)

         # sudo -u hdfs hadoop fs -mkdir -p /user/hive/warehouse

         # sudo -u hdfs hadoop fs -chmod -R 1777 /user

         # sudo -u hdfs hadoop fs -chown -R hive /user/hive

         # sudo -u hdfs hadoop fs -chmod -R 1777 /

五.配置hive(node1、node3、node4节点)

         # cp -p /usr/lib/hive/conf/hive-site.xml /usr/lib/hive/conf/hive-site.xml.20200703

         # vi /usr/lib/hive/conf/hive-site.xml

         变更为:

<configuration>

  <property>

    <name>javax.jdo.option.ConnectionURL</name>

    <value>jdbc:mysql://node3.hadoop.com/metastore</value>

    <description>the URL of the MySQL database</description>

  </property>

  <property>

    <name>javax.jdo.option.ConnectionDriverName</name>

    <value>com.mysql.jdbc.Driver</value>

  </property>

  <property>

    <name>javax.jdo.option.ConnectionUserName</name>

<value>hive</value>

  </property>

  <property>

    <name>javax.jdo.option.ConnectionPassword</name>

    <value>hive</value>

  </property>

  <property>

    <name>datanucleus.autoCreateSchema</name>

    <value>false</value>

  </property>

  <property>

    <name>datanucleus.fixedDatastore</name>

    <value>true</value>

  </property>

  <property>

    <name>datanucleus.autoStartMechanism</name>

    <value>SchemaTable</value>

  </property>

  <property>

    <name>hive.metastore.uris</name>

    <value>thrift://172.26.37.245:9083</value>

    <description>IP address (or fully-qualified domain name) and port of the metastore host</description>

  </property>

  <property>

    <name>hive.metastore.schema.verification</name>

    <value>true</value>

  </property>

</configuration>

六.配置hive-server2(node1、node3、node4节点)

         # vi /usr/lib/hive/conf/hive-site.xml

         增加以下内容:

<property>

  <name>hive.support.concurrency</name>

  <description>Enable Hive's Table Lock Manager Service</description>

  <value>true</value>

</property>

<property>

  <name>hive.zookeeper.quorum</name>

  <description>Zookeeper quorum used by Hive's Table Lock Manager</description>

  <value>node1.hadoop.com,node2.hadoop.com,node4.hadoop.com</value>

</property>

七.数据库中创建metastore需要的用户和库(node3节点)

         # mysql -uroot -p123456

         #创建hive用户

mysql> CREATE USER 'hive'@'%' IDENTIFIED BY 'hive';

mysql> REVOKE ALL PRIVILEGES, GRANT OPTION FROM 'hive'@'%';

mysql> GRANT ALL PRIVILEGES ON metastore.* TO 'hive'@'%';

mysql> grant all on *.* to 'hive'@'%' identified by 'hive';

mysql> FLUSH PRIVILEGES;

mysql> quit;

创建metastore数据库

mysql> CREATE DATABASE metastore;

八.初始化metastore数据库

         # mysql -uroot -p123456

mysql> USE metastore;

mysql> source /usr/lib/hive/scripts/metastore/upgrade/mysql/hive-schema-1.1.0.mysql.sql;

mysql> update  VERSION set SCHEMA_VERSION_V2 = '1.1.0-cdh5.12.0' where VER_ID=1;

九.启动服务

Node1节点

         # service hive-metastore start

         # service hive-metastore status

Node1、Node4节点

         # service hive-server2 start

         # service hive-server2 status

Node1节点

         # ./hive --service metastore &

十.验证安装

         # hive

Logging initialized using configuration in file:/etc/hive/conf.dist/hive-log4j.properties

WARNING: Hive CLI is deprecated and migration to Beeline is recommended.

hive> show tables;

OK

Time taken: 4.26 seconds

hive> show databases;

OK

default

Time taken: 0.086 seconds, Fetched: 1 row(s)

hive>

相关文章

  • Hive安装与简单使用并集成SparkSQL

    Hive环境搭建 hive下载:http://archive-primary.cloudera.com/cdh5/...

  • Hive搭建

    Hive搭建 系统环境: Ubuntu 16 Java 8 Hadoop 2.7 Hive 1.2 MySQL 按...

  • Spark SQL:使用数据源之使用Hive Table

    一.使用Hive Table(把Hive中的数据,读取到Spark SQL 中) 1.首先,搭建Hive的环境(分...

  • hive 相关

    hive 相关 搭建hadoop和hive,mysql的环境,过程截图 1.hadoop install 2.m...

  • 搭建Hive环境

    下载hive:wget http://archive.cloudera.com/cdh5/cdh/5/hive-1...

  • 搭建hive环境

    1.安装java 安装成功后,使用下面的命令来验证是否已经安装java 如果安装成功,则可以看到如下回应: bas...

  • hive环境搭建

    Hive2.1.1安装部署 一、Hive 运行模式 与Hadoop类似,Hive也有 3 种运行模式: 1. 内嵌...

  • Hive环境搭建

    介绍 Hive是运行在Hadoop之上的数据仓库,将结构化的数据文件映射为一张数据库表,提供简单类SQL查询语言,...

  • Hive环境搭建

    hadoop各个节点角色定义 master 和 slavename node 和 data node: name ...

  • Hive环境搭建

    Hive的安装需要提前安装Hadoop和MySQL,Hadoop的安装请参考Hadoop环境搭建,下面是MySQL...

网友评论

      本文标题:10.Hadoop:Hive环境搭建

      本文链接:https://www.haomeiwen.com/subject/zwleqktx.html