美文网首页
Docker 构建 Hive 1.2.2 单节点

Docker 构建 Hive 1.2.2 单节点

作者: 孙瑞锴 | 来源:发表于2020-06-16 16:47 被阅读0次

    1. 借鉴

    Hadoop集群添加新节点
    为什么Hive里，要用mysql?
    centos7下安装mysql5.7（rpm）
    error: Failed dependencies: perl(Data::Dumper) is needed by MySQL-server-5.6.46-1.el7.x8
    如何在DOCKER容器中运行MySQL服务
    rpm 完全卸载mysql
    Tez 0.9安装部署+hive on tez配置 + Tez-UI

    2. 开始

    我们的节点规划如下：

    hive01[172.173.16.22]

    镜像准备

    hive是依托于hadoop的，所以hive机器上需要hadoop环境

    1. docker hub 下载
      docker pull caiserkaiser/hadoop:2.7.2
    1. 构建
      caiser/hadoop:2.7.2 镜像

    创建自定义网络

    Docker 网络操作

    docker network create -d bridge --subnet "172.173.16.0/24" --gateway "172.173.16.1"  datastore_net

    启动容器

    docker run -it -d --network datastore_net --ip 172.173.16.22 --name hive01 caiser/hadoop:2.7.2

    下载并配置hive

    1. 下载hive

    2. 拷贝到容器内

      docker cp ~/Downloads/apache-hive-1.2.2-bin.tar.gz e2d21b78fdb0:/opt/envs

    3. 配置hadoop环境变量

      a. 解压并重命名

      mkdir hive-1.2.2 && tar -zxvf apache-hive-1.2.2-bin.tar.gz -C hive-1.2.2 --strip-components 1

    配置

    hive-env.sh

    • a. 备份

      cp /opt/envs/hive-1.2.2/conf/hive-env.sh.template /opt/envs/hive-1.2.2/conf/hive-env.sh

    • b. 编辑hive-env.sh

       vi /opt/envs/hive-1.2.2/conf/hive-env.sh

    • c. 配置

      ①. 配置 HADOOP_HOME

       export HADOOP_HOME=/opt/envs/hadoop-2.7.2

      ②. 配置 HIVE_CONF_DIR

      export HIVE_CONF_DIR=/opt/envs/hive-1.2.2/conf

    安装which

    不安装which，启动datanode时会报找不到文件的错误

    yum install which

    安装mysql

    为啥需要mysql

    [借鉴引用]
    你在哪路径下，执行hive指令，就在哪路径下生成metastore_db建一套数据库文件，这样是极其不合适的，公司里每个人若不一样，则会显得非常混杂。导致员工之间无法公用交流，也不便于管理。

    • a . 重要如果安装了mysql，先删除，再重新安装

       rpm -qa | grep -i mysql
 rpm -ev --nodeps xxx

 find / -name mysql
 find / -name my.cnf

 rm -rf  xxx/mysql
 rm -rf /etc/my.cnf

    • b. 下载mysql

       MySQL-client-5.6.48-1.el7.x86_64.rpm
 MySQL-server-5.6.48-1.el7.x86_64.rpm

    • c. 拷贝到容器

       docker cp ~/Downloads/MySQL-client-5.6.48-1.el7.x86_64.rpm e2d21b78fdb0:/opt/envs
 docker cp ~/Downloads/MySQL-server-5.6.48-1.el7.x86_64.rpm e2d21b78fdb0:/opt/envs

    • d. 安装mysql服务端

      ① 安装服务端

      rpm -ivh MySQL-server-5.6.48-1.el7.x86_64.rpm

      ② 出现问题:

      /usr/bin/perl is needed by MySQL-server-5.6.48-1.el7.x86_64
libaio.so.1()(64bit) is needed by MySQL-server-5.6.48-1.el7.x86_64
libaio.so.1(LIBAIO_0.1)(64bit) is needed by MySQL-server-5.6.48-1.el7.x86_64
libaio.so.1(LIBAIO_0.4)(64bit) is needed by MySQL-server-5.6.48-1.el7.x86_64
libnuma.so.1()(64bit) is needed by MySQL-server-5.6.48-1.el7.x86_64
libnuma.so.1(libnuma_1.1)(64bit) is needed by MySQL-server-5.6.48-1.el7.x86_64
libnuma.so.1(libnuma_1.2)(64bit) is needed by MySQL-server-5.6.48-1.el7.x86_64
net-tools is needed by MySQL-server-5.6.48-1.el7.x86_64
perl(Data::Dumper) is needed by MySQL-server-5.6.48-1.el7.x86_64

      ③ 安装以下依赖

       yum install -y perl perl-devel autoconf libaio numactl net-tools iproute

      ④ server安装完成[看到以下内容证明完成了]

      A RANDOM PASSWORD HAS BEEN SET FOR THE MySQL root USER !
You will find that password in '/root/.mysql_secret'.

You must change that password on your first connect,
no other statement but 'SET PASSWORD' will be accepted.
See the manual for the semantics of the 'password expired' flag.

Also, the account for the anonymous user has been removed.

In addition, you can run:

/usr/bin/mysql_secure_installation

which will also give you the option of removing the test database.
This is strongly recommended for production servers.

See the manual for more instructions.

Please report any problems at http://bugs.mysql.com/

The latest information about MySQL is available on the web at

  http://www.mysql.com

Support MySQL by buying support/licenses at http://shop.mysql.com

New default config file was created as /usr/my.cnf and
will be used by default by the server when you start it.
You may edit this file to change server settings

    • e. 查看随机生成的密码

      cat /root/.mysql_secret

    • f. 启动mysql

      ① 启动

      /etc/init.d/mysql start
或 service mysql start

      ② 启动成功日志如下：

      Starting MySQL.Logging to '/var/lib/mysql/e2d21b78fdb0.err'.
SUCCESS!

    • g. 安装mysql客户端

       rpm -ivh MySQL-client-5.6.48-1.el7.x86_64.rpm

    • h. 连接mysql，使用root用户

      mysql -uroot -p

    • i. 修改mysql root用户密码

      set password=password('密码');
quit;

    • j. 修改用户表，将host表的host列修改为任意ip可访问，并删除其他host（否则在后面启动时会报没有权限）

      use mysql;

update user set host = '%' where host = 'localhost';

delete from user where host = 'e2d21b78fdb0';
delete from user where Host='127.0.0.1';
delete from user where Host='::1';

flush privileges;

    • k. 安装mysql-connector-java.jar

      ① 下载mysql-connector-java.jar

      ② 将jar包拷贝到 hive-1.2.2/lib 下

    配置Metastore到Mysql

    1. 在hive-1.2.2/conf下创建hive-site.xml

      vi hive-site.xml

    2. 配置如下:

      <?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
   <property>
     <name>javax.jdo.option.ConnectionURL</name>
     <value>jdbc:mysql://hive01:3306/metastore?createDatabaseIfNotExist=true</value>
     <description>JDBC connect string for a JDBC metastore</description>
   </property>
   <property>
     <name>javax.jdo.option.ConnectionDriverName</name>
     <value>com.mysql.jdbc.Driver</value>
     <description>Driver class name for a JDBC metastore</description>
   </property>
   <property>
     <name>javax.jdo.option.ConnectionUserName</name>
     <value>root</value>
     <description>username to use against metastore database</description>
   </property>
   <property>
     <name>javax.jdo.option.ConnectionPassword</name>
     <value>密码</value>
     <description>password to use against metastore database</description>
   </property>
   <!-- 显示当前数据库，显示查询表的头信息 -->
   <property>
     <name>hive.cli.print.header</name>
     <value>true</value>
   </property>
   <property>
     <name>hive.cli.print.current.db</name>
     <value>true</value>
   </property>
</configuration>

    配置运行日志便于查询

    1. 备份

      cp /opt/envs/hive-1.2.2/conf/hive-log4j.properties.template /opt/envs/hive-1.2.2/conf/hive-log4j.properties

    2. 修改

      vi /opt/envs/hive-1.2.2/conf/hive-log4j.properties

    3. 修改hive.log.dir属性

      hive.log.dir=/opt/envs/hive-1.2.2/logs

    保存为镜像并移除容器

    docker commit e2d21b78fdb0 caiser/hive:1.2.2
docker rm e2d21b78fdb0

    启动容器

    docker run -it -d --network datastore_net --ip 172.173.16.22 --name hive01 caiser/hive:1.2.2 bin/bash

    配置ssh免密登录

    1. 进入容器

      docker exec -it hive01 /bin/bash

    2. 到~/.ssh目录下生成秘钥

      ssh-keygen -t rsa

    3. 拷贝秘钥到hadoop01，hadoop02和hadoop03

      a.[如果没开启]开启ssh服务[ps -ef | grep ssh]

      /usr/sbin/sshd -D &

      b. 拷贝秘钥到hadoop01，hadoop02，hadoop03

      ssh-copy-id hadoop01
ssh-copy-id hadoop02
ssh-copy-id hadoop03

    4. hadoop01,hadoop02和hadoop03依次执行

      ssh-copy-id hive01

    向hadoop集群中添加节点

    • 动态
      ① 在hive01机器的hadoop/sbin目录下启动datanode：

      hadoop-daemon.sh start datanode

      ② 在hive01机器的hadoop/sbin目录下启动nodemanager

      yarn-daemon.sh start nodemanager

      ③ jps查看datanode和nodemanager是否已启动
      ④ 回到namenode节点打印集群信息，或网页登录50070端口查看节点数量

      hdfs dfsadmin -report

    • 静态

      ① 停止hadoop集群
      ② 保证各节点hadoop的配置文件内容一致
      ③ 分别配置4个节点hadoop的salves文件

       hadoop01
 hadoop02
 hadoop03
 hive01

      ④ 重启hadoop集群

    启动hive

    1. 启动

      bin/hive

    2. 看到这种就是成了

      Logging initialized using configuration in file:/opt/envs/hive-1.2.2/conf/hive-log4j.properties
hive (default)>

    3. 大功告成

    相关文章

      网友评论

          本文标题：Docker 构建 Hive 1.2.2 单节点

          本文链接：https://www.haomeiwen.com/subject/nkcrxktx.html

          延伸阅读

          深度阅读

          • 您也可以注册成为美文阅读网的作者，发表您的原创作品、分享您的心情！

          栏目导航

          热点阅读

          关于我们|服务条款|联系我们|Docker 构建 Hive 1.2.2 单节点|投稿指南|网站地图|RSS订阅|排版工具|手机版

          提供经典美文摘抄,优美散文欣赏,现代诗歌精选,短篇小说,心情随笔,表白情书范文,故事会在线阅读欣赏

          Copyright © 2014-2023 Haomeiwen.com All Rights Reserved. 好美文阅读网 版权所有

          备案信息：桂公网安备 45052102000051号 · 桂ICP备13007215号-3

          本站所收录作品、热点评论等信息部分来源互联网，目的只是为了系统归纳学习和传递资讯

          所有作品版权归原创作者所有，与本站立场无关，如不慎侵犯了你的权益，请联系我们告知，我们将做删除处理！