美文网首页
搭建Hadoop-HA + ZooKeeper + Yarn +

搭建Hadoop-HA + ZooKeeper + Yarn +

作者: 上杉丶零 | 来源:发表于2019-01-22 18:55 被阅读0次

前提:搭建Hadoop-HA + ZooKeeper + Yarn环境

node01 node02 node03 node04
NameNode01 NameNode02 NameNode03
DataNode01 DataNode02 DataNode03
JournalNode01 JournalNode02 JournalNode03
ZooKeeper01 ZooKeeper02 ZooKeeper03
ZooKeeperFailoverController01 ZooKeeperFailoverController02 ZooKeeperFailoverController03
ResourceManager01 ResourceManager02
NodeManager01 NodeManager02 NodeManager03
MySQL Server MetaStore Server Hive CLI
  1. 配置node01上的MySQL服务

安装MySQL:
yum install mysql-server -y
启动MySQL服务:
service mysqld start
修改MySQL权限:
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123123' WITH GRANT OPTION;
DELETE FROM user WHERE host != '%';
flush privileges;
登陆MySQL:
mysql -u root -p

  1. 安装node03、node04上的Hive

tar -zxvf apache-hive-2.3.4-bin.tar.gz -C /opt/hive/

  1. 配置node03、node04上的Hive

在node03上修改/opt/hive/apache-hive-2.3.4-bin/conf/hive-site.xml
cp /opt/hive/apache-hive-2.3.4-bin/conf/hive-default.xml.template /opt/hive/apache-hive-2.3.4-bin/conf/hive-site.xml
vim /opt/hive/apache-hive-2.3.4-bin/conf/hive-site.xml
添加:

<configuration>
  <property>  
    <name>hive.metastore.warehouse.dir</name>  
    <value>/hive</value>  
  </property>  
  <property>  
    <name>javax.jdo.option.ConnectionURL</name>  
    <value>jdbc:mysql://node01:3306/hive?createDatabaseIfNotExist=true</value>  
  </property>  
  <property>  
    <name>javax.jdo.option.ConnectionDriverName</name>  
    <value>com.mysql.jdbc.Driver</value>  
  </property>     
  <property>  
    <name>javax.jdo.option.ConnectionUserName</name>  
    <value>root</value>  
  </property>  
  <property>  
    <name>javax.jdo.option.ConnectionPassword</name>  
    <value>123123</value>  
  </property>
</configuration>

在node04上修改/opt/hive/apache-hive-2.3.4-bin/conf/hive-site.xml
cp /opt/hive/apache-hive-2.3.4-bin/conf/hive-default.xml.template /opt/hive/apache-hive-2.3.4-bin/conf/hive-site.xml
vim /opt/hive/apache-hive-2.3.4-bin/conf/hive-site.xml
添加:

<configuration>
  <property>  
    <name>hive.metastore.warehouse.dir</name>  
    <value>/hive</value>  
  </property>  
  <property>  
    <name>hive.metastore.uris</name>  
    <value>thrift://node03:9083</value>  
  </property> 
</configuration>
  1. 添加node03上的MySQL驱动:

mv mysql-connector-java-5.1.32-bin.jar /opt/hive/apache-hive-2.3.4-bin/lib/

  1. 配置node03、node04上的环境变量

在node03、node04上修改/etc/profile
vim /etc/profile
添加:

export HIVE_HOME=/opt/hive/apache-hive-2.3.4-bin
export PATH=$PATH:$HIVE_HOME/bin

在node03、node04上运行:
. /etc/profile

  1. 初始化数据库

在node03上运行:
schematool -dbType mysql -initSchema

  1. 启动Hive服务端

在node3上运行:
hive --service metastore

  1. 启动Hive客户端

在node04上运行:
hive

  1. 配置node01、node02、node03、node04上的Hadoop

在node01、node02、node03、node04上修改/opt/hadoop/hadoop-3.1.1/etc/hadoop/core-site.xml
vim /opt/hadoop/hadoop-3.1.1/etc/hadoop/core-site.xml
添加:

<property>
  <name>hadoop.proxyuser.root.groups</name>
  <value>*</value>
</property>
<property>
  <name>hadoop.proxyuser.root.hosts</name>
  <value>*</value>
</property>
  1. 重启Hadoop

在node01、node02、node03上运行:
hdfs dfsadmin -fs hdfs://node01:8020 -refreshSuperUserGroupsConfiguration

  1. 启动Hive服务端

在node3上运行:
hiveserver2

  1. 启动Hive客户端

在node04上运行:
beeline
!connect jdbc:hive2://node03:10000 root 1

  1. 查看进程

在node01、node02、node03、node04上运行:
jps

相关文章

网友评论

      本文标题:搭建Hadoop-HA + ZooKeeper + Yarn +

      本文链接:https://www.haomeiwen.com/subject/pinljqtx.html