大数据环境搭建--03 Centos7安装hadoop2.8集群

作者: M醉逍遥 | 来源:发表于2017-06-20 18:11 被阅读0次

大数据环境搭建--03 Centos7安装hadoop2.8集群
在centos7中部署hadoop2.8分布式集群并安装hbas
CDH5.7.2-Hadoop集群CentOS7下的搭建(离线安
大数据Hadoop集群搭建-04安装配置HDFS
k8s部署kubernetes-dashboard
elasticsearch 集群安装
大数据入门学习之环境搭建：VM、CentOS7安装，jdk1.8
zookeeper集群搭建
symfony4 No1: 运行环境配置之docker环境搭建
macrobase安装过程中的问题

注意：我们将大数据相关组件全部安装在/opt/bigdata目录

1.hadoop2.x 概述

hadoop2中NameNode可以有多个（目前只支持2个）。每一个都有相同的职能。一个是active状态的，一个是standby状态的。当集群运行时，只有active状态的NameNode是正常工作的，standby状态的NameNode是处于待命状态的，时刻同步active状态NameNode的数据。一旦active状态的NameNode不能工作，standby状态的NameNode就可以转变为active状态的，就可以继续工作了。
2个NameNode的数据其实是实时共享的。新HDFS采用了一种共享机制，Quorum Journal Node（JournalNode）集群或者Nnetwork File System（NFS）进行共享。NFS是操作系统层面的，JournalNode是hadoop层面的，我们这里使用JournalNode集群进行数据共享（这也是主流的做法）。JournalNode的架构图如下：

0.jpg

两个NameNode为了数据同步，会通过一组称作JournalNodes的独立进程进行相互通信。当active状态的NameNode的命名空间有任何修改时，会告知大部分的JournalNodes进程。standby状态的NameNode有能力读取JNs中的变更信息，并且一直监控edit log的变化，把变化应用于自己的命名空间。standby可以确保在集群出错时，命名空间状态已经完全同步了。
对于HA集群而言，确保同一时刻只有一个NameNode处于active状态是至关重要的。否则，两个NameNode的数据状态就会产生分歧，可能丢失数据，或者产生错误的结果。为了保证这点，这就需要利用使用ZooKeeper了。首先HDFS集群中的两个NameNode都在ZooKeeper中注册，当active状态的NameNode出故障时，ZooKeeper能检测到这种情况，它就会自动把standby状态的NameNode切换为active状态。

2.hadoop2.8 安装

现在cluster201上将配置文件修改好

[hadoop@cluster201 bigdata]$ pwd
/opt/bigdata
# 下载hadoop2.8安装包
[hadoop@cluster201 bigdata]$ wget http://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.8.0/hadoop-2.8.0.tar.gz
# 解压文件
[hadoop@cluster201 bigdata]$ tar -zvxf hadoop-2.8.0.tar.gz

增加hadoop环境变量，使用root在/etc/profile文件末尾增加如下内容：

#Hadoop
export HADOOP_HOME=/opt/bigdata/hadoop-2.8.0
export PATH=$PATH:$HADOOP_HOME/bin

创建相应目录

[hadoop@cluster201 bigdata]$ mkdir hadoop-2.8.0/tmp
[hadoop@cluster201 bigdata]$ mkdir hadoop-2.8.0/hdfs
[hadoop@cluster201 bigdata]$ mkdir hadoop-2.8.0/hdfs/name
[hadoop@cluster201 bigdata]$ mkdir hadoop-2.8.0/hdfs/data

进入到$HADOOP_HOME/etc/hadoop目录修改配置文件，配置项可参考文档http://hadoop.apache.org/docs/r2.8.0/

image.png

[hadoop@cluster201 bigdata]$ cd hadoop-2.8.0/etc/hadoop/
[hadoop@cluster201 hadoop]$ pwd
/opt/bigdata/hadoop-2.8.0/etc/hadoop
[hadoop@cluster201 hadoop]$ ll
总用量 160
-rw-r--r--. 1 hadoop hadoop  4942 3月  17 13:31 capacity-scheduler.xml
-rw-r--r--. 1 hadoop hadoop  1335 3月  17 13:31 configuration.xsl
-rw-r--r--. 1 hadoop hadoop   318 3月  17 13:31 container-executor.cfg
-rw-r--r--. 1 hadoop hadoop  1148 6月  20 15:22 core-site.xml
-rw-r--r--. 1 hadoop hadoop  3804 3月  17 13:31 hadoop-env.cmd
-rw-r--r--. 1 hadoop hadoop  4682 6月  20 14:56 hadoop-env.sh
-rw-r--r--. 1 hadoop hadoop  2598 3月  17 13:31 hadoop-metrics2.properties
-rw-r--r--. 1 hadoop hadoop  2490 3月  17 13:31 hadoop-metrics.properties
-rw-r--r--. 1 hadoop hadoop  9683 3月  17 13:31 hadoop-policy.xml
-rw-r--r--. 1 hadoop hadoop  1379 6月  20 16:18 hdfs-site.xml
-rw-r--r--. 1 hadoop hadoop  1449 3月  17 13:31 httpfs-env.sh
-rw-r--r--. 1 hadoop hadoop  1657 3月  17 13:31 httpfs-log4j.properties
-rw-r--r--. 1 hadoop hadoop    21 3月  17 13:31 httpfs-signature.secret
-rw-r--r--. 1 hadoop hadoop   620 3月  17 13:31 httpfs-site.xml
-rw-r--r--. 1 hadoop hadoop  3518 3月  17 13:31 kms-acls.xml
-rw-r--r--. 1 hadoop hadoop  1611 3月  17 13:31 kms-env.sh
-rw-r--r--. 1 hadoop hadoop  1631 3月  17 13:31 kms-log4j.properties
-rw-r--r--. 1 hadoop hadoop  5546 3月  17 13:31 kms-site.xml
-rw-r--r--. 1 hadoop hadoop 13661 3月  17 13:31 log4j.properties
-rw-r--r--. 1 hadoop hadoop   951 3月  17 13:31 mapred-env.cmd
-rw-r--r--. 1 hadoop hadoop  1383 3月  17 13:31 mapred-env.sh
-rw-r--r--. 1 hadoop hadoop  4113 3月  17 13:31 mapred-queues.xml.template
-rw-r--r--. 1 hadoop hadoop   862 6月  20 16:18 mapred-site.xml
-rw-r--r--. 1 hadoop hadoop   758 3月  17 13:31 mapred-site.xml.template
-rw-r--r--. 1 hadoop hadoop    33 6月  20 16:05 slaves
-rw-r--r--. 1 hadoop hadoop  2316 3月  17 13:31 ssl-client.xml.example
-rw-r--r--. 1 hadoop hadoop  2697 3月  17 13:31 ssl-server.xml.example
-rw-r--r--. 1 hadoop hadoop  2250 3月  17 13:31 yarn-env.cmd
-rw-r--r--. 1 hadoop hadoop  4567 3月  17 13:31 yarn-env.sh
-rw-r--r--. 1 hadoop hadoop   928 6月  20 16:19 yarn-site.xml

修hadoop-env.sh的JAVA_HOME地址

# The java implementation to use.
export JAVA_HOME=/usr/local/java/jdk1.8.0_121

vim core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://cluster201:9000</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/opt/bigdata/hadoop-2.8.0/tmp</value>
    </property>
    <property>
        <name>ha.zookeeper.quorum</name>
        <value>cluster201:2181,cluster202:2181,cluster203:2181</value>
    </property>
</configuration>

vim hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>/opt/bigdata/hadoop-2.8.0/hdfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.name.dir</name>
        <value>/opt/bigdata/hadoop-2.8.0/hdfs/data</value>
    </property>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.namenode.http-address</name>
        <value>cluster201:50070</value>
    </property>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>cluster201:50090</value>
    </property>
</configuration>

vim mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

vim yarn-site.xml

<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>

<!-- Site specific YARN configuration properties -->
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>cluster201</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
</configuration>

将修改好配置的hadoop安装包压缩拷贝到cluster202，cluster203上，并到对应机器上进行解压

[hadoop@cluster201 hadoop]$ cd /opt/bigdata/
[hadoop@cluster201 bigdata]$ pwd
/opt/bigdata
[hadoop@cluster201 bigdata]$ tar -zvcf hadoop-2.8.0.tar.gz hadoop-2.8.0
[hadoop@cluster201 bigdata]$ scp hadoop-2.8.0.tar.gz cluster202:/opt/bigdata/
[hadoop@cluster201 bigdata]$ scp hadoop-2.8.0.tar.gz cluster203:/opt/bigdata/

3. 启动haoop(先将zookeeper启动好)

# 格式化hdfs
[hadoop@cluster201 bigdata] hadoop namenode -format
# 启动hadoop
[hadoop@cluster201 bigdata] ./hadoop-2.8.0/sbin/start-all.sh

在宿主机windows的hosts文件种增加如下内容(C:\Windows\System32\drivers\etc\hosts)：

192.168.88.201 cluster201
192.168.88.202 cluster202
192.168.88.203 cluster203

hdfs webUI地址：http://cluster201:50070

image.png

yarn webUI地址：http://cluster201:8088

image.png

大数据环境搭建--03 Centos7安装hadoop2.8集群
注意：我们将大数据相关组件全部安装在/opt/bigdata目录 1.hadoop2.x 概述 hadoop2中N...
在centos7中部署hadoop2.8分布式集群并安装hbas
本文详细记录了centos7中安装hadoop2.8分布式集群,之后在hadoop中部署hbase，并简单演示了使...
CDH5.7.2-Hadoop集群CentOS7下的搭建(离线安
CDH5.7.2-Hadoop集群CentOS7下的搭建(离线安装) 标签：cdh大数据hadoopcentos安...
大数据Hadoop集群搭建-04安装配置HDFS
Hadoop集群搭建-03编译安装hadoop Hadoop集群搭建-02安装配置Zookeeper Hadoop...
k8s部署kubernetes-dashboard
部署环境参考centos7搭建k8s集群 2019最新版 Dashboard安装根据上面的信息可以得知dashb...
elasticsearch 集群安装
简介本文讲解如何在centos7 中安装elasticsearch 7 集群，并验证集群是否安装完毕安装环境 ...
大数据入门学习之环境搭建：VM、CentOS7安装，jdk1.8
一：环境搭建：VM、CentOS7安装及网络配置大数据入门首先需要搭建环境，接下来的三篇文章均是环境搭建部分的内...
zookeeper集群搭建
在centos7上安装搭建zookeeper 单机，集群。下载安装包 wget https://mirrors....
symfony4 No1: 运行环境配置之docker环境搭建
系统环境为centos7,使用docker 搭建centos7容器环境,安装php7.1.13 cd; mkdir...
macrobase安装过程中的问题
macrobase需要安装postgresql数据库环境本次安装环境centos7，在centos7环境下安装po...