安装配置Hadoop

[TOC]

一、centos7 卸载open JDK 安装JDK

查询 openjdk 版本

$ java -version

复制到指定目录

$ cp ./jdk-8u172-linux-x64.tar.gz  /usr/env/java/

定位到文件上传的目录

$ cd /usr/env/java/

解压tgz

$ tar -zxvf /usr/env/java/jdk-8u172-linux-x64.tar.gz 

加入环境变量

$ vim /etc/profile

profile #最后一行 加入下面几行

# JAVA_HOME
export JAVA_HOME=/usr/java/jdk1.8.0_172
export CLASSPATH=.:${JAVA_HOME}/jre/lib/rt.jar:${JAVA_HOME}/lib/dt.jar:${JAVA_HOME}/lib/tools.jar
export PATH=$PATH:${JAVA_HOME}/bin

刷新环境变量

$ source /etc/profile

查看当前java 版本

$ java -version

查看当前javac 版本

$ javac -version

二、安装配置Hadoop环境变量

1. 修改主机名

   [root@master hadoop]# vim /etc/hostname
   master
   

2. hadoop.tar.gz 安装配置

   #解压tgz
   $ tar -zxvf /usr/environment/hadoop-2.7.6.tar.gz
   

3. 加入环境变量

   $ vim /etc/profile
       #最后一行 加入下面几行
    #hadoop
    export HADOOP_HOME=/usr/environment/hadoop-2.7.6
    export PATH=$PATH:$HADOOP_HOME/bin

4. 刷新环境变量

$ source /etc/profile

5. 查看当前hadoop 版本

$ hadoop -version
  Hadoop 2.7.6
  Subversion https://shv@git-wip-us.apache.org/repos/asf/hadoop.git -r 085099c66cf28be31604560c376fa282e69282b8
  Compiled by kshvachk on 2018-04-18T01:33Z
  Compiled with protoc 2.5.0
  From source with checksum 71e2695531cb3360ab74598755d036
  This command was run using /usr/environment/hadoop-2.7.6/share/hadoop/common/hadoop-common-2.7.6.jar

6. 创建文件夹

$ mkdir -p /data/hdfs/tmp/
$ mkdir -p /data/hdfs/data/
$ mkdir -p /data/hdfs/name/
$ chown -R hadoop:hadoop /data

7.etc配置
进入hadoop-2.7.6的配置目录：
cd /data/hadoop-2.7.6/etc/hadoop
依次修改core-site.xml、hdfs-site.xml、mapred-site.xml、yarn-site.xml以及slaves文件。

[root@master hadoop]# pwd
/usr/environment/hadoop-2.7.6/etc/hadoop

7.1 core-site.xml 修改

[root@master hadoop]# vim core-site.xml 

core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

  <property>
    <name>hadoop.tmp.dir</name>
    <value>file:/data/hdfs/tmp</value>
    <description>A base for other temporary directories.</description>
  </property>
  <property>
    <name>io.file.buffer.size</name>
    <value>131072</value>
  </property>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://master:9000</value>
  </property>
  <property>
    <name>hadoop.proxyuser.root.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.root.groups</name>
    <value>*</value>
  </property>
</configuration>

7.2 hdfs-site.xml修改

[root@master hadoop]# vim ./hdfs-site.xml 

hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>3</value>
  </property>
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>file:/data/hdfs/name</value>
    <final>true</final>
  </property>
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>file:/data/hdfs/data</value>
    <final>true</final>
  </property>
  <property>
    <name>dfs.namenode.secondary.http-address</name>
    <value>master:9001</value>
  </property>
  <property>
    <name>dfs.webhdfs.enabled</name>
    <value>true</value>
  </property>
  <property>
    <name>dfs.permissions</name>
    <value>false</value>
  </property>
</configuration>

7.3 修改mapred-site.xml

[root@master hadoop]# cp mapred-site.xml.template  mapred-site.xml

mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
</configuration>

7.4 修改 yarn-site.xml

[root@master hadoop]# vim yarn-site.xml 

yarn-site.xml

<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>

<!-- Site specific YARN configuration properties -->


        <property>
          <name>yarn.resourcemanager.address</name>
          <value>master:18040</value>
        </property>
        <property>
          <name>yarn.resourcemanager.scheduler.address</name>
          <value>master:18030</value>
        </property>
        <property>
          <name>yarn.resourcemanager.webapp.address</name>
          <value>master:18088</value>
        </property>
        <property>
          <name>yarn.resourcemanager.resource-tracker.address</name>
          <value>master:18025</value>
        </property>
        <property>
          <name>yarn.resourcemanager.admin.address</name>
          <value>master:18141</value>
        </property>
        <property>
          <name>yarn.nodemanager.aux-services</name>
          <value>mapreduce.shuffle</value>
        </property>
        <property>
          <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
          <value>org.apache.hadoop.mapred.ShuffleHandler</value>
        </property>

</configuration>

7.5 修改slaves

[root@master hadoop]# vi slaves
localhost

删除localhost
添加如下内容：

slave1
slave2
slave3

参考：

1. Linux下Hadoop2.7.1集群环境的搭建（超详细版）

2. hadoop安装教程

3. 【Hadoop】搭建完全分布式的hadoop