Hadoop3伪分布式安装指南

安装环境：CentOS 7，Hadoop3.0

1.安装JDK，设置环境变量

export JAVA_HOME=/opt/soft/jdk1.8.0_162
export PATH=$PATH:$JAVA_HOME/bin

2.解压缩hadoop，设置环境变量

export HADOOP_HOME=/opt/soft/hadoop-3.0.0
export PATH=$PATH:$HADOOP_HOME/bin

3.编辑etc/hadoop/hadoop-env.sh，设置JAVA_HOME

export JAVA_HOME=/usr/java/latest

4.编辑core-site.xml

etc/hadoop/core-site.xml:

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

5.编辑hdfs-site.xml

etc/hadoop/hdfs-site.xml:

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

6.设置ssh

$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
$ chmod 0600 ~/.ssh/authorized_keys
$ ssh localhost

7.初始化hdfs的namenode

$ /opt/soft/hadoop-3.0.0/bin/hdfs namenode -format

8.编辑/opt/soft/hadoop-3.0.0/sbin/start-dfs.sh与/opt/soft/hadoop-3.0.0/sbin/stop-dfs.sh,在顶部添加

HDFS_DATANODE_USER=root
HADOOP_SECURE_DN_USER=hdfs
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root

9.启动hdfs

$ /opt/soft/hadoop-3.0.0/sbin/start-dfs.sh
浏览器访问NameNode - http://localhost:9870/

10.编辑mapred-site.xml

etc/hadoop/mapred-site.xml:

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

11.编辑yarn-site.xml

etc/hadoop/yarn-site.xml:

<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.env-whitelist</name>
        <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
    </property>
</configuration>

12.编辑/opt/soft/hadoop-3.0.0/sbin/start-yarn.sh与/opt/soft/hadoop-3.0.0/sbin/stop-yarn.sh,在顶部添加

YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root

13.开启yarn

$ /opt/soft/hadoop-3.0.0/sbin/start-yarn.sh
浏览器访问ResourceManager - http://localhost:8088/