hadoop集群搭建

一、虚拟机概况
三台centos7.5 4g(内存) 2核 40g

master 192.168.2.146
hadoop01 192.168.2.153
hadoop02 192.168.2.148

二、环境变量以及安装准备
1、/etc/profile.d/hadoop.sh

export JAVA_HOME=/home/james/app/jdk1.8.0_91
export CLASSPATH=$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$JAVA_HOME/bin:$PATH
export HADOOP_HOME=/home/james/app/hadoop-2.6.0-cdh5.7.0
export PATH=$HADOOP_HOME/bin:$PATH 

2、hostname 设置
sudo vi /etc/sysconfig/network

NETWORKING=yes
HOSTNAME=master

hostname 和ip地址的设置 sudo vi /etc/hosts

192.168.2.146    master
192.168.2.153    hadoop01
192.168.2.144    hadoop02

3、各节点关闭防火墙
systemctl stop firewalld.service
4、各节点角色分配
maste: NameNode/DataNode ResourceManager/NodeManager
hadoop01: DataNode NodeManager
hadoop02: DataNode NodeManager

5、ssh 免密码登陆(各节点之间相互免登陆)
在每台机器上运行 ssh-keygen -t rsa
以master机器为主 执行下面的命令
cd ~/.ssh
ssh-copy-id 192.168.2.146
scp /home/hadoop/.ssh/authorized_keys 192.168.2.153:/home/hadoop/.ssh/

在hadoop01上
cd ~/.ssh
ssh-copy-id 192.168.2.153
scp /home/hadoop/.ssh/authorized_keys 192.168.2.144:/home/hadoop/.ssh/

在hadoop02上
cd ~/.ssh
scp /home/hadoop/.ssh/authorized_keys 192.168.2.146:/home/hadoop/.ssh/
scp /home/hadoop/.ssh/authorized_keys 192.168.2.153:/home/hadoop/.ssh/

6、安装jdk
在master:
tar -zxvf jdk-8u91-linux-x64.tar.gz -C ~/app

三、安装hadoop集群
1、在master解压
tar -zxvf hadoop-2.6.0-cdh5.7.0.tar.gz -C ~/app

2、配置文件etc/hadoop编辑
2.1 core-site.xml

<property>
  <name>fs.default.name</name>
  <value>hdfs://master:8020</value>
</property>

2.2 hdfs-site.xml

<property>
  <name>dfs.namenode.name.dir</name>
  <value>/home/james/app/tmp/dfs/name</value>
</property>
<property>
  <name>dfs.datanode.data.dir</name>
  <value>/home/james/app/tmp/dfs/data</value>
</property>
<property>
  <name>dfs.replication</name>
  <value>3</value>
</property>

2.3 yarn-site.xml

<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>master</value>
</property>
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>

2.4 mapred-site.xml

<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>

2.5 slaves

  master
  hadoop01
  hadoop02

3、分发安装包到hadoop01和hadoop02
scp -r ~/app james@hadoop01:~
scp -r ~/app james@hadoop02:~
scp /etc/profile.d/hadoop.sh james@hadoop01:/etc/profile.d/
scp /etc/profile.d/hadoop.sh james@hadoop02:/etc/profile.d/

4、在两台机器上 source /etc/profile.d/hadoop.sh
5、对nameNode(NN）做格式化处理($HADOOP_HOME/bin)
在master就可以了：hdfs namenode -format

四、启动集群
在master就可以了：sbin/start-all.sh
五、验证jps