hadoop 完全分布式部署

作者: Lisong | 来源:发表于2017-06-07 16:51 被阅读41次

hadoop 完全分布式搭建指南
Hadoop框架基础（五）
hadoop学习02
完全分布式部署Hadoop
Hadoop完全分布式部署
hadoop 完全分布式部署
（二）Hadoop集群环境搭建(伪分布式)
Zookeeper分布式安装部署
2018-11-16 hadoop3.1完全分布式部署
大数据与Hadoop 、分布式文件系统、分布式Hadoop

Hdaoop 完全分布式部署

标签（空格分隔）：Hdaoop

一、集群环境

系统版本

Centos6.8 baseServer

hdaoop 版本

hadoop-2.7.3

主机信息

master 192.168.100.11
slave1 192.168.100.12
slave2 192.168.100.13

Hosts 文件

192.168.100.11 master
192.168.100.12 slave1
192.168.100.13 slave2

二、更新yum源

vi /etc/yum.conf
##添加以下内容，不升级内核只升级安装包
exclude=kernel*
exclude=centos-release*

cat >> /etc/yum.conf <<EOF
exclude=kernel*
exclude=centos-release*
EOF
yum upgrade -y

三、hdaoop 集群时间同步

安装Ntp

yum install -y ntp

修改时区

cat > /etc/sysconfig/clock << EOF
ZONE="Asia/Shanghai"
UTC=true
ARC=false
EOF

覆盖系统时间文件

cp -a /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

备份配置文件

test -f /etc/ntp.conf.bak || cp /etc/ntp.conf /etc/ntp.conf.bak
sed -r -i -n -e '/(^[ \t]*#|^[ \t]*$)/d;p' /etc/ntp.conf
cat /etc/ntp.conf

修改配置文件

cat > /etc/ntp.conf <<EOF
driftfile /var/lib/ntp/drift
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
restrict 127.0.0.1 
restrict -6 ::1
restrict 192.168.0.0 mask 255.255.255.0 nomodify notrap
server ntp.api.bz prefer    // prefer优先使用
server 1.cn.pool.ntp.org
server 2.cn.pool.ntp.org 
logfile /var/log/ntp.log
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys
EOF
cat /etc/ntp.conf

开启同步更新hwclock

cat > /etc/sysconfig/ntpd <<EOF
# Drop root to id 'ntp:ntp' by default.
OPTIONS="-u ntp:ntp -p /var/run/ntpd.pid -g"
SYNC_HWCLOCK=yes
EOF

启动ntp服务

service ntpd start

重启ntp服务

service ntpd restart

开机启动ntp服务

chkconfig ntpd on

四、实现ssh互信

各slave 节点也需要配置

cat > ~/hosts.txt<<EOF
root@master a123456!
root@slave1 a123456!
root@slave2 a123456!
EOF
rpm -qa | grep expect || yum install expect -y
cat ~/hosts.txt | while read var_host; do
var_host_name=`echo "${var_host}" | awk '{print $1}'` && echo "${var_host_name}"
var_host_password=`echo "${var_host}" | awk '{print $2}'` && echo "${var_host_password}"
/usr/bin/expect <<END
spawn ssh-copy-id ${var_host_name}
expect {
"*(yes/no)?" {send "yes\r";exp_continue}
"*password:" {send "${var_host_password}\r";exp_continue}
expect eof
}
END
ssh ${var_host_name} <<END
sed -r -i -e '/^[ \t]*#[ \t]* ForwardAgent /c ForwardAgent yes' /etc/ssh/ssh_config
service sshd reload
END
done

五、关闭防火墙和selinux

六、安装JDK

下载jdk

cd /usr/local/src/
下载页面 http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
官网下载链接有时效性，需要自己到官网获取新的下载地址
wget http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jdk-8u131-linux-x64.tar.gz

解压

tar zxvf jdk-8u131-linux-x64.tar.gz -C /usr/local/
cd /usr/local
mv jdk-8u131-linux-x64 java

配置环境变量

##添加如下内容：JAVA_HOME根据实际目录来
cat >> /etc/profile <<EOF
JAVA_HOME=/usr/local/java
CLASSPATH=${JAVA_HOME}/lib/
PATH=$PATH:${JAVA_HOME}/bin
export PATH JAVA_HOME CLASSPAT
EOF
source /etc/profile
java -version  ##查看安装情况

七、安装Hadoop

下载Hdaoop

cd /usr/local/src/
wget http://apache.fayea.com/hadoop/common/hadoop-2.7.3/hadoop-2.7.3.tar.gz

解压

tar -zxvf hadoop-2.7.3.tar.gz -C /usr/local/
cd /usr/local/
mv hadoop-2.7.3 hadoop273
groupadd hadoop
useradd -g hadoop hadoop -s /sbin/nologin
chown -R hadoop:hadoop /usr/local/hadoop273
mkdir -p /usr/local/hadoop273/hdfs/name
mkdir -p /usr/local/hadoop273/hdfs/data

添加环境变

HADOOP_HOME=/usr/local/hadoop273
cat >> /etc/profile <<EOF
export HADOOP_HOME=${HADOOP_HOME}
export PATH=${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:${PATH}
export HADOOP_MAPARED_HOME=${HADOOP_HOME}
export HADOOP_COMMON_HOME=${HADOOP_HOME}
export HADOOP_HDFS_HOME=${HADOOP_HOME}
export HADOOP_YARN_HOME=${HADOOP_HOME}
export YARN_HOME=${HADOOP_HOME}
export YARN_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export HDFS_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export LD_LIBRARY_PATH=${HADOOP_HOME}/lib/native/:$LD_LIBRARY_PATH
export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native
export HADOOP_OPTS="-Djava.library.path=${HADOOP_HOME}/lib/native"
EOF
source /etc/profile
cd $HADOOP_HOME/etc/hadoop
cat >> slaves <<EOF
##添加主机名
slave1
slave2
EOF

vim hadoop-env.sh
##修改JAVA环境变量

export JAVA_HOME=/usr/local/java
source /usr/local/hadoop273/etc/hadoop/hadoop-env.sh

备份配置文件

test -f core-site.xml.bak || cp core-site.xml core-site.xml.bak
test -f hdfs-site.xml.bak || cp hdfs-site.xml hdfs-site.xml.bak
test -f mapred-site.xml.bak || cp mapred-site.xml mapred-site.xml.bak

配置文件修改：
cd /usr/local/hadoop273/etc/hadoop

① core-site.xml

cat > core-site.xml <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
  <name>fs.default.name</name>
  <value>hdfs://master:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop273/hadoop_tmp</value>
</property>
<property>
<name>hadoop.proxyuser.root.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.root.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.sqoop2.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.sqoop2.groups</name>
<value>*</value>
</property>
</configuration>
EOF

② hdfs-site.xml

cat > hdfs-site.xml <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>master:9001</value>
</property>
<property>
   <name>dfs.replication</name>
   <value>1</value>
</property>
<property>
  <name>dfs.tmp.dir</name>
  <value>/usr/local/hadoop273/hadoop_tmp</value>
</property>
<property>
   <name>dfs.namenode.name.dir</name>
   <value>file:/usr/local/hadoop273/hdfs/name</value>
</property>
<property>
    <name>dfs.datanode.data.dir</name>
    <value>file:/usr/local/hadoop273/hdfs/data</value>
</property>
<property>
    <name>dfs.permissions</name>
    <value>false</value>
</property>
<property>
    <name>dfs.webhdfs.enabled</name>
    <value>true</value>
</property>
<property>
<name>dfs.datanode.max.xcievers</name>
<value>4096</value>
</property>
</configuration>
EOF

③ mapred-site.xml
（注：若没有mapred-site.xml，则需要将mapred-site.xml.template改为mapred-site.xml，并设置hadoop为属主属组）

cat > mapred-site.xml <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property> 
<name>mapreduce.framework.name</name> 
<value>yarn</value> 
</property>
<property>
<name>mapreduce.application.classpath</name>
<value>/usr/local/hadoop273/etc/hadoop,/usr/local/hadoop273/share/hadoop/common/*,/usr/local/hadoop273/share/hadoop/common/lib/*,/usr/local/hadoop273/share/hadoop/hdfs/*,/usr/local/hadoop273/share/hadoop/hdfs/lib/*,/usr/local/hadoop273/share/hadoop/mapreduce/*,/usr/local/hadoop273/share/hadoop/mapreduce/lib/*,/usr/local/hadoop273/share/hadoop/yarn/*,/usr/local/hadoop273/share/hadoop/yarn/lib/*</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>1536</value>
</property>
<property>
<name>mapreduce.map.java.opts</name>
<value>-Xmx3072M</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>3072</value>
</property>
<property>
<name>mapreduce.reduce.java.opts</name>
<value>-Xmx6144M</value>
</property>
<property>
<name>mapreduce.cluster.map.memory.mb</name>
<value>-1</value>
</property>
<property>
<name>mapreduce.cluster.reduce.memory.mb</name>
<value>-1</value>
</property>
</configuration>
EOF

④ yarn-site.xml

cat > yarn-site.xml <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8031</value>
</property>   
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master:8033</value>
</property>   
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master:8088</value>
</property>  
<property> 
  <name>yarn.nodemanager.aux-services</name> 
  <value>mapreduce_shuffle</value> 
</property>   
<property> 
  <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> 
  <value>org.apache.hadoop.mapred.ShuffleHandler</value> 
</property>
<property>
<name>mapreduce.application.classpath</name>
<value>/usr/local/hadoop273/etc/hadoop,/usr/local/hadoop273/share/hadoop/common/*,/usr/local/hadoop273/share/hadoop/common/lib/*,/usr/local/hadoop273/share/hadoop/hdfs/*,/usr/local/hadoop273/share/hadoop/hdfs/lib/*,/usr/local/hadoop273/share/hadoop/mapreduce/*,/usr/local/hadoop273/share/hadoop/mapreduce/lib/*,/usr/local/hadoop273/share/hadoop/yarn/*,/usr/local/hadoop273/share/hadoop/yarn/lib/*</value>
</property> 
<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>3</value>
</property>
</configuration>
EOF

八、slave主机配置

将java文件夹、环境变量文件、hadoop文件夹传到slave节点上，保证slave节点上的hadoop用户拥有hadoop文件夹权限：

scp -r /usr/local/hadoop273 root@slave1:/usr/local/
scp /etc/profile root@slave1:/etc/
scp /etc/hosts root@slave1:/etc/
scp -r /usr/local/java root@slave1:/usr/local/

格式化namenode
cd /usr/local/hadoop273
bin/hdfs namenode -format

启动dfs和yarn
sbin/start-dfs.sh

sbin/start-yarn.sh
或直接sbin/start-all.sh都启动

启动history服务，不然在面板中不能打开history链接
sbin/mr-jobhistory-daemon.sh start historyserver

停止集群
sbin/stop-all.sh

使用jps命令查看启动进程：
4504 ResourceManager
4066 DataNode
4761 NodeManager
5068 JobHistoryServer
4357 SecondaryNameNode
3833 NameNode
5127 Jps

http://192.168.100.11:8088/cluster 访问Hadoop

http://192.168.100.11:50070/

hadoop 完全分布式搭建指南
简述 hadoop 集群一共有4种部署模式，详见《hadoop 生态圈介绍》。完全分布式模式将hadoop部署在至...
Hadoop框架基础（五）
** Hadoop框架基础（五）已经部署了Hadoop的完全分布式集群，我们知道NameNode节点的正常运行对...
hadoop学习02
hadoop学习笔记 - HDFS - 完全分布式模式 Hadoop完全分布式模式结构 hadoop01hadoo...
完全分布式部署Hadoop
基本流程：准备3台客户机（关闭防火墙、静态ip、主机名称）安装jdk配置环境变量安装hadoop配置环境变量ssh...
Hadoop完全分布式部署
一、概述概念：是一个可靠的、可伸缩的、分布式计算的开源软件。是一个框架，允许跨越计算机集群的大数据及分布式处理...
hadoop 完全分布式部署
Hdaoop 完全分布式部署标签（空格分隔）：Hdaoop 一、集群环境系统版本 Centos6.8 base...
（二）Hadoop集群环境搭建(伪分布式)
Hadoop部署模式有：本地模式、伪分布模式、完全分布式模式。区分的依据是NameNode、DataNode、Re...
Zookeeper分布式安装部署
分布式安装部署集群规划在hadoop102、hadoop103和hadoop104三个节点上部署Zookeepe...
2018-11-16 hadoop3.1完全分布式部署
Hadoop3.1.0完全分布式集群部署，三台服务器部署结构如下github配置文件源码地址 Pdsh使用方法ht...
大数据与Hadoop 、分布式文件系统、分布式Hadoop
Hadoop Hadoop分布式集群单机安装部署购买云主机主机IP地址配置hadoop1192.168.1....