美文网首页我爱编程
hadoop 完全分布式部署

hadoop 完全分布式部署

作者: Lisong | 来源:发表于2017-06-07 16:51 被阅读41次

    Hdaoop 完全分布式部署

    标签（空格分隔）：Hdaoop

    一、集群环境

    系统版本

    Centos6.8 baseServer

    hdaoop 版本

    hadoop-2.7.3

    主机信息

    master 192.168.100.11
    slave1 192.168.100.12
    slave2 192.168.100.13

    Hosts 文件

    192.168.100.11 master
    192.168.100.12 slave1
    192.168.100.13 slave2

    二、更新yum源

    vi /etc/yum.conf
    ##添加以下内容，不升级内核只升级安装包
    exclude=kernel*
    exclude=centos-release*

    cat >> /etc/yum.conf <<EOF
exclude=kernel*
exclude=centos-release*
EOF
yum upgrade -y

    三、hdaoop 集群时间同步

    安装Ntp

    yum install -y ntp

    修改时区

    cat > /etc/sysconfig/clock << EOF
ZONE="Asia/Shanghai"
UTC=true
ARC=false
EOF

    覆盖系统时间文件

    cp -a /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

    备份配置文件

    test -f /etc/ntp.conf.bak || cp /etc/ntp.conf /etc/ntp.conf.bak
sed -r -i -n -e '/(^[ \t]*#|^[ \t]*$)/d;p' /etc/ntp.conf
cat /etc/ntp.conf

    修改配置文件

    cat > /etc/ntp.conf <<EOF
driftfile /var/lib/ntp/drift
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
restrict 127.0.0.1 
restrict -6 ::1
restrict 192.168.0.0 mask 255.255.255.0 nomodify notrap
server ntp.api.bz prefer    // prefer优先使用
server 1.cn.pool.ntp.org
server 2.cn.pool.ntp.org 
logfile /var/log/ntp.log
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys
EOF
cat /etc/ntp.conf

    开启同步更新hwclock

    cat > /etc/sysconfig/ntpd <<EOF
# Drop root to id 'ntp:ntp' by default.
OPTIONS="-u ntp:ntp -p /var/run/ntpd.pid -g"
SYNC_HWCLOCK=yes
EOF

    启动ntp服务

    service ntpd start

    重启ntp服务

    service ntpd restart

    开机启动ntp服务

    chkconfig ntpd on

    四、实现ssh互信

    各slave 节点也需要配置

    cat > ~/hosts.txt<<EOF
root@master a123456!
root@slave1 a123456!
root@slave2 a123456!
EOF
rpm -qa | grep expect || yum install expect -y
cat ~/hosts.txt | while read var_host; do
var_host_name=`echo "${var_host}" | awk '{print $1}'` && echo "${var_host_name}"
var_host_password=`echo "${var_host}" | awk '{print $2}'` && echo "${var_host_password}"
/usr/bin/expect <<END
spawn ssh-copy-id ${var_host_name}
expect {
"*(yes/no)?" {send "yes\r";exp_continue}
"*password:" {send "${var_host_password}\r";exp_continue}
expect eof
}
END
ssh ${var_host_name} <<END
sed -r -i -e '/^[ \t]*#[ \t]* ForwardAgent /c ForwardAgent yes' /etc/ssh/ssh_config
service sshd reload
END
done

    五、关闭防火墙和selinux

    六、安装JDK

    下载jdk

    cd /usr/local/src/
下载页面 http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
官网下载链接有时效性，需要自己到官网获取新的下载地址
wget http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jdk-8u131-linux-x64.tar.gz

    解压

    tar zxvf jdk-8u131-linux-x64.tar.gz -C /usr/local/
cd /usr/local
mv jdk-8u131-linux-x64 java

    配置环境变量

    ##添加如下内容：JAVA_HOME根据实际目录来
cat >> /etc/profile <<EOF
JAVA_HOME=/usr/local/java
CLASSPATH=${JAVA_HOME}/lib/
PATH=$PATH:${JAVA_HOME}/bin
export PATH JAVA_HOME CLASSPAT
EOF
source /etc/profile
java -version  ##查看安装情况

    七、安装Hadoop

    下载Hdaoop

    cd /usr/local/src/
wget http://apache.fayea.com/hadoop/common/hadoop-2.7.3/hadoop-2.7.3.tar.gz

    解压

    tar -zxvf hadoop-2.7.3.tar.gz -C /usr/local/
cd /usr/local/
mv hadoop-2.7.3 hadoop273
groupadd hadoop
useradd -g hadoop hadoop -s /sbin/nologin
chown -R hadoop:hadoop /usr/local/hadoop273
mkdir -p /usr/local/hadoop273/hdfs/name
mkdir -p /usr/local/hadoop273/hdfs/data

    添加环境变

    HADOOP_HOME=/usr/local/hadoop273
cat >> /etc/profile <<EOF
export HADOOP_HOME=${HADOOP_HOME}
export PATH=${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:${PATH}
export HADOOP_MAPARED_HOME=${HADOOP_HOME}
export HADOOP_COMMON_HOME=${HADOOP_HOME}
export HADOOP_HDFS_HOME=${HADOOP_HOME}
export HADOOP_YARN_HOME=${HADOOP_HOME}
export YARN_HOME=${HADOOP_HOME}
export YARN_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export HDFS_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export LD_LIBRARY_PATH=${HADOOP_HOME}/lib/native/:$LD_LIBRARY_PATH
export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native
export HADOOP_OPTS="-Djava.library.path=${HADOOP_HOME}/lib/native"
EOF
source /etc/profile
cd $HADOOP_HOME/etc/hadoop
cat >> slaves <<EOF
##添加主机名
slave1
slave2
EOF

vim hadoop-env.sh
##修改JAVA环境变量

export JAVA_HOME=/usr/local/java
source /usr/local/hadoop273/etc/hadoop/hadoop-env.sh

    备份配置文件

    test -f core-site.xml.bak || cp core-site.xml core-site.xml.bak
test -f hdfs-site.xml.bak || cp hdfs-site.xml hdfs-site.xml.bak
test -f mapred-site.xml.bak || cp mapred-site.xml mapred-site.xml.bak

    配置文件修改：
    cd /usr/local/hadoop273/etc/hadoop

    ① core-site.xml

    cat > core-site.xml <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
  <name>fs.default.name</name>
  <value>hdfs://master:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop273/hadoop_tmp</value>
</property>
<property>
<name>hadoop.proxyuser.root.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.root.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.sqoop2.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.sqoop2.groups</name>
<value>*</value>
</property>
</configuration>
EOF

    ② hdfs-site.xml

    cat > hdfs-site.xml <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>master:9001</value>
</property>
<property>
   <name>dfs.replication</name>
   <value>1</value>
</property>
<property>
  <name>dfs.tmp.dir</name>
  <value>/usr/local/hadoop273/hadoop_tmp</value>
</property>
<property>
   <name>dfs.namenode.name.dir</name>
   <value>file:/usr/local/hadoop273/hdfs/name</value>
</property>
<property>
    <name>dfs.datanode.data.dir</name>
    <value>file:/usr/local/hadoop273/hdfs/data</value>
</property>
<property>
    <name>dfs.permissions</name>
    <value>false</value>
</property>
<property>
    <name>dfs.webhdfs.enabled</name>
    <value>true</value>
</property>
<property>
<name>dfs.datanode.max.xcievers</name>
<value>4096</value>
</property>
</configuration>
EOF

    ③ mapred-site.xml
    （注：若没有mapred-site.xml，则需要将mapred-site.xml.template改为mapred-site.xml，并设置hadoop为属主属组）

    cat > mapred-site.xml <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property> 
<name>mapreduce.framework.name</name> 
<value>yarn</value> 
</property>
<property>
<name>mapreduce.application.classpath</name>
<value>/usr/local/hadoop273/etc/hadoop,/usr/local/hadoop273/share/hadoop/common/*,/usr/local/hadoop273/share/hadoop/common/lib/*,/usr/local/hadoop273/share/hadoop/hdfs/*,/usr/local/hadoop273/share/hadoop/hdfs/lib/*,/usr/local/hadoop273/share/hadoop/mapreduce/*,/usr/local/hadoop273/share/hadoop/mapreduce/lib/*,/usr/local/hadoop273/share/hadoop/yarn/*,/usr/local/hadoop273/share/hadoop/yarn/lib/*</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>1536</value>
</property>
<property>
<name>mapreduce.map.java.opts</name>
<value>-Xmx3072M</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>3072</value>
</property>
<property>
<name>mapreduce.reduce.java.opts</name>
<value>-Xmx6144M</value>
</property>
<property>
<name>mapreduce.cluster.map.memory.mb</name>
<value>-1</value>
</property>
<property>
<name>mapreduce.cluster.reduce.memory.mb</name>
<value>-1</value>
</property>
</configuration>
EOF

    ④ yarn-site.xml

    cat > yarn-site.xml <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8031</value>
</property>   
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master:8033</value>
</property>   
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master:8088</value>
</property>  
<property> 
  <name>yarn.nodemanager.aux-services</name> 
  <value>mapreduce_shuffle</value> 
</property>   
<property> 
  <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> 
  <value>org.apache.hadoop.mapred.ShuffleHandler</value> 
</property>
<property>
<name>mapreduce.application.classpath</name>
<value>/usr/local/hadoop273/etc/hadoop,/usr/local/hadoop273/share/hadoop/common/*,/usr/local/hadoop273/share/hadoop/common/lib/*,/usr/local/hadoop273/share/hadoop/hdfs/*,/usr/local/hadoop273/share/hadoop/hdfs/lib/*,/usr/local/hadoop273/share/hadoop/mapreduce/*,/usr/local/hadoop273/share/hadoop/mapreduce/lib/*,/usr/local/hadoop273/share/hadoop/yarn/*,/usr/local/hadoop273/share/hadoop/yarn/lib/*</value>
</property> 
<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>3</value>
</property>
</configuration>
EOF

    八、slave主机配置

    将java文件夹、环境变量文件、hadoop文件夹传到slave节点上，保证slave节点上的hadoop用户拥有hadoop文件夹权限：

    scp -r /usr/local/hadoop273 root@slave1:/usr/local/
    scp /etc/profile root@slave1:/etc/
    scp /etc/hosts root@slave1:/etc/
    scp -r /usr/local/java root@slave1:/usr/local/

    格式化namenode
    cd /usr/local/hadoop273
    bin/hdfs namenode -format

    启动dfs和yarn
    sbin/start-dfs.sh

    sbin/start-yarn.sh
    或 直接sbin/start-all.sh都启动

    启动history服务，不然在面板中不能打开history链接
    sbin/mr-jobhistory-daemon.sh start historyserver

    停止集群
    sbin/stop-all.sh

    使用jps命令查看启动进程：
    4504 ResourceManager
    4066 DataNode
    4761 NodeManager
    5068 JobHistoryServer
    4357 SecondaryNameNode
    3833 NameNode
    5127 Jps

    http://192.168.100.11:8088/cluster 访问Hadoop

    http://192.168.100.11:50070/

    相关文章

      网友评论

        本文标题：hadoop 完全分布式部署

        本文链接：https://www.haomeiwen.com/subject/likafxtx.html

        延伸阅读

        深度阅读

        • 您也可以注册成为美文阅读网的作者，发表您的原创作品、分享您的心情！

        栏目导航

        热点阅读

        我爱编程

        关于我们|服务条款|联系我们|hadoop 完全分布式部署|投稿指南|网站地图|RSS订阅|排版工具|手机版

        提供经典美文摘抄,优美散文欣赏,现代诗歌精选,短篇小说,心情随笔,表白情书范文,故事会在线阅读欣赏

        Copyright © 2014-2023 Haomeiwen.com All Rights Reserved. 好美文阅读网 版权所有

        备案信息：桂公网安备 45052102000051号 · 桂ICP备13007215号-3

        本站所收录作品、热点评论等信息部分来源互联网，目的只是为了系统归纳学习和传递资讯

        所有作品版权归原创作者所有，与本站立场无关，如不慎侵犯了你的权益，请联系我们告知，我们将做删除处理！