《日子》.分布式-Hadoop2.6 cluster

一、系统安装及网络环境配制

<p>centos6.5
配制hostname 及 ip
关闭防火墙
3台虚拟机规划
vi /etc/hosts
加入
master.hadoop 192.168.0.10
slave1.hadoop 192.168.0.11
slave2.hadoop 192.168.0.12</p>

二、ssh无密钥登录

参考《日子》.分布式之开篇－SSH无密认证

三、jdk安装及hadoop2.6 拷贝

创建目录 cluster
|- install
|-hadoop-2.6.0

拷贝jdk-7u79-linux-x64.rpm 到install 目录
rpm -ivh jdk-7u79-linux-x64.rpm 安装jdk
拷贝hadoop2.6.0 到cluster目录

四、配制文件修改

core-site.xml

<code><?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/hadoop/tmp</value>
<description>Abase for other temporary directories.</description>
</property>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master.hadoop:9000</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>4096</value>
</property>
</configuration>
</code>

vi hadoop-env.sh 和 yarn-env.sh 在开头添加如下环境变量(一定要添加切勿少了)

export JAVA_HOME=/usr/java/jdk1.7.0_79

mapred-site.xml

<code>
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
<final>true</final>
</property>
<property>
<name>mapreduce.jobtracker.http.address</name>
<value>master.hadoop:50030</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master.hadoop:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master.hadoop:19888</value>
</property>
<property>
<name>mapred.job.tracker</name>
<value>http://master.hadoop:9001</value>
</property>
</configuration>
</code>

yarn-site.xml

<code>
<?xml version="1.0"?>

<configuration>

<property>
<name>yarn.resourcemanager.hostname</name>
<value>master.hadoop</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master.hadoop:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master.hadoop:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master.hadoop:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master.hadoop:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master.hadoop:8088</value>
</property>

</configuration>
</code>

slaves 文件配制

slave1.hadoop
slave2.hadoop

拷贝hadoop2.6.0文件夹到192.168.0.11,192.168.0.12

五、启动集群

格式化namenode
hadoop namenode -formart
启动集群
start-all.sh
访问master

hadoop1.jpg hadoop2.jpg