https://www.zybuluo.com/DFFuture/note/626899
1.查看 hostname
修改主机名 sudo gedit /etc/hostname
重启 reboot
2.编辑host sudo gedit /etc/hosts
删除127.0.1.1那一行，并添加：
192.168.86.140 ubuntu1
192.168.86.143 ubuntu2
192.168.86.144 ubuntu3
ping hadoop01、hadoop02、hadoop03看是否能
3.安装openssh-server sudo apt-get install -y openssh-server
4.生成ssh目录 ssh localhost
在ssh目录下 cd ~./ssh
生成密钥 ssh-keygen -t rsa -P "" 一直按回车即可
将公钥添加到autorized_keys cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
5.将公钥传输到其他节点
scp ~/.ssh/id_rsa.pub edward@hadoop02:~
scp ~/.ssh/id_rsa.pub edward@hadoop03:~
进入hadoop02、hadoop03目录
将id_rea.pub的内容添加进入authorized_keys
cat ~/id_rsa.pub >> ~/.ssh/authorized_keys
删除已经添加的id_rsa.pub
rm -rf id_rsa.pub
如果ssh hadoop01、ssh hadoop02、ssh hadoop03都能无密登陆，即配置ssh成功
无法传输问题：
修改/etc/ssh/ssh_config
在最后添加2行：
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
6.下载 hadoop hadoop-2.7.7.tar.gz
解压到 /usr/local/hadoop
7.配置环境变量 gedit ~/.bashrc
添加
HADOOP_HOME=/usr/local/hadoop/hadoop-2.7.7
PATH添加：
$HADOOP_HOME/bin:$HADOOP_HOME/sbin
source ~/.bashrc hadoop version
8.集群/分布式模式需要修改 /usr/local/hadoop/hadoop-2.7.7/etc/hadoop 中的5个配置文件，
8.1 slaves （配置DataNode主机）
写入 ：
hadoop01
hadoop02
hadoop03
8.2 core-site.xml （基础配置）
写入：
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop02:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/local/hadoop/tmp</value>
</property>
<property>
<name>fs.trash.interval</name>
<value>10080</value>
</property>
</configuration>
8.3 hdfs-site.xml （HDFS配置）
写入：
<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.permissions.enabled</name>
<value>false</value>
</property>
<property>
<name>dfs.namenode.http-address</name>
<value>hadoop02:50070</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>hadoop03:50090</value>
</property>
</configuration>
8.4 mapred-site.xml （MapReduce配置）
复制重命名mapred-site.xml.template cp mapred-site.xml.template mapred-site.xml
写入：
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>hadoop02:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop02:19888</value>
</property>
<property>
<name>mapreduce.job.ubertask.enable</name>
<value>true</value>
</property>
</configuration>
8.5 yarn-site.xml （Yarn配置）
写入：
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop01</value>
</property>
<property>
<name>yarn.web-proxy.address</name>
<value>hadoop01:8888</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>604800</value>
</property>
</configuration>
9.格式化nameNode hadoop namenode -format
异常问题可以添加权限 sudo chmod 777 / usr /local/ hadoop /tmp/
10.将hadoop 复制到其他主机中
scp -r /local/hadoop hadoop@hadoop02:/local/
scp -r /local/hadoop hadoop@hadoop03:/local/
启动
11.hadoop02 /sbin 启动dfs和jobhistory
start-dfs.sh
mr-jobhistory-daemon.sh start historyserver
12.Hadoop01 /sbin 启动yarn和防护进程
start-yarn.sh
yarn-daemon.sh start proxyserver
13.网址
Yarn：192.168.86.136:8088
HDFS：192.168.86.137:50070
关闭
14. hadoop01 /sbin 关闭yarn
stop-yarn.sh
15 hadoop02 /sbin 关闭hdfs和jobhistory
stop-dfs.sh
mr-jobhistory-daemon.sh stop historyserver
报错：
JAVA_HOME is not set and could not be found
https://blog.csdn.net/lxpbs8851/article/details/45225171
$HADOOP_HOME/etc/hadoop/yarn-env.sh 中添加
export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_191
$HADOOP_HOME/libexec/hadoop-config.sh 中添加
export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_191
网友评论