[TOC]
Hadoop
1 环境搭建
1.1 ssh免密登录
-
需要生产ssh key[可以忽略,但是启动的时候,每次都需要输入密码]
首先在用户根目录下ls -a查看包括隐藏的文件夹,应该能发现.ssh,如果没有就安装brew ssh
有了文件.ssh后, ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
ssh: Could not resolve hostname localhost: nodename nor servname provided, or not known
- mac配置ssh远程登录:系统偏好设置-》共享-》远程登录
mac ssh: connect to host localhost port 22: Connection refused
-
需要配置 修改文件
/etc/hosts
,在里面增加对127.0.0.1
的解析[否则second namenode 启动不了,处理方法就是对bogon
做个DNS解析指向127.0.0.1
]127.0.0.1 bogon localhost
1.2 hadoop-env.sh
# 1.跳转到hadoop-3.14/etc/hadoop 目录
cd hadoop
# 2.修改hadoo-env.sh,mac中的open在窗口打开文件,可以设置默认打开软件
vim hadoop-env.sh 或者 open hadoop-env.sh
# 3.配置 JAVA_HOME[在文件对应的位置中配置]
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_271.jdk/Contents/Home
1.2 core-site.xml
vim core-site.xml
<configuration>
<!-- hdfs 端口 -->
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:8020</value>
</property>
<!-- hadoop 临时数据目录 -->
<property>
<name>hadoop.tmp.dir</name>
<value>/Users/ayden/Documents/repository/hadoop/tmp</value>
</property>
</configuration>
1.3 hdfs-site.xml
vim hdfs-site.xml
<configuration>
<!-- hdfs 数据副本数目 -->
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<!-- hdfs web ui界面 -->
<property>
<name>dfs.http.address</name>
<value>0.0.0.0:50070</value>
</property>
<!-- hdfs 存储 fsimage 的地方 -->
<property>
<name>dfs.namenode.name.dir</name>
<value>/Users/ayden/Documents/repository/hadoop/tmp/dfs/name</value>
</property>
<!-- hdfs 数据存放 block 的地方 -->
<property>
<name>dfs.datanode.data.dir</name>
<value>/Users/ayden/Documents/repository/hadoop/tmp/dfs/data</value>
</property>
</configuration>
1.4 Hadoop环境变量
vim ~/.bash_profile
# Hadoop
export HADOOP_HOME=/Users/ayden/Documents/soft/hadoop-3.1.4
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
source ~/.bash_profile
1.5 启动
cd /Users/ayden/Documents/soft/hadoop-3.1.4/sbin
-
启动HDFS
# 启动 start-dfs.sh # 检查 jps # 访问ui http://localhost:50070 # 停止 stop.sh
-
启动HADOOP
# 启动 start-all.sh # 检查 jps # 访问 http://localhost:8088 # 停止 stop-all.sh
1.6 YARN 环境配置
- YARN配置
# 配置mapreduce在yarn上运行
vim mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.application.classpath</name>
<value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
</property>
</configuration>
# 配置yarn中的mapreduce
vim yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.env-whitelist</name>
<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
</property>
</configuration>
- 启动YARN
start-yarn.sh
- 停止YARN
stop-yarn.sh
- YARN 界面
http://localhost:8088/cluster/scheduler
网友评论