一、安装java 8
二、ssh配置
首先确认能够远程登录
系统偏好设置－共享 -远程登录
ssh-keygen -t rsa
Press enter for each line 提示输入直接按回车就好
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod og-wx ~/.ssh/authorized_keys
ssh localhost
如果ssh localhost还需要密码 查看一下你.ssh目录的权限
.ssh的父目录的权限问题（我的问题就出现在这里）：.ssh的父目录文件权限应该是755，即所属用户的 用户文件 （/home下属的一个用户文件）
执行chmod 755 ~/.ssh
三、安装配置hadoop文件
下载地址：http://hadoop.apache.org/releases.html
修改配置文件 配置文件在hadoop目录下的/etc/hadoop
1.修改core-site.xml 文件
<configuration>
<!-- 指定HDFS老大（namenode）的通信地址 -->
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value> </property>
<!-- 指定hadoop运行时产生文件的存储路径 -->
<property>
<name>hadoop.tmp.dir</name>
<value>/Users/chenxun/software/hadoop-2.8.1/temp</value>
</property>
</configuration>
2.修改hdfs-site.xml
默认副本数3，修改为1，dfs.namenode.name.dir指明fsimage存放目录，多个目录用逗号隔开。dfs.datanode.data.dir指定块文件存放目录，多个目录逗号隔开
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name> <value>file:/Users/chenxun/software/hadoop-2.8.1/tmp/hdfs/name</value> </property>
<property>
<name>dfs.datanode.data.dir</name> <value>file:/Users/chenxun/software/hadoop-2.8.1/tmp/hdfs/data</value> </property>
<property>
<name>dfs.namenode.secondary.http-address</name> <value>localhost:9001</value> </property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
</configuration>
3.yarn配置
mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.admin.user.env</name> <value>HADOOP_MAPRED_HOME=$HADOOP_COMMON_HOME</value> </property>
<property>
<name>yarn.app.mapreduce.am.env</name> <value>HADOOP_MAPRED_HOME=$HADOOP_COMMON_HOME</value> </property>
<property>
<name>mapreduce.application.classpath</name>
<value>
/Users/chenxun/software/hadoop-2.8.1/etc/hadoop, /Users/chenxun/software/hadoop-2.8.1/share/hadoop/common/*, /Users/chenxun/software/hadoop-2.8.1/share/hadoop/common/lib/*, /Users/chenxun/software/hadoop-2.8.1/share/hadoop/hdfs/*, /Users/chenxun/software/hadoop-2.8.1/share/hadoop/hdfs/lib/*, /Users/chenxun/software/hadoop-2.8.1/share/hadoop/mapreduce/*, /Users/chenxun/software/hadoop-2.8.1/share/hadoop/mapreduce/lib/*, /Users/chenxun/software/hadoop-2.8.1/share/hadoop/yarn/*, /Users/chenxun/software/hadoop-2.8.1/share/hadoop/yarn/lib/* </value> </property>
</configuration>
yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.application.classpath</name>
<value>
/Users/chenxun/software/hadoop-2.8.1/etc/hadoop, /Users/chenxun/software/hadoop-2.8.1/share/hadoop/common/*, /Users/chenxun/software/hadoop-2.8.1/share/hadoop/common/lib/*, /Users/chenxun/software/hadoop-2.8.1/share/hadoop/hdfs/*, /Users/chenxun/software/hadoop-2.8.1/share/hadoop/hdfs/lib/*, /Users/chenxun/software/hadoop-2.8.1/share/hadoop/mapreduce/*, /Users/chenxun/software/hadoop-2.8.1/share/hadoop/mapreduce/lib/*, /Users/chenxun/software/hadoop-2.8.1/share/hadoop/yarn/*, /Users/chenxun/software/hadoop-2.8.1/share/hadoop/yarn/lib/* </value> </property>
</configuration>
四、配置hadoop环境变量，可能需要重新编译native library
vim ~/.bash_profile
export HADOOP_HOME=/Users/chenxun/software/hadoop-2.8.1
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native/
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native:$HADOOP_COMMON_LIB_NATIVE_DIR"
五、运行hadoop 以及hadoop命令简单的介绍
格式化HDFS
hdfs namenode -format
一次启动hadoop所有进程，sbin下：
start-all.sh
打开 http://localhost:50070 进入hdfs管理页面
打开 http://localhost:8088 进入hadoop进程管理页面
网友评论