最近在研究数据挖掘相关的东西，在本地 Mac 环境搭建了一套伪分布式的 hadoop 开发环境，采用CDH发行版本，中间遇到各种坑，于是还是想把一些过程给记录下来，供大家参考吧。
环境如下：
macOS 10.13.6 (17G65)
build 1.8.0_181-b13
首先安装JDK，这里我省略了，因为我的机器上有IDEA的环境，所以已经有了，没有的同学请自行解决吧，网上搜一下很多教程，唯一需要提的是别忘记添加环境变量，然后source ~/.bash_profile生效哦！
export JAVA_HOME="/Library/Java/JavaVirtualMachines/jdk1.8.x_xx.jdk/Contents/Home"
其实是本地SSH 登录，这里由于我机器上之前已经生成过RSA的秘钥，所以直接执行这一步即可
cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys
image
最后验证 ssh localhost 不弹框直接ok就算是过了。
接下来就是去下载CDH的安装包了 https://archive.cloudera.com/cdh5/cdh/5/
当然，如果是国内用户可以选择这个地址 http://archive-primary.cloudera.com/cdh5/cdh/5/，下载起来速度更快
查找下面四个文件并下载到本地
hadoop-2.6.0-cdh5.9.3.tar.gz
hbase-1.2.0-cdh5.9.3.tar.gz
hive-1.1.0-cdh5.9.3.tar.gz
zookeeper-3.4.5-cdh5.9.3.tar.gz
下一步就是建立 CDH 环境了，打开Terminal，执行以下命令
mkdir -p ~/cloudera/lib ~/cloudera/cdh5.9 ~/cloudera/ops/dn ~/cloudera/ops/logs/hadoop ~/cloudera/ops/logs/hbase ~/cloudera/ops/logs/yarn ~/cloudera/nn ~/cloudera/pids ~/cloudera/tmp ~/cloudera/zk
将上一步下载的四个压缩包拷贝到 ~/cloudera/lib 目录，并打开命令行执行以下命令
tar -xvf hadoop-2.6.0-cdh5.9.3.tar.gz
...
命令行中 cd ~/cloudera/cdh5.9 ，并执行以下命令
ln -s ~/cloudera/lib/hadoop-2.6.0-cdh5.9.3 hadoop
ln -s ~/cloudera/lib/hbase-1.2.0-cdh5.9.3 hbase
ln -s ~/cloudera/lib/hive-1.1.0-cdh5.9.3 hive
ln -s ~/cloudera/lib/zookeeper-3.4.5-cdh5.9.3 zookeeper
然后是编辑配置文件
CDH="cdh5.9"
export JAVA_HOME="/Library/Java/JavaVirtualMachines/jdk1.8.x_xx.jdk/Contents/Home"
export HADOOP_HOME="/Users/你的用户名/cloudera/${CDH}/hadoop"
export HBASE_HOME="/Users/你的用户名/cloudera/${CDH}/hbase"
export HIVE_HOME="/Users/你的用户名/cloudera/${CDH}/hive"
export HCAT_HOME="/Users/你的用户名/cloudera/${CDH}/hive/hcatalog"
export PATH=${JAVA_HOME}/bin:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:${ZK_HOME}/bin:${HBASE_HOME}/bin:${HIVE_HOME}/bin:${HCAT_HOME}/bin:${PATH}
然后开始编辑下列一系列文件，注意这里的路径一定要和你本地的适配起来，这里的用户名是我的，大家记得不要盲目拷贝哦。
$HADOOP_HOME/etc/hadoop/core-site.xml
$HADOOP_HOME/etc/hadoop/hdfs-site.xml
$HADOOP_HOME/etc/hadoop/yarn-site.xml
$HADOOP_HOME/etc/hadoop/mapred-site.xml
这里的hadoop-env.sh要特别注意，配置是我本地修改以后的，因为native部分无法启动，所以我需要忽略这部分的配置
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Set Hadoop-specific environment variables here.
# The only required environment variable is JAVA_HOME. All others are
# optional. When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on
# remote nodes.
# The java implementation to use.
#export JAVA_HOME=${JAVA_HOME}
# The jsvc implementation to use. Jsvc is required to run secure datanodes.
#export JSVC_HOME=${JSVC_HOME}
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"}
# Extra Java CLASSPATH elements. Automatically insert capacity-scheduler.
for f in $HADOOP_HOME/contrib/capacity-scheduler/*.jar; do
if [ "$HADOOP_CLASSPATH" ]; then
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f
else
export HADOOP_CLASSPATH=$f
fi
done
# The maximum amount of heap to use, in MB. Default is 1000.
#export HADOOP_HEAPSIZE=
#export HADOOP_NAMENODE_INIT_HEAPSIZE=""
# Extra Java runtime options. Empty by default.
#export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true -Djava.library.path=/Users/savy.pan/Playground/lib/hadoop/share/hadoop/mapreduce1/lib/native/Mac_OS_X-64"
# Command specific options appended to HADOOP_OPTS when specified
export HADOOP_NAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_NAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-Dhadoop.security.logger=ERROR,RFAS $HADOOP_DATANODE_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_NFS3_OPTS="$HADOOP_NFS3_OPTS"
export HADOOP_PORTMAP_OPTS="-Xmx512m $HADOOP_PORTMAP_OPTS"
# The following applies to multiple commands (fs, dfs, fsck, distcp etc)
export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS"
#HADOOP_JAVA_PLATFORM_OPTS="-XX:-UsePerfData $HADOOP_JAVA_PLATFORM_OPTS"
# On secure datanodes, user to run the datanode as after dropping privileges
export HADOOP_SECURE_DN_USER=${HADOOP_SECURE_DN_USER}
# Where log files are stored. $HADOOP_HOME/logs by default.
export HADOOP_LOG_DIR="/Users/savy.pan/cloudera/ops/logs/hadoop"
export YARN_LOG_DIR="/Users/savy.pan/cloudera/ops/logs/yarn"
# Where log files are stored in the secure data environment.
export HADOOP_SECURE_DN_LOG_DIR=${HADOOP_LOG_DIR}/${HADOOP_HDFS_USER}
# The directory where pid files are stored. /tmp by default.
# NOTE: this should be set to a directory that can only be written to by
# the user that will run the hadoop daemons. Otherwise there is the
# potential for a symlink attack.
export HADOOP_PID_DIR="/Users/savy.pan/cloudera/pids"
export HADOOP_SECURE_DN_PID_DIR=${HADOOP_PID_DIR}
export YARN_PID_DIR=${HADOOP_PID_DIR}
# A string representing this instance of hadoop. $USER by default.
export HADOOP_IDENT_STRING=$USER
$HBASE_HOME/conf/hbase-site.xml
然后还有就是zookeeper的配置了，在conf目录下新建zoo.cfg文件和创建data和logs目录
tickTime=2000
dataDir=/Users/savy.pan/cloudera/ops/logs/zk/data
dataLogDir=/Users/savy.pan/cloudera/ops/logs/zk/logs
clientPort=2181
开始运行!
命令行中执行命令
hdfs namenode -format
这里如果有同学遇到format abort错误，请切记，在系统让你选择Yes Or No的时候一定要注意大小写，我就是被这个坑过了，同时注意权限，后面的log文件中大家可以去看一下，很多时候是本地用户组权限太高导致的。
编辑文件 $HADOOP_HOME/libexec/hadoop-config.sh，查找 # Attempt to set JAVA_HOME if it is not set 到这一行，把下面的四行代码中。
if [ -x /usr/libexec/java_home ]; then
export JAVA_HOME=($(/usr/libexec/java_home))
else
export JAVA_HOME=(/Library/Java/Home)
fi
多余的两对括号去除，变成下面这样
if [ -x /usr/libexec/java_home ]; then
export JAVA_HOME=$(/usr/libexec/java_home)
else
export JAVA_HOME=/Library/Java/Home
fi
这两对括号会导致取值不正确，最后 yarn 中会只用硬编码的 JAVA_HOME=/bin/ 来寻找 java 命令的位置，导致所有的任务都会执行失败。
启动服务
一开始我直接启动hbase发现无法查看网页，查找log以后发现它会去尝试连zookeeper，所以这里在启动之前要先启动zookeeper，这在很多时候文档上都没有写，坑死了！
sh $HADOOP_HOME/sbin/start-dfs.sh
sh $HADOOP_HOME/sbin/start-yarn.sh
./zkServer.sh start
sh $HBASE_HOME/bin/start-hbase.sh
至于停止服务的命令大家自行去了解吧，我这里就不贴了。
服务管理地址
HDFS: http://localhost:50070/dfshealth.html
Yarn Scheduler: http://localhost:8088/cluster
Yarn NodeManager: http://localhost:8042/node
HBase: http://localhost:60010/master.jsp
测试 MR 任务
hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-cdh5.9.3.jar pi 100 100
网友评论