[TOC]

Hadoop

1 环境搭建

1.1 ssh免密登录

• 需要生产ssh key[可以忽略，但是启动的时候，每次都需要输入密码]

  首先在用户根目录下ls -a查看包括隐藏的文件夹，应该能发现.ssh，如果没有就安装brew ssh

  有了文件.ssh后， ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa

  cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

ssh: Could not resolve hostname localhost: nodename nor servname provided, or not known

• mac配置ssh远程登录：系统偏好设置-》共享-》远程登录

mac ssh: connect to host localhost port 22: Connection refused 

• 需要配置 修改文件/etc/hosts,在里面增加对127.0.0.1的解析[否则second namenode 启动不了，处理方法就是对bogon做个DNS解析指向127.0.0.1]

  127.0.0.1    bogon    localhost 
  

1.2 hadoop-env.sh

# 1.跳转到hadoop-3.14/etc/hadoop 目录
cd hadoop

# 2.修改hadoo-env.sh，mac中的open在窗口打开文件，可以设置默认打开软件
vim hadoop-env.sh 或者 open hadoop-env.sh

# 3.配置 JAVA_HOME[在文件对应的位置中配置]
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_271.jdk/Contents/Home

1.2 core-site.xml

vim core-site.xml

<configuration>
    <!-- hdfs 端口 -->
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:8020</value>
    </property>
    <!-- hadoop 临时数据目录 -->
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/Users/ayden/Documents/repository/hadoop/tmp</value>
    </property>
</configuration>

1.3 hdfs-site.xml

vim hdfs-site.xml

<configuration>
    <!-- hdfs 数据副本数目  -->
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <!-- hdfs web ui界面 -->
    <property>
        <name>dfs.http.address</name>
        <value>0.0.0.0:50070</value>
    </property>
    <!-- hdfs 存储 fsimage 的地方  -->
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>/Users/ayden/Documents/repository/hadoop/tmp/dfs/name</value>
    </property>
    <!-- hdfs 数据存放 block 的地方  -->
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>/Users/ayden/Documents/repository/hadoop/tmp/dfs/data</value>
    </property>
</configuration>

1.4 Hadoop环境变量

vim ~/.bash_profile

# Hadoop
export HADOOP_HOME=/Users/ayden/Documents/soft/hadoop-3.1.4
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

source ~/.bash_profile

1.5 启动

cd /Users/ayden/Documents/soft/hadoop-3.1.4/sbin

• 启动HDFS

  # 启动
  start-dfs.sh
  
  # 检查
  jps
  
  # 访问ui
  http://localhost:50070
  
  # 停止
  stop.sh
  

• 启动HADOOP

  # 启动
  start-all.sh 
  
  # 检查
  jps
  
  # 访问
  http://localhost:8088
  
  # 停止
  stop-all.sh
  

1.6 YARN 环境配置

• YARN配置

# 配置mapreduce在yarn上运行
vim mapred-site.xml

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.application.classpath</name>
        <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
    </property>
</configuration>

# 配置yarn中的mapreduce
vim yarn-site.xml

<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.env-whitelist</name>
        <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
    </property>
</configuration>

• 启动YARN

start-yarn.sh

• 停止YARN

stop-yarn.sh

• YARN 界面

http://localhost:8088/cluster/scheduler