mac安装hadoop

Pre-requisite

ssh installed
hadoop 3.1.2
Java installed

Hadoop3.1.2 binary download from
https://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-3.1.2/hadoop-3.1.2.tar.gz

1. Change the hadoop-env.sh
   /usr/local/Cellar/hadoop/3.1.1/libexec/etc/hadoop/hadoop-env.sh

 52 # The java implementation to use. By default, this environment
 53 # variable is REQUIRED on ALL platforms except OS X!
 54 export JAVA_HOME="/Library/Java/JavaVirtualMachines/jdk1.8.0_73.jdk/Contents/Home" 

 92 # export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true -Dsun.security.krb5.debug=true -Dsun.security.spnego.debug"
 93 export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true -Djava.security.krb5.realm= -Djava.security.krb5.kdc="

2. Add Hdfs address and port to core-site.xml

<configuration>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/Users/$UserName/hadoop/tmp</value>
        <description>A base for other temporary directories.</description>
    </property>
    <property>
        <name>fs.default.name</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

3. Add mapreduce top tracker's port to mapred-site.xml

<configuration>
    <property>
        <name>mapred.job.tracker</name>
        <value>localhost:8021</value>
    </property>
</configuration>

4. modify the replication number of hdfs

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

5. Initialize hadoop

$ hdfs namenode --format

Success with below message.

2019-06-28 21:29:52,530 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Yaweis-MacBook-Pro.local/192.168.31.73
************************************************************/

6. hadoop config in .bash_profile

vi ~/.bash_profile

HADOOP_HOME=/Users/$UserName/apps/hadoop/3.1.1
PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export HADOOP_HOME
export PATH

• 问题1

$ start-all.sh
WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
ERROR: Invalid HADOOP_COMMON_HOME