一、准备Linux环境

    使用Vmvare虚拟机安装Linux环境Ubuntu 16.04 LTS

二、JDK安装和配置

2.1 安装JDK

#  apt-get install + 选择的JDK版本

2.2 环境变量配置

    配置/etc/profile

# vim /etc/profile

export JAVA_HOME=/usr/lib/jvm/java-9-openjdk-amd64

export JRE_HOME=$JAVA_HOME/jre

export CLASSPATH=$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH

export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH

    使配置生效

#  source /etc/profile

三、Hadoop安装和配置

3.1  下载Hadoop

   下载Hadoop，并解压到/opt/

 #  wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-1.2.1/hadoop-1.2.1.tar.gz

#  mv hadoop-1.2.1.tar.gz /opt/

#  cd /opt/

 #  tar -zxvf hadoop-1.2.1.tar.gz

3.2 配置Hadoop

    需要配置/conf 目录下的四个文件：hadoop-env.sh，core-site.xml，hdfs-site.xml，mapred-site.xml

# cd  /opt/hadoop-1.2.1/conf

3.2.1 配置hadoop-env.sh

    查看$JAVA_HOME路径

#  echo $JAVA_HOME

/usr/lib/jvm/java-9-openjdk-amd64

    配置hadoop-env.sh文件中的$JAVA_HOME

# vim hadoop-env.sh

export JAVA_HOME=/usr/lib/jvm/java-9-openjdk-amd64

3.2.2 配置core-site.xml

# vim  core-site.xml

<configuration>

<property>

<name> hadoop.tmp.dir </name>

<value>/hadoop</value>

</property>

<property>

<name> dfs.name.dir</name>

<value>/hadoop/name</value>

</property>

<property>

<name> fs.default.name </name>

<value> hdfs://localhost:9000 </value>

</property>

</configuration>

3.2.3 配置hdfs-site.xml

# vim hdfs-site.xml

<configuration>

<property>

<name> dfs.data.dir </name>

<value> /hadoop/data </value>

</property>

</configuration>

3.2.4 配置mapred-site.xml

# vim mapred-site.xml

<configuration>

<property>

<name> mapred.job.tracker </name>

<value> localhost:9001</value>

</property>

</configuration>

3.2.5 配置/etc/profile

# vim /etc/profile

export HADOOP_HOME=/opt/hadoop-1.2.1

export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$HADOOP_HOME/bin:$PATH

# source /etc/profile

3.3  初始化NameNode

    配置好上述文件，运行hadoop命令，结果如下所示

#  hadoop

Warning: $HADOOP_HOME is deprecated.

Usage: hadoop [--config confdir] COMMAND

where COMMAND is one of: 

     namenode -format         format the DFS filesystem 

     secondarynamenode     run the DFS secondary  namenode 

     namenode                      run the DFS namenode 

......

Most commands print help when invoked w/o parameters.

问题描述

1、/etc/profile配置文件在以普通用户登陆时修改后，改为以root用户登录，/etc/profile就失效了。

2、运行hadoop命令，出现警告Warning: $HADOOP_HOME is deprecated.

解决方法

#  cd ~

#  vim .bashrc

export HADOOP_HOME_WARN_SUPPRESS=1

export JAVA_HOME=/usr/lib/jvm/java-9-openjdk-amd64

export JRE_HOME=$JAVA_HOME/jre

export HADOOP_HOME=/opt/hadoop-1.2.1

export CLASSPATH=$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH

export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$HADOOP_HOME/bin:$PATH

#  source .bashrc

    上述问题解决后，对namenode进行格式化操作

# hadoop namenode -format

报错信息

Error: Config file not found: /usr/lib/jvm/java-9-openjdk-amd64/conf/management/ management.properties

解决方法

    将java-9-openjdk-amd64目录下的conf文件删除，并新建一个conf文件夹，在新建的conf文件夹下，新建文件夹management，management下新建空的文件management.properties

# cd /usr/lib/jvm/java-9-openjdk-amd64

# rm conf

# mkdir conf

# cd conf

# mkdir -p management

# touch /usr/lib/jvm/java-9-openjdk-amd64/conf/management/management.properties

    报错解决后，格式化namenode，运行结果如下

18/02/12 04:32:19 INFO namenode.NameNode: STARTUP_MSG:

/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG:  host = ubuntu/127.0.1.1

STARTUP_MSG:  args = [-format]

STARTUP_MSG:  version = 1.2.1

STARTUP_MSG:  build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013

STARTUP_MSG:  java = 9-internal

************************************************************/

Re-format filesystem in /hadoop/dfs/name ? (Y or N) y

Format aborted in /hadoop/dfs/name

18/02/12 04:32:34 INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at ubuntu/127.0.1.1

************************************************************/

3.4 启动Hadoop

    Hadoop安装和配置完成后，在/opt/hadoop-1.2.1/bin目录下可以看到start-all.sh启动命令。运行启动命令

#  start-all.sh

报错信息

localhost: ssh: connect to host localhost port 22: Connection refused

解决方法

下载openssh-server

#  apt-get install openssh-server

重新运行start-all.sh启动命令，输入密码时，有如下报错信息

root@localhost's password: localhost: Permission denied, please try again.

解决方法：

修改root密码

#  sudo passwd root

编辑配置文件，允许 root 用户通过 ssh 登录

#  vim /etc/ssh/sshd_config

将 PermitRootLogin prohibit-password注释掉

添加PermitRootLogin yes

#  service ssh restart

然后ssh localhost ，输入密码后，结果如下所示，则问题解决

# ssh localhost

root@localhost's password:

Welcome to Ubuntu 16.04.3 LTS (GNU/Linux 4.13.0-32-generic x86_64)

* Documentation:  https://help.ubuntu.com

* Management:    https://landscape.canonical.com

* Support:        https://ubuntu.com/advantage

157 packages can be updated.

0 updates are security updates.

重新启动hadoop，运行结果如下所示

# start-all.sh

starting namenode, logging to /opt/hadoop-1.2.1/libexec/../logs/hadoop-root-namenode-ubuntu.out

root@localhost's password:

localhost: starting datanode, logging to /opt/hadoop-1.2.1/libexec/../logs/hadoop-root-datanode-ubuntu.out

root@localhost's password:

localhost: starting secondarynamenode, logging to /opt/hadoop-1.2.1/libexec/../logs/hadoop-root-secondarynamenode-ubuntu.out

starting jobtracker, logging to /opt/hadoop-1.2.1/libexec/../logs/hadoop-root-jobtracker-ubuntu.out

root@localhost's password:

localhost: starting tasktracker, logging to /opt/hadoop-1.2.1/libexec/../logs/hadoop-root-tasktracker-ubuntu.out