前言

直接从官网上下载的hadoop-2.6.0-cdh5.7.0.tar.gz部署后，不支持压缩，生产上是不可接受的，可以用hadoop checknative命令查看hadoop所支持的本地库：

[hadoop@Hadoop001 hadoop]$ bin/hadoop checknative
19/07/08 12:05:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Native library checking:
hadoop:  false 
zlib:    false 
snappy:  false 
lz4:     false 
bzip2:   false 
openssl: false 
19/07/08 12:05:41 INFO util.ExitUtil: Exiting with status 1

可以看出hadoop不支持这些压缩。所以需要从官网上下载源码进行编译让它支持压缩。

编译所需要的环境条件

在源码的文件夹里有个BUILDING.txt文件，讲述了该源码包的编译要求

[hadoop@Hadoop001 hadoop-2.6.0-cdh5.7.0]$ more BUILDING.txt 
Build instructions for Hadoop

----------------------------------------------------------------------------------
Requirements:

* Unix System
* JDK 1.7+
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer (if compiling native code), must be 3.0 or newer on Mac
* Zlib devel (if compiling native code)
* openssl devel ( if compiling native hadoop-pipes )
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)

本文所需要的环境条件如下

组件名称	版本
centos	centos6.5
Hadoop	Hadoop-2.6.0-cdh5.7.0-src.tar.gz
jdk	jdk-7u80-linux-x64.tar.gz
maven	apache-maven-3.6.1-bin.tar.gz
protobuf	protobuf-2.5.0.tar.gz

需要注意的是：编译的JDK版本必须是1.7版本，1.8的JDK会导致编译失败
jdk版本可以安装好多个，需要哪个版本的时候在环境变量里面把注释去掉，把其它版本的注释掉就可以了。
大致过程：把以上软件下载好之后，需要部署安装jdk、maven、protobuf、以及相关的依赖，然后用maven编译Hadoop源码，编译之后，部署Hadoop，验证是否可行

安装依赖库

root用户：

[root@Hadoop001 ~]# yum install -y svn ncurses-devel
[root@Hadoop001 ~]# yum install -y gcc gcc-c++ make cmake
[root@Hadoop001 ~]# yum install -y openssl openssl-devel svn ncurses-devel zlib-devel libtool
[root@Hadoop001 ~]# yum install -y snappy snappy-devel bzip2 bzip2-devel lzo lzo-devel lzop autoconf automake cmake

上传软件到系统上

以前已经存在hadoop用户，没有的话创建一下
hadoop用户

#创建三个文件夹：软件、maven仓库、源码
[hadoop@Hadoop001 ~]$ mkdir software maven_repo source
[hadoop@Hadoop001 software]$ rz -be
[hadoop@Hadoop001 software]$ ll
total 507108
-rw-r--r--  1 hadoop hadoop   9136463 Jul  8 10:13 apache-maven-3.6.1-bin.tar.gz
-rw-r--r--  1 hadoop hadoop  42610549 Jul  8 11:04 hadoop-2.6.0-cdh5.7.0-src.tar.gz
-rw-r--r--  1 hadoop hadoop 153530841 Jul  8 11:03 jdk-7u80-linux-x64.tar.gz
-rw-r--r--  1 hadoop hadoop   2401901 Jun 10 16:07 protobuf-2.5.0.tar.gz

安装JDK

root用户：
解压安装包，安装目录必须是/usr/java,安装后记得修改拥有者为root

[root@Hadoop001 software]# tar -zxvf jdk-7u80-linux-x64.tar.gz -C /usr/java/
[root@Hadoop001 software]# cd /usr/java/
[root@Hadoop001 java]# ll
total 8
drwxr-xr-x  8 uucp  143 4096 Apr 11  2015 jdk1.7.0_80
drwxr-xr-x. 8 root root 4096 Apr 11  2015 jdk1.8.0_45
[root@Hadoop001 java]# chown -R root:root jdk1.7.0_80/
[root@Hadoop001 java]# ll
total 8
drwxr-xr-x  8 root root 4096 Apr 11  2015 jdk1.7.0_80
drwxr-xr-x. 8 root root 4096 Apr 11  2015 jdk1.8.0_45

添加环境变量
因为我的环境之前安装了jdk1.8版本，所以在这里要把这1.8的给注释掉就好了。

[root@Hadoop001 java]# vi /etc/profile  
#添加如下环境变量
#export JAVA_HOME=/usr/java/jdk1.8.0_45
export JAVA_HOME=/usr/java/jdk1.7.0_80
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/lib:$JER_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JER_HOME/bin:$PATH
[root@Hadoop001 java]# source /etc/profile
[root@Hadoop001 java]# java -version
java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)

安装maven

hadoop用户:

[root@Hadoop001 java]# su - hadoop
[hadoop@Hadoop001 software]$ tar -zxvf apache-maven-3.6.1-bin.tar.gz 
[hadoop@Hadoop001 software]$ ln -s /home/hadoop/software/apache-maven-3.6.1 /home/hadoop/app/maven

添加环境变量:

[hadoop@Hadoop001 ~]$ vi .bash_profile 
export MAVEN_HOME=/home/hadoop/app/maven
export MAVEN_OPTS="-Xms1024m -Xmx1024m"
export PATH=$MAVEN_HOME/bin:$PATH
[hadoop@Hadoop001 ~]$ source .bash_profile
[hadoop@Hadoop001 ~]$ which mvn
~/app/maven/bin/mvn

配置maven:

[hadoop@Hadoop001 conf]$  vi /home/hadoop/app/maven/conf/settings.xml
#配置maven的本地仓库位置
<localRepository>/home/hadoop/maven_repo/repo</localRepository>
#添加阿里云中央仓库地址，注意一定要写在<mirrors></mirrors>之间，否则后面编译会报错
<mirror>
     <id>nexus-aliyun</id>
     <mirrorOf>central</mirrorOf>
     <name>Nexus aliyun</name>
     <url>http://maven.aliyun.com/nexus/content/groups/public</url>
</mirror>

小知识点：网络慢可能导致mvn第一次编译时下载需要超长的时间甚至编译失败。如果有别人或者自己已经建立好的本地仓库，可以把maven本地仓库添加到/home/hadoop/maven_repo/中去。

安装protobuf

hadoop用户：
解压

[hadoop@Hadoop001 software]$ tar -zxvf protobuf-2.5.0.tar.gz 
[hadoop@Hadoop001 software]$ ln -s /home/hadoop/software/protobuf-2.5.0 /home/hadoop/app/protobuf
[hadoop@Hadoop001 protobuf]$ ls
aclocal.m4    config.sub        depcomp                       install-sh   Makefile.am          python
autogen.sh    configure         editors                       INSTALL.txt  Makefile.in          README.txt
CHANGES.txt   configure.ac      examples                      java         missing              src
config.guess  CONTRIBUTORS.txt  generate_descriptor_proto.sh  ltmain.sh    protobuf-lite.pc.in  vsprojects
config.h.in   COPYING.txt       gtest                         m4           protobuf.pc.in

编译protobuf:

#./configure  --prefix是指定编译好的包要存放的路径，软件要安装的路径
#make 进行编译，make install进行安装
[hadoop@Hadoop001 protobuf]$ ./configure  --prefix=/home/hadoop/app/protobuf
[hadoop@Hadoop001 protobuf]$ make
[hadoop@Hadoop001 protobuf]$ make install

添加环境变量:

[hadoop@Hadoop001 ~]$ vi .bash_profile 
#追加如下两行内容
export PROTOBUF_HOME=/home/hadoop/app/protobuf
export PATH=$PROTOBUF_HOME/bin:$PATH
#测试是否生效，若出现libprotoc 2.5.0则为生效
[hadoop@Hadoop001 ~]$ source .bash_profile
[hadoop@Hadoop001 ~]$ protoc --version
libprotoc 2.5.0

编译hadoop

hadoop用户：
解压

[hadoop@Hadoop001 software]$ tar -zxvf hadoop-2.6.0-cdh5.7.0-src.tar.gz -C ../source/
[hadoop@Hadoop001 software]$ cd ../source/hadoop-2.6.0-cdh5.7.0/
[hadoop@Hadoop001 hadoop-2.6.0-cdh5.7.0]$ ls
BUILDING.txt       hadoop-client          hadoop-mapreduce1-project  hadoop-project       LICENSE.txt
cloudera           hadoop-common-project  hadoop-mapreduce-project   hadoop-project-dist  NOTICE.txt
dev-support        hadoop-dist            hadoop-maven-plugins       hadoop-tools         pom.xml
hadoop-assemblies  hadoop-hdfs-project    hadoop-minicluster         hadoop-yarn-project  README.txt

编译hadoop使其支持压缩：mvn clean package -Pdist,native -DskipTests -Dtar

#进入hadoop的源码目录
[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$ cd ~/source/hadoop-2.6.0-cdh5.7.0/
[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$ mvn clean package -Pdist,native -DskipTests -Dtar
#进行编译，第一次编译会下载很多依赖的jar包，快慢由网速决定，需耐心等待
#我是把别人的本地仓库拷贝到我的本地仓库，所以很快

若报异常，主要信息如下（无异常跳过）：

2019-07-08_173327.png
需要下载[apache-tomcat-6.0.44](https://archive.apache.org/dist/tomcat/tomcat-6/v6.0.44/bin/(可能需要翻墙)，并解压到[hadoop@Hadoop001 downloads]$ pwd /home/hadoop/source/hadoop-2.6.0-cdh5.7.0/hadoop-common-project/hadoop-kms/downloads目录下

[hadoop@Hadoop001 downloads]$ ll
total 6904
drwxrwxr-x 9 hadoop hadoop    4096 Jul  8 15:29 apache-tomcat-6.0.44
-rw-r--r-- 1 hadoop hadoop 7065218 Jul  8  2019 apache-tomcat-6.0.44.tar.gz

编译成功后会有提示：

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

查看编译后的包：hadoop-2.6.0-cdh5.7.0.tar.gz

[hadoop@Hadoop001 target]$ pwd
/home/hadoop/source/hadoop-2.6.0-cdh5.7.0/hadoop-dist/target
[hadoop@Hadoop001 target]$ ll
total 563992
drwxrwxr-x 2 hadoop hadoop      4096 Jul  8 16:04 antrun
drwxrwxr-x 3 hadoop hadoop      4096 Jul  8 16:04 classes
-rw-rw-r-- 1 hadoop hadoop      1998 Jul  8 16:04 dist-layout-stitching.sh
-rw-rw-r-- 1 hadoop hadoop       690 Jul  8 16:04 dist-tar-stitching.sh
drwxrwxr-x 9 hadoop hadoop      4096 Jul  8 16:04 hadoop-2.6.0-cdh5.7.0
-rw-rw-r-- 1 hadoop hadoop 191863181 Jul  8 16:05 hadoop-2.6.0-cdh5.7.0.tar.gz
-rw-rw-r-- 1 hadoop hadoop      7315 Jul  8 16:04 hadoop-dist-2.6.0-cdh5.7.0.jar
-rw-rw-r-- 1 hadoop hadoop 385596839 Jul  8 16:06 hadoop-dist-2.6.0-cdh5.7.0-javadoc.jar
-rw-rw-r-- 1 hadoop hadoop      4855 Jul  8 16:04 hadoop-dist-2.6.0-cdh5.7.0-sources.jar
-rw-rw-r-- 1 hadoop hadoop      4855 Jul  8 16:04 hadoop-dist-2.6.0-cdh5.7.0-test-sources.jar
drwxrwxr-x 2 hadoop hadoop      4096 Jul  8 16:05 javadoc-bundle-options
drwxrwxr-x 2 hadoop hadoop      4096 Jul  8 16:04 maven-archiver
drwxrwxr-x 3 hadoop hadoop      4096 Jul  8 16:04 maven-shared-archive-resources
drwxrwxr-x 3 hadoop hadoop      4096 Jul  8 16:04 test-classes
drwxrwxr-x 2 hadoop hadoop      4096 Jul  8 16:04 test-dir