最近尝试编译一下Hadoop3.x，发现比2.x简单了很多。
在2.x时代编译Hadoop，编译环境是个难题，需要的工具比较多，对版本也有要求。这次编译Hadoop3.2.0，在官方文档中增加了使用docker容器的方式。官方已经把所需的环境打包到docker镜像中，并且提供了一个脚本。docker确实很强大啊！笑！
开始正题
首先到https://hadoop.apache.org/releases.html下载源码，解压。
[he@docker1 ~]$ cd hadoop-3.2.0-src/
[he@docker1 hadoop-3.2.0-src]$
建议读读官方文档BUILDING.txt
官方文档里提供了一种简单获得编译环境的方法，就是使用docker
The easiest way to get an environment with all the appropriate tools is by means
of the provided Docker config.
This requires a recent version of docker (1.4.1 and higher are known to work).
On Linux:
Install Docker and run this command:
$ ./start-build-env.sh
安装docker1.4.1或者更高版本。具体的安装配置过程就不详细说了。
注意启动好docker。
获得编译环境
[he@docker1 hadoop-3.2.0-src]$ ./start-build-env.sh # 注意不要使用root用户，会失败
这个时候会拉取Apache准备好的docker镜像到本地，然后run。
如果没问题会进入到docker容器中，这个时候所有的编译环境都准备好了。
根据官方文档
Create binary distribution with native code and with documentation:
$ mvn package -Pdist,native,docs -DskipTests -Dtar
如果遇到权限问题，请将docker容器的家目录下.m2和hadoop修改为当前用户的权限。不能用sudo，不然会找不到环境。
我使用下面的命令，可以支持snappy
$ mvn package -Pdist,native,docs -DskipTests -Dtar -Drequire.snappy #添加snappy的支持
接着就是maven下载依赖然后编译了。
编译的时候遇到 hadoop-client-check-test-invariants中enforce-banned-dependencies问题
直接删掉pom里面的hadoop-client-check-test-invariants的依赖，并删掉其文件夹。
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 1.799 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 4.592 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 1.307 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 2.231 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.225 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 2.306 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 3.902 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 1.512 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 6.521 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 2.114 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [01:25 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 3.457 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 3.162 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.046 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 24.206 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [01:09 min]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [ 24.380 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 7.194 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 2.432 s]
[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 27.660 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.040 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [ 0.110 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [ 16.829 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [ 47.036 s]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [ 5.522 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [ 0.089 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 15.826 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 13.686 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [ 3.878 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [ 6.051 s]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [ 5.151 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 21.239 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [ 1.039 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [ 9.162 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [ 3.801 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [ 3.692 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [ 0.087 s]
[INFO] Apache Hadoop YARN TimelineService HBase Common .... SUCCESS [ 5.323 s]
[INFO] Apache Hadoop YARN TimelineService HBase Client .... SUCCESS [ 5.524 s]
[INFO] Apache Hadoop YARN TimelineService HBase Servers ... SUCCESS [ 0.083 s]
[INFO] Apache Hadoop YARN TimelineService HBase Server 1.2 SUCCESS [ 4.771 s]
[INFO] Apache Hadoop YARN TimelineService HBase tests ..... SUCCESS [ 2.367 s]
[INFO] Apache Hadoop YARN Router .......................... SUCCESS [ 5.095 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [ 0.075 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [ 3.256 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [ 2.440 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [ 0.167 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 22.965 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 19.530 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [ 4.496 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [ 7.557 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [ 6.083 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [ 7.069 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 0.997 s]
[INFO] Apache Hadoop YARN Services ........................ SUCCESS [ 0.095 s]
[INFO] Apache Hadoop YARN Services Core ................... SUCCESS [ 1.280 s]
[INFO] Apache Hadoop YARN Services API .................... SUCCESS [ 0.880 s]
[INFO] Apache Hadoop Image Generation Tool ................ SUCCESS [ 2.064 s]
[INFO] Yet Another Learning Platform ...................... SUCCESS [ 4.318 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [ 0.085 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [ 0.077 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [ 8.703 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [ 2.187 s]
[INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [ 5.113 s]
[INFO] Apache Hadoop MapReduce Uploader ................... SUCCESS [ 2.730 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 2.647 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [ 4.161 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 2.567 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 2.500 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [ 1.215 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [ 1.295 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 2.700 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 2.007 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [ 1.372 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [ 1.293 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 0.175 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 2.478 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 4.683 s]
[INFO] Apache Hadoop Kafka Library support ................ SUCCESS [ 1.620 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 4.004 s]
[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [ 1.940 s]
[INFO] Apache Hadoop Client Aggregator .................... SUCCESS [ 3.950 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 3.392 s]
[INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [ 2.602 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 1.636 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 10.198 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.026 s]
[INFO] Apache Hadoop Client API ........................... SUCCESS [01:20 min]
[INFO] Apache Hadoop Client Runtime ....................... SUCCESS [01:08 min]
[INFO] Apache Hadoop Client Packaging Invariants .......... SUCCESS [ 0.305 s]
[INFO] Apache Hadoop Client Test Minicluster .............. SUCCESS [02:11 min]
[INFO] Apache Hadoop Client Packaging Integration Tests ... SUCCESS [ 0.199 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 52.880 s]
[INFO] Apache Hadoop Client Modules ....................... SUCCESS [ 1.087 s]
[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [ 1.696 s]
[INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [ 0.026 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 15:49 min
[INFO] Finished at: 2019-04-08T02:17:25+00:00
[INFO] Final Memory: 192M/941M
[INFO] ------------------------------------------------------------------------
使用docker编译真的比2.x时手动安装编译环境简单了很多。
网友评论