win10编译hadoop3.0.0-beta1源码，获取win

作者: 夜雨辉晨 | 来源:发表于2017-10-24 00:21 被阅读0次

最近在阿里云主机上安装了hadoop3.0.0伪分布式，但是在windows使用eclipse连接时，没找到对应版本的hadoop.dll和winutils.exe，便萌生了自己编译的想法。在此记录一下主要步骤和踩过的坑。

安装需要的编译工具

首先肯定要按照源码根目录中附带的BUILDING文档安装编译所需的各种工具。以下为BUILDING文件中对windows编译的部分要求。建议通读该文档后，再着手hadoop源码的编译。

Building on Windows

Requirements:

Windows System

JDK 1.8+

Maven 3.0 or later

ProtocolBuffer 2.5.0

CMake 3.1 or newer

Windows SDK 7.1 or Visual Studio 2010 Professional

Windows SDK 8.1 (if building CPU rate control for the container executor)

zlib headers (if building native code bindings for zlib)

Internet connection for first build (to fetch all Maven and Hadoop dependencies)

Unix command-line tools from GnuWin32: sh, mkdir, rm, cp, tar, gzip. These
tools must be present on your PATH.

Python ( for generation of docs using 'mvn site')

在安装相关软件的过程中，可以参考Windows 10编译Hadoop 2.6.0源码和在Windows 10 64-bit上安装Windows SDK 7.1和.NET4。这两篇博文已经十分详细，我就不再赘述，只在这里写一下我碰到的问题。

1.关于zlib的编译安装。建议在安装make工具之后，下载到zlib1.2.11源代码，将zlib1.2.11/win32中的Makefile.gcc复制到zlib1.2.11目录下，然后在命令行中使用
make -f makefile.gcc
进行编译，得到zlib1.dll，然后在新建系统变量ZLIB_HOME="你的zlib目录（例如C:\program1\zlib-1.2.11）"。
2.关于Maven仓库的位置。BUILDING文档建议hadoop源码的目录路径尽量短，事实上，最好Maven仓库的位置也设置到根目录。最开始时，由于Maven仓库路径过长，导致我在使用windowsSDK7编译hadoop源码的过程中，在某一处产生的命令行长度超过限制而始终失败。
在此处产生的报错信息为：

Failed to execute goal org.codehaus.mojo:native-maven-plugin:1.0-alpha-8:javah (default) on project hadoop-common: Error running javah command: Error executing command line. Exit code:1 -> [Help 1]

详细报错信息为：

[INFO] --- native-maven-plugin:1.0-alpha-8:javah (default) @ hadoop-common ---
[INFO] cmd.exe /X /C "C:\Java\jdk1.8.0_151\bin\javah -d C:\hadoop\hadoop-common-project\hadoop-common\target\native\javah -classpath C:\hadoop\hadoop-common-project\hadoop-common\target\classes;C:\hadoop\hadoop-common-project\hadoop-annotations\target\hadoop-annotations-3.0.0-beta1.jar;C:\Java\jdk1.8.0_151\jre\..\lib\tools.jar;C:\Users\fyyc\.m2\repository\com\google\guava\guava\11.0.2\guava-11.0.2.jar;C:\Users\fyyc\.m2\repository\commons-cli\commons-cli\1.2\commons-cli-1.2.jar;C:\Users\fyyc\.m2\repository\org\apache\commons\commons-math3\3.1.1\commons-math3-3.1.1.jar;C:\Users\fyyc\.m2\repository\org\apache\httpcomponents\httpclient\4.5.2\httpclient-4.5.2.jar;C:\Users\fyyc\.m2\repository\org\apache\httpcomponents\httpcore\4.4.4\httpcore-4.4.4.jar;C:\Users\fyyc\.m2\repository\commons-codec\commons-codec\1.4\commons-codec-1.4.jar;C:\Users\fyyc\.m2\repository\commons-io\commons-io\2.4\commons-io-2.4.jar;C:\Users\fyyc\.m2\repository\commons-net\commons-net\3.1\commons-net-3.1.jar;C:\Users\fyyc\.m2\repository\commons-collections\commons-collections\3.2.2\commons-collections-3.2.2.jar;C:\Users\fyyc\.m2\repository\javax\servlet\javax.servlet-api\3.1.0\javax.servlet-api-3.1.0.jar;C:\Users\fyyc\.m2\repository\org\eclipse\jetty\jetty-server\9.3.19.v20170502\jetty-server-9.3.19.v20170502.jar;C:\Users\fyyc\.m2\repository\org\eclipse\jetty\jetty-http\9.3.19.v20170502\jetty-http-9.3.19.v20170502.jar;C:\Users\fyyc\.m2\repository\org\eclipse\jetty\jetty-io\9.3.19.v20170502\jetty-io-9.3.19.v20170502.jar;C:\Users\fyyc\.m2\repository\org\eclipse\jetty\jetty-util\9.3.19.v20170502\jetty-util-9.3.19.v20170502.jar;C:\Users\fyyc\.m2\repository\org\eclipse\jetty\jetty-servlet\9.3.19.v20170502\jetty-servlet-9.3.19.v20170502.jar;C:\Users\fyyc\.m2\repository\org\eclipse\jetty\jetty-security\9.3.19.v20170502\jetty-security-9.3.19.v20170502.jar;C:\Users\fyyc\.m2\repository\org\eclipse\jetty\jetty-webapp\9.3.19.v20170502\jetty-webapp-9.3.19.v20170502.jar;C:\Users\fyyc\.m2\repository\org\eclipse\jetty\jetty-xml\9.3.19.v20170502\jetty-xml-9.3.19.v20170502.jar;C:\Users\fyyc\.m2\repository\com\sun\jersey\jersey-core\1.19\jersey-core-1.19.jar;C:\Users\fyyc\.m2\repository\javax\ws\rs\jsr311-api\1.1.1\jsr311-api-1.1.1.jar;C:\Users\fyyc\.m2\repository\com\sun\jersey\jersey-servlet\1.19\jersey-servlet-1.19.jar;C:\Users\fyyc\.m2\repository\com\sun\jersey\jersey-json\1.19\jersey-json-1.19.jar;C:\Users\fyyc\.m2\repository\org\codehaus\jettison\jettison\1.1\jettison-1.1.jar;C:\Users\fyyc\.m2\repository\com\sun\xml\bind\jaxb-impl\2.2.3-1\jaxb-impl-2.2.3-1.jar;C:\Users\fyyc\.m2\repository\javax\xml\bind\jaxb-api\2.2.11\jaxb-api-2.2.11.jar;C:\Users\fyyc\.m2\repository\org\codehaus\jackson\jackson-core-asl\1.9.13\jackson-core-asl-1.9.13.jar;C:\Users\fyyc\.m2\repository\org\codehaus\jackson\jackson-mapper-asl\1.9.13\jackson-mapper-asl-1.9.13.jar;C:\Users\fyyc\.m2\repository\org\codehaus\jackson\jackson-jaxrs\1.9.13\jackson-jaxrs-1.9.13.jar;C:\Users\fyyc\.m2\repository\org\codehaus\jackson\jackson-xc\1.9.13\jackson-xc-1.9.13.jar;C:\Users\fyyc\.m2\repository\com\sun\jersey\jersey-server\1.19\jersey-server-1.19.jar;C:\Users\fyyc\.m2\repository\commons-logging\commons-logging\1.1.3\commons-logging-1.1.3.jar;C:\Users\fyyc\.m2\repository\log4j\log4j\1.2.17\log4j-1.2.17.jar;C:\Users\fyyc\.m2\repository\commons-lang\commons-lang\2.6\commons-lang-2.6.jar;C:\Users\fyyc\.m2\repository\commons-beanutils\commons-beanutils\1.9.3\commons-beanutils-1.9.3.jar;C:\Users\fyyc\.m2\repository\org\apache\commons\commons-configuration2\2.1.1\commons-configuration2-2.1.1.jar;C:\Users\fyyc\.m2\repository\org\apache\commons\commons-lang3\3.4\commons-lang3-3.4.jar;C:\Users\fyyc\.m2\repository\org\slf4j\slf4j-api\1.7.25\slf4j-api-1.7.25.jar;C:\Users\fyyc\.m2\repository\org\slf4j\slf4j-log4j12\1.7.25\slf4j-log4j12-1.7.25.jar;C:\Users\fyyc\.m2\repository\org\apache\avro\avro\1.7.4\avro-1.7.4.jar;C:\Users\fyyc\.m2\repository\com\thoughtworks\paranamer\paranamer\2.3\paranamer-2.3.jar;C:\Users\fyyc\.m2\repository\org\xerial\snappy\snappy-java\1.0.4.1\snappy-java-1.0.4.1.jar;C:\Users\fyyc\.m2\repository\com\google\re2j\re2j\1.1\re2j-1.1.jar;C:\Users\fyyc\.m2\repository\com\google\protobuf\protobuf-java\2.5.0\protobuf-java-2.5.0.jar;C:\Users\fyyc\.m2\repository\com\google\code\gson\gson\2.2.4\gson-2.2.4.jar;C:\hadoop\hadoop-common-project\hadoop-auth\target\hadoop-auth-3.0.0-beta1.jar;C:\Users\fyyc\.m2\repository\com\nimbusds\nimbus-jose-jwt\4.41.1\nimbus-jose-jwt-4.41.1.jar;C:\Users\fyyc\.m2\repository\com\github\stephenc\jcip\jcip-annotations\1.0-1\jcip-annotations-1.0-1.jar;C:\Users\fyyc\.m2\repository\net\minidev\json-smart\2.3\json-smart-2.3.jar;C:\Users\fyyc\.m2\repository\net\minidev\accessors-smart\1.2\accessors-smart-1.2.jar;C:\Users\fyyc\.m2\repository\org\ow2\asm\asm\5.0.4\asm-5.0.4.jar;C:\Users\fyyc\.m2\repository\org\apache\curator\curator-framework\2.12.0\curator-framework-2.12.0.jar;C:\Users\fyyc\.m2\repository\com\jcraft\jsch\0.1.54\jsch-0.1.54.jar;C:\Users\fyyc\.m2\repository\org\apache\curator\curator-client\2.12.0\curator-client-2.12.0.jar;C:\Users\fyyc\.m2\repository\org\apache\curator\curator-recipes\2.12.0\curator-recipes-2.12.0.jar;C:\Users\fyyc\.m2\repository\com\google\code\findbugs\jsr305\3.0.0\jsr305-3.0.0.jar;C:\Users\fyyc\.m2\repository\org\apache\htrace\htrace-core4\4.1.0-incubating\htrace-core4-4.1.0-incubating.jar;C:\Users\fyyc\.m2\repository\org\apache\zookeeper\zookeeper\3.4.9\zookeeper-3.4.9.jar;C:\Users\fyyc\.m2\repository\io\netty\netty\3.10.5.Final\netty-3.10.5.Final.jar;C:\Users\fyyc\.m2\repository\org\apache\commons\commons-compress\1.4.1\commons-compress-1.4.1.jar;C:\Users\fyyc\.m2\repository\org\tukaani\xz\1.0\xz-1.0.jar;C:\Users\fyyc\.m2\repository\org\apache\kerby\kerb-simplekdc\1.0.1\kerb-simplekdc-1.0.1.jar;C:\Users\fyyc\.m2\repository\org\apache\kerby\kerb-client\1.0.1\kerb-client-1.0.1.jar;C:\Users\fyyc\.m2\repository\org\apache\kerby\kerby-config\1.0.1\kerby-config-1.0.1.jar;C:\Users\fyyc\.m2\repository\org\apache\kerby\kerb-core\1.0.1\kerb-core-1.0.1.jar;C:\Users\fyyc\.m2\repository\org\apache\kerby\kerby-pkix\1.0.1\kerby-pkix-1.0.1.jar;C:\Users\fyyc\.m2\repository\org\apache\kerby\kerby-asn1\1.0.1\kerby-asn1-1.0.1.jar;C:\Users\fyyc\.m2\repository\org\apache\kerby\kerby-util\1.0.1\kerby-util-1.0.1.jar;C:\Users\fyyc\.m2\repository\org\apache\kerby\kerb-common\1.0.1\kerb-common-1.0.1.jar;C:\Users\fyyc\.m2\repository\org\apache\kerby\kerb-crypto\1.0.1\kerb-crypto-1.0.1.jar;C:\Users\fyyc\.m2\repository\org\apache\kerby\kerb-util\1.0.1\kerb-util-1.0.1.jar;C:\Users\fyyc\.m2\repository\org\apache\kerby\token-provider\1.0.1\token-provider-1.0.1.jar;C:\Users\fyyc\.m2\repository\org\apache\kerby\kerb-admin\1.0.1\kerb-admin-1.0.1.jar;C:\Users\fyyc\.m2\repository\org\apache\kerby\kerb-server\1.0.1\kerb-server-1.0.1.jar;C:\Users\fyyc\.m2\repository\org\apache\kerby\kerb-identity\1.0.1\kerb-identity-1.0.1.jar;C:\Users\fyyc\.m2\repository\org\apache\kerby\kerby-xdr\1.0.1\kerby-xdr-1.0.1.jar;C:\Users\fyyc\.m2\repository\com\fasterxml\jackson\core\jackson-databind\2.7.8\jackson-databind-2.7.8.jar;C:\Users\fyyc\.m2\repository\com\fasterxml\jackson\core\jackson-annotations\2.7.8\jackson-annotations-2.7.8.jar;C:\Users\fyyc\.m2\repository\com\fasterxml\jackson\core\jackson-core\2.7.8\jackson-core-2.7.8.jar;C:\Users\fyyc\.m2\repository\org\codehaus\woodstox\stax2-api\3.1.4\stax2-api-3.1.4.jar;C:\Users\fyyc\.m2\repository\com\fasterxml\woodstox\woodstox-core\5.0.3\woodstox-core-5.0.3.jar org.apache.hadoop.io.compress.zlib.ZlibCompressor org.apache.hadoop.io.compress.zlib.ZlibDecompressor org.apache.hadoop.security.JniBasedUnixGroupsMapping org.apache.hadoop.io.nativeio.NativeIO org.apache.hadoop.security.JniBasedUnixGroupsNetgroupMapping org.apache.hadoop.io.compress.snappy.SnappyCompressor org.apache.hadoop.io.compress.snappy.SnappyDecompressor org.apache.hadoop.io.compress.zstd.ZStandardCompressor org.apache.hadoop.io.compress.zstd.ZStandardDecompressor org.apache.hadoop.io.compress.lz4.Lz4Compressor org.apache.hadoop.io.compress.lz4.Lz4Decompressor org.apache.hadoop.io.erasurecode.ErasureCodeNative org.apache.hadoop.io.erasurecode.rawcoder.NativeRSRawEncoder org.apache.hadoop.io.erasurecode.rawcoder.NativeRSRawDecoder org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawEncoder org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawDecoder org.apache.hadoop.crypto.OpensslCipher org.apache.hadoop.crypto.random.OpensslSecureRandom org.apache.hadoop.util.NativeCrc32"
命令行太长。
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 27.710 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 15.316 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  2.078 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  3.250 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.392 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  2.499 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  6.374 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  2.876 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [  9.691 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  4.084 s]
[INFO] Apache Hadoop Common ............................... FAILURE [ 14.188 s]
[INFO] Apache Hadoop NFS .................................. SKIPPED
[INFO] Apache Hadoop KMS .................................. SKIPPED
[INFO] Apache Hadoop Common Project ....................... SKIPPED
[INFO] Apache Hadoop HDFS Client .......................... SKIPPED
[INFO] Apache Hadoop HDFS ................................. SKIPPED
[INFO] Apache Hadoop HDFS Native Client ................... SKIPPED
[INFO] Apache Hadoop HttpFS ............................... SKIPPED
[INFO] Apache Hadoop HDFS-NFS ............................. SKIPPED
[INFO] Apache Hadoop HDFS Project ......................... SKIPPED
[INFO] Apache Hadoop YARN ................................. SKIPPED
[INFO] Apache Hadoop YARN API ............................. SKIPPED
[INFO] Apache Hadoop YARN Common .......................... SKIPPED
[INFO] Apache Hadoop YARN Server .......................... SKIPPED
[INFO] Apache Hadoop YARN Server Common ................... SKIPPED
[INFO] Apache Hadoop YARN Registry ........................ SKIPPED
[INFO] Apache Hadoop YARN NodeManager ..................... SKIPPED
[INFO] Apache Hadoop YARN Web Proxy ....................... SKIPPED
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SKIPPED
[INFO] Apache Hadoop YARN Timeline Service ................ SKIPPED
[INFO] Apache Hadoop YARN ResourceManager ................. SKIPPED
[INFO] Apache Hadoop YARN Server Tests .................... SKIPPED
[INFO] Apache Hadoop YARN Client .......................... SKIPPED
[INFO] Apache Hadoop YARN SharedCacheManager .............. SKIPPED
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SKIPPED
[INFO] Apache Hadoop YARN Timeline Service HBase tests .... SKIPPED
[INFO] Apache Hadoop YARN Router .......................... SKIPPED
[INFO] Apache Hadoop YARN Applications .................... SKIPPED
[INFO] Apache Hadoop YARN DistributedShell ................ SKIPPED
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SKIPPED
[INFO] Apache Hadoop YARN Site ............................ SKIPPED
[INFO] Apache Hadoop YARN UI .............................. SKIPPED
[INFO] Apache Hadoop YARN Project ......................... SKIPPED
[INFO] Apache Hadoop MapReduce Client ..................... SKIPPED
[INFO] Apache Hadoop MapReduce Core ....................... SKIPPED
[INFO] Apache Hadoop MapReduce Common ..................... SKIPPED
[INFO] Apache Hadoop MapReduce Shuffle .................... SKIPPED
[INFO] Apache Hadoop MapReduce App ........................ SKIPPED
[INFO] Apache Hadoop MapReduce HistoryServer .............. SKIPPED
[INFO] Apache Hadoop MapReduce JobClient .................. SKIPPED
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SKIPPED
[INFO] Apache Hadoop MapReduce NativeTask ................. SKIPPED
[INFO] Apache Hadoop MapReduce Examples ................... SKIPPED
[INFO] Apache Hadoop MapReduce ............................ SKIPPED
[INFO] Apache Hadoop MapReduce Streaming .................. SKIPPED
[INFO] Apache Hadoop Distributed Copy ..................... SKIPPED
[INFO] Apache Hadoop Archives ............................. SKIPPED
[INFO] Apache Hadoop Archive Logs ......................... SKIPPED
[INFO] Apache Hadoop Rumen ................................ SKIPPED
[INFO] Apache Hadoop Gridmix .............................. SKIPPED
[INFO] Apache Hadoop Data Join ............................ SKIPPED
[INFO] Apache Hadoop Extras ............................... SKIPPED
[INFO] Apache Hadoop Pipes ................................ SKIPPED
[INFO] Apache Hadoop OpenStack support .................... SKIPPED
[INFO] Apache Hadoop Amazon Web Services support .......... SKIPPED
[INFO] Apache Hadoop Kafka Library support ................ SKIPPED
[INFO] Apache Hadoop Azure support ........................ SKIPPED
[INFO] Apache Hadoop Aliyun OSS support ................... SKIPPED
[INFO] Apache Hadoop Client Aggregator .................... SKIPPED
[INFO] Apache Hadoop Mini-Cluster ......................... SKIPPED
[INFO] Apache Hadoop Scheduler Load Simulator ............. SKIPPED
[INFO] Apache Hadoop Azure Data Lake support .............. SKIPPED
[INFO] Apache Hadoop Tools Dist ........................... SKIPPED
[INFO] Apache Hadoop Tools ................................ SKIPPED
[INFO] Apache Hadoop Client API ........................... SKIPPED
[INFO] Apache Hadoop Client Runtime ....................... SKIPPED
[INFO] Apache Hadoop Client Packaging Invariants .......... SKIPPED
[INFO] Apache Hadoop Client Test Minicluster .............. SKIPPED
[INFO] Apache Hadoop Client Packaging Invariants for Test . SKIPPED
[INFO] Apache Hadoop Client Packaging Integration Tests ... SKIPPED
[INFO] Apache Hadoop Distribution ......................... SKIPPED
[INFO] Apache Hadoop Client Modules ....................... SKIPPED
[INFO] Apache Hadoop Cloud Storage ........................ SKIPPED
[INFO] Apache Hadoop Cloud Storage Project ................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:32 min
[INFO] Finished at: 2017-10-24T11:15:25+08:00
[INFO] Final Memory: 96M/848M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:native-maven-plugin:1.0-alpha-8:javah (default) on project hadoop-common: Error running javah command: Error executing command line. Exit code:1 -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :hadoop-common

可以看到，在此处由于命令行太长而导致编译失败。因此需要将maven仓库的默认路径设置为短路径，并且被编译的hadoop源码的路径也尽量短。

3.关于windowsSDK7的安装。可能需要修改注册表，在修改注册表时，可能需要修改该注册表数值的所属用户。具体可以参考在Windows 10 64-bit上安装Windows SDK 7.1和.NET4。

4.在编译时，最好以管理员模式打开编译工具。

进行编译

我使用的是WindowsSDK7,以管理员模式打开，执行如下命令编译。
mvn package -Pdist,native-win -DskipTests -Dtar

获取hadoop.dll和winutils.exe

编译成功后，在"hadoop-common-project\hadoop-common\target\bin"中，可以找到编译好的hadoop.dll和winutils.exe

本人还是学生，所知甚少，文中若有错误，请不吝指出。

win10编译hadoop3.0.0-beta1源码，获取win

安装需要的编译工具

进行编译

获取hadoop.dll和winutils.exe

相关文章

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读

Hadoop