Hadoop之集群运行WordCount

上一篇文章Hadoop之编写WordCount我们在本地搭建的Hadoop运行环境，并在本地运行成功，这篇主要是在上篇的基础上将编写好的WordCount程序打成可执行jar，并在集群上运行。如果你还没有集群环境参考Hadoop集群环境搭建（三台）搭建即可

主要内容：

• 1.修改Job的数据输入和输出文件夹
• 2.打成可执行jar
• 3.提交集群并运行

相关文章：
1.VM12安装配置CentOS7
2.Hadoop集群环境搭建（三台）
3.Hadoop之本地运行WordCount
4.Hadoop之集群运行WordCount
5.Log4j2+Flume+Hdfs日志采集

1.修改Job的数据输入和输出文件夹

由于前面是在本地运行，所以输入文件和输出文件夹都指定在本地

FileInputFormat.setInputPaths(job, "D:\\hadoop\\input");
FileOutputFormat.setOutputPath(job, new Path("D:\\hadoop\\output"));

现在修改为Hdfs上的路径

FileInputFormat.setInputPaths(job, "/input/words.txt");
FileOutputFormat.setOutputPath(job, new Path("/output/wc"));

提前将words.txt上传到hdfs上的input目录下

2.将WordCount打成可执行jar

用Maven打包，在pom.xml里添加如下：

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-jar-plugin</artifactId>
            <version>2.4</version>
            <configuration>
                <excludes>
                    <!-- 过滤指定的文件 -->
                    <exclude>org/**</exclude>
                </excludes>
                <archive>
                    <manifest>
                        <addClasspath>true</addClasspath>
                        <classpathPrefix>lib/</classpathPrefix>
                        <!-- 指定运行的主类 -->
                        <mainClass>me.jinkun.mr.wc.RunWcJob</mainClass>
                    </manifest>
                </archive>
            </configuration>
        </plugin>
    </plugins>
</build>

如果使用idea开发，那么直接在右侧双击package即可

image.png

这时在项目的target下会有名为mapreduce-wc-1.0.jar的jar包

image.png

3.将jar提交集群运行

运行如下命令：

hadoop jar mapreduce-wc-1.0.jar

运行结果如下：

[hadoop@hadoop1 soft-install]$ hadoop jar mapreduce-wc-1.0.jar
18/03/08 17:00:25 INFO client.RMProxy: Connecting to ResourceManager at hadoop1/192.168.2.111:8032
18/03/08 17:00:26 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
18/03/08 17:00:27 INFO input.FileInputFormat: Total input paths to process : 1
18/03/08 17:00:28 INFO mapreduce.JobSubmitter: number of splits:1
18/03/08 17:00:28 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1520498386048_0001
18/03/08 17:00:29 INFO impl.YarnClientImpl: Submitted application application_1520498386048_0001
18/03/08 17:00:29 INFO mapreduce.Job: The url to track the job: http://hadoop1:8088/proxy/application_1520498386048_0001/
18/03/08 17:00:29 INFO mapreduce.Job: Running job: job_1520498386048_0001
18/03/08 17:00:40 INFO mapreduce.Job: Job job_1520498386048_0001 running in uber mode : false
18/03/08 17:00:40 INFO mapreduce.Job:  map 0% reduce 0%
18/03/08 17:00:47 INFO mapreduce.Job:  map 100% reduce 0%
18/03/08 17:00:56 INFO mapreduce.Job:  map 100% reduce 100%
18/03/08 17:00:56 INFO mapreduce.Job: Job job_1520498386048_0001 completed successfully
18/03/08 17:00:56 INFO mapreduce.Job: Counters: 49
        File System Counters
                FILE: Number of bytes read=100
                FILE: Number of bytes written=237705
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=146
                HDFS: Number of bytes written=39
                HDFS: Number of read operations=6
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=2
        Job Counters
                Launched map tasks=1
                Launched reduce tasks=1
                Data-local map tasks=1
                Total time spent by all maps in occupied slots (ms)=4342
                Total time spent by all reduces in occupied slots (ms)=6070
                Total time spent by all map tasks (ms)=4342
                Total time spent by all reduce tasks (ms)=6070
                Total vcore-milliseconds taken by all map tasks=4342
                Total vcore-milliseconds taken by all reduce tasks=6070
                Total megabyte-milliseconds taken by all map tasks=4446208
                Total megabyte-milliseconds taken by all reduce tasks=6215680
        Map-Reduce Framework
                Map input records=4
                Map output records=8
                Map output bytes=78
                Map output materialized bytes=100
                Input split bytes=100
                Combine input records=0
                Combine output records=0
                Reduce input groups=5
                Reduce shuffle bytes=100
                Reduce input records=8
                Reduce output records=5
                Spilled Records=16
                Shuffled Maps =1
                Failed Shuffles=0
                Merged Map outputs=1
                GC time elapsed (ms)=206
                CPU time spent (ms)=1430
                Physical memory (bytes) snapshot=300036096
                Virtual memory (bytes) snapshot=4156841984
                Total committed heap usage (bytes)=141660160
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters
                Bytes Read=46
        File Output Format Counters
                Bytes Written=39

查看结果：
在Hdfs的webui里可以看到如下结果

image.png

其中part-r-00000里就存放的计算结果。

到此我们介绍了2种运行mapreduce的方式，一种本地模式便于本地调试，一种集群模式用于生产环境。