执行一个计算圆周率的任务
cd /usr/local/hadoop/share/hadoop/mapreduce
hadoop jar hadoop-mapreduce-examples-3.3.0.jar pi 2 10 #2表示线程数，10表示采样点数
问题
Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.v2.app.MRAppMaster
Please check whether your <HADOOP_HOME>/etc/hadoop/mapred-site.xml contains the below configuration:
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
</property>
错误信息指出我没有配置以上三个变量，所以在mapred-site.xml文件中追加这三个配置。
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=${HADOOP_COMMON_HOME}</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=${HADOOP_COMMON_HOME}</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=${HADOOP_COMMON_HOME}</value>
</property>
重新运行命令，计算得出结果
root@ALBB:/usr/local/hadoop/share/hadoop/mapreduce# hadoop jar hadoop-mapreduce-examples-3.3.0.jar pi 2 10
Number of Maps = 2
Samples per Map = 10
Wrote input for Map #0
Wrote input for Map #1
Starting Job
2021-05-11 01:23:10,774 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at /0.0.0.0:8032
2021-05-11 01:23:11,450 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1620665185590_0002
2021-05-11 01:23:11,660 INFO input.FileInputFormat: Total input files to process : 2
2021-05-11 01:23:11,753 INFO mapreduce.JobSubmitter: number of splits:2
2021-05-11 01:23:12,241 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1620665185590_0002
2021-05-11 01:23:12,241 INFO mapreduce.JobSubmitter: Executing with tokens: []
2021-05-11 01:23:12,618 INFO conf.Configuration: resource-types.xml not found
2021-05-11 01:23:12,619 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2021-05-11 01:23:12,726 INFO impl.YarnClientImpl: Submitted application application_1620665185590_0002
2021-05-11 01:23:12,779 INFO mapreduce.Job: The url to track the job: http://ALBB:8088/proxy/application_1620665185590_0002/
2021-05-11 01:23:12,780 INFO mapreduce.Job: Running job: job_1620665185590_0002
2021-05-11 01:23:25,132 INFO mapreduce.Job: Job job_1620665185590_0002 running in uber mode : false
2021-05-11 01:23:25,133 INFO mapreduce.Job: map 0% reduce 0%
2021-05-11 01:23:36,275 INFO mapreduce.Job: map 50% reduce 0%
2021-05-11 01:23:37,305 INFO mapreduce.Job: map 100% reduce 0%
2021-05-11 01:23:43,376 INFO mapreduce.Job: map 100% reduce 100%
2021-05-11 01:23:43,386 INFO mapreduce.Job: Job job_1620665185590_0002 completed successfully
2021-05-11 01:23:43,551 INFO mapreduce.Job: Counters: 54
File System Counters
FILE: Number of bytes read=50
FILE: Number of bytes written=793758
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=526
HDFS: Number of bytes written=215
HDFS: Number of read operations=13
HDFS: Number of large read operations=0
HDFS: Number of write operations=3
HDFS: Number of bytes read erasure-coded=0
Job Counters
Launched map tasks=2
Launched reduce tasks=1
Data-local map tasks=2
Total time spent by all maps in occupied slots (ms)=19023
Total time spent by all reduces in occupied slots (ms)=4483
Total time spent by all map tasks (ms)=19023
Total time spent by all reduce tasks (ms)=4483
Total vcore-milliseconds taken by all map tasks=19023
Total vcore-milliseconds taken by all reduce tasks=4483
Total megabyte-milliseconds taken by all map tasks=19479552
Total megabyte-milliseconds taken by all reduce tasks=4590592
Map-Reduce Framework
Map input records=2
Map output records=4
Map output bytes=36
Map output materialized bytes=56
Input split bytes=290
Combine input records=0
Combine output records=0
Reduce input groups=2
Reduce shuffle bytes=56
Reduce input records=4
Reduce output records=0
Spilled Records=8
Shuffled Maps =2
Failed Shuffles=0
Merged Map outputs=2
GC time elapsed (ms)=377
CPU time spent (ms)=1360
Physical memory (bytes) snapshot=599326720
Virtual memory (bytes) snapshot=7957811200
Total committed heap usage (bytes)=332320768
Peak Map Physical memory (bytes)=233218048
Peak Map Virtual memory (bytes)=2649092096
Peak Reduce Physical memory (bytes)=133976064
Peak Reduce Virtual memory (bytes)=2659897344
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=236
File Output Format Counters
Bytes Written=97
Job Finished in 33.008 seconds
Estimated value of Pi is 3.80000000000000000000
网友评论