mapreduce 本地开发环境

本地环境配置

1、解压缩hadoop.*.tar.gz
2、解压缩hadoop.dll、winutils.exe等安装文件，并拷贝到hadopp解压缩文件夹的bin目录下
3、配置环境变量HADOOP_HOME，并将%HADOOP_HOME%\bin 添加到path中
4、重启IDE（eclipse）

问题汇总

winutils不存在

image.png

output文件已存在

Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory file:/d:/output already exists
    at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:146)
    at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:266)
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:139)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1758)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308)
    at com.xuyang.mapreduce.WordCountDriver.main(WordCountDriver.java:37)

image.png