windows环境javaAPI操作hadoop

idea 工具创建java项目,添加hadoop依赖

maven依赖

      <properties>
        <java.version>1.8</java.version>
        <hadoop.version>2.8.5</hadoop.version>
      </properties>
    <dependencies>
       <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-mapreduce-client-core</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
    <dependencies>

windows系统解压hadoop-2.8.5.tar.gz

将winutils下选相应的版本的bin目录下的文件复制到hadoop解压出来的bin目录下

报错处理

hadoop权限问题

org.apache.hadoop.ipc.RemoteException: Permission denied: user=administration, access=WRITE, inode="/":root:supergroup:drwxr-xr-x

解决 :

设置名称,设置hadoop的用户名,我这里是root

System.setProperty("HADOOP_USER_NAME","root");

找不到hadoop.home.dir

16:42:22.156 [main] DEBUG org.apache.hadoop.util.Shell - Failed to detect a valid hadoop home directory
java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset.

解决 :
设置访问路径

//就是hadoop的HADOOP_HOME路径,可能是设置hadoop的path没有生效需要这个设置
System.setProperty("hadoop.home.dir", "D:\\user\\hadoop\\");

注意: 远程的hadoop的配置文件中不能使用localhost或者127.0.0.1,需要换成对应的ip

   <property>
        <name>fs.defaultFS</name>
        <value>hdfs://127.0.0.1:9000</value>
    </property>