项目中需要将hdfs数据导入hbase
其中:
hbase 版本 2.0.2
hadoop 版本3.1.1
使用springboot整合hadoop、hbase的maven如下:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>3.1.1</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
<version>2.0.2</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.hbase/hbase-mapreduce
如果要用到TableMapReduceUtil,需要加载下面的依赖 -->
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-mapreduce</artifactId>
<version>2.0.2</version>
</dependency>
运行报错:
java.lang.IllegalAccessError: class org.apache.hadoop.hdfs.web.HftpFileSystem cannot access its superinterface org.apache.hadoop.hdfs.web.TokenAspect$TokenManagementDelegator
结合网上的方案,这个报错应该是由于依赖包里面引入的hadoop-hdfs有冲突,解决方法是在IDEA导出jar包之前,将依赖包中所有hadoop-hdfs-***.jar包删除,具体可参考:
https://stackoverflow.com/questions/62880009/error-through-remote-spark-job-java-lang-illegalaccesserror-class-org-apache-h
https://blog.csdn.net/u014432433/article/details/109222604
既然是由于包冲突,那么结合maven依赖顺序原则,我们可以在第1个dependency加载hadoop-hdfs
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>3.1.1</version>
</dependency>
实际验证可行,问题解决
网友评论