异常说明
SparkSession初始化一个lazy的HiveClient用于Hive MetaStore连接，一旦使用其内部就会实例化一个org.apache.spark.sql.hive.HiveSessionState对象，执行SessionState.start方法时会根据HiveConf里的配置执行一系列操作，包括创建一些临时的工作目录等。这些目录如果没有正确的权限会导致这个过程失败，致使连接不上Server.
客户端异常
hzyaoqin@hadoop980:~/spark-packages/spark-2.1.0-bin-ne-1.3.0-SNAPSHOT$ bin/beeline -u 'jdbc:hive2://hadoop980.hz.163.org:10001/default;principal=hive/hadoop980.hz.163.org@TEST.MAMMUT.NETEASE.COM;hive.server2.proxy.user=hzyaoqin;spark.sql.warehouse.dir=/user/hzyaoqin/warehouse'
Connecting to jdbc:hive2://hadoop980.hz.163.org:10001/default;principal=hive/hadoop980.hz.163.org@TEST.MAMMUT.NETEASE.COM;hive.server2.proxy.user=hzyaoqin;spark.sql.warehouse.dir=/user/hzyaoqin/warehouse
17/09/06 17:17:14 INFO jdbc.Utils: Supplied authorities: hadoop980.hz.163.org:10001
17/09/06 17:17:14 INFO jdbc.Utils: Resolved authority: hadoop980.hz.163.org:10001
17/09/06 17:17:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/09/06 17:17:15 INFO jdbc.HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://hadoop980.hz.163.org:10001/default;principal=hive/hadoop980.hz.163.org@TEST.MAMMUT.NETEASE.COM;hive.server2.proxy.user=hzyaoqin;spark.sql.warehouse.dir=/user/hzyaoqin/warehouse
Can't overwrite cause with java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':
Beeline version 1.2.1.spark2 by Apache Hive
0: jdbc:hive2://hadoop980.hz.163.org:10001/de (closed)>
其中异常主体为，
Can't overwrite cause with java.lang.IllegalArgumentException:
Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':
， 实际意为初始化HiveSessionState时发生了不可名状的事情；进一步去看服务端日志；
注：HiveSession级别的日志没有做到客户端这边，只有Operation级别的日志可以看哦
服务端异常
Error opening session:
org.apache.spark.SparkException: Failed Init SparkSession for user[hzyaoqin]java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':
at org.apache.spark.sql.hive.thriftserver.multitenancy.SparkHiveSessionImpl.createSparkSession(SparkHiveSessionImpl.scala:135)
at org.apache.spark.sql.hive.thriftserver.multitenancy.SparkHiveSessionImpl.getOrCreateSparkSession(SparkHiveSessionImpl.scala:109)
at org.apache.spark.sql.hive.thriftserver.multitenancy.SparkHiveSessionImpl.open(SparkHiveSessionImpl.scala:186)
at org.apache.spark.sql.hive.thriftserver.multitenancy.ThriftServerSessionManager.openSession(ThriftServerSessionManager.scala:225)
at org.apache.spark.sql.hive.thriftserver.multitenancy.ThriftServerCLIService.openSessionWithImpersonation(ThriftServerCLIService.scala:91)
at org.apache.spark.sql.hive.thriftserver.multitenancy.ThriftClientCLIService.getSessionHandle(ThriftClientCLIService.scala:252)
at org.apache.spark.sql.hive.thriftserver.multitenancy.ThriftClientCLIService.OpenSession(ThriftClientCLIService.scala:266)
at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1253)
at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1238)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:692)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':
at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:1120)
at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:111)
at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:110)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$11.apply(SparkSession.scala:888)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$11.apply(SparkSession.scala:888)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:888)
at org.apache.spark.sql.hive.thriftserver.multitenancy.SparkHiveSessionImpl$$anon$1.run(SparkHiveSessionImpl.scala:124)
at org.apache.spark.sql.hive.thriftserver.multitenancy.SparkHiveSessionImpl$$anon$1.run(SparkHiveSessionImpl.scala:118)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.spark.sql.hive.thriftserver.multitenancy.SparkHiveSessionImpl.createSparkSession(SparkHiveSessionImpl.scala:118)
... 15 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:1117)
... 31 more
Caused by: java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveExternalCatalog':
at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$$reflect(SharedState.scala:183)
at org.apache.spark.sql.internal.SharedState.<init>(SharedState.scala:90)
at org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:102)
at org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:102)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession.sharedState$lzycompute(SparkSession.scala:102)
at org.apache.spark.sql.SparkSession.sharedState(SparkSession.scala:101)
at org.apache.spark.sql.internal.SessionState.<init>(SessionState.scala:157)
at org.apache.spark.sql.hive.HiveSessionState.<init>(HiveSessionState.scala:32)
... 36 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$$reflect(SharedState.scala:180)
... 44 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:264)
at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:367)
at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:271)
at org.apache.spark.sql.hive.HiveExternalCatalog.<init>(HiveExternalCatalog.scala:65)
... 49 more
Caused by: java.lang.RuntimeException: org.apache.hadoop.security.AccessControlException: Permission denied: user=hzyaoqin, access=EXECUTE, inode="/tmp/hive/public/hzyaoqin/9869b00f-0629-4290-ad0f-71f3c9ed0566":hdfs:hadoop:drwxr-x---
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:259)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:205)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1722)
at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:108)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3863)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1012)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:843)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
非常明显,hzyaoqin用户没有用户hdfs所属目录的执行权限；
Permission denied: user=hzyaoqin, access=EXECUTE, inode="/tmp/hive/public/hzyaoqin/9869b00f-0629-4290-ad0f-71f3c9ed0566":hdfs:hadoop:drwxr-x---
这个目录为hive.exec.strachdir（缺省为/tmp/hive/public）下创建username/sessionId的临时目录，我们多租户下执行都是用hzyaoqin这个执行账号去真正执行的，这个目录如果是hdfs用户的，必然不是有本次session创建的，应该是以前的遗留目录
解决方案
- 1、删除/tmp/hive/public/hzyaoqin（如果这目录没啥用），重新连接
- 2、重新配置hive.exec.strachdir到另一个有权限的目录，可以在连接串中动态的session级别指定或classpath中的hive-site.xml进行server级别的全局设置，前者重新连接，后者重启server。
网友评论