一、问题
Hive的用户反馈: 同一段SQL, 在HUE的交互窗口上是可以正常执行并返回结果的,
错误信息
dmp_common_c_cms_page_block initializer failed, vertex=vertex_1556623010488_70481_1_06 [Map 7], org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:354)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:159)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:61)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:211)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:302)
at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:167)
at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:162)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:799)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:193)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:89)
at org.apache.hadoop.hbase.client.MetaScanner.allTableRegions(MetaScanner.java:324)
at org.apache.hadoop.hbase.client.HRegionLocator.getAllRegionLocations(HRegionLocator.java:90)
at org.apache.hadoop.hbase.util.RegionSizeCalculator.init(RegionSizeCalculator.java:94)
at org.apache.hadoop.hbase.util.RegionSizeCalculator.<init>(RegionSizeCalculator.java:81)
at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:256)
at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getSplitsInternal(HiveHBaseTableInputFormat.java:527)
at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getSplits(HiveHBaseTableInputFormat.java:453)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:311)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:413)
at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:155)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:273)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:266)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:266)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
]Vertex killed, vertexName=Map 6, vertexId=vertex_1556623010488_70481_1_07, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1556623010488_70481_1_07 [Map 6] killed/failed due to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Reducer 2, vertexId=vertex_1556623010488_70481_1_10, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1556623010488_70481_1_10 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Map 8, vertexId=vertex_1556623010488_70481_1_01, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1556623010488_70481_1_01 [Map 8] killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:3 (state=08S01,code=2)
Closing: 0: jdbc:hive2://hiveserver02.bigdata.hbh.local:10000/default
Intercepting System.exit(2)
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.Hive2Main], exit code [2]
二、找原因
从堆栈信息也可以看到是hbase的操作失败了.
明明是hive的SQL为什么会涉及到HBASE的内容呢?
SQL里导致报错的这张表是基于HBASE的hive表的视图, 层级关系: Hive view => Hive external table => HBASE table, 结合上面的堆栈信息可以看到是hive在从HBASE中获取表数据的时候发生了异常. 所以应该围绕这个问题去分析.
从网上的搜索报错信息Can't get the location for replica 0来看, 问题原因集中在:
1. hbase中没有需要查找的表
在hive中用show create table查看其中映射的表在HBASE中是否存在, 是不是映射错误.
2. hbase表错误
执行hbase hbck 命令检查是否有表错误
3. zookeeper连接错误
为什么hive连接hbase表查数据会因为zk连接异常而失败呢?
这里需要了解一下客户端读取hbase表的流程, 简单解释一下: hbase的表存储在各个regionserver中, 而HBASE把具体哪个表存在哪个regionserver上的信息存储在zookeeper中, 客户端没有缓存的时候就会去zk获取这个数据, zk地址填写错误或者连接失败的时候也会产生这种报错.
要确定是不是这个问题, 首先要找到相应的报错日志. 上面所截取的报错信息里并没有相关的内容. 原因的查找一度停滞. 后来突然想起我一直在查找oozie - launcher中的日志, 而忽略了oozie - action.
oozie在向Yarn提交任务的时候, 除了会提交相应的需要执行的任务(oozie - action)外, 还会提交一个监控任务(oozie-launcher), 用于监控任务是否执行成功.
后面在oozie-action中的syslog_dag_xxxxxx_post中找到下面的日志, 发现连接的是localhost, 而不是hive-site.xml和hbase-site.xml所配置的地址. 所以检查oozie的sharelib, 但是更新了sharelib中的内容之后依然还是有这个问题.
2019-06-13 13:47:50,267 [WARN] [InputInitializer {Map 10} #0] |zookeeper.ZKUtil|: hconnection-0x7971d4140x0, quorum=localhost:2181, baseZNode=/hbase Unable to set watcher on znode (/hbase/hbaseid)
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1386)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1040)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:220)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:420)
at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:105)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:903)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:648)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getSplitsInternal(HiveHBaseTableInputFormat.java:466)
at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getSplits(HiveHBaseTableInputFormat.java:453)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:311)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:413)
at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:155)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:273)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:266)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:266)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2019-06-13 13:47:50,268 [INFO] [InputInitializer {Map 10} #0] |client.ZooKeeperRegistry|: ClusterId read in ZooKeeper is null
2019-06-13 13:47:50,268 [INFO] [InputInitializer {Map 10} #0-SendThread(localhost:2181)] |zookeeper.ClientCnxn|: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2019-06-13 13:47:50,268 [WARN] [InputInitializer {Map 10} #0-SendThread(localhost:2181)] |zookeeper.ClientCnxn|: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1125)
最终解决
在日志中找到zk的lib列表, 将hive-site.xml和hbase-site.xml软连接到zkclasspath中的一个文件夹中.
2019-06-13 13:46:21,831 [INFO] [InputInitializer {Map 4} #0] |zookeeper.ZooKeeper|: Client environment:java.class.path=/usr/hdp/2.6.4.0-91/hadoop/lib/hadoop-lzo-0.6.0.2.6.4.0-91.jar:/etc/hadoop/conf/secure:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/hive-hcatalog-core.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/htrace-core-3.1.0-incubating.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/hive-exec-1.2.1000.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/phoenix-hive.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/phoenix-4.14.0-HBase-1.1-hive.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/hive-hbase-handler-1.2.1000.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/guava-14.0.1.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/zookeeper-3.4.6.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/hadoop-common-2.7.3.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/hive-exec-1.2.1000.2.6.4.0-91-c70b7ab5a6593801480f931ee6add8f30aea9dacdd4ac4976569250e040da4f7.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/tez-job-analyzer-0.7.0.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/tez-yarn-timeline-history-with-acls-0.7.0.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/tez-history-parser-0.7.0.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/tez-common-0.7.0.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/tez-runtime-internals-0.7.0.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/tez-tests-0.7.0.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/tez-mapreduce-0.7.0.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/tez-yarn-timeline-history-with-fs-0.7.0.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/tez-yarn-timeline-history-0.7.0.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/tez-dag-0.7.0.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/tez-runtime-library-0.7.0.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/tez-yarn-timeline-cache-plugin-0.7.0.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/tez-examples-0.7.0.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/tez-api-0.7.0.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/curator-client-2.7.1.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/api-util-1.0.0-M20.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/hadoop-yarn-server-web-proxy-2.7.3.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/commons-cli-1.2.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/hadoop-yarn-common-2.7.3.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/commons-beanutils-1.7.0.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/stax-api-1.0.1.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/jackson-core-asl-1.9.13.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/commons-lang-2.6.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/hadoop-mapreduce-client-core-2.7.3.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/guava-11.0.2.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/aws-java-sdk-s3-1.10.6.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/servlet-api-2.5.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/okio-1.4.0.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/commons-math3-3.1.1.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/commons-logging-1.1.3.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/commons-collections-3.2.2.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/commons-beanutils-core-1.8.0.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/activation-1.1.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/jackson-annotations-2.2.3.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/xz-1.0.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/hadoop-yarn-client-2.7.3.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/commons-codec-1.4.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/commons-net-3.1.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/joda-time-2.9.4.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/stax-api-1.0-2.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/jersey-client-1.9.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/okhttp-2.4.0.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/api-asn1-api-1.0.0-M20.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/jaxb-api-2.2.2.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/hadoop-azure-datalake-2.7.3.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/azure-keyvault-core-0.8.0.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/jsr305-2.0.3.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/javax.inject-1.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/htrace-core-3.1.0-incubating.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/jackson-core-2.2.3.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/apacheds-i18n-2.0.0-M15.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/httpcore-4.4.4.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/commons-digester-1.8.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/jetty-6.1.26.hwx.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/leveldbjni-all-1.8.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/azure-storage-5.4.0.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/hadoop-annotations-2.7.3.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/avro-1.7.4.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/slf4j-log4j12-1.7.5.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/jersey-core-1.9.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/guice-servlet-3.0.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/jackson-jaxrs-1.9.13.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/hadoop-yarn-server-timeline-pluginstorage-2.7.3.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/xercesImpl-2.9.1.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/protobuf-java-2.5.0.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/hadoop-yarn-server-applicationhistoryservice-2.7.3.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/aopalliance-1.0.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/hadoop-yarn-server-common-2.7.3.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/aws-java-sdk-kms-1.10.6.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/commons-collections4-4.1.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/hadoop-aws-2.7.3.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/hadoop-yarn-registry-2.7.3.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/metrics-core-3.1.0.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/commons-compress-1.4.1.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/curator-recipes-2.7.1.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/fst-2.24.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/jaxb-impl-2.2.3-1.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/azure-data-lake-store-sdk-2.1.4.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/snappy-java-1.0.4.1.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/jetty-sslengine-6.1.26.hwx.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/hadoop-hdfs-2.7.3.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/netty-all-4.0.52.Final.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/objenesis-2.1.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/jetty-util-6.1.26.hwx.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/httpclient-4.5.2.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/hadoop-auth-2.7.3.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/jersey-json-1.9.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/commons-lang3-3.4.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/xmlenc-0.52.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/jcip-annotations-1.0.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/jettison-1.3.4.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/jackson-databind-2.2.3.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/javassist-3.18.1-GA.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/hadoop-azure-2.7.3.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/slf4j-api-1.7.5.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/xml-apis-1.3.04.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/gson-2.2.4.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/jersey-guice-1.9.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/jackson-xc-1.9.13.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/aws-java-sdk-core-1.10.6.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/log4j-1.2.17.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/guice-3.0.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/paranamer-2.3.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/nimbus-jose-jwt-3.9.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/jackson-mapper-asl-1.9.13.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/curator-framework-2.7.1.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/zookeeper-3.4.6.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/hadoop-common-2.7.3.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/netty-3.6.2.Final.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/hadoop-yarn-api-2.7.3.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/commons-configuration-1.6.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/commons-io-2.4.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/servlet-api-2.5-20081211.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/hadoop-mapreduce-client-common-2.7.3.2.6.4.0-91.jar:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001/tezlib/lib/json-smart-1.1.1.jar:/usr/hdp/2.6.4.0-91/hadoop/conf:
2019-06-13 13:46:21,832 [INFO] [InputInitializer {Map 4} #0] |zookeeper.ZooKeeper|: Client environment:java.library.path=/usr/hdp/2.6.4.0-91/hadoop/lib/native:/usr/hdp/2.6.4.0-91/hadoop/lib/native/Linux-amd64-64:./tezlib/lib/native:./tezlib/lib/native/Linux-amd64-64:/grid/0/hadoop/yarn/local/usercache/USER_NAME/appcache/application_1556623010488_72538/container_e237_1556623010488_72538_01_000001:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
application_1556623010488_72538是yarn上oozie-action的appID, 可以看到并没有launher的id(application_1556623010488_72537);
总结
上面所述的问题解决方案并不"优雅", 也没有彻底了解为何oozie没有将上下文环境传递给hbase client所执行的zk环境中. 但是上述方案确实可以解决问题, 具体的问题还需要根据.
另外这里还需要考虑一下, 从hue直接提交SQL运行是直接提交到hiveserver去执行的, 而workflow方式是hue以任务的形式提交到oozie上再触发HiveServer执行的. 所以会导致环境不一致的问题.
网友评论