Hive执行DDL操作后的常见错误，错误分析及处理方式
- 创建一张orc表，并且向该表插入数据，这时候再新增字段，插入新的数据，然后count这张表的时候就会报该异常！报错如下：
2021-03-03 11:13:01,989 INFO [IPC Server handler 24 on 33697] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1584525046250_5922717_m_000001_0: Error: java.lang.RuntimeException: Error creating a batch
at org.apache.hadoop.hive.ql.io.orc.VectorizedOrcInputFormat$VectorizedOrcRecordReader.createValue(VectorizedOrcInputFormat.java:111)
at org.apache.hadoop.hive.ql.io.orc.VectorizedOrcInputFormat$VectorizedOrcRecordReader.createValue(VectorizedOrcInputFormat.java:49)
at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.createValue(CombineHiveRecordReader.java:94)
at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.createValue(CombineHiveRecordReader.java:43)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.createValue(HadoopShimsSecure.java:155)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.createValue(MapTask.java:180)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:459)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: No type found for column type entry 11
at org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx.addScratchColumnsToBatch(VectorizedRowBatchCtx.java:604)
at org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx.createVectorizedRowBatch(VectorizedRowBatchCtx.java:339)
at org.apache.hadoop.hive.ql.io.orc.VectorizedOrcInputFormat$VectorizedOrcRecordReader.createValue(VectorizedOrcInputFormat.java:109)
处理方案：
- 方案一： 在执行sql前执行set hive.vectorized.execution.enabled = false;
- 方案二：修复分区
- 第一种情况：一层分区的情况；执行 MSCK REPAIR TABLE table_name;
- 第二种情况：多层分区情况；执行 set hive.msck.path.validation=ignore; MSCK REPAIR TABLE table_name;
- orc分区表，修改字段类型，查询历史分区数据会报错，报错如下：
Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {2019-12-05, 757664, 390, 10448, 3, 102, 2017-08-05 00:03:54.0, , null, null, , , 0, null, , , null, , , null, 102, 2017-08-05 00:03:54.0, 2017-08-05 00:03:54.0, , 1, haier, 2017-08-04 22:44:07.0, null, 2019-12-05 18:34:59.0, null, 10448, 2020-06-02 15:31:40, 2020-06-02 15:31:40, 2020-06-02 15:42:08, 2020-11-02 18:24:01, null}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:179)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:459)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.access$200(MapOperator.java:100)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(Ma
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {2019-12-05, 757664, 390, 10448, 3, 102, 2017-08-05 00:03:54.0, , null, null, , , 0, null, , , null, , , null, 102, 2017-08-05 00:03:54.0, 2017-08-05 00:03:54.0, , 1, haier, 2017-08-04 22:44:07.0, null, 2019-12-05 18:34:59.0, null, 10448, 2020-06-02 15:31:40, 2020-06-02 15:31:40, 2020-06-02 15:42:08, 2020-11-02 18:24:01, null}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:505)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:170)
... 8 more
Caused by: java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot be cast to org.apache.hadoop.io.IntWritable
at org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableIntObjectInspector.get(WritableIntObjectInspector.java:36)
at org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getLong(PrimitiveObjectInspectorUtils.java:671)
at org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$LongConverter.convert(PrimitiveObjectInspectorConverter.java:180)pOperator.java:496)
at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.convert(ObjectInspectorConverters.java:406)
at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.readRow(MapOperator.java:137)
原因为：单纯修改字段类型，不会涉及到历史分区元数据信息的修改；
处理方案：
- 添加动态分区，执行修改语句
-- 首先开启动态分区
SET hive.exec.dynamic.partition = true;
-- 指定分区修改字段类型
ALTER TABLE tab_name partition(pdate) change oldCol newCol newType comment '';
- 添加级联参数，级联执行
ALTER TABLE tab_name change oldCol newCol newType comment '' CASCADE;
网友评论