前言
dolphinscheduler安装版本:3.1.8
hive版本:2.1.1-cdh6.3.2
发现问题&分析问题
- 添加数据源报错
[ERROR] 2021-01-08 10:55:40.553 org.apache.dolphinscheduler.api.service.DataSourceService:
[433] - Could not establish connection to jdbc:hive2://10.0.4.78:10000/vfs_dw: Required field 'client_protocol' is unset!
Struct:TOpenSessionReq(client_protocol:null, configuration:{use:database=vfs_dw})
java.sql.SQLException: Could not establish connection to jdbc:hive2://10.0.4.78:10000/vfs_dw:
Required field 'client_protocol' is unset!
Struct:TOpenSessionReq(client_protocol:null, configuration:{use:database=vfs_dw})
2.运行任务报错
1.Dolphinscheduler任务启动报错,协议错误
2.Dolphinscheduler任务全部启动异常,所有实例都变成提交成功,但是没有日志,也不会提交
解决办法
解决1:将hive-jdbc-2.1.1-cdh6.3.2-standalone.jar 放到api和work的lib目录,删除多余hive jar包
发现Hive数据可以配置成功,但是ds所有工作流启动不了了,启动就变成提交成功,没有日志。
解决2:将cdh所有hive-*.jar都放到api和work的lib目录,删除多余hive jar包
发现work启动异常,日志出现hive-jdbc-2.1.1-cdh6.3.2-standalone相关报错
Exception in thread "Worker-Server" java.lang.IllegalArgumentException:
LoggerFactory is not a Logback LoggerContext but Logback is on the classpath.
Either remove Logback or the competing implementation
(class org.apache.logging.slf4j.Log4jLoggerFactory loaded from file:/opt/software/dolphinscheduler-3.1.8/worker-server/libs/hive-jdbc-2.1.1-cdh6.3.2-standalone.jar).
If you are using WebLogic you will need to add 'org.slf4j' to prefer-application-packages in WEB-INF/weblogic.xml: org.apache.logging.slf4j.Log4jLoggerFactory
解决3:将cdh hive包选择性替换到api和work的lib目录
成功解决
#api-server
#备份libs
cp -ar /opt/software/dolphinscheduler-3.1.8/api-server/libs/ /opt/software/dolphinscheduler-3.1.8/api-server/libs-bac
#复制到api-server
cp /opt/cloudera/parcels/CDH/jars/hive-jdbc-2.1.1-cdh6.3.2-standalone.jar /opt/software/dolphinscheduler-3.1.8/api-server/libs/
#权限改为dolphinscheduler
chown dolphinscheduler:dolphinscheduler /opt/software/dolphinscheduler-3.1.8/api-server/libs/hive-*.jar
#删除ds自带的所有hive错误版本相关驱动依赖
rm -rf /opt/software/dolphinscheduler-3.1.8/api-server/libs/hive-*2.3.9.jar
#check
ll /opt/software/dolphinscheduler-3.1.8/api-server/libs/|grep hive
#备份libs
cp -ar /opt/software/dolphinscheduler-3.1.8/worker-server/libs/ /opt/software/dolphinscheduler-3.1.8/worker-server/libs-bac
#worker-server
#删除ds自带的所有hive错误版本相关驱动依赖
rm -rf /opt/software/dolphinscheduler-3.1.8/worker-server/libs/hive-*.jar
#复制到worker-server
cp -n /opt/cloudera/parcels/CDH/jars/hive-common*.jar /opt/software/dolphinscheduler-3.1.8/worker-server/libs/
cp -n /opt/cloudera/parcels/CDH/jars/hive-jdbc-2.1.1-cdh6.3.2.jar /opt/software/dolphinscheduler-3.1.8/worker-server/libs/
cp -n /opt/cloudera/parcels/CDH/jars/hive-metastore*.jar /opt/software/dolphinscheduler-3.1.8/worker-server/libs/
cp -n /opt/cloudera/parcels/CDH/jars/hive-serde*.jar /opt/software/dolphinscheduler-3.1.8/worker-server/libs/
cp -n /opt/cloudera/parcels/CDH/jars/hive-service*.jar /opt/software/dolphinscheduler-3.1.8/worker-server/libs/
cp -n /opt/cloudera/parcels/CDH/jars/hive-service-rpc*.jar /opt/software/dolphinscheduler-3.1.8/worker-server/libs/
cp -n /opt/cloudera/parcels/CDH/jars/hive-storage-api*.jar /opt/software/dolphinscheduler-3.1.8/worker-server/libs/
#权限改为dolphinscheduler
chown dolphinscheduler:dolphinscheduler /opt/software/dolphinscheduler-3.1.8/worker-server/libs/hive-*.jar
#check
ll /opt/software/dolphinscheduler-3.1.8/worker-server/libs/|grep hive
网友评论