美文网首页
Dolphinscheduler3.1.8兼容Hive2.1.1

Dolphinscheduler3.1.8兼容Hive2.1.1

作者: 圆企鹅i | 来源:发表于2023-11-22 10:20 被阅读0次

前言

dolphinscheduler安装版本:3.1.8
hive版本:2.1.1-cdh6.3.2

发现问题&分析问题

  1. 添加数据源报错
[ERROR] 2021-01-08 10:55:40.553 org.apache.dolphinscheduler.api.service.DataSourceService:
[433] - Could not establish connection to jdbc:hive2://10.0.4.78:10000/vfs_dw: Required field 'client_protocol' is unset! 
Struct:TOpenSessionReq(client_protocol:null, configuration:{use:database=vfs_dw})
java.sql.SQLException: Could not establish connection to jdbc:hive2://10.0.4.78:10000/vfs_dw: 
Required field 'client_protocol' is unset! 
Struct:TOpenSessionReq(client_protocol:null, configuration:{use:database=vfs_dw})

2.运行任务报错
1.Dolphinscheduler任务启动报错,协议错误
2.Dolphinscheduler任务全部启动异常,所有实例都变成提交成功,但是没有日志,也不会提交

解决办法

解决1:将hive-jdbc-2.1.1-cdh6.3.2-standalone.jar 放到api和work的lib目录,删除多余hive jar包
发现Hive数据可以配置成功,但是ds所有工作流启动不了了,启动就变成提交成功,没有日志。

解决2:将cdh所有hive-*.jar都放到api和work的lib目录,删除多余hive jar包
发现work启动异常,日志出现hive-jdbc-2.1.1-cdh6.3.2-standalone相关报错

Exception in thread "Worker-Server" java.lang.IllegalArgumentException: 
LoggerFactory is not a Logback LoggerContext but Logback is on the classpath. 
Either remove Logback or the competing implementation 
(class org.apache.logging.slf4j.Log4jLoggerFactory loaded from file:/opt/software/dolphinscheduler-3.1.8/worker-server/libs/hive-jdbc-2.1.1-cdh6.3.2-standalone.jar). 
If you are using WebLogic you will need to add 'org.slf4j' to prefer-application-packages in WEB-INF/weblogic.xml: org.apache.logging.slf4j.Log4jLoggerFactory

解决3:将cdh hive包选择性替换到api和work的lib目录
成功解决

#api-server 
#备份libs
cp -ar /opt/software/dolphinscheduler-3.1.8/api-server/libs/ /opt/software/dolphinscheduler-3.1.8/api-server/libs-bac
#复制到api-server 
cp /opt/cloudera/parcels/CDH/jars/hive-jdbc-2.1.1-cdh6.3.2-standalone.jar /opt/software/dolphinscheduler-3.1.8/api-server/libs/
#权限改为dolphinscheduler 
chown dolphinscheduler:dolphinscheduler /opt/software/dolphinscheduler-3.1.8/api-server/libs/hive-*.jar
#删除ds自带的所有hive错误版本相关驱动依赖
rm -rf /opt/software/dolphinscheduler-3.1.8/api-server/libs/hive-*2.3.9.jar
#check 
ll /opt/software/dolphinscheduler-3.1.8/api-server/libs/|grep hive

#备份libs
cp -ar /opt/software/dolphinscheduler-3.1.8/worker-server/libs/ /opt/software/dolphinscheduler-3.1.8/worker-server/libs-bac

#worker-server
#删除ds自带的所有hive错误版本相关驱动依赖 
rm -rf /opt/software/dolphinscheduler-3.1.8/worker-server/libs/hive-*.jar

#复制到worker-server
cp -n /opt/cloudera/parcels/CDH/jars/hive-common*.jar /opt/software/dolphinscheduler-3.1.8/worker-server/libs/
cp -n /opt/cloudera/parcels/CDH/jars/hive-jdbc-2.1.1-cdh6.3.2.jar /opt/software/dolphinscheduler-3.1.8/worker-server/libs/
cp -n /opt/cloudera/parcels/CDH/jars/hive-metastore*.jar /opt/software/dolphinscheduler-3.1.8/worker-server/libs/
cp -n /opt/cloudera/parcels/CDH/jars/hive-serde*.jar /opt/software/dolphinscheduler-3.1.8/worker-server/libs/
cp -n /opt/cloudera/parcels/CDH/jars/hive-service*.jar /opt/software/dolphinscheduler-3.1.8/worker-server/libs/
cp -n /opt/cloudera/parcels/CDH/jars/hive-service-rpc*.jar /opt/software/dolphinscheduler-3.1.8/worker-server/libs/
cp -n /opt/cloudera/parcels/CDH/jars/hive-storage-api*.jar /opt/software/dolphinscheduler-3.1.8/worker-server/libs/

#权限改为dolphinscheduler
chown dolphinscheduler:dolphinscheduler /opt/software/dolphinscheduler-3.1.8/worker-server/libs/hive-*.jar
#check 
ll /opt/software/dolphinscheduler-3.1.8/worker-server/libs/|grep hive

相关文章

  • hive环境搭建

    Hive2.1.1安装部署 一、Hive 运行模式 与Hadoop类似,Hive也有 3 种运行模式: 1. 内嵌...

  • hive2.1.1部署记录

    参考了网上很多的文档,终于部署成功,记录下来方便大家本文衔接《hadoop2.7.3部署记录》http://www...

  • Hive2.1.1集群搭建

    软件环境: 主机配置: 一共m1, m2, m3这五部机, 每部主机的用户名都为centos 集群搭建: 一.搭建...

  • CentOS6.5安装hive2.1.1

    一、安装hadoop集群,这里准备两台hadoop服务器,hd01,hd02 二、安装MySQL数据库 三、下载h...

  • 兼容测试

    兼容性测试 兼容性测试分为软件本身向前或向后兼容、不同版本兼容、数据共享兼容、浏览器兼容、操作系统兼容、分辨率兼容...

  • 向前兼容和向后兼容

    向前兼容 向前兼容:= 向上兼容(Upward Compatibility) 向前兼容(Forward Compa...

  • 记录 - 兼容相关(主要IE)

    事件兼容 页面坐标兼容 过滤输入兼容

  • 模块兼容

    模块兼容 兼容commonjs规范 nodeJs 兼容amd规范 requireJs 兼容browser规范 wi...

  • 兼容性问题

    CSS兼容性问题PC页面布局兼容JS兼容问题移动页面兼容问题 CSS兼容问题.header {_width:100...

  • 前端10

    浏览器前缀 -ms-兼容IE-moz-兼容firefox-o-兼容opera-webkit-兼容谷歌和苹果 JS ...

网友评论

      本文标题:Dolphinscheduler3.1.8兼容Hive2.1.1

      本文链接:https://www.haomeiwen.com/subject/hewiwdtx.html