Spark JDBC 自定义方言

前言

在大数据开发中，少不了有对各种数据库的读写。经常会出现一些字段类型与数据库期望不一致报错。Spark 默认的方言对象都存放在org.apache.spark.sql.jdbc 包下。Spark 通过这些方言配置Java Bean ，scala BeanType 与对应数据库字段类型的关系。

问题产生背景

由于老业务系统使用的是SqlServer,在将Boolean类型数据回写数据时产生 异常（bit 类型 不允许设置 宽度）。

问题分析

很明显的类型设置错误，把 scala boolean 类型翻译成SqlServer 成 bit 并且为他设置长度，所以导致报错。
查看源码发现spark 并没有特殊处理 Boolean 这种数据类型(至少在 2.1.0 版本如此)。 源码如下

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.spark.sql.jdbc

import org.apache.spark.sql.types._


private object MsSqlServerDialect extends JdbcDialect {

  override def canHandle(url: String): Boolean = url.startsWith("jdbc:sqlserver")

  override def getCatalystType(
      sqlType: Int, typeName: String, size: Int, md: MetadataBuilder): Option[DataType] = {
    if (typeName.contains("datetimeoffset")) {
      // String is recommend by Microsoft SQL Server for datetimeoffset types in non-MS clients
      Option(StringType)
    } else {
      None
    }
  }

  override def getJDBCType(dt: DataType): Option[JdbcType] = dt match {
    case TimestampType => Some(JdbcType("DATETIME", java.sql.Types.TIMESTAMP))
    case _ => None
  }

  override def isCascadingTruncateTable(): Option[Boolean] = Some(false)
}

解决思路

查看源码发现spark 对时间类型做了特殊处理，直接参考现成的例子创建一个类。并重写getJDBCType
由于MsSqlServerDialect 是私有的，所以我们只能继承JdbcDialect。具体实现类如下

package cn.harsons.bd.common
import org.apache.spark.sql.jdbc.{JdbcDialect, JdbcType}
import org.apache.spark.sql.types._

/**
  *
  * @author liyabin
  * @date 2020/3/24 0024
  */
object SQLServerJdbcDialect extends JdbcDialect {

  override def canHandle(url: String): Boolean = url.startsWith("jdbc:sqlserver")

  override def getCatalystType(
                                sqlType: Int, typeName: String, size: Int, md: MetadataBuilder): Option[DataType] = {
    if (typeName.contains("datetimeoffset")) {
      // String is recommend by Microsoft SQL Server for datetimeoffset types in non-MS clients
      Option(StringType)
    }  else {
      None
    }
  }

  override def getJDBCType(dt: DataType): Option[JdbcType] = dt match {
    case TimestampType => Some(JdbcType("DATETIME", java.sql.Types.TIMESTAMP))
    case BooleanType => Some(JdbcType("tinyint",java.sql.Types.BOOLEAN))
    case _ => None
  }

  override def isCascadingTruncateTable(): Option[Boolean] = Some(false)
}

上面只是把方言对象定义好了，如果让方言生效？
JdbcDialects.registerDialect(SQLServerJdbcDialect) 可以直接让方言生效，大家也不必担心spark会不会读默认的MsSqlServerDialect 毕竟他们canHandle 条件都是一样的，一般情况下都是配置大于约定。Spark会优先加载自定义的。