spark远程读写hive数据库

作者: iE简 | 来源:发表于2020-12-04 21:39 被阅读0次

spark远程读写hive数据库
Spark之HiveSupport连接（spark-shell和
Hive数据源实战
spark sql use hive
8.Spark学习(Python版本)：连接Hive读写数据（D
Spark下连接星环Hive2失败
Spark On Hive 部署和配置
Spark SQL
Hive On Spark搭建(cdh)
Hive on Spark参数调优姿势小结

实现spark远程连接hive数据库，需要将服务端mysql数据库里的hive数据表DBS和SDS里的localhost改为可以访问到的IP地址或域名。

image.png

更改方式参考：hive修改localhost
本机spark访问服务端应采用域名的方式。

配置hostname

作者采用dbhive作为统一访问hostname
服务端修改/etc/hosts:

nano /etc/hosts

# 10.211.55.2是服务端内网地址
10.211.55.2     dbhive

本机修改/etc/hosts:

nano /etc/hosts

# 119.23.5.xx是服务端外网地址
119.23.5.xx     dbhive

服务端配置

hadoop下的core-site.xml:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
        <name>fs.defaultFS</name>
        <value>hdfs://0.0.0.0:9000</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/Users/zhi/Documents/app/hadoopdata</value>
    </property>
    <property>
        <name>hadoop.proxyuser.root.hosts</name>
        <value>*</value>
    </property>
    <property>
        <name>hadoop.proxyuser.root.groups</name>
        <value>*</value>
    </property>
</configuration>

fs.defaultFS: 注意将ip设为0.0.0.0，允许外部访问。

hadoop下的hdfs-site.xml:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.datanode.hostname</name>
        <value>dbhive</value>
    </property>
    <property>
        <name>dfs.client.use.datanode.hostname</name>
        <value>true</value>
    </property>
    <property>
        <name>dfs.datanode.use.datanode.hostname</name>
        <value>true</value>
    </property>
</configuration>

hive下的hive-site.xml:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>hive.metastore.local</name>
        <value>false</value>
    </property>
    <property>  
        <name>hive.metastore.uris</name>  
        <value>thrift://0.0.0.0:9083</value>  
        <description>Thrift uri for the remote metastore. Used by metastore client to connect to remote metastore.</description>  
    </property>
    <property>
        <name>hive.server2.thrift.bind.host</name>
        <value>localhost</value>
    </property>
    <property>
        <name>hive.server2.thrift.port</name>
        <value>10000</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://localhost:3306/hivedata?characterEncoding=UTF-8</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.jdbc.Driver</value>
    </property>
    <property>
        <name>hive.server2.enable.doAs</name>
        <value>false</value> 
    </property>
    <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>hive</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>228228</value>
    </property>
    <property>
        <name>hive.metastore.schema.verification</name>
        <value>false</value>
    </property>
    <property>
        <name>hive.server2.thrift.client.user</name>
        <value>hive</value>
        <description>Username to use against thrift client</description>
    </property>
    <property>
        <name>hive.server2.thrift.client.password</name>
        <value>hive228228</value>
        <description>Password to use against thrift client</description>
    </property> 
</configuration>

注意将hive.metastore.uris的ip段设为0.0.0.0。

本机spark配置文件:

在spark/conf下有三个文件core-site.xml、hdfs-site.xml、hive-site.xml(从服务端拷贝过来)。
core-site.xml:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://dbhive:9000</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/Users/zhi/Documents/app/hadoopdata</value>
    </property>
</configuration>

fs.defaultFS: 服务端的hdfs地址；
hadoop.tmp.dir: 和服务端保持一致。

hdfs-site.xml:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.datanode.hostname</name>
        <value>dbhive</value>
    </property>
    <property>
        <name>dfs.datanode.use.datanode.hostname</name>
        <value>true</value>
    </property>
    <property>
        <name>dfs.client.use.datanode.hostname</name>
        <value>true</value>
    </property>
</configuration>

dfs.datanode.hostname: 设置datanode的hostname；
dfs.datanode.use.datanode.hostname: datanode通讯采用域名方式；
dfs.client.use.datanode.hostname: 客户端访问datanode采用域名方式。

hive-site.xml:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>hive.metastore.uris</name>
        <value>thrift://dbhive:9083</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://dbhive:3306/hivedata?characterEncoding=UTF-8</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.jdbc.Driver</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>hive</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>123456</value>
    </property>
</configuration>

hive.metastore.uris: 访问metastore采用域名方式；
javax.jdo.option.ConnectionURL: 服务器mysql地址；
javax.jdo.option.ConnectionUserName: 数据库用户；
javax.jdo.option.ConnectionPassword: 数据库密码。

之后本机spark连接远程的metastore就可以读写了。

配置远程访问最重要的是要配置hadoop下的hdfs-site.xml，设置dfs.datanode.hostname域名，以及开启dfs.client.use.datanode.hostname和dfs.datanode.use.datanode.hostname开启域名访问。hdfs和metastore地址设为0.0.0.0.

spark远程读写hive数据库
实现spark远程连接hive数据库，需要将服务端mysql数据库里的hive数据表DBS和SDS里的localh...
Spark之HiveSupport连接（spark-shell和
本文介绍了使用Spark连接Hive的两种方式，spark-shell和IDEA远程连接。 1.spark-she...
Hive数据源实战
Spark SQL支持对Hive中存储的数据进行读写。操作Hive中的数据时，必须创建HiveContext，而不...
spark sql use hive
spark sql 通过Spark SQL 查询hive中的数据要求提供操作截图切换数据库，选择指定的表；操...
8.Spark学习(Python版本)：连接Hive读写数据（D
Step1. 让Spark包含Hive支持为了让Spark能够访问Hive，必须为Spark添加Hive支持。按...
Spark下连接星环Hive2失败
错误内容解决删除spark的spark-hive_2.11-2.3.0.jar spark-hive-thr...
Spark On Hive 部署和配置
Spark On Hive，通过spark sql模块访问和使用Hive，默认Spark预编译(pre-built...
Spark SQL
Spark on hive (Spark SQL):hive只是作为了存储的角色;Spark Sql作为计算的角色...
Hive On Spark搭建(cdh)
hive 和 spark版本之前有强对应关系 apache hive 和 spark 对应关系表 cdh hive...
Hive on Spark参数调优姿势小结
前言 Hive on Spark是指使用Spark替代传统MapReduce作为Hive的执行引擎，在HIVE-7...

spark远程读写hive数据库

配置hostname

服务端配置

本机spark配置文件:

相关文章

spark远程读写hive数据库

Spark之HiveSupport连接（spark-shell和

Hive数据源实战

spark sql use hive

8.Spark学习(Python版本)：连接Hive读写数据（D

Spark下连接星环Hive2失败

Spark On Hive 部署和配置

Spark SQL

Hive On Spark搭建(cdh)

Hive on Spark参数调优姿势小结

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读