大数据之Spark

spark下载地址: https://archive.apache.org/dist/spark/

一、配置Spark环境变量及将jars中 以下jar包拷贝至hive lib目录中
spark-core_2.12-3.0.0.jar
spark-kvstore_2.12-3.0.0.jar 
spark-launcher_2.12-3.0.0.jar 
spark-network-common_2.12-3.0.0.jar
spark-network-shuffle_2.12-3.0.0.jar 
spark-tags_2.12-3.0.0.jar 
spark-unsafe_2.12-3.0.0.jar

二、在Hive中创建spark配置文件 
1、vim /opt/module/hive/conf/spark-defaults.conf  
spark.master yarn
spark.eventLog.enabled          true
spark.eventLog.dir              hsfs://hadoop102:8020/spark-history spark.executor.memory          1g
spark.driver.memory            1g 
2、hadoop fs -mkdir /spark-history

三、上传纯净版spark jar包至hdfs 
1、hadoop fs -mkdir /spark-jars 
2、hadoop fs -put /opt/module/spark-3.0.0-bin-without-hadoop/jars/*  /spark-jars

四、修改Hive中hive-site.xml  
<!--spark依赖包(注:端口号必须和hadoop中namenode端口号一致) -->
<property>

<name>spark.yarn.jars</name>

<value>hdfs://hadoop102:8020/spark-jars/*</value>

</property>

<!--hive执行引擎 -->

<property>

<name>hive.execution.engine</name>

<value>spark</value>

</property>

<!--hive和spark连接超时时间 -->

<property>

<name>hive.spark.client.connect.timeout</name>

<value>10000ms</value>

</property>