美文网首页
在Jupyter notebook配置spark

在Jupyter notebook配置spark

作者: 吵吵人 | 来源:发表于2022-01-27 17:12 被阅读0次

from pyspark.sql import SparkSession
from pyspark.sql import SparkSession
from pyspark.sql.types import *
import pyspark.sql.functions as F
spark = SparkSession.builder.
config('spark.executor.memory', '8g').
config('spark.executor.cores', '4').
config('spark.driver.memory','8g').
config('spark.executor.instances', '2').
config('spark.sql.execution.arrow.enabled', 'true').
config('spark.driver.maxResultSize', '8g').
config('spark.default.parallelism', '480').
config("hive.exec.dynamici.partition",'true').
config("hive.exec.dynamic.partition.mode","nonstrict").
appName('order_task_auto_hitch_modelD_online').
enableHiveSupport().getOrCreate()
print("Spark启动完成;")

相关文章

网友评论

      本文标题:在Jupyter notebook配置spark

      本文链接:https://www.haomeiwen.com/subject/pquahrtx.html