block replica默认随机存储:
BlockManager.scala def initialize(appId: String): Unit =
val priorityClass = conf.get("spark.storage.replication.policy", classOf[RandomBlockReplicationPolicy].getName)
replication policy在BlockReplicationPolicy.scala中定义,如果是hdfs,需要配置成BasicBlockReplicationPolicy,一份在rank,一份在out of rank,其它随机选择。
Spark rank配置(TopologyMapper.scala):
spark.storage.replication.topologyMapper配置map 类:
org.apache.spark.storage.FileBasedTopologyMapper和DefaultTopologyMapper(无topo设置,默认配置项)
spark.storage.replication.topologyFile配置map config,为java properties配置文档
网友评论