美文网首页
Spark block replica设置(Rank设置)

Spark block replica设置(Rank设置)

作者: clive0x | 来源:发表于2019-06-27 21:32 被阅读0次

block replica默认随机存储:

BlockManager.scala def initialize(appId: String): Unit = 

val priorityClass = conf.get("spark.storage.replication.policy", classOf[RandomBlockReplicationPolicy].getName)

replication policy在BlockReplicationPolicy.scala中定义,如果是hdfs,需要配置成BasicBlockReplicationPolicy,一份在rank,一份在out of rank,其它随机选择。

Spark rank配置(TopologyMapper.scala):

spark.storage.replication.topologyMapper配置map 类:

org.apache.spark.storage.FileBasedTopologyMapper和DefaultTopologyMapper(无topo设置,默认配置项)

spark.storage.replication.topologyFile配置map config,为java properties配置文档

相关文章

网友评论

      本文标题:Spark block replica设置(Rank设置)

      本文链接:https://www.haomeiwen.com/subject/hopdcctx.html