美文网首页
Q:Assuming connection is dead; p

Q:Assuming connection is dead; p

作者: 点点渔火 | 来源:发表于2017-11-20 21:44 被阅读0次

case:
ERROR TransportChannelHandler: Connection to /172.31.47.51:29662 has been quiet for 500000 ms while there are outstanding requests. Assuming connection is dead; please adjust spark.network.timeout if this is wrong.


图片.png

官网对spark.network.timeout的解释, drive端在500000ms(spark.network.timeout 设置的是500) 时间内没有收到excutor端的响应, 断开网络连接, 然后后面用到这个excutor的temp数据时再报File does not exist。

解决方案:
参考 - http://blog.csdn.net/wesleyflagon/article/details/52839165

https://stackoverflow.com/questions/37260230/spark-cluster-full-of-heartbeat-timeouts-executors-exiting-on-their-own

一个方案就是增加spark.network.timeout的大小
另一个方案是在数据处理的时候repartition减少task的数量

相关文章

网友评论

      本文标题:Q:Assuming connection is dead; p

      本文链接:https://www.haomeiwen.com/subject/ggubvxtx.html