美文网首页
[Spark]本地调试遇到的坑

[Spark]本地调试遇到的坑

作者: 帅可儿妞 | 来源:发表于2019-09-18 23:35 被阅读0次
  1. scc.checkpoint 设置成本地路径, 报错:
    • Exception in thread "main" java.io.FileNotFoundException: File file:/sparklink/statis/checkpoint/XXXX/does not exist
    • scc.checkpoint在配置文件中设置成自己创建的目录, 但是报错:
      Exception in thread "main" java.io.FileNotFoundException: File file:/sparklink/statis/checkpoint/test/dacad09a-6a0e-4ee8-9820-cb832a9cd515 does not exist
          at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:537)
          at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:750)
          at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:527)
          at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:409)
          at org.apache.spark.SparkContext$$anonfun$setCheckpointDir$2.apply(SparkContext.scala:2118)
          at org.apache.spark.SparkContext$$anonfun$setCheckpointDir$2.apply(SparkContext.scala:2114)
          at scala.Option.map(Option.scala:146)
          at org.apache.spark.SparkContext.setCheckpointDir(SparkContext.scala:2114)
          at org.apache.spark.streaming.StreamingContext.checkpoint(StreamingContext.scala:241)
          at com.xylink.ai.consumer.Kafka2HiveXxx$.main(Kafka2HiveXxx.scala:27)
          at com.xylink.ai.consumer.Kafka2HiveXxx.main(Kafka2HiveXxx.scala)
      
    • 解决: 创建的目录的权限问题: chown -R shrekernil:admin /sparklink

相关文章

网友评论

      本文标题:[Spark]本地调试遇到的坑

      本文链接:https://www.haomeiwen.com/subject/ofaiyctx.html