美文网首页
[Spark]本地调试遇到的坑

[Spark]本地调试遇到的坑

作者: 帅可儿妞 | 来源:发表于2019-09-18 23:35 被阅读0次
    1. scc.checkpoint 设置成本地路径, 报错:
      • Exception in thread "main" java.io.FileNotFoundException: File file:/sparklink/statis/checkpoint/XXXX/does not exist
      • scc.checkpoint在配置文件中设置成自己创建的目录, 但是报错:
        Exception in thread "main" java.io.FileNotFoundException: File file:/sparklink/statis/checkpoint/test/dacad09a-6a0e-4ee8-9820-cb832a9cd515 does not exist
            at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:537)
            at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:750)
            at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:527)
            at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:409)
            at org.apache.spark.SparkContext$$anonfun$setCheckpointDir$2.apply(SparkContext.scala:2118)
            at org.apache.spark.SparkContext$$anonfun$setCheckpointDir$2.apply(SparkContext.scala:2114)
            at scala.Option.map(Option.scala:146)
            at org.apache.spark.SparkContext.setCheckpointDir(SparkContext.scala:2114)
            at org.apache.spark.streaming.StreamingContext.checkpoint(StreamingContext.scala:241)
            at com.xylink.ai.consumer.Kafka2HiveXxx$.main(Kafka2HiveXxx.scala:27)
            at com.xylink.ai.consumer.Kafka2HiveXxx.main(Kafka2HiveXxx.scala)
        
      • 解决: 创建的目录的权限问题: chown -R shrekernil:admin /sparklink

    相关文章

      网友评论

          本文标题:[Spark]本地调试遇到的坑

          本文链接:https://www.haomeiwen.com/subject/ofaiyctx.html