美文网首页
spark stream

spark stream

作者: Hystrix_Hu | 来源:发表于2017-12-01 11:58 被阅读0次

    Dstream 是一个 rdd的队列。
    当spark stream 窗口函数的间隔不是batchDuration的倍数时会报错。

    Exception in thread "main" java.lang.Exception: The window duration of windowed DStream (10000 ms) must be a multiple of the slide duration of parent DStream (3000 ms)
       at org.apache.spark.streaming.dstream.WindowedDStream.<init>(WindowedDStream.scala:35)
       at org.apache.spark.streaming.dstream.DStream$$anonfun$window$1.apply(DStream.scala:766)
       at org.apache.spark.streaming.dstream.DStream$$anonfun$window$1.apply(DStream.scala:766)
       at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
       at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
       at org.apache.spark.SparkContext.withScope(SparkContext.scala:679)
       at org.apache.spark.streaming.StreamingContext.withScope(StreamingContext.scala:264)
       at org.apache.spark.streaming.dstream.DStream.window(DStream.scala:765)
    

    相关文章

      网友评论

          本文标题:spark stream

          本文链接:https://www.haomeiwen.com/subject/gqnibxtx.html