Sparkstreaming 消费kafka报警告Offsets

Sparkstreaming程序在消费kafka时报了一个告警

23/02/06 11:03:08 WARN scheduler.TaskSetManager: Lost task 1.0 in stage 344.0 (TID 8615, hadoop07, executor 5): org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets out of range with no configured reset policy for partitions: {GZJY-3=30970547102}
    at org.apache.kafka.clients.consumer.internals.Fetcher.parseCompletedFetch(Fetcher.java:990)
    at org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:491)
    at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1269)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1200)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1135)
    at org.apache.spark.streaming.kafka010.InternalKafkaConsumer.poll(KafkaDataConsumer.scala:200)
    at org.apache.spark.streaming.kafka010.InternalKafkaConsumer.get(KafkaDataConsumer.scala:129)
    at org.apache.spark.streaming.kafka010.KafkaDataConsumer$class.get(KafkaDataConsumer.scala:36)
    at org.apache.spark.streaming.kafka010.KafkaDataConsumer$CachedKafkaDataConsumer.get(KafkaDataConsumer.scala:212)
    at org.apache.spark.streaming.kafka010.KafkaRDDIterator.next(KafkaRDD.scala:261)
    at org.apache.spark.streaming.kafka010.KafkaRDDIterator.next(KafkaRDD.scala:229)
    at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
    at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:149)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
    at org.apache.spark.scheduler.Task.run(Task.scala:121)
    at org.apache.spark.executor.Executor$TaskRunner$$anonfun$11.apply(Executor.scala:407)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1408)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:413)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

命令查看对应消费id的消费情况

kafka-consumer-groups  --bootstrap-server hadoop02:9092,hadoop03:9092,hadoop04:9092,hadoop05:9092,hadoop06:9092,hadoop07:9092,hadoop08:9092,hadoop09:9092,hadoop10:9092,hadoop11:9092,hadoop12:9092 --describe --group pangu_GDT003_online108

返回

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                     HOST            CLIENT-ID
GZJY            10         21102297666     21102810249     512583          consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19   consumer-1
GZJY            3          30971455431     30962503292     -8952139        consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19   consumer-1
GZJY            2          31581677989     31582209958     531969          consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19   consumer-1
GZJY            9          21063437255     21063932876     495621          consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19   consumer-1
GZJY            4          27081104241     27081622964     518723          consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19   consumer-1
GZJY            8          31544897443     31545428226     530783          consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19   consumer-1
GZJY            6          31479796535     31480294328     497793          consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19   consumer-1
GZJY            7          31523391532     31523930639     539107          consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19   consumer-1
GZJY            0          31344827282     31345323448     496166          consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19   consumer-1
GZJY            1          31097505976     31098074369     568393          consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19   consumer-1
GZJY            5          31478597056     31479107234     510178          consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19   consumer-1

可以看出来3号确实有些问题

通过命令修改3号为最新

kafka-consumer-groups --bootstrap-server hadoop02:9092,hadoop03:9092,hadoop04:9092,hadoop05:9092,hadoop06:9092,hadoop07:9092,hadoop08:9092,hadoop09:9092,hadoop10:9092,hadoop11:9092,hadoop12:9092 --group pangu_GDT003_online108 --topic GZJY --execute --reset-offsets --to-earliest

再次查看返回

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                     HOST            CLIENT-ID
GZJY            10         21102967356     21103534428     567072          consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19   consumer-1
GZJY            3          30972117204     30972666789     549585          consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19   consumer-1
GZJY            2          31582359667     31582930138     570471          consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19   consumer-1
GZJY            9          21064090203     21064628633     538430          consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19   consumer-1
GZJY            4          27081771694     27082333571     561877          consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19   consumer-1
GZJY            8          31545568770     31546140879     572109          consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19   consumer-1
GZJY            6          31480442205     31481007582     565377          consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19   consumer-1
GZJY            7          31524089705     31524647802     558097          consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19   consumer-1
GZJY            0          31345452158     31345995112     542954          consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19   consumer-1
GZJY            1          31098218435     31098770724     552289          consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19   consumer-1
GZJY            5          31479258366     31479815080     556714          consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19   consumer-1

正常

观察Sparkstreaming没有报警告。