美文网首页
One error fix process

One error fix process

作者: 泓礼 | 来源:发表于2019-03-07 14:24 被阅读0次

    ## todo monitor direct memory to avoid OOM,sometimes when read column, ARROW realloc cause OOM:

    vector, can't allocate more direct Mem

    ERROR [entry-prefetch-1024-44-1:AbstractVectorReader@380] - getNextValueVector f        ail when data is available

    18850188org.apache.arrow.memory.OutOfMemoryException: Failure allocating buffer.

    18850189        at io.netty.buffer.PooledByteBufAllocatorL.allocate(PooledByteBufAllocatorL.java:57)

    18850190        at org.apache.arrow.memory.AllocationManager.<init>(AllocationManager.java:93)

    18850191        at org.apache.arrow.memory.BaseAllocator.bufferWithoutReservation(BaseAllocator.java:315)

    18850192        at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:284)

    18850193        at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:255)

    18850194        at org.apache.bookkeeper.common.util.ArrowSerializer.deserializeRecordBatch(ArrowSerializer.java:1        11)

    18850195        at org.apache.bookkeeper.stream.storage.impl.column.AbstractVectorReader.loadData2Vector(AbstractV        ectorReader.java:958)

    solution: to check whether release?!

    可能跟writer占用了大量的direct mem有关

    getNextValueVector        fail when data is available

    31648489org.apache.arrow.memory.OutOfMemoryException: Failure while allocating memory.

    31648490        at org.apache.arrow.vector.BaseFixedWidthVector.allocateNew(BaseFixedWidthVector.java:257)

    31648491        at org.apache.bookkeeper.stream.storage.impl.column.FloatVectorReader.getNextValueVector(FloatVect        orReader.java:67)

    31648492        at org.apache.bookkeeper.stream.storage.impl.column.AbstractVectorReader.notifyDataAvailable(Abstr        actVectorReader.java:308)

    31648493        at org.apache.bookkeeper.stream.storage.impl.column.AbstractVectorReader.lambda$updateEntriesAndSc        hedulePrefetch$6(AbstractVectorReader.java:290)

    31648494        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)

    31648495        at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:778)

    31648496        at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2140)

    31648497        at org.apache.bookkeeper.stream.storage.impl.column.AbstractVectorReader.updateEntriesAndScheduleP        refetch(AbstractVectorReader.java:282)

    31648498        at org.apache.bookkeeper.stream.storage.impl.column.AbstractVectorReader.lambda$new$0(AbstractVect        orReader.java:163)

    31648499        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

    31648500        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)

    31648501        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThread        PoolExecutor.java:180)

    31648502        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExe        cutor.java:294)

    31648503        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

    31648504        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

    31648505        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

    相关文章

      网友评论

          本文标题:One error fix process

          本文链接:https://www.haomeiwen.com/subject/bducpqtx.html