spark在hadoop的yarn上运行报内存分配不足

运行错误如下：

Paste_Image.png

解决方法：
hadoop yarn 在分配资源的时候，会进行验证

参考这里
I found what the problem was: in part of my code, each of the mappers had to access a local lmdb database. When an lmdb database starts, it reserves 1 TB of virtual memory, this caused Hadoop to think that I was using this much memory while in fact I wasn't.
I solved the issue by setting yarn.nodemanager.vmem-check-enabled to false in yarn-site.xml, which prevents Hadoop from checking the virtual memory limits. Note that you shouldn't use that unless you're sure of it, because Hadoop is trying to protect you from memory leaks and similar issues by this check. I only used it because I was sure it wasn't a memory leak

但是不建议这么做，这样做只是偷懒的方式，具体如何操作，可以参考这里

YARN资源管理设置

stack over flow 参考