beyond virtual memory limits

查看错误日志

不一定能在 Client日志 里面看到错误，可以打开AM的WEB页面查看：

Application appattempt

Diagnostics Info:   
AM Container for appattempt_1465808152295_0002_000001 exited with exitCode: -103
For more detailed output, check application tracking page:http://bigdatamgr1:8088/proxy/application_1465808152295_0002/Then, click on links to logs of each attempt.
Diagnostics: Container [pid=68226,containerID=container_1465808152295_0002_01_000001] is running beyond virtual memory limits. Current usage: 346.6 MB of 1 GB physical memory used; 2.5 GB of 2.1 GB virtual memory used. Killing container. 

查找解决办法

Google： 【2.5 GB of 2.1 GB virtual memory used. Killing container.】

http://stackoverflow.com/questions/21005643/container-is-running-beyond-memory-limits

http://stackoverflow.com/questions/14110428/am-container-is-running-beyond-virtual-memory-limits

https://issues.apache.org/jira/browse/YARN-4714

也可以调整opts和memory两个配置。

修改配置

[eshore@bigdatamgr1 ~]$ cat hadoop/etc/hadoop/yarn-site.xml 
...
<property>
  <name>yarn.nodemanager.vmem-check-enabled</name>
  <value>false</value>
  <description>Whether virtual memory limits will be enforced for containers</description>
</property>
<property>
  <name>yarn.nodemanager.vmem-pmem-ratio</name>
  <value>4</value>
  <description>Ratio between virtual memory to physical memory when setting memory limits for containers</description>
</property>

同步到所有nodemanager，然后重启yarn。