跑etl任务,报错:

Container [pid=31306,containerID=container_1479290736535_0004_01_000003] is running beyond physical memory limits. Current usage: 2.7 GB of 2.5 GB physical memory used; 4.4 GB of 7.5 GB virtual memory used. Killing container.


发现是一个小时任务打开了太多文件(大约7000多个,每个进程占用的buffer太多,导致内存不足)


解决办法:

修改程序设置,将每个buffer设置占用更小内存,以下是修改后成功运行mr程序的配置

   "orc.strip.size":1024*1024,

   "orc.block.size":16*1024*1024,

   "orc.row.index.stride":500,

   "orc.compress.size":8*1024,