一、问题现象:
在 9.104 环境运行会员系统后,tomcat 每隔 2小时就自动停掉 tomcat,并报 :
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f9220720000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /usr/local/tomcat/webapps/hs_err_pid851855.log
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
# [ timer expired, abort... ]
二、解决方案:
1、开始以为jvm 中 heap 配置小了,加大 heap内存,还是报错。
2、后面才抓到关键错误信息:Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
不太清楚 mmap 是啥东东,只知道是 linux 系统级的东西。
Google:mmap是一种内存映射文件的方法,即将一个文件或者其它对象映射到进程的地址空间,实现文件磁盘地址和进程虚拟地址空间中一段虚拟地址的一一对映关系。
3、去看 线程dump错误信息:hs_err_pid851855.log:
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 65536 bytes for committing reserved memory.
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
# Out of Memory Error (os_linux.cpp:2640), pid=267752, tid=0x00007fecd03b5700
# JRE version: Java(TM) SE Runtime Environment (8.0_151-b12) (build 1.8.0_151-b12)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.151-b12 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
max_map_count : 进程中内存映射区域的最大数量。在调用malloc,直接调用mmap和mprotect和加载共享库时会产生内存映射区域。
虽然大多数程序需要的内存映射区域不超过1000个,但是特定的程序,特别是malloc调试器,可能需要很多,例如每次分配都会产生一到两个内存映射区域。默认值是65536。
4、看上去是 map 值小了,于是: sysctl -a |grep map
vm.max_map_count = 65530
vm.min_unmapped_ratio = 1
vm.mmap_min_addr = 4096
vm.max_map_count = 65530 设置小了,于是加大点:
echo "vm.max_map_count = 131072" >> /etc/sysctl.conf
sysctl -p
5、运行104系统后,问题解决。