JVM调优实战

原创

隔岸无名 2020-10-12 09:06:48 ©著作权

文章标签 性能压测 JVM调优 文章分类 软件测试

©著作权归作者所有：来自51CTO博客作者隔岸无名的原创作品，请联系作者获取转载授权，否则将追究法律责任

对项目 i..net 网站进行了接口的压测，压测过程中发现各微服务的 heap 内存设置过小（-Xms64m –Xmx128m），导致压测过程中服务假死（频繁fullgc导致服务假死）和发生OOM。

1、xx接口的压测情况如下，150并发进行压测，发现响应时间在小于1s一段时间后，响应时间陡升到3s。

2、 TPS 也是从420 陡降到 60 左右。

3、trace 调用链跟踪发现 xxxxsevice中的 xxxxWorktime方法调用耗时3S

4、继续跟踪该方法的调用链，发现调用 attend-xxxx项目中的接口耗时3S

5、查看后台日志发现：atten-xxxx 服务发生了 OOM；通过telnet命令连接xxxx的端口，发现无法连接端口，服务处于假死状态。

6、查看该服务的heap使用情况，基本上被耗尽

7、jstat 发现发生了频繁的 FullGC

8、压测过程中发现在进行资源争夺时，有时还把mongo 给拖死：

ERROR c.y.a.sign.exception.BaseControllerException - attend-xxxx统一异常捕获... org.springframework.dao.DataAcce***esourceFailureException: Timed out after 30000 ms while waiting for a server that matches WritableServerSelector. Client view of cluster state is Caused by: com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches WritableServerSelector. Client view of cluster state is {type=REPLICA_SET, servers=[{address=xx.xx.0.40:47020, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=xx.xx.0.40:47020, type=REPLICA_SET_ARBITER, roundTripTime=0.6 ms, state=CONNECTED}, {address=xx.xx.0.40:47020, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}] at com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:377) at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:104) at com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.<init>(ClusterBinding.java:75) at com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.<init>(ClusterBinding.java:71) at com.mongodb.binding.ClusterBinding.getWriteConnectionSource(ClusterBinding.java:68) at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:411) at com.mongodb.operation.BaseWriteOperation.execute(BaseWriteOperation.java:133) at com.mongodb.operation.BaseWriteOperation.execute(BaseWriteOperation.java:60) at com.mongodb.Mongo.execute(Mongo.java:845) at com.mongodb.Mongo$2.execute(Mongo.java:828) at com.mongodb.DBCollection.executeWriteOperation(DBCollection.java:342) at com.mongodb.DBCollection.insert(DBCollection.java:337) at com.mongodb.DBCollection.insert(DBCollection.java:328) at com.mongodb.DBCollection.insert(DBCollection.java:298) at com.mongodb.DBCollection.insert(DBCollection.java:264) at com.mongodb.DBCollection.insert(DBCollection.java:186) at com.mongodb.DBCollection.save(DBCollection.java:407) at com.mongodb.DBCollection.save(DBCollection.java:382)

9、数据库在压测过程中被压死：

综上所述，调优建议如下：

1、微服务的heap 设置太小了，需要扩容内存，建议设置为（-Xms512m –Xmx1024m）

2、数据库服务服务器建议单独部署物理机，且数据库内存建议设置64G以上；建议采用多主从或分片部署。