crm系统突然不能登录,查看后台tcp状态,发现有大量的close_wait产生, 服务器内存飙升至98%左右;

tcp        0      0 172.16.112.72:8060      0.0.0.0:*               LISTEN      7234/python     
tcp 0 0 172.16.112.72:8060 172.16.118.75:30740 SYN_RECV -
tcp 876 0 172.16.112.72:8060 172.16.111.17:53656 CLOSE_WAIT -
tcp 764 0 172.16.112.72:8060 172.16.110.225:54383 CLOSE_WAIT -
tcp 0 0 172.16.112.72:8060 172.16.111.13:49246 ESTABLISHED 7234/python
tcp 849 0 172.16.112.72:8060 172.16.110.225:54355 CLOSE_WAIT -
tcp 871 0 172.16.112.72:8060 172.16.111.17:53663 CLOSE_WAIT -
tcp 876 0 172.16.112.72:8060 172.16.111.17:53629 CLOSE_WAIT -
tcp 787 0 172.16.112.72:8060 172.16.111.247:63299 CLOSE_WAIT -
tcp 1040 0 172.16.112.72:8060 172.16.111.248:51508 CLOSE_WAIT -
tcp 854 0 172.16.112.72:8060 172.16.110.225:54402 CLOSE_WAIT -
tcp 0 0 172.16.112.72:8060 172.16.111.13:49247 ESTABLISHED -
tcp 351 0 172.16.112.72:8060 172.16.111.95:52611 ESTABLISHED -
tcp 810 0 172.16.112.72:8060 172.16.111.17:53680 CLOSE_WAIT -
tcp 849 0 172.16.112.72:8060 172.16.110.225:54364 CLOSE_WAIT -
tcp 876 0 172.16.112.72:8060 172.16.111.17:53648 CLOSE_WAIT -
tcp 876 0 172.16.112.72:8060 172.16.111.17:53637 CLOSE_WAIT -
tcp 854 0 172.16.112.72:8060 172.16.110.225:54401 CLOSE_WAIT -
tcp 876 0 172.16.112.72:8060 172.16.111.17:53641 CLOSE_WAIT -
tcp 849 0 172.16.112.72:8060 172.16.110.225:54376 CLOSE_WAIT -
tcp 876 0 172.16.112.72:8060 172.16.111.17:53626 CLOSE_WAIT -
tcp 861 0 172.16.112.72:8060 172.16.111.247:63300 CLOSE_WAIT -
tcp 849 0 172.16.112.72:8060 172.16.110.225:54395 CLOSE_WAIT -
tcp 418 0 172.16.112.72:8060 172.16.110.225:54455 CLOSE_WAIT -
tcp 790 0 172.16.112.72:8060 172.16.110.225:54373 CLOSE_WAIT -
tcp 764 0 172.16.112.72:8060 172.16.110.225:54391 CLOSE_WAIT -
tcp 849 0 172.16.112.72:8060 172.16.110.225:54396 CLOSE_WAIT -
tcp 764 0 172.16.112.72:8060 172.16.110.225:54385 CLOSE_WAIT -
tcp 764 0 172.16.112.72:8060 172.16.110.225:54380 CLOSE_WAIT -
tcp 876 0 172.16.112.72:8060 172.16.111.17:53639 CLOSE_WAIT -
tcp 876 0 172.16.112.72:8060 172.16.111.17:53651 CLOSE_WAIT -
tcp 840 0 172.16.112.72:8060 172.16.111.247:63312 CLOSE_WAIT -
tcp 351 0 172.16.112.72:8060 172.16.111.172:52131 ESTABLISHED -
tcp 876 0 172.16.112.72:8060 172.16.111.17:53649 CLOSE_WAIT -
tcp 873 0 172.16.112.72:8060 172.16.111.17:53635 CLOSE_WAIT -
tcp 872 0 172.16.112.72:8060 172.16.111.17:53634 CLOSE_WAIT -
tcp 871 0 172.16.112.72:8060 172.16.111.17:53661 CLOSE_WAIT -
tcp 764 0 172.16.112.72:8060 172.16.110.225:54351 CLOSE_WAIT -
tcp 764 0 172.16.112.72:8060 172.16.110.225:54384 CLOSE_WAIT -
tcp 790 0 172.16.112.72:8060 172.16.110.225:54371 CLOSE_WAIT -
tcp 0 0 172.16.112.72:8060 172.16.111.13:49249 ESTABLISHED -
tcp 418 0 172.16.112.72:8060 172.16.110.225:54448 CLOSE_WAIT -
tcp 876 0 172.16.112.72:8060 172.16.111.17:53638 CLOSE_WAIT -
tcp 849 0 172.16.112.72:8060 172.16.110.225:54378 CLOSE_WAIT -
tcp 401 0 172.16.112.72:8060 172.16.111.13:49378 CLOSE_WAIT -
tcp 764 0 172.16.112.72:8060 172.16.110.225:54390 CLOSE_WAIT -
tcp 854 0 172.16.112.72:8060 172.16.110.225:54403 CLOSE_WAIT -
tcp 395 0 172.16.112.72:8060 172.16.110.225:54432 CLOSE_WAIT -
tcp 854 0 172.16.112.72:8060 172.16.110.225:54366 CLOSE_WAIT -
tcp 810 0 172.16.112.72:8060 172.16.111.17:53689 CLOSE_WAIT -
tcp 764 0 172.16.112.72:8060 172.16.110.225:54389 CLOSE_WAIT -
tcp 790 0 172.16.112.72:8060 172.16.110.225:54372 CLOSE_WAIT -
tcp 418 0 172.16.112.72:8060 172.16.110.225:54440 CLOSE_WAIT -
tcp 876 0 172.16.112.72:8060 172.16.111.17:53642 CLOSE_WAIT -
tcp 764 0 172.16.112.72:8060 172.16.110.225:54358 CLOSE_WAIT -
tcp 828 0 172.16.112.72:8060 172.16.110.225:54365 CLOSE_WAIT -
tcp 876 0 172.16.112.72:8060 172.16.111.17:53657 CLOSE_WAIT -
tcp 764 0 172.16.112.72:8060 172.16.110.225:54352 CLOSE_WAIT -
tcp 0 0 172.16.112.72:8060 172.16.111.13:49243 ESTABLISHED 19180/python
tcp 854 0 172.16.112.72:8060 172.16.110.225:54398 CLOSE_WAIT -
tcp 384 0 172.16.112.72:8060 172.16.111.247:63341 CLOSE_WAIT -
tcp 876 0 172.16.112.72:8060 172.16.111.17:53645 CLOSE_WAIT -
tcp 858 0 172.16.112.72:8060 172.16.111.17:53647 CLOSE_WAIT -
tcp 876 0 172.16.112.72:8060 172.16.111.17:53624 CLOSE_WAIT -
tcp 876 0 172.16.112.72:8060 172.16.111.17:53652 CLOSE_WAIT -
tcp 828 0 172.16.112.72:8060 172.16.110.225:54387 CLOSE_WAIT -
tcp 790 0 172.16.112.72:8060 172.16.110.225:54369 CLOSE_WAIT -
tcp 764 0 172.16.112.72:8060 172.16.110.225:54354 CLOSE_WAIT -
tcp 390 0 172.16.112.72:8060 172.16.111.247:63325 CLOSE_WAIT -
tcp 764 0 172.16.112.72:8060 172.16.110.225:54356 CLOSE_WAIT -
tcp 764 0 172.16.112.72:8060 172.16.110.225:54350 CLOSE_WAIT -
tcp 790 0 172.16.112.72:8060 172.16.110.225:54368 CLOSE_WAIT -
tcp 401 0 172.16.112.72:8060 172.16.111.13:49375 CLOSE_WAIT -
tcp 876 0 172.16.112.72:8060 172.16.111.17:53630 CLOSE_WAIT -
tcp 876 0 172.16.112.72:8060 172.16.111.17:53625 CLOSE_WAIT -
tcp 0 0 172.16.112.72:8060 172.16.111.13:49248 ESTABLISHED -
tcp 389 0 172.16.112.72:8060 172.16.111.247:63349 ESTABLISHED -
tcp 876 0 172.16.112.72:8060 172.16.111.17:53653 CLOSE_WAIT -
tcp 854 0 172.16.112.72:8060 172.16.110.225:54399 CLOSE_WAIT -
tcp 764 0 172.16.112.72:8060 172.16.110.225:54408 CLOSE_WAIT -
tcp 764 0 172.16.112.72:8060 172.16.110.225:54392 CLOSE_WAIT -
tcp 876 0 172.16.112.72:8060 172.16.111.17:53655 CLOSE_WAIT -
tcp 876 0 172.16.112.72:8060 172.16.111.17:53628 CLOSE_WAIT -
tcp 764 0 172.16.112.72:8060 172.16.110.225:54379 CLOSE_WAIT -
tcp 764 0 172.16.112.72:8060 172.16.110.225:54359 CLOSE_WAIT -
tcp 876 0 172.16.112.72:8060 172.16.111.17:53659 CLOSE_WAIT -
tcp 854 0 172.16.112.72:8060 172.16.110.225:54362 CLOSE_WAIT -
tcp 854 0 172.16.112.72:8060 172.16.111.247:63320 CLOSE_WAIT -
tcp 810 0 172.16.112.72:8060 172.16.111.17:53673 CLOSE_WAIT -
tcp 873 0 172.16.112.72:8060 172.16.111.17:53633 CLOSE_WAIT -
tcp 876 0 172.16.112.72:8060 172.16.111.17:53627 CLOSE_WAIT -
tcp 764 0 172.16.112.72:8060 172.16.110.225:54382 CLOSE_WAIT -
tcp 787 0 172.16.112.72:8060 172.16.111.247:63297 CLOSE_WAIT -
tcp 849 0 172.16.112.72:8060 172.16.110.225:54377 CLOSE_WAIT -
tcp 876 0 172.16.112.72:8060 172.16.111.17:53623 CLOSE_WAIT -
tcp 790 0 172.16.112.72:8060 172.16.110.225:54367 CLOSE_WAIT -
tcp 871 0 172.16.112.72:8060 172.16.111.17:53662 CLOSE_WAIT -
tcp 787 0 172.16.112.72:8060 172.16.111.247:63296 CLOSE_WAIT -
tcp 858 0 172.16.112.72:8060 172.16.111.247:63311 CLOSE_WAIT -
tcp 764 0 172.16.112.72:8060 172.16.110.225:54381 CLOSE_WAIT -
tcp 812 0 172.16.112.72:8060 172.16.111.248:51507 CLOSE_WAIT -
tcp 876 0 172.16.112.72:8060 172.16.111.17:53654 CLOSE_WAIT -
tcp 871 0 172.16.112.72:8060 172.16.111.17:53665 CLOSE_WAIT -
tcp 854 0 172.16.112.72:8060 172.16.110.225:54361 CLOSE_WAIT -
tcp 871 0 172.16.112.72:8060 172.16.111.17:53660 CLOSE_WAIT -
tcp 854 0 172.16.112.72:8060 172.16.110.225:54397 CLOSE_WAIT -
tcp 787 0 172.16.112.72:8060 172.16.110.225:54412 CLOSE_WAIT -
tcp 854 0 172.16.112.72:8060 172.16.110.225:54363 CLOSE_WAIT -
tcp 871 0 172.16.112.72:8060 172.16.111.17:53664 CLOSE_WAIT -
tcp 810 0 172.16.112.72:8060 172.16.111.17:53684 CLOSE_WAIT -
tcp 876 0 172.16.112.72:8060 172.16.111.17:53650 CLOSE_WAIT -
tcp 849 0 172.16.112.72:8060 172.16.110.225:54374 CLOSE_WAIT -
tcp 828 0 172.16.112.72:8060 172.16.110.225:54386 CLOSE_WAIT -
tcp 849 0 172.16.112.72:8060 172.16.110.225:54394 CLOSE_WAIT -
tcp 854 0 172.16.112.72:8060 172.16.110.225:54400 CLOSE_WAIT -
tcp 764 0 172.16.112.72:8060 172.16.110.225:54393 CLOSE_WAIT -
tcp 764 0 172.16.112.72:8060 172.16.110.225:54353 CLOSE_WAIT -
tcp 876 0 172.16.112.72:8060 172.16.111.17:53644 CLOSE_WAIT -
tcp 395 0 172.16.112.72:8060 172.16.110.225:54452 CLOSE_WAIT -
tcp 351 0 172.16.112.72:8060 172.16.111.83:52791 ESTABLISHED -
tcp 790 0 172.16.112.72:8060 172.16.110.225:54370 CLOSE_WAIT -
tcp 828 0 172.16.112.72:8060 172.16.110.225:54388 CLOSE_WAIT -
tcp 876 0 172.16.112.72:8060 172.16.111.17:53643 CLOSE_WAIT -
tcp 849 0 172.16.112.72:8060 172.16.110.225:54375 CLOSE_WAIT -
tcp 764 0 172.16.112.72:8060 172.16.110.225:54357 CLOSE_WAIT -
tcp 876 0 172.16.112.72:8060 172.16.111.17:53640 CLOSE_WAIT -
tcp 764 0 172.16.112.72:8060 172.16.110.225:54349 CLOSE_WAIT -

这个问题深入研究了一下:如果我们的服务器程序处于close_wait状态,说明套接字是被动关闭的;

关闭TCP连接共需要四个packet:

Client     --->  FIN ---> Server

Client     <---  ACK <--- Server   此时client端处于fin_wait_2状态;而server程序处于close_wait状态;

Client     <---  FIN  <--- Server  此时server发送fin给client,server就为last_ack状态;

Client     --->  ACK ---> Server  client回应ack,此时server的套接字才会真正的为closed状态;


server程序处于close_wait状态,而不是last_ack状态,说明此时还没有发fin给client,那么可能是在关闭连接之前还有很多数据要发送或者其他事要做,导致这个fin没有发送。

close_wait特别消耗资源,通常,一个close_wait会维持至少2个小时;如果一个人特地写了一个程序,给你造成一堆的close_wait,消耗你的资源,那么通常等不到释放的那一刻,系统就已经崩溃了。