TCP的连接状态对于我们web服务器来说是至关重要的,尤其是并发量ESTAB;或者是syn_recv值,假如这个值比较大的话我们可以认为是不是受到了攻击,或是是time_wait值比较高的话,我们要考虑看我们内核是否需要调优,太高的time_wait值的话会占用太多端口,要是端口少的话后果不堪设想
一、TCP状态介绍
1、man netstat查看TCP的各种状态信息描述:
LISTEN -侦听来自远方TCP端口的连接请求;
SYN-SENT -在发送连接请求后等待匹配的连接请求;
SYN-RECEIVED -在收到和发送一个连接请求后等待对连接请求的确认;
ESTABLISHED -代表一个打开的连接,数据可以传送给用户;
FIN-WAIT-1 -等待远程TCP的连接中断请求,或先前的连接中断请求的确认;
FIN-WAIT-2 -从远程TCP等待连接中断请求;
CLOSE-WAIT -等待从本地用户发来的连接中断请求;
CLOSING -等待远程TCP对连接中断的确认;
LAST-ACK -等待原来发向远程TCP的连接中断请求的确认;
TIME-WAIT -等待足够的时间以确保远程TCP接收到连接中断请求的确认;
CLOSED -没有任何连接状态;
2、监控原理
[root@Node1 ~]# /bin/netstat -an|awk '/^tcp/{++S[$NF]}END{for(a in S) print a,S[a]}' //通过netstat获取相关值
LISTEN 10
ESTABLISHED 1
TIME_WAIT 178
[root@Node1 ~]#
二、监控脚本编写
1、编写脚本,放于 /etc/zabbix/zabbix_agentd.d/目录下
1 [root@Node1 zabbix_agentd.d]# cat tcp_status.sh
2 #!/bin/bash
3 #This script is used to get tcp and udp connetion status
4 #tcp status
5 metric=$1
6 tmp_file=/tmp/tcp_status.txt
7 /bin/netstat -an|awk '/^tcp/{++S[$NF]}END{for(a in S) print a,S[a]}' > $tmp_file
8 case $metric in
9 closed)
10 output=$(awk '/CLOSED/{print $2}' $tmp_file)
11 if [ "$output" == "" ];then
12 echo 0
13 else
14 echo $output
15 fi
16 ;;
17 listen)
18 output=$(awk '/LISTEN/{print $2}' $tmp_file)
19 if [ "$output" == "" ];then
20 echo 0
21 else
22 echo $output
23 fi
24 ;;
25 synrecv)
26 output=$(awk '/SYN_RECV/{print $2}' $tmp_file)
27 if [ "$output" == "" ];then
28 echo 0
29 else
30 echo $output
31 fi
32 ;;
33 synsent)
34 output=$(awk '/SYN_SENT/{print $2}' $tmp_file)
35 if [ "$output" == "" ];then
36 echo 0
37 else
38 echo $output
39 fi
40 ;;
41 established)
42 output=$(awk '/ESTABLISHED/{print $2}' $tmp_file)
43 if [ "$output" == "" ];then
44 echo 0
45 else
46 echo $output
47 fi
48 ;;
49 timewait)
50 output=$(awk '/TIME_WAIT/{print $2}' $tmp_file)
51 if [ "$output" == "" ];then
52 echo 0
53 else
54 echo $output
55 fi
56 ;;
57 closing)
58 output=$(awk '/CLOSING/{print $2}' $tmp_file)
59 if [ "$output" == "" ];then
60 echo 0
61 else
62 echo $output
63 fi
64 ;;
65 closewait)
66 output=$(awk '/CLOSE_WAIT/{print $2}' $tmp_file)
67 if [ "$output" == "" ];then
68 echo 0
69 else
70 echo $output
71 fi
72 ;;
73 lastack)
74 output=$(awk '/LAST_ACK/{print $2}' $tmp_file)
75 if [ "$output" == "" ];then
76 echo 0
77 else
78 echo $output
79 fi
80 ;;
81 finwait1)
82 output=$(awk '/FIN_WAIT1/{print $2}' $tmp_file)
83 if [ "$output" == "" ];then
84 echo 0
85 else
86 echo $output
87 fi
88 ;;
89 finwait2)
90 output=$(awk '/FIN_WAIT2/{print $2}' $tmp_file)
91 if [ "$output" == "" ];then
92 echo 0
93 else
94 echo $output
95 fi
96 ;;
97 *)
98 echo -e "\e[033mUsage: sh $0 [closed|closing|closewait|synrecv|synsent|finwait1|finwait2|listen|established|lastack|timewait]\e[0m"
99 esac
赋予脚本执行权限:
[root@Node1 ~]# chmod a+x /etc/zabbix/zabbix_agentd.d/tcp_status.sh //赋予执行权限
[root@Node1 ~]# ll /etc/zabbix/zabbix_agentd.d/tcp_status.sh
-rwxr-xr-x 1 root root 2502 Jan 18 09:48 /etc/zabbix/zabbix_agentd.d/tcp_status.sh
[root@Node1 ~]#
2、添加zabbix配置文件,放于 /etc/zabbix/zabbix_agentd.d/目录下(agent的配置文件 /etc/zabbix/zabbix_agentd.conf 中定义了其他key的包含目录)创建配置文件tcp_status.conf
[root@Node1 ~]# cat /etc/zabbix/zabbix_agentd.d/tcp_status.conf
UserParameter=tcp.status[*],/etc/zabbix/zabbix_agentd.d/tcp_status.sh "$1" //脚本路径
[root@Node1 ~]#
3、确保配置Agent配置文件开启自定义参数UnsafeUserParameters=1
[root@Node1 ~]# grep -n "^[a-Z]" /etc/zabbix/zabbix_agentd.conf
13:PidFile=/var/run/zabbix/zabbix_agentd.pid
32:LogFile=/var/log/zabbix/zabbix_agentd.log
43:LogFileSize=0
57:DebugLevel=3
97:Server=172.17.21.208
138:ServerActive=172.17.21.208
149:Hostname=Node1.contoso.com
267:Include=/etc/zabbix/zabbix_agentd.d/*.conf
286:UnsafeUserParameters=1 //1代表允许,0代表关闭
4、重启zabbix-agent服务
[root@Node1 ~]# systemctl restart zabbix-agent.service
备注:因为脚本是把tcp的一些信息存放在/tmp/下,为了zabbix可以读取到我们设置zabbix可以读的权限,确保属主与属组都为zabbix即可
[root@Node1 ~]# chown zabbix.zabbix /tmp/tcp_status.txt //改变属主与属主
[root@Node1 ~]# ll /tmp/tcp_status.txt
-rw-rw-r-- 1 zabbix zabbix 38 Jan 18 11:32 /tmp/tcp_status.txt
5、在zabbix servere服务器上测试,是否能正常获取数据
[root@Node3 ~]# zabbix_get -s 172.17.21.206 -p 10050 -k "tcp.status[listen]"
0
[root@Node3 ~]# zabbix_get -s 172.17.21.206 -p 10050 -k "tcp.status[timewait]"
183
[root@Node3 ~]# zabbix_get -s 172.17.21.206 -p 10050 -k "tcp.status[established]"
2
[root@Node3 ~]#
三、WEB界面配置
1、导入事先准备好的TCP Connection Status模板,Configuration-->Templates-->Import,选择相应的模板,点击Import即可
2、关联模板,Configuration-->Hosts-->Node1--Templates,选择模板,点击Updtae即可
3、查看TCP状态:
Monitoring-->Graphs-->Host(Node1)-->Grahp(TCP Status)
附模板:Template_TCP_Status.xml
<?xml version="1.0" encoding="UTF-8"?>
<zabbix_export>
<version>3.4</version>
<date>2018-01-18T11:10:22Z</date>
<groups>
<group>
<name>Templates</name>
</group>
</groups>
<templates>
<template>
<template>Template TCP Connection Status</template>
<name>Template TCP Connection Status</name>
<description/>
<groups>
<group>
<name>Templates</name>
</group>
</groups>
<applications>
<application>
<name>TCP Status</name>
</application>
</applications>
<items>
<item>
<name>CLOSED</name>
<type>0</type>
<snmp_community/>
<snmp_oid/>
<key>tcp.status[closed]</key>
<delay>60s</delay>
<history>90d</history>
<trends>365d</trends>
<status>0</status>
<value_type>3</value_type>
<allowed_hosts/>
<units/>
<snmpv3_contextname/>
<snmpv3_securityname/>
<snmpv3_securitylevel>0</snmpv3_securitylevel>
<snmpv3_authprotocol>0</snmpv3_authprotocol>
<snmpv3_authpassphrase/>
<snmpv3_privprotocol>0</snmpv3_privprotocol>
<snmpv3_privpassphrase/>
<params/>
<ipmi_sensor/>
<authtype>0</authtype>
<username/>
<password/>
<publickey/>
<privatekey/>
<port/>
<description/>
<inventory_link>0</inventory_link>
<applications>
<application>
<name>TCP Status</name>
</application>
</applications>
<valuemap/>
<logtimefmt/>
<preprocessing/>
<jmx_endpoint/>
<master_item/>
</item>
<item>
<name>CLOSE_WAIT</name>
<type>0</type>
<snmp_community/>
<snmp_oid/>
<key>tcp.status[closewait]</key>
<delay>60s</delay>
<history>90d</history>
<trends>365d</trends>
<status>0</status>
<value_type>3</value_type>
<allowed_hosts/>
<units/>
<snmpv3_contextname/>
<snmpv3_securityname/>
<snmpv3_securitylevel>0</snmpv3_securitylevel>
<snmpv3_authprotocol>0</snmpv3_authprotocol>
<snmpv3_authpassphrase/>
<snmpv3_privprotocol>0</snmpv3_privprotocol>
<snmpv3_privpassphrase/>
<params/>
<ipmi_sensor/>
<authtype>0</authtype>
<username/>
<password/>
<publickey/>
<privatekey/>
<port/>
<description/>
<inventory_link>0</inventory_link>
<applications>
<application>
<name>TCP Status</name>
</application>
</applications>
<valuemap/>
<logtimefmt/>
<preprocessing/>
<jmx_endpoint/>
<master_item/>
</item>
<item>
<name>CLOSING</name>
<type>0</type>
<snmp_community/>
<snmp_oid/>
<key>tcp.status[closing]</key>
<delay>60s</delay>
<history>90d</history>
<trends>365d</trends>
<status>0</status>
<value_type>3</value_type>
<allowed_hosts/>
<units/>
<snmpv3_contextname/>
<snmpv3_securityname/>
<snmpv3_securitylevel>0</snmpv3_securitylevel>
<snmpv3_authprotocol>0</snmpv3_authprotocol>
<snmpv3_authpassphrase/>
<snmpv3_privprotocol>0</snmpv3_privprotocol>
<snmpv3_privpassphrase/>
<params/>
<ipmi_sensor/>
<authtype>0</authtype>
<username/>
<password/>
<publickey/>
<privatekey/>
<port/>
<description/>
<inventory_link>0</inventory_link>
<applications>
<application>
<name>TCP Status</name>
</application>
</applications>
<valuemap/>
<logtimefmt/>
<preprocessing/>
<jmx_endpoint/>
<master_item/>
</item>
<item>
<name>ESTABLISHED</name>
<type>0</type>
<snmp_community/>
<snmp_oid/>
<key>tcp.status[established]</key>
<delay>60s</delay>
<history>90d</history>
<trends>365d</trends>
<status>0</status>
<value_type>3</value_type>
<allowed_hosts/>
<units/>
<snmpv3_contextname/>
<snmpv3_securityname/>
<snmpv3_securitylevel>0</snmpv3_securitylevel>
<snmpv3_authprotocol>0</snmpv3_authprotocol>
<snmpv3_authpassphrase/>
<snmpv3_privprotocol>0</snmpv3_privprotocol>
<snmpv3_privpassphrase/>
<params/>
<ipmi_sensor/>
<authtype>0</authtype>
<username/>
<password/>
<publickey/>
<privatekey/>
<port/>
<description/>
<inventory_link>0</inventory_link>
<applications>
<application>
<name>TCP Status</name>
</application>
</applications>
<valuemap/>
<logtimefmt/>
<preprocessing/>
<jmx_endpoint/>
<master_item/>
</item>
<item>
<name>FIN_WAIT1</name>
<type>0</type>
<snmp_community/>
<snmp_oid/>
<key>tcp.status[finwait1]</key>
<delay>60s</delay>
<history>90d</history>
<trends>365d</trends>
<status>0</status>
<value_type>3</value_type>
<allowed_hosts/>
<units/>
<snmpv3_contextname/>
<snmpv3_securityname/>
<snmpv3_securitylevel>0</snmpv3_securitylevel>
<snmpv3_authprotocol>0</snmpv3_authprotocol>
<snmpv3_authpassphrase/>
<snmpv3_privprotocol>0</snmpv3_privprotocol>
<snmpv3_privpassphrase/>
<params/>
<ipmi_sensor/>
<authtype>0</authtype>
<username/>
<password/>
<publickey/>
<privatekey/>
<port/>
<description/>
<inventory_link>0</inventory_link>
<applications>
<application>
<name>TCP Status</name>
</application>
</applications>
<valuemap/>
<logtimefmt/>
<preprocessing/>
<jmx_endpoint/>
<master_item/>
</item>
<item>
<name>FIN_WAIT2</name>
<type>0</type>
<snmp_community/>
<snmp_oid/>
<key>tcp.status[finwait2]</key>
<delay>60s</delay>
<history>90d</history>
<trends>365d</trends>
<status>0</status>
<value_type>3</value_type>
<allowed_hosts/>
<units/>
<snmpv3_contextname/>
<snmpv3_securityname/>
<snmpv3_securitylevel>0</snmpv3_securitylevel>
<snmpv3_authprotocol>0</snmpv3_authprotocol>
<snmpv3_authpassphrase/>
<snmpv3_privprotocol>0</snmpv3_privprotocol>
<snmpv3_privpassphrase/>
<params/>
<ipmi_sensor/>
<authtype>0</authtype>
<username/>
<password/>
<publickey/>
<privatekey/>
<port/>
<description/>
<inventory_link>0</inventory_link>
<applications>
<application>
<name>TCP Status</name>
</application>
</applications>
<valuemap/>
<logtimefmt/>
<preprocessing/>
<jmx_endpoint/>
<master_item/>
</item>
<item>
<name>LAST_ACK</name>
<type>0</type>
<snmp_community/>
<snmp_oid/>
<key>tcp.status[lastack]</key>
<delay>60s</delay>
<history>90d</history>
<trends>365d</trends>
<status>0</status>
<value_type>3</value_type>
<allowed_hosts/>
<units/>
<snmpv3_contextname/>
<snmpv3_securityname/>
<snmpv3_securitylevel>0</snmpv3_securitylevel>
<snmpv3_authprotocol>0</snmpv3_authprotocol>
<snmpv3_authpassphrase/>
<snmpv3_privprotocol>0</snmpv3_privprotocol>
<snmpv3_privpassphrase/>
<params/>
<ipmi_sensor/>
<authtype>0</authtype>
<username/>
<password/>
<publickey/>
<privatekey/>
<port/>
<description/>
<inventory_link>0</inventory_link>
<applications>
<application>
<name>TCP Status</name>
</application>
</applications>
<valuemap/>
<logtimefmt/>
<preprocessing/>
<jmx_endpoint/>
<master_item/>
</item>
<item>
<name>LISTEN</name>
<type>0</type>
<snmp_community/>
<snmp_oid/>
<key>tcp.status[listen]</key>
<delay>60s</delay>
<history>90d</history>
<trends>365d</trends>
<status>0</status>
<value_type>3</value_type>
<allowed_hosts/>
<units/>
<snmpv3_contextname/>
<snmpv3_securityname/>
<snmpv3_securitylevel>0</snmpv3_securitylevel>
<snmpv3_authprotocol>0</snmpv3_authprotocol>
<snmpv3_authpassphrase/>
<snmpv3_privprotocol>0</snmpv3_privprotocol>
<snmpv3_privpassphrase/>
<params/>
<ipmi_sensor/>
<authtype>0</authtype>
<username/>
<password/>
<publickey/>
<privatekey/>
<port/>
<description/>
<inventory_link>0</inventory_link>
<applications>
<application>
<name>TCP Status</name>
</application>
</applications>
<valuemap/>
<logtimefmt/>
<preprocessing/>
<jmx_endpoint/>
<master_item/>
</item>
<item>
<name>SYN_RECV</name>
<type>0</type>
<snmp_community/>
<snmp_oid/>
<key>tcp.status[synrecv]</key>
<delay>60s</delay>
<history>90d</history>
<trends>365d</trends>
<status>0</status>
<value_type>3</value_type>
<allowed_hosts/>
<units/>
<snmpv3_contextname/>
<snmpv3_securityname/>
<snmpv3_securitylevel>0</snmpv3_securitylevel>
<snmpv3_authprotocol>0</snmpv3_authprotocol>
<snmpv3_authpassphrase/>
<snmpv3_privprotocol>0</snmpv3_privprotocol>
<snmpv3_privpassphrase/>
<params/>
<ipmi_sensor/>
<authtype>0</authtype>
<username/>
<password/>
<publickey/>
<privatekey/>
<port/>
<description/>
<inventory_link>0</inventory_link>
<applications>
<application>
<name>TCP Status</name>
</application>
</applications>
<valuemap/>
<logtimefmt/>
<preprocessing/>
<jmx_endpoint/>
<master_item/>
</item>
<item>
<name>SYN_SENT</name>
<type>0</type>
<snmp_community/>
<snmp_oid/>
<key>tcp.status[synsent]</key>
<delay>60s</delay>
<history>90d</history>
<trends>365d</trends>
<status>0</status>
<value_type>3</value_type>
<allowed_hosts/>
<units/>
<snmpv3_contextname/>
<snmpv3_securityname/>
<snmpv3_securitylevel>0</snmpv3_securitylevel>
<snmpv3_authprotocol>0</snmpv3_authprotocol>
<snmpv3_authpassphrase/>
<snmpv3_privprotocol>0</snmpv3_privprotocol>
<snmpv3_privpassphrase/>
<params/>
<ipmi_sensor/>
<authtype>0</authtype>
<username/>
<password/>
<publickey/>
<privatekey/>
<port/>
<description/>
<inventory_link>0</inventory_link>
<applications>
<application>
<name>TCP Status</name>
</application>
</applications>
<valuemap/>
<logtimefmt/>
<preprocessing/>
<jmx_endpoint/>
<master_item/>
</item>
<item>
<name>TIME_WAIT</name>
<type>0</type>
<snmp_community/>
<snmp_oid/>
<key>tcp.status[timewait]</key>
<delay>60s</delay>
<history>90d</history>
<trends>365d</trends>
<status>0</status>
<value_type>3</value_type>
<allowed_hosts/>
<units/>
<snmpv3_contextname/>
<snmpv3_securityname/>
<snmpv3_securitylevel>0</snmpv3_securitylevel>
<snmpv3_authprotocol>0</snmpv3_authprotocol>
<snmpv3_authpassphrase/>
<snmpv3_privprotocol>0</snmpv3_privprotocol>
<snmpv3_privpassphrase/>
<params/>
<ipmi_sensor/>
<authtype>0</authtype>
<username/>
<password/>
<publickey/>
<privatekey/>
<port/>
<description/>
<inventory_link>0</inventory_link>
<applications>
<application>
<name>TCP Status</name>
</application>
</applications>
<valuemap/>
<logtimefmt/>
<preprocessing/>
<jmx_endpoint/>
<master_item/>
</item>
</items>
<discovery_rules/>
<httptests/>
<macros/>
<templates/>
<screens/>
</template>
</templates>
<triggers>
<trigger>
<expression>{Template TCP Connection Status:tcp.status[listen].last()}>500</expression>
<recovery_mode>0</recovery_mode>
<recovery_expression/>
<name>There are too many TCP LISTEN status</name>
<correlation_mode>0</correlation_mode>
<correlation_tag/>
<url/>
<status>0</status>
<priority>4</priority>
<description/>
<type>0</type>
<manual_close>0</manual_close>
<dependencies/>
<tags/>
</trigger>
<trigger>
<expression>{Template TCP Connection Status:tcp.status[timewait].last()}>10000</expression>
<recovery_mode>0</recovery_mode>
<recovery_expression/>
<name>There are too many TCP TIME_WAIT status</name>
<correlation_mode>0</correlation_mode>
<correlation_tag/>
<url/>
<status>0</status>
<priority>4</priority>
<description/>
<type>0</type>
<manual_close>0</manual_close>
<dependencies/>
<tags/>
</trigger>
</triggers>
<graphs>
<graph>
<name>TCP Status</name>
<width>900</width>
<height>200</height>
<yaxismin>0.0000</yaxismin>
<yaxismax>100.0000</yaxismax>
<show_work_period>1</show_work_period>
<show_triggers>1</show_triggers>
<type>0</type>
<show_legend>1</show_legend>
<show_3d>0</show_3d>
<percent_left>0.0000</percent_left>
<percent_right>0.0000</percent_right>
<ymin_type_1>0</ymin_type_1>
<ymax_type_1>0</ymax_type_1>
<ymin_item_1>0</ymin_item_1>
<ymax_item_1>0</ymax_item_1>
<graph_items>
<graph_item>
<sortorder>0</sortorder>
<drawtype>0</drawtype>
<color>C80000</color>
<yaxisside>0</yaxisside>
<calc_fnc>2</calc_fnc>
<type>0</type>
<item>
<host>Template TCP Connection Status</host>
<key>tcp.status[closed]</key>
</item>
</graph_item>
<graph_item>
<sortorder>1</sortorder>
<drawtype>0</drawtype>
<color>00C800</color>
<yaxisside>0</yaxisside>
<calc_fnc>2</calc_fnc>
<type>0</type>
<item>
<host>Template TCP Connection Status</host>
<key>tcp.status[closewait]</key>
</item>
</graph_item>
<graph_item>
<sortorder>2</sortorder>
<drawtype>0</drawtype>
<color>0000C8</color>
<yaxisside>0</yaxisside>
<calc_fnc>2</calc_fnc>
<type>0</type>
<item>
<host>Template TCP Connection Status</host>
<key>tcp.status[closing]</key>
</item>
</graph_item>
<graph_item>
<sortorder>3</sortorder>
<drawtype>0</drawtype>
<color>C800C8</color>
<yaxisside>0</yaxisside>
<calc_fnc>2</calc_fnc>
<type>0</type>
<item>
<host>Template TCP Connection Status</host>
<key>tcp.status[established]</key>
</item>
</graph_item>
<graph_item>
<sortorder>4</sortorder>
<drawtype>0</drawtype>
<color>00C8C8</color>
<yaxisside>0</yaxisside>
<calc_fnc>2</calc_fnc>
<type>0</type>
<item>
<host>Template TCP Connection Status</host>
<key>tcp.status[finwait1]</key>
</item>
</graph_item>
<graph_item>
<sortorder>5</sortorder>
<drawtype>0</drawtype>
<color>C8C800</color>
<yaxisside>0</yaxisside>
<calc_fnc>2</calc_fnc>
<type>0</type>
<item>
<host>Template TCP Connection Status</host>
<key>tcp.status[finwait2]</key>
</item>
</graph_item>
<graph_item>
<sortorder>6</sortorder>
<drawtype>0</drawtype>
<color>C8C8C8</color>
<yaxisside>0</yaxisside>
<calc_fnc>2</calc_fnc>
<type>0</type>
<item>
<host>Template TCP Connection Status</host>
<key>tcp.status[lastack]</key>
</item>
</graph_item>
<graph_item>
<sortorder>7</sortorder>
<drawtype>0</drawtype>
<color>960000</color>
<yaxisside>0</yaxisside>
<calc_fnc>2</calc_fnc>
<type>0</type>
<item>
<host>Template TCP Connection Status</host>
<key>tcp.status[listen]</key>
</item>
</graph_item>
<graph_item>
<sortorder>8</sortorder>
<drawtype>0</drawtype>
<color>009600</color>
<yaxisside>0</yaxisside>
<calc_fnc>2</calc_fnc>
<type>0</type>
<item>
<host>Template TCP Connection Status</host>
<key>tcp.status[synrecv]</key>
</item>
</graph_item>
<graph_item>
<sortorder>9</sortorder>
<drawtype>0</drawtype>
<color>000096</color>
<yaxisside>0</yaxisside>
<calc_fnc>2</calc_fnc>
<type>0</type>
<item>
<host>Template TCP Connection Status</host>
<key>tcp.status[synsent]</key>
</item>
</graph_item>
<graph_item>
<sortorder>10</sortorder>
<drawtype>0</drawtype>
<color>960096</color>
<yaxisside>0</yaxisside>
<calc_fnc>2</calc_fnc>
<type>0</type>
<item>
<host>Template TCP Connection Status</host>
<key>tcp.status[timewait]</key>
</item>
</graph_item>
</graph_items>
</graph>
</graphs>
</zabbix_export>
View Code