prothemus 监控hadoop haproxy监控

转载

mob64ca14031c97 2024-04-28 15:32:37

文章标签 prothemus 监控hadoop mysql sed 连接数 文章分类 Hadoop 大数据

前言：

1、线上业务使用了几十上百台haproxy服务，需要针对这些服务做监控(包含haproxy端口对应的TCP连接数监控，haproxy服务监控等)。

2、普通的在zabbix_agentd.conf里面配置"UserParameter"的方式太繁琐，这里使用建表(收集数据,汇总数据)的方式。

3、将数据汇总到表里面，通过第三方脚本对数据做一个处理，拿到处理结果并监控。

一、创建数据表(收集haproxy服务的信息)

1.1、在zabbix_DB上创建表：t_3party_server

create table t_3party_server(
app varchar(64) NOT NULL,  
server varchar(64) NOT NULL,                                          
division varchar(64) NOT NULL,                                       
node varchar(64) NOT NULL,                                             
port varchar(64) NOT NULL,                                              
remark int(10) unsigned NOT NULL,                                   
PRIMARY KEY (app,server,division,node,port)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

注释：

app          #这里写的是项目名称(MOBA项目)
server       #服务名称(haproxy服务)
division     #由于起了多个haproxy实例，名称也不一样，所以这里做了区分
node         #haproxy服务IP地址
port         #haproxy服务端口,多个实例端口不同  
remark       #备注作用(由于线上某些haproxy服务安装路径不一致，所以此处用来区分路径，也可以区分其他)
PRIMARY KEY (app,server,division,node,port)  #添加主键

1.2、创建完毕后效果

mysql> desc t_3party_server;
+----------+-------------+------+-----+---------+-------+
| Field    | Type        | Null | Key | Default | Extra |
+----------+-------------+------+-----+---------+-------+
| app      | varchar(64) | NO   | PRI | NULL    |       |
| server   | varchar(64) | NO   | PRI | NULL    |       |
| division | varchar(64) | NO   | PRI | NULL    |       |
| node     | varchar(64) | NO   | PRI | NULL    |       |
| port     | varchar(64) | NO   | PRI | NULL    |       |
| remark   | text        | NO   |     | NULL    |       |
+----------+-------------+------+-----+---------+-------+
6 rows in set (0.00 sec)

1.3、然后往数据表里面插入数据：

1.3.1、在Navicat上添加

insert into t_3party_server(app,server,division,node,port,status) values('MOBA','haproxy','haproxy-login','10.117.16.242','30021','0');

insert into t_3party_server(app,server,division,node,port,status) values('MOBA','haproxy','haproxy-match','10.117.16.242','30032','0');

insert into t_3party_server(app,server,division,node,port,status) values('MOBA','haproxy','haproxy-test','10.117.16.242','30035','0');

1.3.2、在MySQL数据库客户端上添加：

INSERT INTO `zabbix`.`t_3party_server` (`app`, `server`, `division`, `node`, `port`, `remark`) VALUES ('MOBA', 'haproxy', 'haproxy-test', '10.117.16.242', '30035', '0');

1.4、最终数据生成图

prothemus 监控hadoop haproxy监控_sed

二、针对生成的数据表做监控

2.1、监控脚本：放在zabbix_server下 "/install_path/zabbix_server/share/zabbix/externalscripts"

#!/usr/bin/php
<?php

if ($_SERVER["argc"] != 2) {
	echo "Usage: " . $_SERVER["argv"][0] . " host\n";
	return;
}

$host = $_SERVER["argv"][1];

$mysqlconn = new mysqli("10.116.207.188", "moba", "moba2016", "zabbix", 3306);
if (mysqli_connect_errno()) {
	echo "fail to connect mysql: " . mysqli_connect_error() . "\n";
	return;
}

$result = $mysqlconn->query("SELECT * FROM t_3party_server WHERE node=\"" . $mysqlconn->real_escape_string($host) . "\"");

$hostvalue = array(
	"data" => array()
);
while ($row = $result->fetch_assoc()) {
	$app = $row["app"];
	$server = $row["server"];
	$division = $row["division"];
	$node = $row["node"];
        $port = $row["port"];
        $remark = $row["remark"];

        if ($remark == "/usr/local")
        {
            $hostvalue["data"][] = array(
                "{#APP}" => $app,
                "{#SERVER}" => $server,
                "{#DIVISION}" => $division,
                "{#PORT}" => $port,
                "{#PATH}" => "/usr/local/" . $division . "/sbin",
            );

             continue;
        }

	$server = substr($server, 0, 15);

	$hostvalue["data"][] = array(
		"{#APP}" => $app,
		"{#SERVER}" => $server,
		"{#DIVISION}" => $division,
                "{#PORT}" => $port,
		"{#PATH}" => "/data/app/" . $division . "/sbin",
		#"{#MFW_PATH}" => "/data/app/" . $division . "/" . ($division ? $division . "/" : "") . $server,
	);
}

echo json_encode($hostvalue);
echo "\n";

2.1.1、脚本生成的数据：

./mfw_discovery_haproxy.php 10.117.16.222
{"data":[{"{#APP}":"MOBA","{#SERVER}":"haproxy","{#DIVISION}":"haproxy-login","{#PORT}":"30021","{#PATH}":"\/data\/app\/haproxy-login\/sbin"}]}

2.1.2、转换为Json格式如下：

{
    "data":[
        {
            "{#APP}":"MOBA",
            "{#SERVER}":"haproxy",
            "{#DIVISION}":"haproxy-login",
            "{#PORT}":"30021",
            "{#PATH}":"/data/app/haproxy-login/sbin"
        }
    ]
}

2.2、在haproxy服务机器上配置

1、配置：zabbix_agentd.conf

vim /data/app/zabbix_agent/etc/zabbix_agentd.conf

LogRemoteCommands=1
LogFileSize=1
DebugLevel=3

#Haproxy    #用来监控TCP连接数
UserParameter=haproxy.tcp.[*],/usr/sbin/ss -ant sport eq :"$1" | awk 'NR>1{++s[$$1]} END {for(k in s){print k,s[k]}}' |grep -w "ESTAB" |awk '{print $$2}'

2、配置：haproxy服务守护脚本 #此处可不配(根据业务选择)

启动脚本：

#!/bin/bash

old_path=`pwd`
cd `dirname $0`
path=`pwd`
server=sbin/haproxy


if [ -n "`ps -ef|grep "$path/$server"|grep -v "grep"`" ]
then
    echo -e "\033[32m $path/$server was running!.... \033[0m"
    ps -ef|grep "$path/$server"|grep -v "grep"
else
    echo -e "\033[32m start $path/$server .... \033[0m"
    $path/$server -f $path/conf/haproxy.cfg -p /tmp/$RANDOM-haproxy.pid  -n 20000 -N 20000
    if [ -n "`ps -ef|grep "$path/$server"|grep -v "grep"`" ]
        then
        echo -e "\033[32m start $path/$server ok .... \033[0m"
        ps -ef|grep "$path/$server"|grep -v "grep"
    else
        echo -e "\033[31m start $path/$server faild .... \033[0m"
        fi
fi

关闭脚本：

#!/bin/bash

old_path=`pwd`
cd `dirname $0`
path=`pwd`
server=sbin/haproxy

if [ "$(ps -ef|grep "$path/$server"|grep -v "grep" | awk '{print $2}')" == "" ];then
    echo -e "\033[32m can not find $path/$server process .... \033[0m"

    else
    killall $path/$server 2>/dev/null

until [ -z "`ps -ef|grep "$path/$server"|grep -v "grep"`" ]
do
    sleep 1
done

ps -ef|grep "$path/$server"|grep -v "grep"
echo -e "\033[31m stop $path/$server ok .... \033[0m"

fi

ps -ef|grep "$path/$server"|grep -v "grep"

killall $path/$server 2>/dev/null

重载脚本：

#!/bin/bash

old_path=`pwd`
cd `dirname $0`
path=`pwd`
server=sbin/haproxy
hpid=`ps -ef|grep "$path/$server"|grep -v "grep" |awk '{print $ 12}'`

if [ -n "`ps -ef|grep "$path/$server"|grep -v "grep"`" ]
then
    echo -e "\033[32m $path/$server is running!.... \033[0m"
    $path/$server -f $path/conf/haproxy.cfg -p $hpid  -n 20000 -N 20000 -sf $(cat $hpid)
    ps -ef|grep "$path/$server"|grep -v "grep"
else
    echo -e "\033[31m $path/$server was not running.... \033[0m"
    echo -e "\033[32m start $path/$server process.... \033[0m"
    $path/$server -f $path/conf/haproxy.cfg -p /tmp/$RANDOM-haproxy.pid  -n 20000 -N 20000
    if [ -n "`ps -ef|grep "$path/$server"|grep -v "grep"`" ]
        then
        echo -e "\033[32m start $path/$server ok .... \033[0m"
        ps -ef|grep "$path/$server"|grep -v "grep"
    else
        echo -e "\033[31m start $path/$server faild .... \033[0m"
        fi
fi

守护进程脚本：

#!/bin/sh

#chkconfig: 2345 59 81
#description: The Haproxy service daemon script.

if [ -d /usr/local/haproxy/ ];then
SCRIPT=`/bin/find /usr/local/*haproxy*  -name 'start.sh'`
else
SCRIPT=`/bin/find /data/app/*haproxy* -name 'start.sh'`
fi

PROC_NAME=haproxy
PID=0

function proc_num()                                              
{  
    NUM=`ps -ef | grep ${PROC_NAME} | grep -v grep |wc -l`  
    return $NUM
}  

function proc_id()                                             
{  
    PID=`ps -ef | grep ${PROC_NAME} | grep -v grep | awk '{print $2}'`  
    return $PID
} 

function shell(){

    echo "${SCRIPT}" | /bin/sed -n '1p' | xargs /bin/sh  >>/dev/null
    sleep 0.3
    echo "${SCRIPT}" | /bin/sed -n '2p' | xargs /bin/sh  >>/dev/null
    sleep 0.3
    echo "${SCRIPT}" | /bin/sed -n '3p' | xargs /bin/sh  >>/dev/null
}

function main(){
  proc_num
  #proc_id

#if [ ${NUM} -eq 0 ] || [ ${PID} -eq 0 ];then
if [ ${NUM} -eq 0 ];then
  shell
fi
}

main

创建定时任务：

*/2 * * * * /data/tools/script/daemon_Haproxy.sh > /dev/null 2>&1

2.3、在zabbix上添加监控

2.3.1、创建模板-->自动发现规则

prothemus 监控hadoop haproxy监控_连接数_02

定义宏变量：宏的值为json导出后的值

prothemus 监控hadoop haproxy监控_sed_03

2.3.2、添加item和trigger

Item1：检查haproxy进程是否存在

Key：proc.num["{#SERVER}",,,"{#PATH}"]

prothemus 监控hadoop haproxy监控_prothemus 监控hadoop_04

Trigger1：对应Item1

{Template moba server:proc.num["{#SERVER}",,,"{#PATH}"].last()} < 1

prothemus 监控hadoop haproxy监控_连接数_05

Item2：检查haproxy端口对应的TCP连接数

Key：haproxy.tcp.[{#PORT}] 这个key需要在haproxy机器上的zabbix_agent.conf手动配置UserParameter (前面有介绍)

prothemus 监控hadoop haproxy监控_mysql_06

Trigger2：对应Item2

{Template moba server:haproxy.tcp.[{#PORT}].last()}>40000

prothemus 监控hadoop haproxy监控_连接数_07

三、之后新增haproxy服务添加到监控的流程如下

1、在zabbixDB上的t_3party_server表里添加数据：
INSERT INTO `zabbix`.`t_3party_server` (`app`, `server`, `division`, `node`, `port`, `remark`) VALUES ('MOBA', 'haproxy', 'haproxy', '10.111.152.15', '30032', '0');

2、在zabbix上 Host groups-->Haproxy_Group 组内添加IP   #自定义(可不要)

3、在机器上添加配置参数：
vim /data/app/zabbix_agent/etc/zabbix_agentd.conf

LogRemoteCommands=1
LogFileSize=1
DebugLevel=3

#Haproxy
UserParameter=haproxy.tcp.[*],/usr/sbin/ss -ant sport eq :"$1" | awk 'NR>1{++s[$$1]} END {for(k in s){print k,s[k]}}' |grep -w "ESTAB" |awk '{print $$2}'
             
4、在机器上添加守护脚本：
/data/tools/script/daemon_Haproxy.sh

5、添加定时任务
*/2 * * * * /data/tools/script/daemon_Haproxy.sh > /dev/null 2>&1

6、把守护进程脚本放到  /data/app/zabbix_agent/share/zabbix/externalscripts/ 这个目录内：   #这一步主要是给zabbix上执行远程命令使用的【Configuration--》Actions--》WeChat--》Operations--》--》2	Run remote commands on host groups: Haproxy_Group】  #自定义(可不要)

cp -ap /data/tools/script/daemon_Haproxy.sh /data/app/zabbix_agent/share/zabbix/externalscripts/

7、配置完毕后重启 zabbix_agentd
pkill zabbix_agentd
/data/app/zabbix_agent/sbin/zabbix_agentd

8、到zabbix上监控haproxy服务是否被监控

本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。