Environment
- Rackspace
- Ubuntu 12.04
- Wordpress
- MySql
The issue
I have been experiencing quite serious out-memory-issues in the last couple of days.
While I resolved one possible cause the issue I still get a very suspicious activity of sendmail.
Any recommendations on how to tackle this issue? I think that must be some malware, but I no experience on resolving this kind of attacks.
htop
1 [||||||||||||||||||||||||| 27.0%] Tasks: 101, 50 thr; 1 running
2 [||||||||||||||||||||||||||||||||||||||||| 45.7%] Load average: 12.96 12.55 11.95
Mem[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||1183/1995MB] Uptime: 09:53:28
Swp[|||| 93/2047MB]
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
19704 root 20 0 120M 25328 2896 S 2.0 1.2 0:46.16 sendmail: MTA: ./s6HH4rLv009027 gmail.co.: user open
3298 root 20 0 99M 5612 1684 S 2.0 0.3 2:46.31 sendmail: MTA: s6OABpf4003298 localhost [127.0.0.1]: DATA
3301 root 20 0 99M 5544 1684 S 2.0 0.3 2:40.89 sendmail: MTA: s6OAGAAh003301 localhost [127.0.0.1]: DATA
19510 root 20 0 26488 2568 1212 R 2.0 0.1 0:23.73 htop
771 syslog 20 0 244M 3892 516 S 1.0 0.2 2:22.43 rsyslogd -c5
1226 smmsp 20 0 133M 56328 1396 S 0.0 2.8 1:56.85 sendmail: MSP: ./s6K1OdvJ030780 [127.0.0.1]: client DATA status
32488 root 20 0 102M 7168 2748 S 0.0 0.4 0:00.02 sendmail: MTA: ./s6OAcr6I032488 aspmx.l.google.com.: client EHLO
31723 www-data 39 19 448M 72676 47276 S 0.0 3.6 0:01.14 /usr/sbin/apache2 -k start
29624 root 20 0 120M 25916 2884 S 0.0 1.3 0:29.65 sendmail: MTA: ./s6NHPdHs002287 todito.com.: user open
898 mysql 20 0 1315M 105M 3296 S 0.0 5.3 23:25.23 /usr/sbin/mysqld
30966 root 20 0 101M 5092 460 D 0.0 0.2 0:01.52 sendmail: MTA: running queue: /var/spool/mqueue
5013 mysql 20 0 1315M 105M 3296 S 0.0 5.3 0:25.58 /usr/sbin/mysqld
25504 root 20 0 120M 25904 2900 S 0.0 1.3 0:24.57 sendmail: MTA: ./s6JHcEdS028616 hotamil.com.: user open
1033 root 20 0 630M 6228 2356 S 0.0 0.3 1:17.85 /usr/local/bin/driveclient --daemon
1062 root 20 0 630M 6228 2356 S 0.0 0.3 0:12.50 /usr/local/bin/driveclient --daemon
1082 newrelic 20 0 107M 1576 1072 S 0.0 0.1 0:46.81 /usr/sbin/nrsysmond -c /etc/newrelic/nrsysmond.cfg -p /var/run/nrsysmond.pid
1089 newrelic 20 0 107M 1576 1072 S 0.0 0.1 0:46.80 /usr/sbin/nrsysmond -c /etc/newrelic/nrsysmond.cfg -p /var/run/nrsysmond.pid
822 syslog 20 0 244M 3892 516 S 0.0 0.2 1:35.12 rsyslogd -c5
1061 root 20 0 630M 6228 2356 S 0.0 0.3 0:12.80 /usr/local/bin/driveclient --daemon
8532 root 20 0 105M 9444 460 D 0.0 0.5 0:06.40 sendmail: MTA: running queue: /var/spool/mqueue
31711 www-data 39 19 445M 75316 52764 S 0.0 3.7 0:01.50 /usr/sbin/apache2 -k start
27927 root 20 0 120M 25904 2900 S 0.0 1.3 0:32.35 sendmail: MTA: ./s6NKLEhE005721 yahoo.co.: user open
13821 mysql 20 0 1315M 105M 3296 S 0.0 5.3 2:25.39 /usr/sbin/mysqld
31924 mysql 20 0 1315M 105M 3296 S 0.0 5.3 0:49.12 /usr/sbin/mysqld
31713 www-data 39 19 446M 68484 45496 S 0.0 3.4 0:00.79 /usr/sbin/apache2 -k start
4195 mysql 20 0 1315M 105M 3296 S 0.0 5.3 0:29.08 /usr/sbin/mysqld
9799 mysql 20 0 1315M 105M 3296 S 0.0 5.3 2:29.95 /usr/sbin/mysqld
2664 smmsp 20 0 133M 56424 1476 D 0.0 2.8 1:52.68 sendmail: MSP: ./s6K3MC7s027126 [127.0.0.1]: client DATA status
853 syslog 20 0 244M 3892 516 S 0.0 0.2 0:47.23 rsyslogd -c5
31714 www-data 39 19 446M 68404 45420 S 0.0 3.3 0:00.73 /usr/sbin/apache2 -k start
31903 mysql 20 0 1315M 105M 3296 S 0.0 5.3 0:47.96 /usr/sbin/mysqld
1063 root 20 0 630M 6228 2356 S 0.0 0.3 0:12.40 /usr/local/bin/driveclient --daemon
31600 www-data 39 19 448M 71340 46228 S 0.0 3.5 0:00.92 /usr/sbin/apache2 -k start
4308 mysql 20 0 1315M 105M 3296 S 0.0 5.3 0:28.28 /usr/sbin/mysqld
1064 root 20 0 630M 6228 2356 S 0.0 0.3 0:12.41 /usr/local/bin/driveclient --daemon
31727 www-data 39 19 447M 70324 45756 S 0.0 3.4 0:00.84 /usr/sbin/apache2 -k start
31725 www-data 39 19 447M 70340 45756 S 0.0 3.4 0:00.86 /usr/sbin/apache2 -k start
31724 www-data 39 19 447M 70548 45932 S 0.0 3.5 0:00.84 /usr/sbin/apache2 -k start
1715 mysql 20 0 1315M 105M 3296 S 0.0 5.3 3:05.00 /usr/sbin/mysqld
23774 root 39 19 425M 6636 4676 S 0.0 0.3 0:06.00 /usr/sbin/apache2 -k start
1065 root 20 0 630M 6228 2356 S 0.0 0.3 0:12.35 /usr/local/bin/driveclient --daemon
1060 root 20 0 630M 6228 2356 S 0.0 0.3 0:12.43 /usr/local/bin/driveclient --daemon
F1Help F2Setup F3SearchF4FilterF5Tre
huge /var/mail
root@web:/var/mail# ls -alh
total 1.2G
drwxrwsrwt 2 root mail 4.0K Jul 24 10:51 .
drwxr-xr-x 15 root root 4.0K Jul 24 00:45 ..
-rw-rw---- 1 munin mail 83K Jul 19 18:48 munin
-rw------- 1 root mail 1.1G Jul 24 10:51 root
-rw-rw---- 1 www-data mail 98M Jul 23 22:34 www-data
My root mail account is continuously sending emails
NOTE: I have replace my domain by filtered.com
Return-Path: <MAILER-DAEMON>
Received: from localhost (localhost)
by web.filtered.com (8.14.4/8.14.4/Debian-2ubuntu2) id s6OAqpuv010033;
Thu, 24 Jul 2014 10:52:51 GMT
Date: Thu, 24 Jul 2014 10:52:51 GMT
From: Mail Delivery Subsystem <MAILER-DAEMON>
Message-Id: <201407241052.s6OAqpuv010033@web.filtered.com>
To: <kara_velazquez@filtered.com>
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
boundary="s6OAqpuv010033.1406199171/web.filtered.com"
Subject: Returned mail: see transcript for details
Auto-Submitted: auto-generated (failure)
--s6OAqpuw010033.1406199172/web.filtered.com--
From MAILER-DAEMON Thu Jul 24 10:52:53 2014
Return-Path: <MAILER-DAEMON>
Received: from localhost (localhost)
by web.filtered.com (8.14.4/8.14.4/Debian-2ubuntu2) id s6OAqq6J010047;
Thu, 24 Jul 2014 10:52:53 GMT
Date: Thu, 24 Jul 2014 10:52:53 GMT
From: Mail Delivery Subsystem <MAILER-DAEMON>
Message-Id: <201407241052.s6OAqq6J010047@web.filtered.com>
To: postmaster
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
boundary="s6OAqq6J010047.1406199173/web.filtered.com"
Subject: Postmaster notify: see transcript for details
Auto-Submitted: auto-generated (postmaster-notification)
This is a MIME-encapsulated message
--s6OAqq6J010047.1406199173/web.filtered.com
The original message was received at Thu, 24 Jul 2014 10:52:52 GMT
from localhost
with id s6OAqq6I010047
----- The following addresses had permanent fatal errors -----
<audra_gray@filtered.com>
(reason: 550-5.1.1 The email account that you tried to reach does not exist. Please try)
----- Transcript of session follows -----
... while talking to aspmx.l.google.com.:
>>> RCPT To:<audra_gray@filtered.com>
<<< 550-5.1.1 The email account that you tried to reach does not exist. Please try
<<< 550-5.1.1 double-checking the recipient's email address for typos or
<<< 550-5.1.1 unnecessary spaces. Learn more at
<<< 550 5.1.1 http://support.google.com/mail/bin/answer.py?answer=6596 sq8si14059110obc.83 - gsmtp
550 5.1.1 <audra_gray@filtered.com>... User unknown
>>> DATA
<<< 503 5.5.1 RCPT first. sq8si14059110obc.83 - gsmtp
--s6OAqq6J010047.1406199173/web.filtered.com
Content-Type: message/delivery-status
Reporting-MTA: dns; web.filtered.com
Received-From-MTA: DNS; localhost
Arrival-Date: Thu, 24 Jul 2014 10:52:52 GMT
Final-Recipient: RFC822; audra_gray@filtered.com
Action: failed
Status: 5.1.1
Remote-MTA: DNS; aspmx.l.google.com
Diagnostic-Code: SMTP; 550-5.1.1 The email account that you tried to reach does not exist. Please try
Last-Attempt-Date: Thu, 24 Jul 2014 10:52:53 GMT
--s6OAqq6J010047.1406199173/web.filtered.com
Content-Type: text/rfc822-headers
Return-Path: <MAILER-DAEMON>
Received: from localhost (localhost)
by web.filtered.com (8.14.4/8.14.4/Debian-2ubuntu2) id s6OAqq6I010047;
Thu, 24 Jul 2014 10:52:52 GMT
Date: Thu, 24 Jul 2014 10:52:52 GMT
From: Mail Delivery Subsystem <MAILER-DAEMON>
Message-Id: <201407241052.s6OAqq6I010047@web.filtered.com>
To: <audra_gray@filtered.com>
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
boundary="s6OAqq6I010047.1406199172/web.filtered.com"
Subject: Returned mail: see transcript for details
Auto-Submitted: auto-generated (failure)
--s6OAqq6J010047.1406199173/web.filtered.com--
ps -ef | grep sendmail
root@web:/var/mail# ps -ef | grep sendmail
smmsp 1226 1 0 00:45 ? 00:02:04 sendmail: MSP: ./s6KKDDVU014035 [127.0.0.1]: client DATA status
smmsp 2644 2641 0 01:00 ? 00:00:00 /bin/sh -c test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp
smmsp 2647 2644 0 01:00 ? 00:00:00 /bin/sh /usr/share/sendmail/sendmail cron-msp
smmsp 2664 2647 0 01:00 ? 00:01:58 sendmail: MSP: [127.0.0.1]: idle
root 3298 1 1 07:57 ? 00:03:16 sendmail: MTA: s6OB1dam003298 localhost [127.0.0.1]: DATA
root 3301 1 1 07:57 ? 00:03:05 sendmail: MTA: server localhost [127.0.0.1] cmd read
root 19675 1 0 11:20 ? 00:00:00 sendmail: MTA: ./s6OBKJuv019675 aspmx.l.google.com.: client DATA 354
root 19689 1 0 11:20 ? 00:00:00 sendmail: MTA: ./s6OBKLuv019689 aspmx.l.google.com.: client DATA 354
root 19800 1 0 11:20 ? 00:00:00 sendmail: MTA: ./s6OBKbuv019800 aspmx.l.google.com.: client DATA 354
root 20178 1 0 11:21 ? 00:00:00 sendmail: MTA: ./s6OBLSuv020178 aspmx.l.google.com.: client DATA 354
root 20270 1 0 11:21 ? 00:00:00 sendmail: MTA: ./s6OBLZuv020270 aspmx.l.google.com.: client DATA 354
root 20537 1 0 11:21 ? 00:00:00 sendmail: MTA: ./s6OBM0uv020537 aspmx.l.google.com.: client DATA 354
root 20646 1 0 11:22 ? 00:00:00 sendmail: MTA: ./s6OBM5uv020646 aspmx.l.google.com.: client DATA 354
root 21006 1 0 11:22 ? 00:00:00 sendmail: MTA: ./s6OBMZ6I021006 aspmx.l.google.com.: client DATA 354
root 21015 1 0 11:22 ? 00:00:00 sendmail: MTA: ./s6OBMZ6I021015 aspmx.l.google.com.: client DATA 354
root 21027 1 0 11:22 ? 00:00:00 sendmail: MTA: ./s6OBMauv021027 aspmx.l.google.com.: client DATA 354
root 21036 1 0 11:22 ? 00:00:00 sendmail: MTA: ./s6OBMb6I021036 aspmx.l.google.com.: client DATA 354
root 21063 1 0 11:22 ? 00:00:00 sendmail: MTA: ./s6OBMeuv021063 aspmx.l.google.com.: client DATA 354
root 21065 1 0 11:22 ? 00:00:00 sendmail: MTA: ./s6OBMg6I021065 aspmx.l.google.com.: client DATA 354
root 21086 1 1 11:22 ? 00:00:00 sendmail: MTA: ./s6OBMg6I021086 aspmx.l.google.com.: client DATA 354
root 21094 1 0 11:22 ? 00:00:00 sendmail: MTA: ./s6OBMg6I021094 aspmx.l.google.com.: client DATA 354
root 21098 1 2 11:22 ? 00:00:00 sendmail: MTA: ./s6OBMg6I021098 aspmx.l.google.com.: client DATA 354
root 21103 1 1 11:22 ? 00:00:00 sendmail: MTA: ./s6OBMg6I021103 aspmx.l.google.com.: client DATA 354
root 21105 1 1 11:22 ? 00:00:00 sendmail: MTA: ./s6OBMguv021105 aspmx.l.google.com.: client DATA 354
root 21108 1 0 11:22 ? 00:00:00 sendmail: MTA: ./s6OB1dag003298 mx-eu.mail.am0.yahoodns.net.: client MAIL
root 21111 1 0 11:22 ? 00:00:00 sendmail: MTA: ./s6OBMg6I021111 aspmx.l.google.com.: client RCPT
root 21113 1 0 11:22 ? 00:00:00 sendmail: MTA: ./s6OAsOi1003301 mx-ha03.web.de.: client greeting
root 21117 1 1 11:22 ? 00:00:00 sendmail: MTA: ./s6OAsOi3003301 gmail-smtp-in.l.google.com.: client DATA status
root 21123 1 0 11:22 ? 00:00:00 sendmail: MTA: ./s6OAsOi5003301 gmail-smtp-in.l.google.com.: client EHLO
root 21127 18604 0 11:22 pts/0 00:00:00 grep --color=auto sendmail
Sendmail status
root@web:/var/mail# /etc/init.d/sendmail status
MSP: is run via cron (20m)
MTA: is not running
QUE: Same as MTA
/var/spool/mqueue
root@web:/var/spool# ls -alh
total 48M
drwxr-xr-x 7 root root 4.0K Mar 29 2013 .
drwxr-xr-x 15 root root 4.0K Jul 24 00:45 ..
drwxr-xr-x 5 root root 4.0K May 1 2012 cron
lrwxrwxrwx 1 root root 7 May 1 2012 mail -> ../mail
drwxr-s--- 2 smmta smmsp 14M Jul 24 11:44 mqueue
drwxrws--- 2 smmsp smmsp 34M Jul 24 12:25 mqueue-client
drwxr-xr-x 2 root root 4.0K Apr 13 2012 plymouth
drwxr-xr-x 2 root root 4.0K Mar 30 2012 rsyslog
root@web:/var/spool# du -h -d 1
4.0K ./plymouth
1.6G ./mqueue <=====
4.0K ./rsyslog
root@web:/var/spool/mqueue# more qfs6OBTUZY003298
V8
T1406201622
K1406201622
N1
P120781
I202/1/476577
MDeferred: 421 4.7.1 : (DYN:T1) http://postmaster.info.aol.com/errors/421dynt1.html
Fbs
$_localhost [127.0.0.1]
$rESMTP
$sweb.anybots.com
${daemon_flags}
${if_addr}127.0.0.1
S<patty_jennings@anybots.com>
MDeferred: 421 4.7.1 : (DYN:T1) http://postmaster.info.aol.com/errors/421dynt1.html
rRFC822; neve7@aim.com
RPFD:<neve7@aim.com>
H?P?Return-Path: <?g>
H??Received: from web.anybots.com (localhost [127.0.0.1])
by web.anybots.com (8.14.4/8.14.4/Debian-2ubuntu2) with ESMTP id s6OBTUZY003298
for <neve7@aim.com>; Thu, 24 Jul 2014 11:33:42 GMT
H??Received: (from www-data@localhost)
by web.anybots.com (8.14.4/8.14.4/Submit) id s6JHVJId026134;
Sat, 19 Jul 2014 17:31:19 GMT
H??Date: Sat, 19 Jul 2014 17:31:19 GMT
H??Message-Id: <201407191731.s6JHVJId026134@web.anybots.com>
H??X-Authentication-Warning: web.anybots.com: www-data set sender to patty_jennings@anybots.com using -f
H??To: neve7@aim.com
H??Subject: Fw: Hi Generic Drugs Online Products
H??X-PHP-Originating-Script: 33:dirs.php
H??From: "Patty Jennings" <patty_jennings@anybots.com>
H??Reply-To:"Patty Jennings" <patty_jennings@anybots.com>
H??X-Priority: 3 (Normal)
H??MIME-Version: 1.0
H??Content-Type: text/html; charset="iso-8859-1"
H??Content-Transfer-Encoding: 8bit
.
