One of the ubuntu 16.04 servers I operate is being abused for sending spam. There are no user accounts, nobody other than me has shell access.
What I know so far is that this was caused by php scripts being injected via a www-data writeable directory in the web folders (an unfortunate necessity for a web app that's hosted there). So far no existing files have been compromised. A bunch of scripts having showed up in directories where www-data has write permissions, and the server started sending spam.
Rather then killing this machine and setting it up anew with improved security, I'm trying to figure out exactly what happened first, but I'm hitting a wall trying to find where the spam scripts are hidden. Perhaps someone with better sysadmin skills can help me try and locate them?
This is an amazon tiny VM with both packet filter set up to only allow ports 80/443 from anywhere, as well as iptables set up to do the same (to protect the internal interface as well). I only noticed something was wrong because OSSEC told me there were files being written to the web directory that shouldn't be there. A little later, amazon sent an abuse report stating my server had been used to send spam. I've removed the offending scripts as well as any trace I could find of scripts that might be used to trigger hidden spam scripts on the system, but I can't find anything further and the spam keeps re-occurring.
I've poked around the process stack when the spammer scripts are active. I am unable to located the script itself as it appears to remove itself after launch, and it also manages to change its own process name as listed in e.g. ps to exim as well as httpd and possibly others:
www-data 4568 0.1 0.7 38440 14492 ? Ss 21:47 0:00 httpd
www-data 5036 0.1 0.4 32964 9056 ? Ss 21:50 0:00 exim
exim isn't even installed on this server. Looking at details for the exim process:
sudo lsof | grep 5036:
exim 5036 www-data cwd DIR 202,1 4096 2 /
exim 5036 www-data rtd DIR 202,1 4096 2 /
exim 5036 www-data txt REG 202,1 1907192 927 /usr/bin/perl
exim 5036 www-data mem REG 202,1 27168 6706 /usr/lib/x86_64-linux-gnu/perl/5.22.1/auto/File/Glob/Glob.so
exim 5036 www-data mem REG 202,1 102064 6721 /usr/lib/x86_64-linux-gnu/perl/5.22.1/auto/POSIX/POSIX.so
exim 5036 www-data mem REG 202,1 18696 6704 /usr/lib/x86_64-linux-gnu/perl/5.22.1/auto/Fcntl/Fcntl.so
exim 5036 www-data mem REG 202,1 43464 6727 /usr/lib/x86_64-linux-gnu/perl/5.22.1/auto/Socket/Socket.so
exim 5036 www-data mem REG 202,1 18768 6713 /usr/lib/x86_64-linux-gnu/perl/5.22.1/auto/IO/IO.so
exim 5036 www-data mem REG 202,1 39224 396841 /lib/x86_64-linux-gnu/libcrypt-2.23.so
exim 5036 www-data mem REG 202,1 1864888 396824 /lib/x86_64-linux-gnu/libc-2.23.so
exim 5036 www-data mem REG 202,1 138696 399454 /lib/x86_64-linux-gnu/libpthread-2.23.so
exim 5036 www-data mem REG 202,1 1088952 396860 /lib/x86_64-linux-gnu/libm-2.23.so
exim 5036 www-data mem REG 202,1 14608 396822 /lib/x86_64-linux-gnu/libdl-2.23.so
exim 5036 www-data mem REG 202,1 162632 396436 /lib/x86_64-linux-gnu/ld-2.23.so
exim 5036 www-data 0r CHR 1,3 0t0 6 /dev/null
exim 5036 www-data 1w CHR 1,3 0t0 6 /dev/null
exim 5036 www-data 2w CHR 1,3 0t0 6 /dev/null
exim 5036 www-data 3u IPv4 7357569 0t0 TCP *:27450 (LISTEN)
exim 5036 www-data 22r FIFO 0,10 0t0 4428903 pipe
exim 5036 www-data 23w FIFO 0,10 0t0 4428903 pipe
exim 5036 www-data 24r FIFO 0,10 0t0 4428904 pipe
exim 5036 www-data 25w FIFO 0,10 0t0 4428904 pipe
So we see this is apparently a perl script masquerading its name as exim, listening on TCP port 27450. That should be fruitless though as there is a physical firewall only allowing ports 80 and 443 from anywhere. The only other port allowed is 22 and that is limited to the admin location using an IP mask.
Further to this, there are iptables set up with the same rules, only allow 80/443 from anywhere, the rest is totally locked down.
In other words, whatever is instructing these scripts to do anything, it comes in via HTTP/HTTPS.
So looking at the httpd process which seems more suspect, we find that this again is a perl process that was launched from /tmp/phpnMv330:
ubuntu@XXX:~/bin$ sudo ls -la /proc/4678/exe
lrwxrwxrwx 1 www-data www-data 0 Dec 24 22:00 /proc/4678/exe -> /usr/bin/perl
Also, this seems to be the more active process:
sudo lsof | grep 4678
httpd 4678 www-data cwd DIR 202,1 4096 2 /
httpd 4678 www-data rtd DIR 202,1 4096 2 /
httpd 4678 www-data txt REG 202,1 1907192 927 /usr/bin/perl
httpd 4678 www-data mem REG 202,1 27168 6706 /usr/lib/x86_64-linux-gnu/perl/5.22.1/auto/File/Glob/Glob.so
httpd 4678 www-data mem REG 202,1 102064 6721 /usr/lib/x86_64-linux-gnu/perl/5.22.1/auto/POSIX/POSIX.so
httpd 4678 www-data mem REG 202,1 18696 6704 /usr/lib/x86_64-linux-gnu/perl/5.22.1/auto/Fcntl/Fcntl.so
httpd 4678 www-data mem REG 202,1 43464 6727 /usr/lib/x86_64-linux-gnu/perl/5.22.1/auto/Socket/Socket.so
httpd 4678 www-data mem REG 202,1 18768 6713 /usr/lib/x86_64-linux-gnu/perl/5.22.1/auto/IO/IO.so
httpd 4678 www-data mem REG 202,1 39224 396841 /lib/x86_64-linux-gnu/libcrypt-2.23.so
httpd 4678 www-data mem REG 202,1 1864888 396824 /lib/x86_64-linux-gnu/libc-2.23.so
httpd 4678 www-data mem REG 202,1 138696 399454 /lib/x86_64-linux-gnu/libpthread-2.23.so
httpd 4678 www-data mem REG 202,1 1088952 396860 /lib/x86_64-linux-gnu/libm-2.23.so
httpd 4678 www-data mem REG 202,1 14608 396822 /lib/x86_64-linux-gnu/libdl-2.23.so
httpd 4678 www-data mem REG 202,1 162632 396436 /lib/x86_64-linux-gnu/ld-2.23.so
httpd 4678 www-data 0r CHR 1,3 0t0 6 /dev/null
httpd 4678 www-data 1w CHR 1,3 0t0 6 /dev/null
httpd 4678 www-data 2w CHR 1,3 0t0 6 /dev/null
httpd 4678 www-data 4w FIFO 0,10 0t0 721333 pipe
httpd 4678 www-data 5r FIFO 0,10 0t0 721334 pipe
httpd 4678 www-data 7u IPv4 7451383 0t0 TCP ip-XXX-XXX-XXX-XXX.us-west-2.compute.internal:60598->etb-4.mail.tiscali.it:smtp (ESTABLISHED)
httpd 4678 www-data 8u IPv4 7450674 0t0 TCP ip-XXX-XXX-XXX-XXX.us-west-2.compute.internal:48986->mta-v3.mail.vip.gq1.yahoo.com:smtp (ESTABLISHED)
httpd 4678 www-data 9u IPv4 7450754 0t0 TCP ip-XXX-XXX-XXX-XXX.us-west-2.compute.internal:49146->mta-v3.mail.vip.gq1.yahoo.com:smtp (ESTABLISHED)
httpd 4678 www-data 10u IPv4 7450546 0t0 TCP ip-XXX-XXX-XXX-XXX.us-west-2.compute.internal:48730->mta-v3.mail.vip.gq1.yahoo.com:smtp (ESTABLISHED)
httpd 4678 www-data 11u IPv4 7450866 0t0 TCP ip-XXX-XXX-XXX-XXX.us-west-2.compute.internal:49370->mta-v3.mail.vip.gq1.yahoo.com:smtp (ESTABLISHED)
httpd 4678 www-data 14u IPv4 7450853 0t0 TCP ip-XXX-XXX-XXX-XXX.us-west-2.compute.internal:49344->mta-v3.mail.vip.gq1.yahoo.com:smtp (ESTABLISHED)
httpd 4678 www-data 15u IPv4 7450480 0t0 TCP ip-XXX-XXX-XXX-XXX.us-west-2.compute.internal:48598->mta-v3.mail.vip.gq1.yahoo.com:smtp (ESTABLISHED)
httpd 4678 www-data 16u IPv4 7450824 0t0 TCP ip-XXX-XXX-XXX-XXX.us-west-2.compute.internal:49286->mta-v3.mail.vip.gq1.yahoo.com:smtp (ESTABLISHED)
httpd 4678 www-data 17u IPv4 7450508 0t0 TCP ip-XXX-XXX-XXX-XXX.us-west-2.compute.internal:48654->mta-v3.mail.vip.gq1.yahoo.com:smtp (ESTABLISHED)
httpd 4678 www-data 22r FIFO 0,10 0t0 114904 pipe
httpd 4678 www-data 23w FIFO 0,10 0t0 114904 pipe
httpd 4678 www-data 24r FIFO 0,10 0t0 114905 pipe
httpd 4678 www-data 25w FIFO 0,10 0t0 114905 pipe
httpd 4678 www-data 32u REG 202,1 115271 633 /tmp/phpnMv33O (deleted)
I've replaced /usr/sbin/sendmail with a wrapper (as per this post: Mail spam sending script injected) to report which account is sending the emails, simply to confirm that it is www-data. However, the sendmail wrapper never gets triggered by the scripts, yet all mail ends up in the mail queue. I wonder how that occurs? Sending mail manually using e.g. mail -s "xxx" to@me.com
I can't find any suspicious files in any of the web directories now, I've cleaned them up. I ran maldet which located a few of the bad scripts, but missed at least half of them. There's only one web directory the www-data user has write permissions to. I've also cleaned up /tmp and /var/tmp, both if which had been harbouring spam scripts. OSSEC does not report any new files being written to the web directory www-data can write to. The www-data cronjob that used to run a spam script has been disabled as well (and the target executable removed - which was en ELF binary btw, not a script).
EDIT: I just located a suspicious POST in the apache logs in a old and static wordpress site on the same server which has no write/upload/update permissions anywhere, i.e. is a crippled WP. The POST was to a file that's been there since early August this year. That POST coincided with with spam generation being triggered using a binary located in /var/tmp.
Lessons: Spammers appear to sometimes take their time between injection and abuse of a system. I'm still interested in learning other sysadmin's tricks of the trade to locate such abuses more swiftly. I'm also still stumped as to how it is possible for a perl process to rename itself in ps...
