1

A linux server I manage is sending out hundreds of spam messages from a specific user account every 5 minutes.

I found, within the user's account (which runs Wordpress), a couple of PHP exploit scripts. One of these was the Meyhem dropper.

However, I can't find any signs on the system that the droppers have compromised the system. The files they reference don't exist, there are no running "host" processes, nothing in crontab, no unusual listening or connections on port 80, etc. I went through all running processes and didn't see anything unusual (may have missed something).

However the only sign of a problem is that the system sends out a ton of messages every five minutes, using the same "from" username. (the username of the compromised user mentioned above).

I can monitor /var/log/maillog and every 5 minutes these messages drop into the queue.

I can't figure out how to tell which process is doing this. I have stopped httpd, crond, atd, and disabled the customer's website so nothing would be hitting the malicious PHP files. I also renamed the PHP files. But still, every 5 minutes the log file shows another boatload of messages from this user being sent out.

Can anyone please point me in the right direction to find this exploit/malware/etc?

If there is a way to watch all newly spawned processes, I could see which one spawns at the time the spam goes out. Any ideas?

Excerpt from maillog:

Jan 28 18:12:13 xyz postfix/qmgr[6829]: C4BF3206075: from=<username@xyz.[domain name here].net>, size=1199, nrcpt=1 (queue active)
Jan 28 18:12:13 xyz postfix/qmgr[6829]: C0421227061: from=<username@xyz.[domain name here].net>, size=1222, nrcpt=1 (queue active)
Jan 28 18:12:13 xyz postfix/qmgr[6829]: C05082060C7: from=<username@xyz.[domain name here].net>, size=1180, nrcpt=1 (queue active)
Jan 28 18:12:13 xyz postfix/qmgr[6829]: BA6D922629A: from=<username@xyz.[domain name here].net>, size=1232, nrcpt=1 (queue active)
Jan 28 18:12:13 xyz postfix/qmgr[6829]: BC58A226B0F: from=<username@xyz.[domain name here].net>, size=1224, nrcpt=1 (queue active)
Jan 28 18:12:13 xyz postfix/qmgr[6829]: BB1C6227574: from=<username@xyz.[domain name here].net>, size=1216, nrcpt=1 (queue active)
Jan 28 18:12:13 xyz postfix/qmgr[6829]: B896B226EB3: from=<username@xyz.[domain name here].net>, size=1194, nrcpt=1 (queue active)
Jan 28 18:12:13 xyz postfix/qmgr[6829]: BC7532266B8: from=<username@xyz.[domain name here].net>, size=1186, nrcpt=1 (queue active)
Ryan Griggs
  • 885
  • 2
  • 12
  • 27
  • post the maillog here – masegaloeh Jan 28 '15 at 22:37
  • First, @HopelessN00b, I need to confirm that it *is* compromised. Then I have justification for wiping it. But as I stated, I can't find any indicators that would point to an infection. The only thing is the repeated emails being sent every 5 min. – Ryan Griggs Jan 28 '15 at 23:22
  • I just looked in the postfix deferred spool directories and found hundreds of files sitting there. I deleted all these and now the server is not attempting to send mail every 5 minutes. I'm thinking that during the time the compromised scripts existed on that user's wordpress account, they were using it as a mail relay. – Ryan Griggs Jan 28 '15 at 23:23
  • Looks like those messages queued up in 'deferred' and were possibly being retried every 5 min by postfix itself. Does that sound logical? – Ryan Griggs Jan 28 '15 at 23:24
  • well, you can start by issuing command `grep BC58A226B0F /var/log/maillog` to know, where the email enter postfix... – masegaloeh Jan 28 '15 at 23:28
  • Here's the first line for that message. It came through the postfix pickup: Jan 26 19:23:06 xyz postfix/pickup[26636]: BC58A226B0F: uid=642 from= – Ryan Griggs Jan 28 '15 at 23:34

3 Answers3

3

In php.ini find mail.log. Activate logging PHP mails.

mail.log = /var/log/phpmaillog

Also activate x-mail-headers

mail.add_x_header = 1

Reboot httpd to activate the PHP changes.

Than start monitoring this log.

tail -f /var/log/phpmaillog

Delete those PHP files. Install iThemes Wordpress plugin.

borayeris
  • 213
  • 1
  • 9
1

If you also have a rootkit exploit then you will not be able to see the processes, network connections, etc. from the compromised machine. This is because the executables you use have been compromised and adjusted not to show the other compromised tools and services.

In this kind of situation you must, at the very least, boot from a verifiable separate OS (something like SystemRescueCD would be my starting point). Then you can verify all the executables against installation checksums and reinstall anything dubious. Ideally you would tear the system down and reinstall.

I'm sure there are many helpful articles about recovering from rootkits on https://security.stackexchange.com/

Community
  • 1
roaima
  • 1,567
  • 13
  • 26
  • 1
    Does Linode offer any kind of rescue mode for systems that are borked and won't boot? (My own hosting provder does, hence the question.) – roaima Jan 28 '15 at 23:21
  • Two problems: 1) this is a virtual server hosted at linode, so I can't boot it from any other media, and 2) I checked for many indicators of infection/rootkit per documentation and found none. – Ryan Griggs Jan 28 '15 at 23:25
  • YES it does! Cool, I had not thought of that. https://www.linode.com/docs/troubleshooting/rescue-and-rebuild – Ryan Griggs Jan 28 '15 at 23:26
  • What would be the most effective way to verify file integrity? Can I run a DIFF against a repository, or checksum important files, or is there something else that's more automated? – Ryan Griggs Jan 28 '15 at 23:26
  • You _should_ be able to verify against your installation repository. It's not something I'd want to have to do, though, as it will take a lot of (your) time. HopelessN00b's reference to the "[probable duplicate](http://serverfault.com/questions/218005/how-do-i-deal-with-a-compromised-server)" article contains more detail on this and is worth reviewing. – roaima Jan 28 '15 at 23:29
0

Perhaps TCPdump can give you an insight. Watch the NIC card sending traffic to that target i.e

tcpdump -i eth0 -p smtp -Z root (or add the IP, instead of SMTP)

Joe
  • 13
  • 4