Odd log entries and insane bandwidth usage

Question

I have a CentOS cloud server I use for testing and noticed that in the past 30 days, bandwidth out has gone through the roof (16,000 GB to be precise)

When I initially attempted to check on the server it was completely inaccessible (no response on SSH, web, even console (console just showed a bunch of errors, no login prompt)

I bounced the server and started going through logs and logins. No SSH logins in ages, nothing strange in the logs except this every minute:

Jan 17 02:20:01 wwwdev crond[21971]: (apache) CMD (/tmp/.../.shy/update >/dev/null 2>&1)
Jan 17 02:21:01 wwwdev crond[21976]: (apache) CMD (/tmp/.../.shy/update >/dev/null 2>&1)
Jan 17 02:22:01 wwwdev crond[21985]: (apache) CMD (/tmp/.../.shy/update >/dev/null 2>&1)
Jan 17 02:23:01 wwwdev crond[21990]: (apache) CMD (/tmp/.../.shy/update >/dev/null 2>&1)
Jan 17 02:24:01 wwwdev crond[22000]: (apache) CMD (/tmp/.../.shy/update >/dev/null 2>&1)
Jan 17 02:25:01 wwwdev crond[22006]: (apache) CMD (/tmp/.../.shy/update >/dev/null 2>&1)
Jan 17 02:26:01 wwwdev crond[22015]: (apache) CMD (/tmp/.../.shy/update >/dev/null 2>&1)
Jan 17 02:27:01 wwwdev crond[22024]: (apache) CMD (/tmp/.../.shy/update >/dev/null 2>&1)
Jan 17 02:28:01 wwwdev crond[22029]: (apache) CMD (/tmp/.../.shy/update >/dev/null 2>&1)
Jan 17 02:29:01 wwwdev crond[22034]: (apache) CMD (/tmp/.../.shy/update >/dev/null 2>&1)

Whatever was actually in /tmp is long gone since I had to reboot but I would really like to know what the heck happened.

If you don't mind me asking - 16tb of traffic - how much did that just cost you? — Mark Henderson, Jan 17 '12 at 08:10
@MarkHenderson more than I care to think about; $3,000. A contracter installed phpMyAdmin, I should have kept an eye on that stuff (expensive lesson) — Code Magician, Jan 17 '12 at 08:39
Bugger. I've always hated PHPMyAdmin and this is just another reason to add to the list why ;) — Mark Henderson, Jan 17 '12 at 10:12

score 5 · Accepted Answer · edited Apr 13 '17 at 12:14

I don't know for sure but that description smells very much like a server compromise that sent out email spam. It could also have been a worm searching for more hosts to compromise but the email spam sounds more likely.

If you look up your server's IP address at an email reputation website such as SenderScore or SenderBase or simply search for it in Google you might find some evidence of this and even examples of the spam. You can also check in DNS Blacklists to see whether your server is blacklisted.

The username that the cron job was running as suggests the compromise (if that's what it was) happened via your website. Your web server access and error logs in the days before the cron jobs started would be the best place to start looking.

Nearly forgot the obligatory link to My server's been hacked EMERGENCY. Plenty of good advice there about what to do and how to clean up. Don't forget to remove or restrict phpMyAdmin.

That appears to be the case, looks like PhpMyAdmin was on there and it got hit with a remote code execution exploit. — Code Magician, Jan 17 '12 at 08:06
I believe this particular worm tries to find insecure SIP endpoints that bridge to the regular phone network and make them dial premium numbers. — David Schwartz, Jan 17 '12 at 12:37

Odd log entries and insane bandwidth usage

1 Answers1