how to check if my root server was compromised

Question

My provider sent me an email stating that my root server seems to have been misused for attacking other systems.

How do I check if this is true and my system is compromised?

I was told I have 4 days to fix the problem and all the info I was supplied with are the logs further down from my provider.

The server runs Debian Squeeze and is always up-to-date. Very few users have ssh access and only via a jailkit so there isn't much they can do. The web server, apache2, is running via suexec and FastCGI so if one site is compromised, the other are still safe. Interesting enough, the IP reported to have been 85.214.249.*** is my second IP which I only received and activated a week ago.

rkhunter is running daily on my server, I just did a full check again followed by a clamscan. No results. I compared the log entries provided with my own syslog entries but nothing. All I have is a ton of incoming traffic that apf-firewall does block since it is not legitimate traffic. I can't find either any of these dreamhost servers nor any of their IPs in any of my logs.

| Attacker's IP | Timestamp (Pacific Time)      | Targeted Server       | Attack ID     | Attack Information    |
 -------------------------------------------------------------------------------------------------------
| 85.214.249.***        | 2012-04-16 11:15:01   | johnson.dreamhost.com | 28775675      | e107 BBCode Arbitrary PHP Code Execution Vulnerability        |
| 85.214.249.***        | 2012-04-16 11:12:55   | unuk.dreamhost.com    | 28802766      | e107 BBCode Arbitrary PHP Code Execution Vulnerability        |
| 85.214.249.***        | 2012-04-16 10:50:29   | nationals.dreamhost.com | 28784913      | e107 BBCode Arbitrary PHP Code Execution Vulnerability        |
| 85.214.249.***        | 2012-04-16 11:03:23   | lakers.dreamhost.com  | 28776910      | e107 BBCode Arbitrary PHP Code Execution Vulnerability        |
| 85.214.249.***        | 2012-04-16 11:02:27   | univox.dreamhost.com  | 28803414      | e107 BBCode Arbitrary PHP Code Execution Vulnerability        |

@David schwartz: you misread this. Supposedly attacks to that particular exploit have been launched from my IP towards those hosts you see listed above. I need to know if it is true and how to figure out from where/who did it... — Ovidiu Pacuraru, Apr 18 '12 at 10:31
@OvidiuPacuraru either case it means your system is compromised, hence nuke it from orbit. — Lucas Kauffman, Apr 18 '12 at 10:37
@LucasKauffman: That will be the final step, as soon as I can, but I was wondering as I have only the email of my ISP as proof and nothing in any logs of the server. Since I can be sure nobody got root access, the only way the server can be compromised is via an exploit on the sites I host and that should be picked up by rkhunter and/or clamscan, right? — Ovidiu Pacuraru, Apr 18 '12 at 10:45
If they have root they might be able to change Rkhunter behavior or just the way md5sum works. — Lucas Kauffman, Apr 18 '12 at 10:46
I would recommend running `tcpdump -n -w log.pcap src 85.214.249.x` for a while (as long as it takes) to see if you are still causing any attack traffic. The `x` in the IP address I used should match your second IP address. If you find some, you definitely have a problem... if not, you still may have a problem. — Ladadadada, Apr 18 '12 at 10:48
Has one of your users been online at that specific time? If they have network access, they could try out new scripts they found elsewhere (the log looks like it). — ott--, Apr 18 '12 at 11:31

score 1 · Answer 1 · edited Apr 13 '17 at 12:14

1

First of all if your system is compromised you can't trust your logs. Rootkits are there to make you think everything is running normally. The only way to detect if there is illegitimate traffic residing from your server, is to sniff it once it has left your machine (this can be done with a repeater port on a switch).

So unfortunately this might mean:

edited Apr 13 '17 at 12:14

Community

1

answered Apr 18 '12 at 10:45

Lucas Kauffman

16,818
9
57
92

I'd even -1 it - While true, it's a no-brainer answer. Nuking gives no insight on what exactly happened and how the system was compromised, if one simply nukes and reinstalls, system will be compromised again. It's a MUST to know what happened. – Sandman4 Apr 18 '12 at 11:10
1

I never said you can't image it to go look for the culprit. I said you need to take it out of production. Also the first part isn't quite a no brainer imo. – Lucas Kauffman Apr 18 '12 at 11:34
Just to clarify matters: I am pretty sure the root login isn't compromised. SSH login as root is disabled, I use sudo su which is only allowed for my user and needs the password besides which I get daily log file stats about attempted logins to ssh and successful logins too. Too many attempts to login via ssh results in banning the IP and I don't use ssh from other machines than my own. The purpose of the question was to learn how to debug and find out if/how the server was compromised and not to be told to nuke it regardless whether that will be the result of my investigation or not. – Ovidiu Pacuraru Apr 18 '12 at 11:58

score 0 · Answer 2 · answered Apr 18 '12 at 11:47

tcpdump all the outgoing traffic that goes from the IP in question to destination ports 80 and 443. On most systems there normally will be not much of such traffic. If system is still actively trying to attack others, you will probably see a lot of outgoing packets to different hosts.

If you see no traffic, you can be pretty sure that CURRENTLY it isn't attacking, but from that to conclusion that there's no compromise is a very long way. Probably the only way to be sure is to find what WAS the source of the attack traffic.

Since you mention that you got IP only recently, is there a possibility that your ISP still have two machines with that IP ? - Previous owner could configure it statically. Maybe you disable that IP on your system (as you have two IPs, it shoudn't be a problem - configure one IP statically and siable DHCP) and then try to ping the disabled IP. (Though I'm not 100% sure I'm saying anything sane here - have little experience on this topic.)

So would you be using tcpdump on the host? Because most rootkits these days are made in such a way that tools like tcpdump get modified to only display certain information. — Lucas Kauffman, Apr 18 '12 at 12:02
Using this method and tweaking my clamscan with some parameters (turns out I was not using the command line scanner properly) I tracked down infected files inside one wordpress based site.It was a plugin that used an old timthumb.php version and that was the point used to compromise that particular website.The infected files were deleted, the vulnerable file replaced with a newer version. Currently checking if anything else is infected but it is highly unlikely as the compromised website ran using FASTCgi + suexec so the attacker only inherited the rights of that heavily restricted user/group. — Ovidiu Pacuraru, Apr 18 '12 at 14:14

how to check if my root server was compromised

2 Answers2