Compromised Debian server sending DDOS attack

0

I have a server online that apparently has been compromised and it sends DDOS attacks to another ip. The system is a debian 7 server, hosting a few wordpress, joomla websites, and airtime to host a radio. Fail2ban is installed and configured.

I have access in rescue mode, but i could not find anything suspicious in logs. As soon as i boot the system in normal mode it starts sending packets and i cannot remote login (ssh).

The following image reflects the packets sent as soon as i attempt to boot the system.

The following image reflects the packets sent as soon as the system attempts to boot in normal mode

Any help appreciated.

spacebiker

Posted 2014-08-05T18:07:52.377

Reputation: 310

Do you have physical access to the server and its NIC? Is it your own server or a hosted server? Can you use Wireshark to monitor between the server NIC and the network to see what's going on? – Kinnectus – 2014-08-05T20:02:09.090

It is a hosted server with online.net so i do not have physical access to the server. When i reboot normally from the online.net console pings do not respond, i cannot login via ssh with my user password as i used to do before, and the statistics page (also in online.net console) traces those packet sent peaks as shown in attached image – spacebiker – 2014-08-05T20:22:20.320

I think you may need to involve your hosting company if you cannot access the server to properly identify the problem origin... – Kinnectus – 2014-08-05T20:29:09.020

Well, i can ssh on "rescue mode" and mount the problematic file system, I checked logs in the server, also searched for any file modified last week, checked the system for rootkits or virus but i found nothing suspicious. I moved the /var/www folder completely to a new location to avoid any possible wordpress , joomla vulnerability but as i told you, when i reboot the system normally, pinging does not respond, therefor the ssh login is not possible and the server is sending a lot of outbound packets. Is there a way to modify grub so that only the very default is loaded at boot? – spacebiker – 2014-08-05T20:40:49.850

1

Nuke it from orbit. http://serverfault.com/questions/218005/how-do-i-deal-with-a-compromised-server

– Zoredache – 2014-08-05T22:15:38.297

Answers

0

I finally solved it, i should remark that the image attached in the question was showing outbound packets sent, not due to any ddos attack but because of the traffic generated when rebooting in "rescue mode".

These are the steps i took to solve it:

1- Start on rescue mode fron the online.net console.

2- ssh on "rescue mode" and mount the problematic file system, note that your device could differ from the following sample command:

$ mount /dev/sda2 /mnt

3- Check logs in the server:

$ cat /var/log/syslog
$ cat /var/log/dmesg
$ cat /var/log/messages
$ cat /var/log/fail2ban.log
$ cat /var/log/auth.log

Additionally your apache2 access.log, and/or any other critical logs in your system.

4- Search for any file modified a day or two before the attack:

$ find /mnt -mtime -3 -ls

5- Check the system for rootkits (rkhunter) or virus (clamav)

As I could not find anything suspicious I checked if there was any kind of remote access service in the online.net console ( i wish i had thought of this before) and so it was. Then i reboot in normal mode and i could check that the system was not booting due to errors in /dev/sda2. To solve it i rebooted in rescue mode and did a

$ fsck2 /dev/sda2

There were some corrupted inodes, i replied Yes to repair them all.

6- Then i rebooted the system again, but the boot process got stock at "cleaning temporary files". To solve it i changed TMPTIME to 60 in the following file:

/etc/default/rcS

Note: Remember to resolve this issue and put back this setting to 0 on successful login, leaving TMPTIME to remove tmp files older than 60 days is a security risk

7- After a few minor errors i could finally login, put the needed settings back to defaults, and rescan the entire system with rkhunter and clamav, apt-get update && apt-get upgrade and a last reboot.

The system now is clean and running again.

spacebiker

Posted 2014-08-05T18:07:52.377

Reputation: 310