-1

I just discovered my debian server has been hacked. I noticed when I did apt-get update, it failed because /var/log/apt directory did not exist.

I investigated and found the following:

  • Almost all files in /var/log/ were owned by the group "smile".
  • Most of /var/log files were zero bites and dated Oct 10 (presume time of attack)
  • There was an empty file /var/log/apt (preventing directory apt)
  • There was a new user on the server called "smile" with associated group.
  • Ran rkhunter, gave warnings:
    • Possible rootkit: Xzibit Rootkit (web search returns probably false positive)
    • Warning: Account 'smile' is root equivalent (UID = 0)
  • Neither ps ax| grep smile, or w or who showed anything

I then did 

# userdel smile
userdel: user smile is currently logged in
# userdel -f smile

Then

# grep -R smile /var/log
./auth.log:Nov 20 03:02:05 server sshd[9228]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=211.  user=smile
./auth.log:Nov 20 03:02:07 server sshd[9228]: Failed password for smile from 211.... port 58701 ssh2
./auth.log:Nov 20 04:20:32 server sshd[10819]: Invalid user smileing from 211....
./auth.log:Nov 20 04:20:34 server sshd[10819]: Failed password for invalid user smileing from 211.... port 33223 ssh2

So I can assume the user/bot has been logging in multiple times.

I restored the permissions on /var/log, and mkdir /var/log/apt. I then did apt-get update; apt-get upgrade (it had been a while).

My questions are:

  • How can I investigate further to find out what the attackers did/changed?
  • What else can I do to secure the server? Is it even possible or will I have to do a fresh install?

It is a head node on a small experimental cluster. It is doing NIS, NFS, Torque PBS and Ganglia and would be very inconvenient to reinstall.

Thank you

davepc
  • 3
  • 2
  • 1
    Should be on serverfault. I flagged it for moderator attention, it'll probably be migrated soon – Jon Nov 21 '13 at 02:33
  • 1
    Why investigate. Install all updates for all software running on the server. It might be an inconvenience to reinstall the server but its the ONLY WAY to be sure its CLEAN – Ramhound Nov 21 '13 at 02:42
  • I want to investigate in increase my knowledge of server security, discover as much as possible what they did, and possibly stop any ongoing malicious services. I fully acknowledge that to be 100% sure, a fresh install is required, but like I said it is an experimental cluster, there are no mission critical services running. – davepc Nov 21 '13 at 03:06

1 Answers1

3
  1. Rebuild the system. Don't try to repair it. You won't succeed.
  2. Don't trust anything on the system. Anything could be compromised.
  3. Replace all SSL keys. Revoke existing certificates. Get new ones.
  4. Make sure your system is always fully up to date. Always install all security updates. Always run a supported release.
  5. Use a host integrity monitoring system. Some options are samhain, osiris, tripwire, aide, suricata.
  6. Use a configuration management system (preferably an auto healing one). Some options are cfengine, puppet, chef.
  7. Don't run services you don't need.
  8. Don't run services with a track record of frequent security vulnerabilities. Find alternatives with a better track record for security.
bahamat
  • 6,193
  • 23
  • 28