2

My system is CentOS 6.1, /var/log directory was gone, what are the possible problems? I created a new /var/log, and 6 hours later it was gone again.

Is there any system log that I can inspect?

pat.inside
  • 123
  • 2
  • 3
    I agree with the general feeling that it's likely the server has been compromised. You may wish to look at http://serverfault.com/questions/218005/my-servers-been-hacked-emergency which is the canonical answer on recovering from intrusion. – MadHatter Dec 29 '11 at 12:18

3 Answers3

9

Most probable answers:

  • your server was hacked
  • you installed a program that removes files
  • you wrote a program that removes files

The first answer is much more likely...

koollman
  • 131
  • 2
5

Looks like your server is compromised. I would recommend to check the following:

  • crontab
  • output of ps

Immediately disable root logins. Each and everything requiring superuser privilege should be run using sudo. commands executed via sudo are logged with the username and other details at syslog or you can define the log path in /etc/sudoers. I recommend to log it outside /var/log since your /var/log is vanishing.

As a final measure you can rename /bin/rm to /bin/myrm or something else and tell trusted users to use it instead of rm. This way rouge scripts executing rm will not be able to do anything.

Aditya Patawari
  • 1,065
  • 8
  • 23
  • 1
    What makes you think they're executing rm, and not just using the rm syscall? – Tom O'Connor Dec 29 '11 at 12:35
  • 7
    Bugger disabling anything. Turn it off at the wall. – Tom O'Connor Dec 29 '11 at 12:35
  • of course I cannot be sure that they are not using the system call. It is just something I would have checked. – Aditya Patawari Dec 29 '11 at 12:48
  • What @TomO'Connor said -- Rip the plug out and do proper forensics later, but if it smells compromised get it the hell off the network. – voretaq7 Dec 29 '11 at 20:23
  • 2
    Also note that if you suspect the system is compromised ***NONE*** of the software installed on it can be assumed to be telling the truth -- You cannot "fix" a compromised system because you cannot trust anything on it (right up to and including the kernel - Remember Linux lets you hot-patch running kernels, so an attacker can bury stuff that will hose even "known good" binaries run off a CD). – voretaq7 Dec 29 '11 at 20:26
1

Booting this system off a rescue CD (after power off) should be ok, as long as you don`t run any binaries off that system. If you have a backup of your server before it misbehaved compare the files, inspect crontab, look at tmpwatch, ...

Nils
  • 7,657
  • 3
  • 31
  • 71