0

Today I received a call from security officer telling me that the /etc/passwd file of one machine was truncated (0 byte). The same happened with /etc/group a couple of days before.

I tried to trace the cause but I had no success... can someone please give me a clue where should I look for?

I'm running Red Hat 5.7 x86_64.

84104
  • 12,698
  • 6
  • 43
  • 75
jyz
  • 155
  • 6
  • 1
    My personal guess is that you have someone with root access that doesn't understand output redirection as well as they should. – Andrew B Feb 15 '13 at 19:43
  • Also, does your company implement any form of configuration management that might be pushing out updates to system files like these? – Andrew B Feb 15 '13 at 19:49
  • @AndrewB yes, they run eTrust client on each machine. Do you think this might be the problem? – jyz Feb 18 '13 at 14:32
  • Yes, config management platforms are notorious for blowing away files when misconfigured. Have you found a solution to this in the meantime, or has it stopped happening? Please either provide an answer of your own and self-accept, or accept the answer that was closest to solving your problem. Thanks! – Andrew B Jun 16 '13 at 20:09

3 Answers3

6

If you think you've been hacked, I strongly recommend How do I deal with a compromised server? .

Otherwise, check root's .bash_history, looking for things like output redirection onto /etc/passwd. If you use sudo to control root access, check the sudo logs around the time (the modification time on the file will tell you when it was zeroed out, most likely).

And, for goodness' sake, get off Red Hat 5.7. You're running an OS that's nearly a year's worth of patches out-of-date, and there's no sane reason to. Red Hat's patching policy is that no version of any package is ever bumped inside a major version of the OS; patches are backported instead. The whole point of this is that you should be able to keep your RHEL5 constantly in patch without endangering any functionality.

In many ways, there isn't even any such thing as "Red Hat 5.7"; RHEL 5.7 isn't really an OS release version, it's simply a line drawn through the current state of patch of RHEL5 on 12/3/2012. When you say "I'm running RHEL 5.7" what you're really saying is "I'm running RHEL 5 and I'm a year out of date".

Community
  • 1
MadHatter
  • 78,442
  • 20
  • 178
  • 229
  • I'm not sure I'm being hacked because this is internal server, just few people access it. Of course I can't discard this option, but I prefer to think in another options. Those servers will be upgraded later, there's already a plan going on. – jyz Feb 15 '13 at 18:12
  • **Remember:** Unless timestamps were enabled in the history files, events are not necessarily in chronological order. They are large blobs of commands that were appended to the file in the order that the shells terminated. – Andrew B Feb 15 '13 at 19:31
  • 1
    jyzuz, I'm not saying upgrade them - I'm saying **patch** them, is all. – MadHatter Feb 15 '13 at 20:55
4

It will be a lot easier to find out the cause, if you have auditd daemon properly configured to watch on critical files. In addition to the suggestions forwarded by others, you can try to look for hints in the /var/log/audit/audit.log file and use the aureport and ausearch commands.

For instance, you can run ausearch -f passwd to list all audit information related to the passwd file.

In the future, you might need to follow the security guidelines set in the CIS Red Hat Enterprise Linux 5 Benchmark on how to configure audit daemon to watch (-w ) certain files. Here are some lines you need to add to /etc/audit/audit.rules -

Things that affect identity:

   -w /etc/group -p wa -k identity
   -w /etc/passwd -p wa -k identity
   -w /etc/gshadow -p wa -k identity
   -w /etc/shadow -p wa -k identity
   -w /etc/security/opasswd -p wa -k identity
Daniel t.
  • 9,061
  • 1
  • 32
  • 36
3

@MadHatter offers a good start. Beyond that, here's a trick I keep in my toolbelt for days like these:

find /filesystem1 /filesystem2 -xdev -printf '%T@ %t %p\n' | sort

%T@ Modification time, epoch seconds (for sorting)

%t Modification time, human readable

%p Full file path

Season to taste. Run on one mounted filesystem at a time or as many as you think is relevant.

What this does is give you a timestamp sorted audit of the most recent modifications to files. This can occasionally give you insight into other things that were happening at the time of the unknown event. It's certainly not bulletproof, and may not reveal anything at all, but I occasionally find this extremely useful in identifying what work was being performed at the time that the unknown event happened. (say, a someone performing a rm -Rf on /lib instead of Tomcat's lib...)

Andrew B
  • 31,858
  • 12
  • 90
  • 128