17

I just got a network alert that I've never seen before, on one of the few Ubuntu boxes that we have:

The following monitoring trigger has been fired:

/vmlinuz has been changed on server XXXXX: PROBLEM
2012.09.19 06:24:33
Trigger key: vfs.file.cksum[/vmlinuz]
Value: 3397367448
Host: XXXXX

The checksum of vmlinuz changed. I see from Wikipedia that this has something to do with the kernel.

Should I care that its checksum has changed? This particular server does run Wordpress which is known for vulnerabilities in its 3rd party plugins, so I tend to take alerts from it pretty seriously.

I'm making the conclusion that this server has been compromised. Better safe than sorry, as /var/log/apache2/access.log is 0 bytes, and there should be a bit (not much, but a bit) of data in there, and it clearly looks like something (a bot most likely) covering their tracks. Time to pull out last nights backup :)

Mark Henderson
  • 68,316
  • 31
  • 175
  • 255
  • On Ubuntu systems `/vmlinuz` should be a symbolic to a kernel under `/boot/vmlinux-?.?.?-???`, unless this is some kind of hosted VM. – Zoredache Sep 18 '12 at 22:06
  • @Zoredache - yes, `lrwxrwxrwx 1 root root 34 Sep 18 19:52 /vmlinuz -> boot/vmlinuz-2.6.32-43-generic-pae` – Mark Henderson Sep 18 '12 at 22:07

4 Answers4

13

This is the compressed kernel and you should care if it ever changed without you knowing about it, because if the kernel was replaced, you could be open to any attack. It may have been a legitimate reason, but unless you are sure, you should not trust the changed kernel.

johnshen64
  • 5,747
  • 23
  • 17
8

I see from Wikipedia that this has something to do with the kernel

That is an understatement: The vmlinuz file is the kernel itself. It is this file which gets loaded when you boot your server, then it gets uncompressed (hence the 'z'), and then started.

If you recompiled or installed a new kernel then there is nothing to worry about. If you did no such thing then look closely at this file, or replace it with a know good version.

Making this file read-only with chattr and disallowing root to change this until after a reboot is also a good idea.

Hennes
  • 4,772
  • 1
  • 18
  • 29
5

It is not something that has to do with your kernel, it is your kernel. If you reboot, and that file is corrupt, proverbial shit is going to hit the proverbial fan.

Did you have a kernel update at the time mentioned in the message?

wzzrd
  • 10,269
  • 2
  • 32
  • 47
  • Ok queue next dumbass question (I deal with 99% Windows machines), how can I check for a kernel update? This server is almost never logged in to, so I very much doubt anything was manually triggered. – Mark Henderson Sep 18 '12 at 22:09
  • check if someone logged in yesterday to upgrade the kernel : last -i and history (look for apt-get/aptitude update and upgrade). Check if some automatic updates are on (iirc ubuntu has some https://help.ubuntu.com/community/AutomaticSecurityUpdates). –  Sep 18 '12 at 22:18
  • @MarkHenderson Check the access, modify and change dates with ''stat /vmlinuz''. You should probably be able to see updates in ''/var/log/dpkg.log''. However, if the machine is not configured for automatic updates, that should show very little. – wzzrd Sep 19 '12 at 08:01
  • Check the cron jobs too, some package managers will automatically do updates via cron. –  Aug 19 '13 at 14:42
3

That is the compressed (hence the "z") kernel image. It should have not changed short of you performing a kernel upgrade.

I'd guess that you are wise in your suspicion that this may be due to a vulnerability, but as you know, it could also be due to underlying disk or fs issues, in which case you should be seeing other file system error logs. Either way, it's something to check into.

EEAA
  • 108,414
  • 18
  • 172
  • 242