0

It has come to my attention that my CentOS 5.6 5.11 VPS has over 16,000 files with a timestamp during a 3-day period in September this year. I can't figure out why that is, as I did not log in via SSH during that time (nor were there any other logins, which would have been very worrisome) and there are no records of actions in Webmin during that period. Those are the only two ways I would have done anything to change system files. The server appears to run fine, but no one else is supposed to have access (other than the hoster, of course), so I want to know what's going on.

I captured the files with those dates in a file thus:

touch -t 201409160000 start.tmp
touch -t 201409190000 stop.tmp
find / -type f -newer start.tmp \! -newer stop.tmp -exec ls -la {} \; > sep-16-18.txt

But what do I do with this information? The volume is crazy!

[root](21:05:55)[~]$ grep "Sep 16" -c sep-16-18.txt
6079
[root](21:06:30)[~]$ grep "Sep 17" -c sep-16-18.txt
2580
[root](21:06:40)[~]$ grep "Sep 18" -c sep-16-18.txt
7653

The whole rest of the month of September was less than 500 files total, so something radical happened in the middle of the month.

Almost all the files were in the /usr/ directory:

/usr/:  16130
/lib/:     49
/sbin/:    49
/etc/:     45
/lib64/:   37
All others: 3

But the breakdown of subdirectories under /usr/ is spread across the board pretty widely:

/usr/lib/:     6514
/usr/include/: 5109
/usr/share/:   3438
/usr/lib64/:    853
/usr/bin/:      105
/usr/sbin/:      61
/usr/libexec/:   36
/usr/kerberos/:  14

Perhaps the file with the earliest timestamp might shed some light (perhaps it started the cascade), but I can't seem to sort the find results by time. I could repeatedly reset start.tmp and stop.tmp to smaller and smaller time periods, re-running find until I got a reasonable number of results to look through manually, but I would probably have no idea what I'm looking at - I'm really not a Linux expert anyway. Perhaps someone can give me some tips on what "questions" I should be asking my server. It looks as if Linux was reinstalled or something, but...

UPDATE: For those who are asking, here are the first several files with timestamps on the 16th. The first one seems unrelated, but then starting at 15:04, for the next couple minutes over 5000 files were updated (or created, I suppose):

1410846366 Tue Sep 16 14:46:06 2014 /etc/default/nss
1410847466 Tue Sep 16 15:04:26 2014 /usr/include/features.h
1410847466 Tue Sep 16 15:04:26 2014 /usr/include/gnu-versions.h
1410847466 Tue Sep 16 15:04:26 2014 /usr/include/limits.h
1410847466 Tue Sep 16 15:04:26 2014 /usr/include/values.h
1410847466 Tue Sep 16 15:04:26 2014 /usr/lib64/libc.so
1410847467 Tue Sep 16 15:04:27 2014 /usr/include/bits/xopen_lim.h
1410847467 Tue Sep 16 15:04:27 2014 /usr/include/gnu/libc-version.h
1410847467 Tue Sep 16 15:04:27 2014 /usr/include/gnu/lib-names.h
1410847467 Tue Sep 16 15:04:27 2014 /usr/include/gnu/stubs.h
1410847468 Tue Sep 16 15:04:28 2014 /usr/include/gconv.h
1410847468 Tue Sep 16 15:04:28 2014 /usr/include/iconv.h
1410847473 Tue Sep 16 15:04:33 2014 /usr/lib64/gconv/gconv-modules
1410847475 Tue Sep 16 15:04:35 2014 /usr/include/bits/locale.h
1410847475 Tue Sep 16 15:04:35 2014 /usr/include/langinfo.h
1410847475 Tue Sep 16 15:04:35 2014 /usr/include/locale.h
1410847475 Tue Sep 16 15:04:35 2014 /usr/include/xlocale.h
1410847476 Tue Sep 16 15:04:36 2014 /usr/share/i18n/charmaps/ANSI_X3.110-1983.gz
1410847476 Tue Sep 16 15:04:36 2014 /usr/share/i18n/charmaps/ANSI_X3.4-1968.gz
1410847476 Tue Sep 16 15:04:36 2014 /usr/share/i18n/charmaps/ARMSCII-8.gz
1410847476 Tue Sep 16 15:04:36 2014 /usr/share/i18n/charmaps/ASMO_449.gz

After that, it goes on with character maps and locale-related files at some length, then more include files, then after about 5500 of that type of file it takes a break for a little while before more .so files and other things that look pretty normal. Does this tell anyone anything? None of I have seen in the list look suspicious; I'm no expert, but it looks more like an innocent update. But I don't know how how an update would happen besides a shell login or Webmin.

ewwhite
  • 194,921
  • 91
  • 434
  • 799
OsakaWebbie
  • 183
  • 1
  • 11
  • Um, what files changed? Wouldn't some example file names help? – ewwhite Nov 14 '14 at 13:48
  • 2
    Could it have been a software update? is there anything in the syslog that corresponds to that time? Are there any entries in any other logs which give you any more info? – NickW Nov 14 '14 at 14:01
  • 1
    when was this VPS installed ? – Hrvoje Špoljar Nov 14 '14 at 14:03
  • No software updates in that period? `rpm -qa --last` – HBruijn Nov 14 '14 at 14:26
  • @ewwhite: I don't know where to begin listing filenames when there are thousands. The one I noticed first, causing me to do the investigation, was /etc/httpd/conf/magic - I was not aware of it before, so might be new. But it looks harmless - I don't remember installing mod_mime_magic, but it's possible. – OsakaWebbie Nov 15 '14 at 02:14
  • @Dennis Kaarsemaker: I don't know yet whether it has been compromised at all - that's what I'm trying to determine first. Other than file timestamps, I don't have any suspicions (no odd behavior that I have noticed), but as I said, I'm not very good at this stuff. – OsakaWebbie Nov 15 '14 at 02:17
  • @NickW: Yes, that is very possible, but OS's don't normally themselves. /var/log/yum.log doesn't have anything in the time period. There are 10 things named syslog: 6 are directories and the other 4 have older timestamps. Can you suggest a specific file to check? – OsakaWebbie Nov 15 '14 at 02:18
  • @Hrvoje: The VPS was installed in 2011, as CentOS 5.5. Later I upgraded it to 5.6 (I don't remember when, but not this year for sure). Most recently I did a yum update in response to learning about Shellshock, but that was apparently Oct. 1st (or maybe started late Sept. 30). – OsakaWebbie Nov 15 '14 at 02:21
  • @HBruijn: Nope, there is nothing in `rpm -qa --last` between Jan. 1st and Oct. 1st. – OsakaWebbie Nov 15 '14 at 02:22
  • Downvoted? I'm doing my best to answer everyone's questions and give as much information as I know how to give. I know that @ewwhite asked for specific files, but I can't list thousands of files. I'm currently trying to find the first files that changed, but it's a slow process. I'm not a serverfault veteran, so please bear with me. I'll edit my question when I have additional info. – OsakaWebbie Nov 15 '14 at 04:51
  • What does this server do? You mention “Webmin” but is it a web server? I actually just cleaned up a machine that was infected by the `bash` shellshock bug and Webmin’s port 10000 played a role in it. I would check the `/tmp` directory for anything odd. If there are files in there, look at the modification dates on those files and whether they have execution rights. – Giacomo1968 Nov 15 '14 at 06:52
  • 3
    The downvote is probably because this question borders on too broad. You have to focus on specific challenges on Serverfault, and those challenges should not be duplicates of existing questions. It might have fared better if you had kept the question focused on *how* you perform mtime forensics, which is the question I answer below. If you're looking for someone to tell you what exactly happened in these circumstances (very situation specific, too many possibilities), this question is probably going to continue on its way to being closed. – Andrew B Nov 15 '14 at 06:55
  • 2
    I was focussing hard on software updates, but I noticed the OP mentioned the box is C5.6 - which means it's wildly out of date, can't have been patched recently *and* is ripe for exploitation. But the OP also mentions a `yum update` in early Oct, which would explain those timestamps but not the OS version number. @OsakaWebbie, **please clarify whether the system really is C5.6**. The contents of `/etc/redhat-release` would be **really** helpful. – MadHatter Nov 15 '14 at 07:47
  • @JakeGould: It is indeed a webserver. Everything in /tmp is newer than Nov 5th, all look normal (PHP session files and things my code produces); none are executable. – OsakaWebbie Nov 15 '14 at 10:00
  • @MadHatter: Wow, you're right - it's currently 5.11! I did `yum update` periodically without really looking at the version numbers - I didn't know that major versions were that easy. Or perhaps they aren't - perhaps the hoster upgraded me in September, and then the Oct. 1st one that I did (in response to shellshock) was just a minor update. I submitted a support ticket to ask my hoster if they did something - I'll report back their answer. – OsakaWebbie Nov 15 '14 at 10:07
  • [Administration panels are off topic](http://serverfault.com/help/on-topic). [Even the presence of an administration panel on a system,](http://meta.serverfault.com/q/6538/118258) because they [take over the systems in strange and non-standard ways, making it difficult or even impossible for actual system administrators to manage the servers normally](http://meta.serverfault.com/a/3924/118258), and tend to indicate low-quality questions from *users* with insufficient knowledge for this site. – HopelessN00b Apr 09 '15 at 13:33
  • This is an old thread, so I should probably not even bother responding. But just to clear the air for those who might read it later: Webmin is not heavy-handed like Plesk or cPanel - I added it afterwards to a server that was installed with no control panel at all, and it does not change anything about the configuration, but only adds an additional interface for administrating it. – OsakaWebbie Apr 09 '15 at 19:24

2 Answers2

6

This question's on its way to getting closed as a duplicate of How do I deal with a compromised server, and I think that's a mistake, because the very correct advice in that question is to nuke the server and restore from known-good media. However, there's absolutely no evidence that this system has been compromised.

OsakaWebbie, you have admitted that you were wrong about the OS version; the system's at 5.11. That given, the files you list are all part of the glibc-headers package, with the exception of /usr/lib64/libc.so which is part of glibc-devel, a package that travels in lockstep with glibc-headers and is generally built at the same time.

I would have done this on a C5 system, but I don't have one to hand. On an up-to-date C6 system, the information about glibc-headers is

[me@lory 2012]$ rpm -qi glibc-headers
Name        : glibc-headers                Relocations: (not relocatable)
Version     : 2.12                              Vendor: CentOS
Release     : 1.149.el6                     Build Date: Wed 15 Oct 2014 03:00:58 BST
Install Date: Fri 24 Oct 2014 20:42:04 BST      Build Host: c6b9.bsys.dev.centos.org
[...]

Note the build and install dates. Now let's look at the files in question:

[me@lory 2012]$ for file in `rpm -ql glibc-headers`; do ls -la $file ; done
[...]
-rw-r--r--. 1 root root 2416 Oct 15 02:44 /usr/include/gnu-versions.h
-rw-r--r--. 1 root root 3057 Oct 15 02:44 /usr/include/gnu/lib-names.h
-rw-r--r--. 1 root root 1337 Oct 15 02:44 /usr/include/gnu/libc-version.h
-rw-r--r--. 1 root root 315 Oct 15 02:44 /usr/include/gnu/stubs.h
-rw-r--r--. 1 root root 6991 Oct 15 02:44 /usr/include/grp.h
-rw-r--r--. 1 root root 4611 Oct 15 02:44 /usr/include/gshadow.h
-rw-r--r--. 1 root root 1949 Oct 15 02:44 /usr/include/iconv.h
-rw-r--r--. 1 root root 4990 Oct 15 02:44 /usr/include/ieee754.h
[...]

Note how the files all have the same timestamp, and it's a few minutes before the Build Date baked into the RPM. From this we conclude that the modification time of an RPM-controlled system file is the modification time it had on the build system where the RPM was created; since a build happens as a monolithic process, it is completely reasonable that all the files in an RPM should have very similar mtimes, and indeed that those in associated packages should have them also, and that those mtimes will not correspond to the date on which the packages were installed.

If you can confirm that your system is indeed at C5.10/5.11, you can stop panicking; you have presented no evidence that anything bad has happened to your system.

Edit: OsakaWebbie, you having confirmed that the system is now at 5.11, I submit this is a non-issue. The modification times on the files you've shown are exactly as you'd expect them to be.

For reference - and it's quite important that you understand this - C5.11 isn't a major release of C5. C6 is a different major release to C5, but C5.11 is just a fully-patched C5.6 (or any other version of C5). You can read about this at some length in an earlier answer I wrote on C/RH versioning if you'd like, but in all probability you brought the system to 5.11 when you did that yum update in late Sep / early Oct. You can confirm that with grep centos-release /var/log/yum.log*, if you like.

Community
  • 1
MadHatter
  • 78,442
  • 20
  • 178
  • 229
  • Thank you for the explanation and reassurance. As you can see in a comment on the OP (I saw your comment there before I saw this answer), I did discover that I'm at 5.11. Sorry for that mistake! I wasn't actually panicking - I figured that it was more likely that there was a reasonable explanation that I was simply ignorant of. BTW, I'm not relying on my memory for dates - /var/log/yum.log has entries on 1/1 and then 10/1, so I know my `yum update` after learning about Shellshock was on 10/1, not in the middle of Sept. And `last` says there was no login between 9/10 and 9/19. – OsakaWebbie Nov 15 '14 at 10:28
  • So what that there was no login between those dates? **Please read what I wrote above** - the mtimes on those files are inherent to the RPM. That fact that you installed the RPM in early October makes no difference to the mtimes; they remain as the RPM sets them. – MadHatter Nov 15 '14 at 10:32
  • Oh, after your second comment, I think I understand what you meant (sorry I'm slow). The RPM files were dated in mid-Sept., packed in a .gz or whatever, and then unpacked onto the server when I updated. That makes total sense now. Thank you! I can't upvote anything because I don't have 15 reputation points yet, but it did let me select it as THE answer. Thanks again! – OsakaWebbie Nov 15 '14 at 10:36
1

I'm going to largely C&P an answer of mine from an unrelated question. This is a better home for it I think. The instructions below are provided with the assumption that you have already followed the instructions for How do I deal with a compromised server? if applicable. Always disconnect the network cable when in doubt.

find /filesystem1 /filesystem2 -xdev -printf '%T@ %t %p\n' | sort -n

%T@ Modification time, epoch seconds (for sorting)

%t Modification time, human readable

%p Full file path

Season to taste. Run on one mounted filesystem at a time or as many as you think is relevant.

What this does is give you a timestamp sorted audit of the most recent modifications to files. This can occasionally give you insight into other things that were happening at the time of the unknown event. It's certainly not bulletproof, and may not reveal anything at all, but I occasionally find this extremely useful in identifying what work was being performed at the time that the unknown event happened. (i.e. someone performing rm -Rf /lib instead of rm -Rf lib)

Asides:

  • If the output of this command contains a wall of common libraries and executables that cannot be associated with an OS update, the most likely cause is a rootkit being overlaid on top of your operating system.

  • Another possibility (that you should be less inclined to consider) is a clueless admin running a greedy rsync from another system to this one. It's unlikely to be a restore from an archival copy (backup software, tar, etc.) as those are generally good about honoring the original mtime.

  • If you can't figure it out, you have no choice but to treat this as an intrusion.

Community
  • 1
Andrew B
  • 31,858
  • 12
  • 90
  • 128
  • Thanks! I had tried about a dozen ways to sort a find result list by time, with no success. As for the "hacked server" advice from you and Dennis, I actually think it's unlikely that I'm hacked, and even if I am, I can't unplug, reformat, or most of the other first-responder things suggested in that other thread - it's not my server. I'm on a hosted VPS. Hmm, I wonder if the hoster did something to my stuff in September - I'll write to support and ask. But if they upgraded my stuff and didn't tell me, I should probably change hosters. – OsakaWebbie Nov 15 '14 at 08:03