0

Possible Duplicate:
How do I know if my Linux server has been hacked?
My server's been hacked EMERGENCY

I am running CentOS 5.3 and here is the result of "chkrootkit": 

Possible t0rn v8 \(or variation\) rootkit installed

Warning: Possible Showtee Rootkit installed
 /usr/include/file.h /usr/include/proc.h
Warning: `//root/.mysql_history' file size is zero
INFECTED (PORTS:  465)
You have    61 process hidden for readdir command
You have    62 process hidden for ps command
chkproc: Warning: Possible LKM Trojan installed
 The tty of the following user process(es) were not found
 in /var/run/utmp !
! RUID          PID TTY    CMD
! root         3040 tty2   /sbin/mingetty tty2
! root         3041 tty3   /sbin/mingetty tty3
! root         3042 tty4   /sbin/mingetty tty4
! root         3043 tty5   /sbin/mingetty tty5
! root         3046 tty6   /sbin/mingetty tty6

I don't understand what the warnings mean.

Is the server infected or in danger?

Edit:

Let me add that I first got strange message on the command line:

Unknown HZ value! (##) Assume 100

Then I followed this great instructions and replaced my hacked files with new ones. I replaced: 

/sbin/ifconfig
/bin/netstat
/usr/bin/pstree
/usr/bin/top

They were all repored as infected by the "chkrootkit".

Now I re-ran "chkrootkit" and got the above output. How to proceed to get rid off all the warnings?

Edit 2:

After checking rpm integrity with: rpm -vVa | grep 'S\.5\.\.\.\.\T' > rpmverify.txt this is what I got:

S.5....T  c /etc/mail/spamassassin/local.cf
S.5....T  c /etc/pam.d/system-auth
S.5....T  c /etc/sudoers
S.5....T  c /etc/samba/smb.conf
S.5....T    /opt/drweb/lib/drweb32.dll
S.5....T    /var/drweb/bases/drw50000.vdb
S.5....T    /var/drweb/bases/drw50001.vdb
S.5....T    /var/drweb/bases/drw50002.vdb
S.5....T    /var/drweb/bases/drw50003.vdb
S.5....T    /var/drweb/bases/drw50004.vdb
S.5....T    /var/drweb/bases/drw50005.vdb
S.5....T    /var/drweb/bases/drw50006.vdb
S.5....T    /var/drweb/bases/drw50007.vdb
S.5....T    /var/drweb/bases/drw50008.vdb
S.5....T    /var/drweb/bases/drw50009.vdb
S.5....T    /var/drweb/bases/drw50010.vdb
S.5....T    /var/drweb/bases/drw50011.vdb
S.5....T    /var/drweb/bases/drw50012.vdb
S.5....T    /var/drweb/bases/drw50013.vdb
S.5....T    /var/drweb/bases/drw50014.vdb
S.5....T    /var/drweb/bases/drw50015.vdb
S.5....T    /var/drweb/bases/drw50016.vdb
S.5....T    /var/drweb/bases/drw50017.vdb
S.5....T    /var/drweb/bases/drw50018.vdb
S.5....T    /var/drweb/bases/drw50019.vdb
S.5....T    /var/drweb/bases/drw50020.vdb
S.5....T    /var/drweb/bases/drw50021.vdb
S.5....T    /var/drweb/bases/drw50022.vdb
S.5....T    /var/drweb/bases/drw50023.vdb
S.5....T    /var/drweb/bases/drw50024.vdb
S.5....T    /var/drweb/bases/drw50025.vdb
S.5....T    /var/drweb/bases/drw50026.vdb
S.5....T    /var/drweb/bases/drw50027.vdb
S.5....T    /var/drweb/bases/drw50028.vdb
S.5....T    /var/drweb/bases/drw50029.vdb
S.5....T    /var/drweb/bases/drwebase.vdb
S.5....T    /var/drweb/bases/drwnasty.vdb
S.5....T    /var/drweb/bases/drwrisky.vdb
S.5....T    /var/drweb/bases/drwtoday.vdb
S.5....T    /var/drweb/bases/dwn50001.vdb
S.5....T    /var/drweb/bases/dwn50002.vdb
S.5....T    /var/drweb/bases/dwntoday.vdb
S.5....T    /var/drweb/bases/dwr50001.vdb
S.5....T    /var/drweb/bases/dwrtoday.vdb
S.5....T    /bin/basename
S.5....T    /bin/cat
S.5....T    /bin/chgrp
S.5....T    /bin/chmod
S.5....T    /bin/chown
S.5....T    /bin/cp
S.5....T    /bin/cut
S.5....T    /bin/dd
S.5....T    /bin/df
S.5....T    /bin/env
S.5....T    /bin/false
S.5....T    /bin/link
S.5....T    /bin/ln
S.5....T  c /etc/proftpd.conf
S.5....T  c /root/.bash_profile
S.5....T  c /etc/httpd/conf.d/mailman.conf
S.5....T    /usr/lib/mailman/Mailman/mm_cfg.pyc
S.5....T  c /etc/drweb/drweb32.ini
S.5....T    /opt/drweb/ldwrap.sh
S.5....T  c /etc/drweb/users.conf
S.5....T    /usr/share/psa-horde/imp/compose.php
S.5....T    /usr/share/psa-horde/imp/contacts.php
S.5....T    /usr/local/psa/admin/plib/api-common/cuMail.php
S.5....T    /usr/local/psa/admin/sbin/autoinstaller
S.5....T    /usr/local/psa/admin/htdocs/modules/watchdog/stats-graph.php
S.5....T    /usr/local/psa/etc/modules/watchdog/monitrc
S.5....T    /usr/local/psa/etc/modules/watchdog/wdcollect.inc.php
S.5....T    /usr/local/psa/var/modules/watchdog/lib/rkhunter/lib/rkhunter      /db/backdoorports.dat
S.5....T    /usr/local/psa/var/modules/watchdog/lib/rkhunter/lib/rkhunter/db/mirrors.dat
S.5....T    /usr/local/psa/var/modules/watchdog/lib/rkhunter/lib/rkhunter/db/programs_bad.dat
S.5....T    /usr/local/psa/var/modules/watchdog/lib/rkhunter/lib/rkhunter/db/suspscan.dat
S.5....T  c /etc/courier-imap/imapd.cnf
S.5....T  c /etc/php.ini
S.5....T  c /etc/ssh/sshd_config
S.5....T  c /etc/syslog.conf
S.5....T  c /etc/sysconfig/named
S.5....T  c /etc/httpd/conf.d/ssl.conf
S.5....T  c /etc/smartd.conf
S.5....T  c /etc/vsftpd/vsftpd.conf
S.5....T    /usr/share/psa-horde/util/icon_browser.php
S.5....T  c /etc/init.d/psa
S.5....T    /usr/lib/plesk-9.0/key-handler
S.5....T    /usr/local/psa/admin/htdocs/domains/databases/phpMyAdmin/librari/config.default.php
S.5....T    /usr/local/psa/admin/plib/class.ComponentsChecker.php
S.5....T    /usr/local/psa/admin/plib/class.ComponentsShow.php
S.5....T    /usr/local/psa/admin/plib/class.RestartServForm.php
S.5....T    /usr/local/psa/admin/plib/class.ServiceControl.php
S.5....T    /usr/local/psa/admin/sbin/packagemng
S.5....T    /usr/local/psa/admin/plib/backup/BackupCreateBackupNowForm.php
S.5....T  c /etc/samba/smbusers
S.5....T  c /etc/pam.d/ekshell
S.5....T  c /etc/pam.d/kshell
S.5....T  c /etc/printcap
S.5....T  c /etc/my.cnf
S.5....T    /usr/bin/spf_example_static
S.5....T    /usr/bin/spfd_static
S.5....T    /usr/bin/spfquery_static
S.5....T    /usr/bin/spftest_static
S.5....T    /usr/lib/libspf2.so.2.1.0
S.5....T  c /etc/awstats/awstats.model.conf
S.5....T    /usr/local/sso/base/Cookie.php
S.5....T  c /etc/httpd/conf/httpd.conf
S.5....T    /usr/sbin/suexec

Does that help?

Edit 3:

Here is rpm check result after core utils have been reinstalled:

S.5....T  c /etc/mail/spamassassin/local.cf
S.5....T  c /etc/pam.d/system-auth
S.5....T  c /etc/sudoers
S.5....T  c /etc/samba/smb.conf
S.5....T    /opt/drweb/lib/drweb32.dll
S.5....T    /var/drweb/bases/drw50000.vdb
S.5....T    /var/drweb/bases/drw50001.vdb
S.5....T    /var/drweb/bases/drw50002.vdb
S.5....T    /var/drweb/bases/drw50003.vdb
S.5....T    /var/drweb/bases/drw50004.vdb
S.5....T    /var/drweb/bases/drw50005.vdb
S.5....T    /var/drweb/bases/drw50006.vdb
S.5....T    /var/drweb/bases/drw50007.vdb
S.5....T    /var/drweb/bases/drw50008.vdb
S.5....T    /var/drweb/bases/drw50009.vdb
S.5....T    /var/drweb/bases/drw50010.vdb
S.5....T    /var/drweb/bases/drw50011.vdb
S.5....T    /var/drweb/bases/drw50012.vdb
S.5....T    /var/drweb/bases/drw50013.vdb
S.5....T    /var/drweb/bases/drw50014.vdb
S.5....T    /var/drweb/bases/drw50015.vdb
S.5....T    /var/drweb/bases/drw50016.vdb
S.5....T    /var/drweb/bases/drw50017.vdb
S.5....T    /var/drweb/bases/drw50018.vdb
S.5....T    /var/drweb/bases/drw50019.vdb
S.5....T    /var/drweb/bases/drw50020.vdb
S.5....T    /var/drweb/bases/drw50021.vdb
S.5....T    /var/drweb/bases/drw50022.vdb
S.5....T    /var/drweb/bases/drw50023.vdb
S.5....T    /var/drweb/bases/drw50024.vdb
S.5....T    /var/drweb/bases/drw50025.vdb
S.5....T    /var/drweb/bases/drw50026.vdb
S.5....T    /var/drweb/bases/drw50027.vdb
S.5....T    /var/drweb/bases/drw50028.vdb
S.5....T    /var/drweb/bases/drw50029.vdb
S.5....T    /var/drweb/bases/drwebase.vdb
S.5....T    /var/drweb/bases/drwnasty.vdb
S.5....T    /var/drweb/bases/drwrisky.vdb
S.5....T    /var/drweb/bases/drwtoday.vdb
S.5....T    /var/drweb/bases/dwn50001.vdb
S.5....T    /var/drweb/bases/dwn50002.vdb
S.5....T    /var/drweb/bases/dwntoday.vdb
S.5....T    /var/drweb/bases/dwr50001.vdb
S.5....T    /var/drweb/bases/dwrtoday.vdb
S.5....T  c /etc/proftpd.conf
S.5....T  c /etc/profile.d/colorls.csh
S.5....T  c /etc/profile.d/colorls.sh
S.5....T  c /root/.bash_profile
S.5....T  c /etc/httpd/conf.d/mailman.conf
S.5....T    /usr/lib/mailman/Mailman/mm_cfg.pyc
S.5....T  c /etc/drweb/drweb32.ini
S.5....T    /opt/drweb/ldwrap.sh
S.5....T  c /etc/drweb/users.conf
S.5....T    /usr/share/psa-horde/imp/compose.php
S.5....T    /usr/share/psa-horde/imp/contacts.php
S.5....T    /usr/local/psa/admin/plib/api-common/cuMail.php
S.5....T    /usr/local/psa/admin/sbin/autoinstaller
S.5....T    /usr/local/psa/admin/htdocs/modules/watchdog/stats-graph.php
S.5....T    /usr/local/psa/etc/modules/watchdog/monitrc
S.5....T    /usr/local/psa/etc/modules/watchdog/wdcollect.inc.php
S.5....T    /usr/local/psa/var/modules/watchdog/lib/rkhunter/lib/rkhunter /db/backdoorports.dat
S.5....T    /usr/local/psa/var/modules/watchdog/lib/rkhunter/lib/rkhunter/db/mirrors.dat
S.5....T    /usr/local/psa/var/modules/watchdog/lib/rkhunter/lib/rkhunter/db/programs_bad.dat
S.5....T    /usr/local/psa/var/modules/watchdog/lib/rkhunter/lib/rkhunter/db/suspscan.dat
S.5....T  c /etc/courier-imap/imapd.cnf
S.5....T  c /etc/php.ini
S.5....T  c /etc/ssh/sshd_config
S.5....T  c /etc/syslog.conf
S.5....T  c /etc/sysconfig/named
S.5....T  c /etc/httpd/conf.d/ssl.conf
S.5....T  c /etc/smartd.conf
S.5....T  c /etc/vsftpd/vsftpd.conf
S.5....T    /usr/share/psa-horde/util/icon_browser.php
S.5....T  c /etc/init.d/psa
S.5....T    /usr/lib/plesk-9.0/key-handler
S.5....T    /usr/local/psa/admin/htdocs/domains/databases/phpMyAdmin/libraries/config.default.php
S.5....T    /usr/local/psa/admin/plib/class.ComponentsChecker.php
S.5....T    /usr/local/psa/admin/plib/class.ComponentsShow.php
S.5....T    /usr/local/psa/admin/plib/class.RestartServForm.php
S.5....T    /usr/local/psa/admin/plib/class.ServiceControl.php
S.5....T    /usr/local/psa/admin/sbin/packagemng
S.5....T    /usr/local/psa/admin/plib/backup/BackupCreateBackupNowForm.php
S.5....T  c /etc/samba/smbusers
S.5....T  c /etc/pam.d/ekshell
S.5....T  c /etc/pam.d/kshell
S.5....T  c /etc/printcap
S.5....T  c /etc/my.cnf
S.5....T    /usr/bin/spf_example_static
S.5....T    /usr/bin/spfd_static
S.5....T    /usr/bin/spfquery_static
S.5....T    /usr/bin/spftest_static
S.5....T    /usr/lib/libspf2.so.2.1.0
S.5....T  c /etc/awstats/awstats.model.conf
S.5....T    /usr/local/sso/base/Cookie.php
S.5....T  c /etc/httpd/conf/httpd.conf
S.5....T    /usr/sbin/suexec
Community
  • 1
Danijel
  • 256
  • 5
  • 18
  • 4
    That means it's time to format the hard disk(s) and reinstall an OS. – HopelessN00b Oct 04 '12 at 11:18
  • @HopelessN00b "The only way to be sure!™" – Aaron Copley Oct 04 '12 at 11:22
  • @AaronCopley along the same lines as... `Which is why it's better to put too many bullets in the head than too few.` Overkill >> underkill. – HopelessN00b Oct 04 '12 at 11:24
  • Anything more constructive guys? I'm in panic. – Danijel Oct 04 '12 at 11:37
  • 2
    The moment there's a rootkit, there's no way to find out what *actually* is tained, and what isn't - thats the whole point of a rootkit. Telling you to format, reinstall and restore from backups *is* constructive. You simply cannot trust that install any more. – Journeyman Geek Oct 04 '12 at 11:42
  • @Danijel Implement your disaster recovery plan and reinstall the OS. Restore your data from known good backup. (That's as constructive as it gets.) Unless you're in a position where you need to perform forensics, there simply isn't any point to spending any more time thinking about what you should do. – Aaron Copley Oct 04 '12 at 11:50
  • I do have a "rsnapshot" from few days ago. Should that do? – Danijel Oct 04 '12 at 11:59
  • 1
    I wouldn't know.. You test your backups lately? If not, it'd be a good idea to do the rebuild on spare hardware. Also, when you rebuild you'll need to make sure you don't reintroduce the same vulnerability. At some level you'll need to know how it got compromised so you can learn from it. Unpatched binaries? Weak password? SQL injection? XSS? What did the server do? You might need to call in some professional help on this if you don't feel comfortable on your own. – Aaron Copley Oct 04 '12 at 12:14
  • @Danijel You should reinstall the OS, restore data from your backups, and patch the security hole that allowed your server to get hacked in the first place. – Chris S Oct 04 '12 at 13:51

1 Answers1

3

It's a CentOS system. I typically repair these rootkits, but the chances you'll detect/get everything, having not done this before, is slim...

You could start with an RPM verification...

Run rpm -vVa | grep 'S\.5\.\.\.\.\T' > rpmverify.txt

Then examine the output in rpmverify.txt. That would allow you to check to see which binaries and config files to not match the checksums from the RPM database. It's the first place I start on fixing these systems (after making sure no unauthorized network daemons/services are running).

Edit:

I see the output of your RPM verify command. If your yum still works, run yum install yum-utils in order to gain access to the yumdownloader command.

Based on your output, your coreutils and possibly the httpd package has been compromised (cat, df, dd, chown, cp, etc.). Run yumdownloader coreutils to obtain the rpm. It'll download into your current directory. I'd force the reinstallation of the RPM (rpm -ivh --force coreutils*) and re-run the verify I suggested above.

Update:

Hackers/rootkits will often replace binaries with Trojaned versions and set the immutable flag on the file to prevent them from being removed.

Please take a look at the attributes on the /bin/ls binary by running lsattr /bin/ls.

You will possibly see an "a", "u", "i" and "s" in the output. Running chattr -uisa on the same file should remove the immutable flag and allow you to run the rpm installation.

The attributes should look like:

[root@kitteh ~]# lsattr /bin/ls
------------- /bin/ls

Repeat for any other files that fail in the RPM installation. You may need to also change/remove those attributes on the enclosing directory...

ewwhite
  • 194,921
  • 91
  • 434
  • 799
  • If you're going to do this, you should do it from a rescue environment (otherwise you can't trust the results.) – Aaron Copley Oct 04 '12 at 12:03
  • This is true, although rootkits rarely target RPM itself (even though they should). It's a good quick way to get an assessment. – ewwhite Oct 04 '12 at 12:13
  • I ran the command, please see Edit 2 added above. – Danijel Oct 04 '12 at 12:16
  • @Danijel See my edit. – ewwhite Oct 04 '12 at 12:26
  • OK, tried and... That `rpm` command failed: `error: unpacking of archive failed on file /bin/ls: cpio: rename failed - Operation not permitted`. Any idea? – Danijel Oct 04 '12 at 13:24
  • Also, `lsattr /bin/ls` gives `s---ia------- /bin/ls`. The attributes need to be fixed? – Danijel Oct 04 '12 at 13:32
  • @Danijel See my edit. – ewwhite Oct 04 '12 at 13:34
  • Total 7 files had attributes like this: `s---ia------`. Those are: /bin/ps, /bin/ls, /usr/bin/dir, /usr/bin/find, /usr/bin/md5sum, /sbin/ttyload, and /sbin/ttymon. Does that tell anything? I have fixed this 7. Which other folders should I check for attributes? – Danijel Oct 04 '12 at 13:43
  • 2
    @Danijel This is really beyond the scope of what I can provide in a Server Fault answer. I was just trying to help point you in the right direction. – ewwhite Oct 04 '12 at 13:47
  • OK, sure I understand. – Danijel Oct 04 '12 at 13:58
  • Btw, I find this thread very usefull for future people who get into same kind of trouble. That's why I posted all those commands and tried to give as much detail as possible. Flaging this thread as duplicate with those two - really a nonsense. This one is by far more concrete. – Danijel Oct 04 '12 at 15:05
  • 1
    @Danijel well, it still can be helpful as a closed question, and by linking it to two other similar questions, I'd argue it's made more useful, because it now contains easily accessible linkage to two similar questions, even if they are more generic. A closed question isn't necessarily a bad thing. I have one myself, and am actually rather proud of it, and thinks it makes a contribution as well. – HopelessN00b Oct 04 '12 at 15:13
  • Yes, but this problem is only half way resolved - but closed. – Danijel Oct 04 '12 at 20:35