8

So, we host a geoservice webserver thing at the office.

Someone apparently broke into this box (probably via ftp or ssh), and put some kind of irc-managed rootkit thing.

Now I'm trying to clean the whole thing up, I found the process pid who tries to connect via irc, but i can't figure out who's the invoking process (already looked with ps, pstree, lsof) The process is a perl script owned by www user, but ps aux |grep displays a fake file path on the last column.

Is there another way to trace that pid and catch the invoker?

Forgot to mention: the kernel is 2.6.23, which is exploitable to become root, but I can't touch this machine too much, so I can't upgrade the kernel

EDIT: lsof might help:

lsof -p 9481
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAMEss
perl 9481 www cwd DIR 8,2 608 2 /ss
perl 9481 www rtd DIR 8,2 608 2 /ss
perl 9481 www txt REG 8,2 1168928 38385 /usr/bin/perl5.8.8ss
perl 9481 www mem REG 8,2 135348 23286 /lib64/ld-2.5.soss
perl 9481 www mem REG 8,2 103711 23295 /lib64/libnsl-2.5.soss
perl 9481 www mem REG 8,2 19112 23292 /lib64/libdl-2.5.soss
perl 9481 www mem REG 8,2 586243 23293 /lib64/libm-2.5.soss
perl 9481 www mem REG 8,2 27041 23291 /lib64/libcrypt-2.5.soss
perl 9481 www mem REG 8,2 14262 23307 /lib64/libutil-2.5.soss
perl 9481 www mem REG 8,2 128642 23303 /lib64/libpthread-2.5.soss
perl 9481 www mem REG 8,2 1602809 23289 /lib64/libc-2.5.soss
perl 9481 www mem REG 8,2 19256 38662 /usr/lib64/perl5/5.8.8/x86_64-linux-threa d-multi/auto/IO/IO.soss
perl 9481 www mem REG 8,2 21328 38877 /usr/lib64/perl5/5.8.8/x86_64-linux-threa d-multi/auto/Socket/Socket.soss
perl 9481 www mem REG 8,2 52512 23298 /lib64/libnss_files-2.5.soss
perl 9481 www 0r FIFO 0,5 1068892 pipess
perl 9481 www 1w FIFO 0,5 1071920 pipess
perl 9481 www 2w FIFO 0,5 1068894 pipess
perl 9481 www 3u IPv4 130646198 TCP 192.168.90.7:60321->www.****.net:ircd (SYN_SENT)

paul.ago
  • 201
  • 2
  • 7
  • 2
    Unless you upgrade the kernel, what's to stop the hacker repeating the hack as soon as you remove the rootkit? There may well be a trojan kernel module that hides processes. – pjc50 Nov 12 '10 at 11:06
  • this looks very similar to the irc ddos bot i just cleaned off my vps: http://serverfault.com/questions/639699/how-can-i-find-out-more-about-this-perl-process – Hayden Thring Oct 26 '14 at 12:01

5 Answers5

38

If I can give you any advice, it is to stop wasting your time cleaning up. Make an image of the OS for forensic stuff for later, and just reinstall the server.

Sorry, but its the only secure way to resolving yourself from being rootkitted.

Later you can check the image, for certain reasons, why it happened.

From my own personal experience, I did this, and later found an internal user which had a SSH key containing the flaw of openssl in 2008.

I hope, it clears up the things.

Note:
If you are going to image/backup the server before reinstalling, be very careful, how you do this. As @dfranke said, boot from a trusted medium to backup.

You shouldn't connect to other machines from a rooted server, as great rootkits are known to be able to spread through trusted sessions such as SSH.

peterh
  • 4,914
  • 13
  • 29
  • 44
Arenstar
  • 3,592
  • 2
  • 24
  • 34
  • 11
    I strongly second this advice. You've found one rootkit, but you have absolutely no idea what else the attacker has tampered with. Once rooted, always rooted. Boot from a trusted medium, zero out the drive. Fdisk, format, re-install, doo dah, doo dah. – dfranke Nov 11 '10 at 17:06
  • Yah... Good root kits run under your kernel.. No chance of deleting them, with out ALOT of effort – Arenstar Nov 11 '10 at 17:47
  • 4
    +1 For any production system (really, any system) there is no reason to clean. Kill it with fire and rebuild. – phoebus Nov 11 '10 at 18:03
  • I'd +100 if I could. A re-install is the only way I could sleep at night if I was in your situation. – egorgry Nov 11 '10 at 18:19
  • Absolutely do this. Once a hacker has installed a rootkit it simply isn't your computer any more, end of discussion... – Rob Moir Nov 11 '10 at 22:41
  • 1
    +1 to reinstall. The rootkit probably messed with your binaries or kernel and everything you are seeing (fake paths, etc...) is probably smoke courtain from the rootkit to hide itself. – coredump Nov 18 '10 at 00:25
  • 1
    [Oblig. quote](http://www.imdb.com/title/tt0090605/quotes): "I say we take off and nuke the entire site from orbit. It's the only way to be sure." – Charles Stewart Nov 19 '10 at 11:16
  • http://nukeitfromorbit.com/ – Martijn Heemels Nov 22 '10 at 00:39
1

The commandline can be changed if the process alters argv[0]. Try ls -l /proc/[pid]/exe

From man 5 proc

this file is a symbolic link containing the actual pathname of the executed command. This symbolic link can be dereferenced normally; attempting to open it will open the executable. You can even type /proc/[number]/exe to run another copy of the same executable as is being run by process [number]. In a multithreaded process, the contents of this symbolic link are not available if the main thread has already terminated

ps auxwf | less gives you the "forest view" of processes which can show you what process launched this process (unless the rootkit is hiding it, or the app's parent has exited and it's been reparented to init).

This would be mostly academic and probably just a timewaster, but strings -n 10 /proc/[pid]/mem might be fun to watch scroll past. You could also echo 0x7 > /proc/[pid]/coredump_filter and use gdb gcore to force a coredump with everything possible in it, but then the process dies, which could block further analysis.

But definitely take Arenstar's advice. Back up data only, restore everything executable from backups, and start over. You should probably restore the website from backups as well, there could be malicious javascript added to every html or php file. If you're considering legal action, you'll want to just set the machine aside, unplug it from the net, and stop whatever it is you're doing until forensic experts have done their job.

DerfK
  • 19,313
  • 2
  • 35
  • 51
  • Really great answer. – paul.ago Nov 12 '10 at 09:21
  • unfortunately ls -l /proc/[pid]/exe returns the perl5.8.8 bin path, and ps/pstree says the process father is init, this thing looks really well hidden. I would definitely start over with a fresh install, but the original developer of the application running on it is out of the country for a while, so I was looking for a temporary solution. Really great answer by the way – paul.ago Nov 12 '10 at 10:09
0

Try "cat /proc/[process id]/cmdline" Though if it is a true rootkit it may modify the kernel to hide itself better.

mfarver
  • 2,576
  • 13
  • 16
  • it gives me the same of ps aux "/usr/sbin/apache/logs" which is fake. Neither the /usr/sbin/apache directory exists – paul.ago Nov 11 '10 at 16:39
0

You should reinstall, I agree. Have you tried escaping the characters in the path? Maybe one of those slashes is actually part of a filename and not a directory. At the very least you should use iptables to block outbound traffic to that host or IRC general until fixed. Check netstat as well.

sinping
  • 2,055
  • 14
  • 12
  • nope, the path seems "real". Sadly iptables seems installed but the kernel module is missing, so I must recompile the kernel. I would probably go that way to fix the thing for a few days until I get in touch with the guy who has the app source code, so I can reinstall the server – paul.ago Nov 12 '10 at 10:20
0

I'd think by now you have reinstalled. Your justing wasting time trying to track the processes and do forensics as chances of anything legally developing from it would be very minute and chances of finding the hacker would be futile anyway. Unless it just interests you to research and reverse rootkits which can be fun:)

null
  • 1