-1

Possible Duplicate:
How do I deal with a compromised server?

This process is running on my Linux server and is sending spam (connecting to port 25 and port 80 with random IP's).

top - 12:28:42 up 108 days,  2:24,  1 user,  load average: 0.08, 0.22, 0.37
Tasks: 116 total,   2 running, 114 sleeping,   0 stopped,   0 zombie
Cpu(s): 12.3% us,  1.7% sy,  0.0% ni, 86.1% id,  0.0% wa,  0.0% hi,  0.0% si
Mem:    499540k total,   477096k used,    22444k free,     7916k buffers
Swap:  1015800k total,     1664k used,  1014136k free,   231476k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
27850 <userid>  15   0  9808 5616 1400 S  9.3  1.1  47:14.62 perl

Pressing c while top is running shows:

27850 <userid>  15   0  9808 5616 1400 S  9.3  1.1  47:31.52 ./mail

I know the userid that is running the process. It appears to be an automated process that starts periodically (but is not running all of the time).

What can I do to find out exactly from where it's being launched (what script file, etc.)? If I look in /proc with the process ID, it shows:

-r--r--r--    1 <userid> users 0 Jan  5 12:22 statm
-r--r--r--    1 <userid> users 0 Jan  5 12:22 stat
-r--r--r--    1 <userid> users 0 Jan  5 12:23 cmdline
-r--r--r--    1 <userid> users 0 Jan  5 12:31 wchan
dr-xr-xr-x    3 <userid> users 0 Jan  5 12:31 task
-r--r--r--    1 <userid> users 0 Jan  5 12:31 status
-r--------    1 <userid> users 0 Jan  5 12:31 smaps
lrwxrwxrwx    1 <userid> users 0 Jan  5 12:31 root -> /
-r--------    1 <userid> users 0 Jan  5 12:31 mountstats
-r--r--r--    1 <userid> users 0 Jan  5 12:31 mounts
-rw-------    1 <userid> users 0 Jan  5 12:31 mem
-r--r--r--    1 <userid> users 0 Jan  5 12:31 maps
-rw-r--r--    1 <userid> users 0 Jan  5 12:31 loginuid
-r--------    1 <userid> users 0 Jan  5 12:31 limits
-r--r--r--    1 <userid> users 0 Jan  5 12:31 io
dr-x------    2 <userid> users 0 Jan  5 12:31 fd
lrwxrwxrwx    1 <userid> users 0 Jan  5 12:31 exe -> /usr/bin/perl
-r--------    1 <userid> users 0 Jan  5 12:31 environ
lrwxrwxrwx    1 <userid> users 0 Jan  5 12:31 cwd -> /
-rw-r--r--    1 <userid> users 0 Jan  5 12:31 coredump_filter
-r--------    1 <userid> users 0 Jan  5 12:31 auxv
dr-xr-xr-x    2 <userid> users 0 Jan  5 12:31 attr
Community
  • 1
user1873584
  • 1

2 Answers2

1

In all the commands below, <PID> should be replaced with the process ID of the spam-sending process. Before you start, read the related canonical answer in the comments.

First, stop the spam, either with iptables by blocking ports 25 and 80 outbound or by pausing the process with kill -STOP <PID>.

Find out how it was started with cat /proc/<PID>/cmdline. That will tell you exactly what command was executed to start the process. The contents of ls -l /proc/<PID>/fd/ are usually very interesting too.

The next step is to find what started it with ps -ef | grep <PID>. It's normal for these sorts of spam-sending scripts to create a file that does the sending, execute it and then delete it. That way, although the working directory is / and the command line is ./mail, you won't find a /mail file on your filesystem and if you did, deleting it wouldn't help. The second column of the output is the PID, the third column is the PID of its parent. Find the command line of the parent and figure out how it started the process. It may be something like the cron daemon, in which case there should be some logging in /var/log/cron to indicate which crontab is responsible.

Once you know how the process is being started up each time, you can trace it further back in time and figure out how that happened. Eventually, once you have traced far enough back, you should get to the initial compromise. This is the primary goal. At this point, you can prevent the compromise happening again (upgrade the affected service or firewall it off or change its configuration to disallow whatever allowed the compromise).

The last step is to wipe your server and re-install from backups. Since you now know when the initial compromise happened, you can confidently restore from a clean backup that was taken before the compromise happened.

Although you can just clean up the files and remove the scripts, you can't ever be certain you have removed everything and you won't be any better prepared for the next compromise. The wip-and-re-install route is the smart option.

Ladadadada
  • 25,847
  • 7
  • 57
  • 90
0

Try using lsof on the process. This will give at least the inodes of all open files (and full names if possible). This will help you find out where the program is located.

lsof is not always installed on a Linux installation, so you might have to get the package and install it on your system.

Also be sure to analyze the whole process tree, not necessarily the single process itself.

mdpc
  • 11,698
  • 28
  • 51
  • 65