0

First of all, a disclamer, I am not a sys admin, I was just asked to check why a VM of ours is using a lot of the cpu.
It turns out that this process is running for ever on 200%
$top

PID   USER     PR NI VIRT RES SHR S %CPU %MEM   TIME+ COMMAND
11058 www-data 20 0 269564 5812 384 S  185.5     0.1  2:42.95   .resyslogd

After some research I figured out that this is a logger (actually I found that the logger is the rsyslog but i gess resyslog is the same thing.
So i noticed that there are alot of connection attempts (from bots I am guessing) that was triggering the logger to write. So I decided to stop allowing passwords for loging in and switched to key authentication. That did nothing for the cpu usage, even though the records on the logger where now less than before. for the random IPs that i was seeing on the logger, I added them on /etc/hosts.deny.
After all these the cpu usage has not droped at all. the same process is using more than 100% of the cpu core.
I know that the www-data is a web server, assuming apache, and dont know if it actually used for any reason.

Any help is highly appreciated.

Skaros Ilias
  • 131
  • 5
  • 1
    `.resyslogd` is almost certainly a malware infection with a thin layer of "obfuscation" for the most casual of looks. – Sven May 22 '18 at 11:18
  • @Sven that was my first thought, but googling the name only comes up the logger, rsyslog, so I figured that this was it, since no one else seems to have the same problem. Will look at the post you mention as duplicated. thanks – Skaros Ilias May 22 '18 at 11:43

1 Answers1

1

Looks very fishy. rsyslogd should definitely not be running as user www-data!

.resyslogd is definitely NOT the same thing as rsyslogd, but it's trying to pretend it's a logger process to avoid suspicions.

This is definitely not a logger. It's probably a coin miner or something similar, that has wormed its way into your server by using a weakness in your web server, or one of the scripts (CGIs, PHP, Perl, whatever) the web server is configured to run.

If you have root access (or can switch to www-data user), you could stop and analyze this worm. If not, you should get the system administrator to help you.

To stop it from wasting any more of your CPU, run kill -STOP 11058 (either as root or as www-data). This signal cannot be intercepted by the program: it simply tells the kernel to not allocate any more CPU time for this process until specifically told otherwise. Effectively, it freezes the process in its tracks.

Then use the /proc filesystem to dump the actual program code used by the worm, by using /proc/<PID of the suspicious process>/exe. In your case, the PID is 11058, so:

 sudo cat /proc/11058/exe >/tmp/dirty_worm

This should work even if the worm has deleted its executable file (which it probably has done, to hide itself).

You can then use whatever commands you like to analyze the executable, like

strings /tmp/dirty_worm | less

to see if it contains any readable text that might help in identifying it and its purpose.

Then, you can inspect all the available information under /proc to see what the process was doing. For example, ls -l /proc/11058/fd should reveal which files it had open, ls -l /proc/11058/map_files should list any libraries and other memory-mapped files it was using, etc.

Once you've gathered all the information you can, you can finally remove this process with kill -9 11058. This is one of the few situations where using kill -9 is definitely justified.

Since it looks like this process got into your server through the web server, you should shut down the web server, check its configuration, remove everything (web applications, CGI/PHP/Perl scripts, etc.) from it that is no longer used, and make sure everything you do use is up to date and secure.

telcoM
  • 4,153
  • 12
  • 23