Fake httpd process masked under perl execution

Question

I have apparently malicious php script somewhere in my webserver, Debian 8.3, Apache 2.4 Legal Apache process can be seen like this:

16169 www-data  20   0  482156  59180  41952 S  7.3  1.5   0:01.08 apache2
14791 www-data  20   0  483288  63456  45088 S  0.7  1.6   0:02.08 apache2

The illegal httpd process can be seen like this:

19214 www-data  20   0  142700 112156   2052 S  0.7  2.8   0:00.89 httpd

We can see user, which launched the process and image name, which is obviously faked, because when I do some tracking via PID:

root@VS1:~/skripty# lsof -p 19214
COMMAND   PID     USER   FD   TYPE   DEVICE SIZE/OFF    NODE NAME
httpd   19214 www-data  cwd    DIR      8,1     4096       2 /
httpd   19214 www-data  rtd    DIR      8,1     4096       2 /
httpd   19214 www-data  txt    REG      8,1    10416 3278761 /usr/bin/perl
httpd   19214 www-data  mem    REG      8,1    27136 3281419 /usr/lib/x86_64-linux-gnu/perl/5.20.2/auto/File/Glob/Glob.so
httpd   19214 www-data  mem    REG      8,1    89368 3281423 /usr/lib/x86_64-linux-gnu/perl/5.20.2/auto/POSIX/POSIX.so
httpd   19214 www-data  mem    REG      8,1    18632 3281416 /usr/lib/x86_64-linux-gnu/perl/5.20.2/auto/Fcntl/Fcntl.so
httpd   19214 www-data  mem    REG      8,1    43408 3281417 /usr/lib/x86_64-linux-gnu/perl/5.20.2/auto/Socket/Socket.so
httpd   19214 www-data  mem    REG      8,1    18728 3281425 /usr/lib/x86_64-linux-gnu/perl/5.20.2/auto/IO/IO.so
httpd   19214 www-data  mem    REG      8,1    35176 1313952 /lib/x86_64-linux-gnu/libcrypt-2.19.so
httpd   19214 www-data  mem    REG      8,1  1738176 1313950 /lib/x86_64-linux-gnu/libc-2.19.so
httpd   19214 www-data  mem    REG      8,1   137440 1313946 /lib/x86_64-linux-gnu/libpthread-2.19.so
httpd   19214 www-data  mem    REG      8,1  1051056 1313954 /lib/x86_64-linux-gnu/libm-2.19.so
httpd   19214 www-data  mem    REG      8,1    14664 1313953 /lib/x86_64-linux-gnu/libdl-2.19.so
httpd   19214 www-data  mem    REG      8,1  1827768 3281415 /usr/lib/x86_64-linux-gnu/libperl.so.5.20.2
httpd   19214 www-data  mem    REG      8,1   140928 1313947 /lib/x86_64-linux-gnu/ld-2.19.so
httpd   19214 www-data    0r   CHR      1,3      0t0    5593 /dev/null
httpd   19214 www-data    1w   CHR      1,3      0t0    5593 /dev/null
httpd   19214 www-data    2w   CHR      1,3      0t0    5593 /dev/null
httpd   19214 www-data    3u  IPv4 29921164      0t0     TCP *:27451 (LISTEN)

Please note, that this server is right now behind internal firewall, so there is no way to comunicate with outside world through SMTP on port 25. If I unblock this firewall rule, LSOF and NETSTAT will show me tens, or even hundreds of active SMTP connections. This malware is apparently designed for spamming. What I can not see from LSOF is the origin of perl script. I can kill the process but it wil reapear again. It is even shutting down itself.

So what I allready did:

blocked any outside communication on port 25 to preserve me from banning
searched for any suspitious code blocks like eval, base64_decode and handfully checked scripts for suspitious code blocks.
In php.ini I blocked following: disable_functions = pcntl_alarm,pcntl_fork,pcntl_waitpid,pcntl_wait,pcntl_wifexited,pcntl_wifstopped,pcntl_wifsignaled,pcntl_wexitstatus,pcntl_wtermsig,pcntl_wstopsig,pcntl_signal,pcntl_signal_dispatch,pcntl_get_last_error,pcntl_strerror,pcntl_sigprocmask,pcntl_sigwaitinfo,pcntl_sigtimedwait,pcntl_exec,pcntl_getpriority,pcntl_setpriority,exec,shell_exec,system,passthru,
Hardened file/folder permissions to make the php applications (wordpresses, expression engine). Any php file has owner of root and permissions set to 444. Any folder owner is www-data with 500 permissions. For special directories, like uploads, there is 700 rule. To fix that, I set up cronjob for every folder which is running every 1 minute and cleans up any *.php file and puts harmless index.php file into every folder/subfolder. I needed to exclude som cache folders and I suspect that that might be my current attack vector.

Important note: When I stop Apache 2.4 server and manually kill fake httpd process, no new malicious httpd processes will spawn.

This leads me to a conclusion, to manualy enable/disable my virtualhosts (10 at this time) to se, which one is causing launching of malicious httpd process.

If you have better idea, I will gladly follow your advices.

Thanks

EDIT: This is allready a fresh new server but when I migrated on some old webapps, which I cleaned by automatic tools and eyeballing the source code, I apparently missed something. In the old server, there were a lot of other issues (like litecoin mining, image data for webshops which do not belonged to me, thousands of html files generated by robots). Right now I have no other known issue except this fake process. Server is hard (or at least) hardened enough then before. This is my last problem with him. I need to track perl script executing.

EDIT 2

root@VS1:~# pstree -p -u
systemd(1)─┬─acpid(461)
           ├─agetty(751)
           ├─apache2(28369)─┬─apache2(26542,www-data)
           │                ├─apache2(27556,www-data)
           │                ├─apache2(27895,www-data)
           │                ├─apache2(28231,www-data)
           │                ├─apache2(28233,www-data)
           │                ├─apache2(30282,www-data)
           │                ├─apache2(30283,www-data)
           │                ├─apache2(30285,www-data)
           │                ├─apache2(30286,www-data)
           │                └─apache2(30288,www-data)
           ├─atd(447,daemon)
           ├─cron(450)
           ├─dbus-daemon(454,messagebus)
           ├─httpd(30628,www-data)
           ├─init(29268,www-data)
           ├─logger(15714)
           ├─mysqld_safe(15713)─┬─logger(15853)
           │                    └─mysqld(15852,mysql)─┬─{mysqld}(15854)
           │                                          ├─{mysqld}(15855)
           │                                          ├─{mysqld}(15856)
           │                                          ├─{mysqld}(15857)
           │                                          ├─{mysqld}(15858)
           │                                          ├─{mysqld}(15859)
           │                                          ├─{mysqld}(15860)
           │                                          ├─{mysqld}(15861)
           │                                          ├─{mysqld}(15862)
           │                                          ├─{mysqld}(15863)
           │                                          ├─{mysqld}(15864)
           │                                          ├─{mysqld}(15866)
           │                                          ├─{mysqld}(15867)
           │                                          ├─{mysqld}(15868)
           │                                          ├─{mysqld}(15869)
           │                                          ├─{mysqld}(15870)
           │                                          ├─{mysqld}(15871)
           │                                          ├─{mysqld}(15872)
           │                                          ├─{mysqld}(15873)
           │                                          ├─{mysqld}(15874)
           │                                          ├─{mysqld}(15875)
           │                                          ├─{mysqld}(15876)
           │                                          ├─{mysqld}(28177)
           │                                          ├─{mysqld}(28253)
           │                                          ├─{mysqld}(28259)
           │                                          ├─{mysqld}(28275)
           │                                          ├─{mysqld}(30162)
           │                                          ├─{mysqld}(30166)
           │                                          ├─{mysqld}(30172)
           │                                          └─{mysqld}(30173)
           ├─named(2130,bind)─┬─{named}(2132)
           │                  ├─{named}(2133)
           │                  └─{named}(2134)
           ├─rpc.idmapd(446)
           ├─rpc.statd(432,statd)
           ├─rpcbind(423)
           ├─rsyslogd(460)─┬─{in:imklog}(498)
           │               ├─{in:imuxsock}(497)
           │               └─{rs:main Q:Reg}(499)
           ├─sshd(14432)───sshd(28238)───bash(28240)───pstree(30964)
           ├─systemd-journal(142)
           ├─systemd-logind(452)
           ├─systemd-udevd(145)
           └─vmtoolsd(448)

Forgive me but **How do I deal with a compromised server?** is just a bunch of theory how to deal with threats like this. Although good point for starters, it doesn`t have any speciffic conclusions. It is only a compilation of "good advices" but nothing exact. — almiralower, Apr 06 '16 at 13:09
Yes. Long ago we, as a community decided that the only advice we would give regarding compromised servers is encapsulated in the dupe and that the précis of this is Nuke from orbit. On a personal level, cleaning up after a compromise isn't really Q&A it's Tech Support and SF isn't meant to be free Tech Support for the internets. — user9517, Apr 06 '16 at 15:12
@lain: so I am on my own. If I found a sollution to this particullar malware, would you consider it as a contribution to the community? Regards — almiralower, Apr 06 '16 at 15:52
I don;t think so, your question will likely be closed soon anyway, http://meta.serverfault.com/a/8691/9517. Excellent work on finding a solution but you still don't know if the original exploit left stuff that you haven't found. — user9517, Apr 06 '16 at 16:21
I'd recommend running `pstree` to figure out how the thing's getting started. Might not lead anywhere, ofc (the thing could've been added as an `at` job or service or something), but that might let you know which subsystem was the culprit. You can also learn more about the malware by copying the perl script and running it through a few virus scanners, maybe seeing whether there's a bot with a known M.O. which deploys that malware. — Parthian Shot, Apr 06 '16 at 19:08
If the worm that installed it is known, you may be able to figure out how the worm got in by looking up how the worm spreads itself and then running a similar scanner (or a deweaponized version of the worm) against a fresh install of your stuff. — Parthian Shot, Apr 06 '16 at 19:09
Also, aside from spamming, does that malware do any scanning? If you get a packet capture of what it does when it finds another server configured the same way you may actually be able to see a successful attack in real time and do a post-mortem with that packet capture alone. — Parthian Shot, Apr 06 '16 at 19:11
@Parthian Shot: please see my updated question. The problematic proces is **httpd(30628,www-data)** I am beginning with packet capturing, it will take some time. Thank You — almiralower, Apr 07 '16 at 08:17
After upgrading SW packages, including kernel, the malicious process is not spawning anymore. — almiralower, Apr 07 '16 at 09:43

Fake httpd process masked under perl execution

0 Answers0