`ps` and `bin/bash su` in ps aux output

Question

I have a shared server (amazon ec2 micro powered by bitnami) with a web application (Wordpress), which is showing low performance lately (with relatively stable traffic and usage). The only suspitious lines in apache2/logs/access.log I see are of form

$ tail --lines=1000 /opt/bitnami/apache2/logs/access_log
93.120.84.31 - - [06/Nov/2013:03:02:54 +0000] "POST /cgi-bin/php-cgi?%2D%64+%61%6C%6C%6F%77%5F%75%72%6C%5F%69%6E%63%6C%75%64%65%3D%6F%6E+%2D%64+%73%61%66%65%5F%6D%6F%64%65%3D%6F%66%66+%2D%64+%73%75%68%6F%73%69%6E%2E%73%69%6D%75%6C%61%74%69%6F%6E%3D%6F%6E+%2D%64+%64%69%73%61%62%6C%65%5F%66%75%6E%63%74%69%6F%6E%73%3D%22%22+%2D%64+%6F%70%65%6E%5F%62%61%73%65%64%69%72%3D%6E%6F%6E%65+%2D%64+%61%75%74%6F%5F%70%72%65%70%65%6E%64%5F%66%69%6C%65%3D%70%68%70%3A%2F%2F%69%6E%70%75%74+%2D%64+%63%67%69%2E%66%6F%72%63%65%5F%72%65%64%69%72%65%63%74%3D%30+%2D%64+%63%67%69%2E%72%65%64%69%72%65%63%74%5F%73%74%61%74%75%73%5F%65%6E%76%3D%30+%2D%6E HTTP/1.1" 504 249

and in ps aux output

daemon   29369  0.0  0.2  17688  1500 ?        S    10:52   0:00 /bin/bash ./su 57.54
daemon   29377 30.5  0.0   1780   544 ?        R    10:52   3:14 ./ps 57.54 22

what are those processes, and if this is a [known] exploit/vulnerability, how do I close it? I googled this mention, but can't find out whether this is applicable in my case.

If this question belongs to other SE site, please help me ask it where appropriate.

Many thanks in advance!

UPDATE

Followed tips from @Ladadadada, results are below. Yep, it's intrusion. Server stopped, restarted from weekly backup (all db and other things were on separate drives and I hope wasn't compromized) and being monitored. All suggestions on what's going on and how to prevent this in future are appreciated.

for ps proc:

$ sudo ls -l /proc/30633/fd
total 0
lr-x------ 1 daemon daemon 64 2013-11-06 13:05 0 -> /dev/null
l-wx------ 1 daemon daemon 64 2013-11-06 13:05 1 -> pipe:[2010646]
l-wx------ 1 daemon daemon 64 2013-11-06 13:05 2 -> pipe:[1918347]
lr-x------ 1 daemon daemon 64 2013-11-06 13:05 255 -> /tmp/.ssh_auth/.b/.ssh_auth/.b/su
lr-x------ 1 daemon daemon 64 2013-11-06 13:05 3 -> /opt/bitnami/apache2/cgi-bin/php-cgi

for /bin/bash su proc

$ sudo ls -l /proc/30641/fd | more
total 0
lr-x------ 1 daemon daemon 64 2013-11-06 13:04 0 -> /dev/null
l-wx------ 1 daemon daemon 64 2013-11-06 13:04 1 -> pipe:[2010646]
lrwx------ 1 daemon daemon 64 2013-11-06 13:04 10 -> socket:[5538939]
lrwx------ 1 daemon daemon 64 2013-11-06 13:04 100 -> socket:[5539029]
lrwx------ 1 daemon daemon 64 2013-11-06 13:04 101 -> socket:[5539030]
lr-x------ 1 daemon daemon 64 2013-11-06 13:04 3 -> /opt/bitnami/apache2/cgi-bin/php-cgi

and many more. Contents of temp folder:

/tmp/.ssh_auth/.b/.ssh_auth/.b$ ll
total 1848
drwxr-xr-x 2 daemon daemon    4096 2013-11-06 13:04 ./
drwxr-xr-x 3 daemon daemon    4096 2013-11-06 01:34 ../
-rw-r--r-- 1 daemon daemon       0 2013-11-06 13:04 246.120.pscan.22
-rw-r--r-- 1 daemon daemon       0 2013-11-06 03:42 .a
-rwxr-xr-x 1 daemon daemon 1384518 2005-06-05 20:24 brute*
-rwxr-xr-x 1 daemon daemon    1161 2013-03-13 20:22 dns-pool*
-rwxr-xr-x 1 daemon daemon      73 2013-11-05 04:14 pass.txt*
-rwxr-xr-x 1 daemon daemon     154 2013-11-05 04:13 print*
-rwxr-xr-x 1 daemon daemon   16071 2012-08-12 16:19 ps*
-rwxr-xr-x 1 daemon daemon  453972 2011-03-21 15:15 ss*
-rwxr-xr-x 1 daemon daemon     520 2013-11-06 01:33 su*

UPDATE 2

Winner by split second with Amazon, just after instance shutdown:

Hello,
We have detected that your instance(s):
i-xxxxxxxx
have been behaving in the following way that is against our AWS Customer Agreement:
Port Scanning

be careful: if they got in once, they will go in again quickly unless you patch the vulnerability that made them go in in the first place... For example you need to cleanup *at least* the /tmp/.ssh_auth/.b directory . But I bet they put also other backdoors in place. beware: some could be easy to find, and to make you "relax" when you found&cleaned them... while other backdoors, much better hidden, could still exist.. — Olivier Dulac, Nov 06 '13 at 14:56
since you've traced this to an intrusion you now need to figure out how they got in (that's a tall order and one we really can't help with: we can't do a forensic examination in a Q&A format). beyond that the standard advice is "wipe the machine and reinstall" — voretaq7, Nov 29 '13 at 03:14

score 9 · Accepted Answer · edited Apr 13 '17 at 12:14

That long string in your access logs is URL encoded and decodes to this:

-d allow_url_include=on -d safe_mode=off -d suhosin.simulation=on -d disable_functions="" -d open_basedir=none -d auto_prepend_file=php://input -d cgi.force_redirect=0 -d cgi.redirect_status_env=0

It looks to me like it's passing a bunch of options to php-cgi that turn off all the security features such as disallowing including URLs, disallowing certain functions and safe mode. It also allows PHP code to be passed in on STDIN which probably allows the attacker to simply supply code to be run as the POST parameters.

That's almost certainly malicious.

The vulnerability in this case is probably that php-cgi is in a web-accessible directory and/or can be requested directly.

The two lines in your ps output are also suspicious. You might investigate them further by looking in the /proc filesystem.

sudo ls -l /proc/29377/fd

Or by using strace on the PID:

sudo strace -f -p 29377 -s400

The output from strace can be quite large and difficult to read but it shouldn't be too hard to spot any spam or network connection attempts in there. You can limit the output with options like -e trace=file or -e trace=file,network.

Once you determine that it is malicious and you know how they got in, you should follow the instructions in this question to clean up afterwards. Simply removing the files you find is not enough because the attacker could have done anything to hide himself in the system.

`ps` and `bin/bash su` in ps aux output

1 Answers1