DDoS Virus infection (as a unix service) on a Debian 8 VM Webserver

Question

I maintain a (fully updated) Wordpress for a student team on a Virtual Machine on ~okeanos service for a couple of years. Today, the helpdesk informed me that i am conducting DDoS attacks, which - of course - I am not (this service has my academic credentials connected..). After they suspended the machine and I flamed their mailing system I tried to find out what has happend.

First of all, I run a ps -ej to checkout what is running:

root@snf-25181:~# ps -ej
1545 1545 1545 ? 00:00:00 console-kit-dae
1618 1057 1057 ? 00:00:00 gdm-session-wor
1632 1632 1632 ? 00:01:40 rghuoywvrf
1767 1767 1767 ? 00:00:00 sshd
1769 1769 1769 ? 00:00:00 systemd
1770 1769 1769 ? 00:00:00 (sd-pam)
1775 1767 1767 ? 00:00:00 sshd
1776 1776 1776 pts/0 00:00:00 bash
1849 1849 1776 pts/0 00:00:00 su
1870 1870 1776 pts/0 00:00:00 bash
2246 0 0 ? 00:00:00 kworker/0:0
2797 839 839 ? 00:00:00 apache2
3158 3158 3158 ? 00:00:00 bvxktwwnsb
3162 3162 3162 ? 00:00:00 bvxktwwnsb
3163 3163 3163 ? 00:00:00 bvxktwwnsb
3164 3164 3164 ? 00:00:00 bvxktwwnsb
3165 3165 1776 pts/0 00:00:00 ps

Notice the bvxktwwnsb and the rguoywvrf

Then I did a ps aux to get the services (again, a tail):

Debian-+  1629  0.0  0.0 178300  4444 ?        Sl   16:53   0:00 /usr/lib/dconf/dconf-service
root      1667  0.0  0.0  30744  4436 ?        Ss   16:53   0:00 /sbin/wpa_supplicant -u -s -O /run/wpa_supplicant
root      1670  0.0  0.1 299588  9884 ?        Ssl  16:53   0:00 /usr/lib/packagekit/packagekitd
root      1674  0.0  0.1 1055004 6168 ?        Ssl  16:53   0:00 /usr/sbin/console-kit-daemon --no-daemon
www-data  1923  0.0  0.1 240964  8112 ?        S    16:53   0:00 /usr/sbin/apache2 -k start
pankgeo+  5656  0.0  0.0  27416  3424 ?        Ss   17:03   0:00 /lib/systemd/systemd --user
pankgeo+  5657  0.0  0.0 143108  2408 ?        S    17:03   0:00 (sd-pam)   
root      5893  0.0  0.1 102420  6428 ?        Ss   17:04   0:00 sshd: pankgeorg [priv]
pankgeo+  5904  0.1  0.0 102560  4128 ?        S    17:04   0:02 sshd: pankgeorg@pts/0
pankgeo+  5905  0.2  0.1  16816  6388 pts/0    Ss+  17:04   0:04 -bash      
root      7443  0.0  0.1 102420  6496 ?        Ss   17:07   0:00 sshd: pankgeorg [priv]
pankgeo+  7448  0.0  0.0 102552  4160 ?        S    17:07   0:00 sshd: pankgeorg@pts/1
pankgeo+  7449  0.0  0.1  16468  6228 pts/1    Ss+  17:07   0:01 -bash      
root     17351  0.0  0.0      0     0 ?        S    17:15   0:00 [kworker/0:0]
root     18446  0.0  0.0      0     0 ?        S    17:18   0:00 [kworker/0:2]
root     18488  0.1  0.0      0     0 ?        S    17:18   0:01 [kworker/1:1]
root     22680  1.5  0.0      0     0 ?        S    17:28   0:08 [kworker/1:0]
root     24173  0.0  0.1 102420  6416 ?        Ss   17:31   0:00 sshd: pankgeorg [priv]
pankgeo+ 24181  0.3  0.0 102420  3360 ?        S    17:31   0:01 sshd: pankgeorg@pts/2
pankgeo+ 24182  0.0  0.0  16480  6112 pts/2    Ss   17:31   0:00 -bash      
root     25316  2.3  0.0      0     0 ?        S    17:33   0:06 [kworker/1:2]
root     26777  0.0  0.0      0     0 ?        S    17:35   0:00 [kworker/0:1]
root     26778  0.0  0.0      0     0 ?        S    17:35   0:00 [kworker/0:3]
root     27300  0.0  0.0   1424  1040 ?        Ss   17:38   0:00 cat resolv.conf  #note                        
root     27306  0.0  0.0   1424  1036 ?        Ss   17:38   0:00 gnome-terminal   #from                     
root     27307  0.0  0.0   1424  1036 ?        Ss   17:38   0:00 ifconfig eth0    #here                    
root     27308  0.0  0.0   1424  1040 ?        Ss   17:38   0:00 id               #(DDOS?)         
root     27309  0.0  0.0   1424  1040 ?        Ss   17:38   0:00 ifconfig                        
pankgeo+ 27315  0.0  0.0  11136  2044 pts/2    R+   17:38   0:00 ps aux     

Note the items[-4:-1]. Then I found online about chkconfig --list so I run that and this popped out:

root@snf-25181:/home/pankgeorg# chkconfig --list
acdnfhruvx 0:off 1:off 2:off 3:off 4:off 5:off 6:off
flyymwddwn 0:off 1:off 2:off 3:off 4:off 5:off 6:off

1 to 5 where on but I turned them off. Then I restarted and it has changed name. Then I located the acdnfhruvx and this popped out:

root@snf-25181:~# locate acdnfhruvx
/etc/init.d/acdnfhruvx
/etc/rc1.d/S01acdnfhruvx
/etc/rc2.d/S01acdnfhruvx
/etc/rc3.d/S01acdnfhruvx
/etc/rc4.d/S01acdnfhruvx
/etc/rc5.d/S01acdnfhruvx

The contents of one of them (they are all the same): root@snf-25181:~# cat /etc/init.d/acdnfhruvx #!/bin/sh

chkconfig: 12345 90 90
description: acdnfhruvx
BEGIN INIT INFO
Provides: acdnfhruvx
Required-Start:
Required-Stop:
Default-Start: 1 2 3 4 5
Default-Stop:
Short-Description: acdnfhruvx
END INIT INFO
case $1 in
start)
/bin/acdnfhruvx
;;
stop)
;;
*)
/bin/acdnfhruvx   
;;
esac    

This was found after a restart, so /bin/acdnfhruvx was nowhere. Later I found exes (ELF Formatted) at /usr/bin (I think I can share it if there is a brave man among you)

An extensive list of the commands I saw the machine executing without knowing origin (from successive ps -ejs and ps auxes:

root     27755  0.0  0.0   1424  1036 ?        Ss   17:40   0:00 ifconfig                        
root     27759  0.0  0.0   1424  1036 ?        Ss   17:40   0:00 who                        
root     27760  0.0  0.0   1424  1040 ?        Ss   17:40   0:00 echo "find"                        
root     27761  0.0  0.0   1424  1036 ?        Ss   17:40   0:00 top                        
root     27762  0.0  0.0   1424  1036 ?        Ss   17:40   0:00 id                        
root     27805  0.0  0.0   1424  1036 ?        Ss   17:40   0:00 gnome-terminal                        
root     27809  0.0  0.0   1424  1040 ?        Ss   17:40   0:00 ifconfig                        
root     27810  0.0  0.0   1424  1044 ?        Ss   17:40   0:00 sh                        
root     27811  0.0  0.0   1424  1040 ?        Ss   17:40   0:00 sleep 1                        
root     27822  0.0  0.0   1424  1040 ?        Ss   17:40   0:00 netstat -an                        
root     27826  0.0  0.0   1424  1036 ?        Ss   17:40   0:00 top                        
root     27829  0.0  0.0   1424  1040 ?        Ss   17:40   0:00 bash                        
root     27833  0.0  0.0   1424  1040 ?        Ss   17:40   0:00 cd /etc                        
root     27834  0.0  0.0   1424  1040 ?        Ss   17:40   0:00 whoami                        
root     27822  0.0  0.0   1424  1040 ?        Ss   17:40   0:00 netstat -an                        
root     27826  0.0  0.0   1424  1036 ?        Ss   17:40   0:00 top                        
root     27829  0.0  0.0   1424  1040 ?        Ss   17:40   0:00 bash                        
root     27833  0.0  0.0   1424  1040 ?        Ss   17:40   0:00 cd /etc                        
root     27834  0.0  0.0   1424  1040 ?        Ss   17:40   0:00 whoami                        

pkilling is pointless, since it always forks, removing files from /etc/init.d/ and /{usr/,}bin is also pointless since after restart there is a new (identical) version of the executable. After all this info, I have two questions: Can I find out HOW I was infected? Can I get rid of this? Thank you in advance!

Answer 1

We suffered a similar infection on Suse, probably through ssh brute force login.

The steps to clean are:

1. Check the file /etc/crontab. You probably have an entry to call the virus every 3 minutes

   */3 * * * * root /etc/cron.hourly/cron.sh
   

   Delete this line.

2. Identify the parent process of the virus. The rguoywvrf in your ps -ej. The other proceses are created and killed continously.
3. Stop it, dont kill it, with kill -STOP 1632
4. Check with another ps -ej that only the parent lives, the children should die quickly
5. Now you can delete the files in /usr/bin and /etc/init.d. There are variants of the virus that also uses /boot or /bin. Use ls -lt | head to look for files that have been modified recently.
6. Check the script in /etc/cron.hourly/cron.sh. In our server it was calling another copy of the virus on /lib/libgcc.so. Delete both files.
7. Now you can kill definitely the rguoywvrf process.

Answer 2

To answer your questions:

1. Without the necessary precautions (off site syslog, IDS, log monitoring, etc) you'll probably never find out what happened.
2. I would have to agree with Matt. You'll invest time to get a machine running which you will never really trust. In my opinion, the best solution is to move the data off site & redo the machine.

Of course, for what it is worth, this is only my opinion. Though, when redoing the machine, you can of course take the necessary precautions & protect yourself better in the future.

Answer 3

this a threat that generates allot of problems because launch a DDOS attack and generate thousands of connections to externals servers on port 80, but i do not if intentionally or not, it tends to overload your connection until the routers/firewalls freeze if there are not DDOS attack rules.

now , how can you remove this threat?

1. find your threat , use

Centos/redhat

ps -ely 

Debian

ps -ej

you will see:

3158 3158 3158 ? 00:00:00 bvxktwwnsb
3162 3162 3162 ? 00:00:00 bvxktwwnsb
3163 3163 3163 ? 00:00:00 bvxktwwnsb
3164 3164 3164 ? 00:00:00 bvxktwwnsb

the "bvxktwwnsb" is your target

2. then you need to boot your linux server on single user mode , make any changes in multiuser mode is pointless, usually you can switch with the following command:

   telinit S

3. after that you need to delete files run at startup

in Centos/Redhat the procedure is

Step a)

cd /etc/init.d          
ll -tr 

the last command order your files in reverse date, you are going to see a last 1 or 2 files at the end with named like

acdnfhruvx
kmrkuwbrng
gqpjiestmf
bvxktwwnsb

you need to see the content

cat /etc/init.d/gqpjiestmf

normally you will see the execution of a file located in /bin or /usr/sbin with same name

you need to delete both files.

Step b)

cd /etc/
ll -tr 

check if your crontab file is recently changed, look at its contents, search for a line

*/3 * * * * root /etc/cron.hourly/udev.sh

or

*/3 * * * * root /etc/cron.hourly/crontab.sh 

you need to edit the file and remove that line.

check the content of udev.sh or crontab.sh and you will see something like this

#!/bin/sh
PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/usr/X11R6/bin
cp /lib/libgcc4.so /lib/libgcc4.4.so
/lib/libgcc4.4.so

you need to remove the "libgcc4.4.so" file or any other mentioned there (changing the permissions would also work, for instance chmod a-x libgcc.so )

reboot your server and everything should be ok.

For debian/ubuntu and relatives use:

locate bvxktwwnsb

and delete the files found in /etc and /bin

hope this help many people.

Answer 4

I found something!!!

look for /etc/crontab

In my server, there is a cronjob at every 3 minutes for executing something:

*/3 * * * * root /etc/cron.hourly/cron.sh

cat cron.sh

#!/bin/sh
PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/usr/X11R6/bin
for i in `cat /proc/net/dev|grep :|awk -F: {'print $1'}`; do ifconfig $i up& done
cp /lib/libgcc.so /lib/libgcc.so.bak
/lib/libgcc.so.bak

My solution:

1. disable permission (rwx 000) for: /etc/init.d/ {/usr}/bin/ /lib/libgcc.so
2. remove cronjob entry in /etc/crontab
3. remove cron script in /etc/cron.hourly/cron.sh
4. restart the server

note: file locations may vary

Answer 5

Additional trick complementary to Serhii solution. Stopping all the processes can be difficult as this thing spams network and cpu. Therefore add this line to your /etc/crontab to automatically STOP all the nasty processes (stops all processes with 10 character in the name every three minutes):

*/3 * * * * root pstree -ap | grep -E -- '-[a-z]{10},' | cut -d, -f2 | xargs kill -STOP 2>/dev/null

This is a good thing to do after cleanup to make sure the process does not return. Run it for a while until you are sure your box is clean.